I am learning Hadoop using the Hadoop for Dummies. I have successfully run thr first steps like installation, creating HDFS and storing Data, running Pig scripts etc.
I have created the FlightsByCarries.jar File.
When I run the code as given in the book, I am getting an error:
[root#localhost totalmiles]# hadoop jar /home/tom/Desktop/totalmiles/FlightsByCarrier.jar FlightsByCarrier /user/root/airline-data/2008.csv /user/root/output/flightsCount
Exception in thread "main" java.lang.ClassNotFoundException: FlightsByCarrier
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)
Could someone please help me correct this?
it is because your compiler is not getting your main method in the jar FlightsByCarrier. may be you haven't copy the file to hdfs or your file path is not appropriate...check once..and then try
You JAR file is unable to locate the main class name, if you are using any package name in your main class then specify complete name along with package while running the program.
Ex:
hadoop jar FlightsByCarrier.jar com.dev.FlightsByCarrier /input /output
You can also include main class while creating JAR file.
While creating a JAR file after selecting the path then click next --> asks for selecting main class --> browse the name and select --> click OK
Run the command without class name in this case.
hadoop jar FlightsByCarrier.jar /input /output
Hope this solves your problem.
Related
I am trying to solve a VRP using google or tools. I use java for my program and everything works fine locally.
After deploying the program on lambda I get an error
java.lang.IllegalArgumentException: Path component should be '/'
2021-08-13T12:40:59.915+04:00 at java.base/sun.nio.fs.UnixFileSystemProvider.checkUri(Unknown Source)
2021-08-13T12:40:59.915+04:00 at java.base/sun.nio.fs.UnixFileSystemProvider.newFileSystem(Unknown Source)
2021-08-13T12:40:59.915+04:00 at java.base/java.nio.file.FileSystems.newFileSystem(Unknown Source)
2021-08-13T12:40:59.915+04:00 at java.base/java.nio.file.FileSystems.newFileSystem(Unknown Source)
2021-08-13T12:40:59.916+04:00 at com.google.ortools.Loader.unpackNativeResources(Loader.java:88)
2021-08-13T12:40:59.916+04:00 at com.google.ortools.Loader.loadNativeLibraries(Loader.java:106)
this is caused by Loader.loadNativeLibraries() method call.
I built the jar package using maven with shade plugin.
I am using Ubuntu 20.04 if that is related.
I use com.google.ortools:ortools-java
Thank you in advance.
UPDATE :
I manually downloaded native library files and put the full path with System.load(path) . This gave the following error when executing on Lambda
Can't load library: /var/task/src/main/resources/darwin-x86-64/libjniortools.so: java.lang.UnsatisfiedLinkError
java.lang.UnsatisfiedLinkError: Can't load library: /var/task/src/main/resources/darwin-x86-64/libjniortools.so
I am using AWS SDK to upload files to S3 bucket when set java classpath to "aws-java-sdk-1.10.49.jar", the program works fine, but when replacing jar file to a later version "aws-java-sdk-1.11.35.jar", I get an exception "AmazonServiceException class not found as below."
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.AmazonServiceException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
any help?
here we have three possible problems:
A simple example of NoClassDefFoundError is class belongs to a missing JAR file or JAR was not added into classpath or sometimes jar's name has been changed by someone like in my case one of my colleagues has changed tibco.jar into tibco_v3.jar and the program is failing with java.lang.NoClassDefFoundError and I were wondering what's wrong.
The class is not in Classpath, there is no sure shot way of knowing it but many times you can just have a look to print System.getproperty("java.classpath") and it will print the classpath from there you can at least get an idea of your actual runtime classpath.
Just try to run with explicitly -classpath option with the classpath you think will work and if it's working then it's a sure short sign that someone is overriding java classpath.
Take a look at this article
On a completely fresh install of Developer Studio 3.6 (not the plugin) I have created a very simple ESB config project and tried to create a car from the pom editor (using the button in the top right hand corner (as per doc). I get a failure message in the UI "an error occurred while creating the carbon archive file. For more details view the log". The log just shows "java.io.FileNotFoundException: C:\wso2workspace\AthenaHRInterfaceApplication\target\AthenaHRInterfaceApplication_0.1.car (The system cannot find the file specified)".
If I view the target directory as I do this, one can see the file created halfway through the process (i.e. when selecting which directory to save .car file to), but then on completion (ie. OK button), the error is displayed and the file size of .car file goes to 0kb
This is really frustrating and doesn't seem to make any sense. I did get this to work (not with this project) before, but this one just doesn't want to play ball. I can get the car file to be generated external to Dev Studio through maven, but then it isn't deployable from within Studio and appears to be affected by the above error in any case.
Any suggestions to where I may be going wrong?
This is the stacktrace in my log file...
!ENTRY org.wso2.developerstudio.eclipse.distribution.project 4 0 2014-05-14 08:55:30.677
!MESSAGE An error occured while creating the carbon archive file
!STACK 0
java.io.FileNotFoundException: C:\wso2workspace\AthenaHRInterfaceApplication\AthenaHRInterfaceCAP\target\AthenaHRInterfaceCAP_0.1.car (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at org.wso2.developerstudio.eclipse.utils.file.FileUtils.copy(FileUtils.java:227)
at org.wso2.developerstudio.eclipse.distribution.project.editor.DistProjectEditorPage.exportCar(DistProjectEditorPage.java:791)
at org.wso2.developerstudio.eclipse.distribution.project.editor.DistProjectEditorPage$8.run(DistProjectEditorPage.java:755)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:499)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:584)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:501)
at org.eclipse.jface.action.ActionContributionItem$6.handleEvent(ActionContributionItem.java:452)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1057)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4170)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3759)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1113)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:997)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:140)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:611)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:567)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:636)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:591)
at org.eclipse.equinox.launcher.Main.run(Main.java:1450)
at org.eclipse.equinox.launcher.Main.main(Main.java:1426)
is this a bug - it looks like it. Happens with a fresh install of DS and new project.
Not that anyone appears to be following this other than the on comments on java version, but for the sake of completeness, I will keep adding to this thread.
It appears that the error was being caused by me selecting the location of the 'target' directory, I suspect that behind the scenes DS is using the same location (as in maven) which is trying to write to the same directory with the same file name which is causing the file not found exception. Saving to a different directory appears to at least get past this error. However, the new issue I have is that Registry Resources aren't being included in the resultant .car file using the button at the top right hand corner of the Composite Application POM editor. If I mvn install (from either command line or from Run(ing) the pom within DS, the correct .car file is generated.
I think I should create another thread with this specific issue and answer this one myself.
Views?
I am new in jetty. I am trying to run Jetty with IKVM. However, it throws exception. I am not sure what should I do.
alex#AlexUbuntu:/usr/share/jetty$ ikvm -jar start.jar
5 [main] INFO org.mortbay.log - Logging to org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Method.java:
at org.mortbay.start.Main.invokeMain(Main.java:179)
at org.mortbay.start.Main.start(Main.java:508)
at org.mortbay.start.Main.start(Main.java:439)
at org.mortbay.start.Main.main(Main.java:99)
Caused by: cli.System.TypeLoadException: Could not load type 'org.apache.xerces.util.NamespaceSupport' from assembly 'ikvm_dynamic_assembly__40326550, Version=2011.611.1039.16726, Culture=neutral, PublicKeyToken=null'.
at org.apache.xerces.parsers.XIncludeAwareParserConfiguration.<init>(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:517)
at java.lang.Class.newInstance0(Class.java:333)
at java.lang.Class.newInstance(Class.java:320)
at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.<init>(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)
at org.mortbay.xml.XmlParser.setValidating(XmlParser.java)
at org.mortbay.xml.XmlParser.<init>(XmlParser.java:68)
at org.mortbay.xml.XmlConfiguration.initParser(XmlConfiguration.java)
at org.mortbay.xml.XmlConfiguration.<init>(XmlConfiguration.java:105)
at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:958)
... 5 more
It seems that I need to install some external libraries in order to make it works. But what should I need to install.
The environment is fresh and clean:
Ubuntu 11.04
IKVM 0.40.0.1
Java 1.6.0_22
Mono 2.6.7
Update on 28 June 2010
I think I make it works. But I haven't try loading .NET classes in jetty. By the way, I used a dirty method that I replaced /usr/bin/java and /usr/lib/jvm/default-jvm/java with ikvm.exe. So everytime when I type java that actually is IKVM.
I will try to load .NET classes in jetty. But I am not familiar with jetty so I may take sometime.
Update on 1 July 2010
I have tried to load a .NET class. However, finally I got an error message.
HTTP ERROR 500
Problem accessing /hello/servlet. Reason:
ikvmstub generated stubs can only be used on IKVM.NET
Caused by:
java.lang.UnsatisfiedLinkError: ikvmstub generated stubs can only be used on IKVM.NET
at cli.CSharpClass.<init>(Unknown Source)
at HelloServlet.doPost(HelloServlet.java:28)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Can you run Jetty with 'java -jar start.jar'?
I suspect you need at least a few base JAR files for jetty even if it's classloader dynamically loads everything that is needed. It looks like it is failing in the log initialization.
Add the xerces JAR file to the classpath when running IKVM.
ikvm -cp .:xerces.jar -jar startup.jar
Update
I just looked through jetty.sh and there are a few things the script file sets up. You'll need to go through that file and determine what you need out of it, or replace all the instances of java with ikvm and be aware that Jetty also uses tools.jar
The jar files that you generate with ikvmstub are only for the java compiler and not for the runtime. The java compiler can nor work with .NET dlls. For the runtime you need to use the dlls directly.
We use the jetty without problems with IKVM but we use a newer version 0.46. The simplest is you build all jar files in one step with a shared classloader. See the ikvm wiki for details.
Following the instructions in http://en.wikibooks.org/wiki/Compojure/Getting_Started, I:
downloaded http://github.com/weavejester/compojure/tarball/0.3.2
extracted to ~/compojure
chdir to there
changed permission to executable
run ant deps
ant
run export CLASSPATH=~/compojure/compojure.jar
created compojure shell script
Clojure source file
chdir to there
run ./compojure hello.clj
Then, I got:
Exception in thread "main" java.lang.NoClassDefFoundError: clojure/lang/Script
Caused by: java.lang.ClassNotFoundException: clojure.lang.Script
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: clojure.lang.Script. Program will exit.
I tried changing Compojure shell script to use full classpath, but still didn't work.
Why not? How to install Compojure?
That page is outdated.
You should follow these instructions from compojure developers:
https://github.com/weavejester/compojure/wiki/Getting-Started