Home Forums HDP on Windows – Installation Running oozie job fails JA018 [java.lang.NoClassDefFoundError]

This topic contains 6 replies, has 3 voices, and was last updated by  chandra 3 months, 3 weeks ago.

  • Creator
    Topic
  • #28050

    Bill Carroll
    Participant

    When I run a oozie job that executes a Pig action it fails on HDP 1.1 on Windows with the stack below. It looks like oozie does not have the classpath to Pig? If I run the Pig scrip manually without oozie it is successful.

    >>> Invoking Pig command line now >>>

    Run pig script using Main.main() for Pig version before 0.8

    <<< Invocation of Main class completed <<<

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], main() threw exception, org/apache/pig/Main
    java.lang.NoClassDefFoundError: org/apache/pig/Main
    at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:324)
    at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:219)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
    at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.Child.main(Child.java:265)
    Caused by: java.lang.ClassNotFoundException: org.apache.pig.Main
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    … 17 more

    Oozie Launcher failed, finishing Hadoop job gracefully

    Oozie Launcher ends

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #56545

    chandra
    Participant

    Hi,
    Whats the solution here to overcome NoClassDefFoundError while running oozie workflow for Hive/Pig/MR?
    I a getting below error:
    java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.JavaMain not found

    thanks for your help in advance!

    thanks
    Chandra

    Collapse
    #28474

    Seth Lyubich
    Keymaster

    Hi Bill,

    Thanks for the follow up. Great to hear that your issue is resolved.

    Thanks,
    Seth

    Collapse
    #28450

    Bill Carroll
    Participant

    Thanks Seth,

    That did it. I did two things. I put the C:\hdp\oozie-3.2.0-incubating\oozie-win-distro\share\lib jar files to HDFS under /user/hadoop/share/lib keeping the same directory strucutre. User hadoop is running the oozie service. I added the following to my job.properties file oozie.use.system.libpath=true

    After that the pig action was executed by oozie.

    Bill

    Collapse
    #28408

    Seth Lyubich
    Keymaster

    Hi Bill,

    I think you can try to use C:\hdp\oozie-3.2.0-incubating\oozie-win-distro\share\lib for Sharelib setup. In general you need to make sure that required jars are located in HDFS folder /user/oozie/shared/lib.

    Some more information on running example can be found here:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-Win-1.1.0/bk_dataintegration/content/ch_using-oozie.html

    Hope this helps.

    Thanks,
    Seth

    Collapse
    #28158

    Bill Carroll
    Participant

    Hi Seth,

    Thanks for the help. Here is the job information. I am not familiar with Sharelib, so I don’t believe I am using it.

    Microsoft Windows [Version 6.2.9200]
    (c) 2012 Microsoft Corporation. All rights reserved.

    C:\Users\wcarroll>hdfs fs -lsr /user/wcarroll/congress
    -rw-r–r– 2 wcarroll supergroup 618 2013-06-17 12:42 /user/wcarroll/congress/whitehouse.pig
    -rw-r–r– 2 wcarroll supergroup 952 2013-06-20 06:53 /user/wcarroll/congress/workflow.xml

    C:\Users\wcarroll>hdfs fs -cat /user/wcarroll/congress/whitehouse.pig
    –whitehouse.pig
    –pig -f whitehouse.pig

    visits = LOAD ‘/user/wcarroll/labs/WhiteHouseVisits.csv’ USING PigStorage(‘,’);

    congress = FILTER visits BY ($25 MATCHES ‘.* CONGRESS .*’);

    project_congress = FOREACH congress GENERATE
    $0 AS lname:chararray,
    $1 AS fname:chararray,
    $2 AS middle:chararray,
    $6 AS time_of_arrival:chararray,
    $11 AS appt_scheduled_time:chararray,
    $19 AS visitee_lname:chararray,
    $20 AS visitee_fname:chararray,
    $21 AS location:chararray,
    $25 AS comment:chararray ;

    STORE project_congress INTO ‘/user/wcarroll/congress/project_congress';

    #C:\Users\wcarroll>hdfs fs -cat /user/wcarroll/congress/workflow.xml

    ${jobTracker}
    ${nameNode}

    mapred.job.queue.name
    ${queueName}

    whitehouse.pig

    Job failed, error
    message[${wf:errorMessage(wf:lastErrorNode())}]

    ============================================================
    #c:\public\oozie\pig\job.properties

    oozie.wf.application.path=hdfs://wpch1:8020/user/wcarroll/congress
    #Hadoop mapred.job.tracker
    jobTracker=wpch1:50300
    #Hadoop fs.default.name
    nameNode=hdfs://wpch1:8020/
    #Hadoop mapred.queue.name
    queueName=default
    ============================================================

    C:\Users\wcarroll>hdfs fs -mkdir /user/wcarroll/congress/project_congress

    C:\Users\wcarroll>hdfs fs -lsr /user/wcarroll/labs
    -rw-r–r– 1 wcarroll supergroup 183292776 2013-05-26 21:49 /user/wcarroll/labs/WhiteHouseVisits.csv

    C:\Users\wcarroll>oozie job -config c:\public\oozie\pig\job.properties -run
    job: 0000001-130624112145582-oozie-hado-W

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], main() threw exception, org/apache/pig/Main
    java.lang.NoClassDefFoundError: org/apache/pig/Main
    at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:324)

    Collapse
    #28149

    Seth Lyubich
    Keymaster

    Hi Bill,

    Can you please let us know what Pig job you are trying to run, command that you used, and if you setup Sharelib in /user/oozie/share/lib ?

    Thanks,
    Seth

    Collapse
Viewing 6 replies - 1 through 6 (of 6 total)