HDP on Windows – Installation Forum

Running oozie job fails JA018 [java.lang.NoClassDefFoundError]

  • #28050
    Bill Carroll

    When I run a oozie job that executes a Pig action it fails on HDP 1.1 on Windows with the stack below. It looks like oozie does not have the classpath to Pig? If I run the Pig scrip manually without oozie it is successful.

    >>> Invoking Pig command line now >>>

    Run pig script using Main.main() for Pig version before 0.8

    <<< Invocation of Main class completed <<<

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], main() threw exception, org/apache/pig/Main
    java.lang.NoClassDefFoundError: org/apache/pig/Main
    at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:324)
    at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:219)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
    at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.Child.main(Child.java:265)
    Caused by: java.lang.ClassNotFoundException: org.apache.pig.Main
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    … 17 more

    Oozie Launcher failed, finishing Hadoop job gracefully

    Oozie Launcher ends

to create new topics or reply. | New User Registration

  • Author
  • #28149
    Seth Lyubich

    Hi Bill,

    Can you please let us know what Pig job you are trying to run, command that you used, and if you setup Sharelib in /user/oozie/share/lib ?


    Bill Carroll

    Hi Seth,

    Thanks for the help. Here is the job information. I am not familiar with Sharelib, so I don’t believe I am using it.

    Microsoft Windows [Version 6.2.9200]
    (c) 2012 Microsoft Corporation. All rights reserved.

    C:\Users\wcarroll>hdfs fs -lsr /user/wcarroll/congress
    -rw-r–r– 2 wcarroll supergroup 618 2013-06-17 12:42 /user/wcarroll/congress/whitehouse.pig
    -rw-r–r– 2 wcarroll supergroup 952 2013-06-20 06:53 /user/wcarroll/congress/workflow.xml

    C:\Users\wcarroll>hdfs fs -cat /user/wcarroll/congress/whitehouse.pig
    –pig -f whitehouse.pig

    visits = LOAD ‘/user/wcarroll/labs/WhiteHouseVisits.csv’ USING PigStorage(‘,’);

    congress = FILTER visits BY ($25 MATCHES ‘.* CONGRESS .*’);

    project_congress = FOREACH congress GENERATE
    $0 AS lname:chararray,
    $1 AS fname:chararray,
    $2 AS middle:chararray,
    $6 AS time_of_arrival:chararray,
    $11 AS appt_scheduled_time:chararray,
    $19 AS visitee_lname:chararray,
    $20 AS visitee_fname:chararray,
    $21 AS location:chararray,
    $25 AS comment:chararray ;

    STORE project_congress INTO ‘/user/wcarroll/congress/project_congress’;

    #C:\Users\wcarroll>hdfs fs -cat /user/wcarroll/congress/workflow.xml




    Job failed, error


    #Hadoop mapred.job.tracker
    #Hadoop fs.default.name
    #Hadoop mapred.queue.name

    C:\Users\wcarroll>hdfs fs -mkdir /user/wcarroll/congress/project_congress

    C:\Users\wcarroll>hdfs fs -lsr /user/wcarroll/labs
    -rw-r–r– 1 wcarroll supergroup 183292776 2013-05-26 21:49 /user/wcarroll/labs/WhiteHouseVisits.csv

    C:\Users\wcarroll>oozie job -config c:\public\oozie\pig\job.properties -run
    job: 0000001-130624112145582-oozie-hado-W

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], main() threw exception, org/apache/pig/Main
    java.lang.NoClassDefFoundError: org/apache/pig/Main
    at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:324)

    Seth Lyubich

    Hi Bill,

    I think you can try to use C:\hdp\oozie-3.2.0-incubating\oozie-win-distro\share\lib for Sharelib setup. In general you need to make sure that required jars are located in HDFS folder /user/oozie/shared/lib.

    Some more information on running example can be found here:

    Hope this helps.


    Bill Carroll

    Thanks Seth,

    That did it. I did two things. I put the C:\hdp\oozie-3.2.0-incubating\oozie-win-distro\share\lib jar files to HDFS under /user/hadoop/share/lib keeping the same directory strucutre. User hadoop is running the oozie service. I added the following to my job.properties file oozie.use.system.libpath=true

    After that the pig action was executed by oozie.


    Seth Lyubich

    Hi Bill,

    Thanks for the follow up. Great to hear that your issue is resolved.



    Whats the solution here to overcome NoClassDefFoundError while running oozie workflow for Hive/Pig/MR?
    I a getting below error:
    java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.JavaMain not found

    thanks for your help in advance!


You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.