The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Oozie Forum

Oozie and LZO problems

  • #31091
    Marc A.
    Member

    Hello,
    I am running HDP 1.2 and used Oozie successfully to run Java MR, Pig and Hive jobs.

    Then I installed Lzo support which now seems to break my Oozie.

    I have the same situation as documented here:
    http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html

    Oozie can’t start any MR jobs and reports:
    ClassNotFound error (java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found)

    I tried several things that Oozie finds the library hadoop-lzo-0.5.0.jar:
    (1) cp /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar /var/lib/oozie
    (2) cp /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar /usr/lib/oozie/lib

    Can you help me how to fix this?

    Another question:
    Do you know what of purpose of /usr/lib/oozie/share/lib is?
    There are lots of jars in it.
    Additionally it is also in hdfs at /user/oozie/share/lib

    I appreciate any help.

  • Author
    Replies
  • #31171
    Sasha J
    Moderator

    You should add LZO codec to oozie .war file
    Use ooze-setupsh or addtowar.sh script for this.
    Take a look to the Hortoworks docs:
    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap8-1.html

    Thank you!
    Sasha

    #31236
    Jayashankar VS
    Participant

    As pointed out, here is the exact command from the documentation to add hadop-lzo.jar to ozzie.war. There is also one other thing with this command and the tricky part is you need to set the oozie PID in ozzie-env.sh

    export CATALINA_PID=/var/run/oozie/oozie.pid

    /usr/lib/oozie/bin/oozie-setup.sh -hadoop 0.20.200 /usr/lib/hadoop -extjs /usr/share/HDP-oozie/ext-2.2.zip -jars /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar

    Executing the above command will bring the below output.

    /usr/lib/oozie/bin/oozie-setup.sh -hadoop 0.20.200 /usr/lib/hadoop -extjs /usr/share/HDP-oozie/ext-2.2.zip -jars /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar
    setting OOZIE_CONFIG=/etc/oozie/conf
    setting OOZIE_DATA=/var/lib/oozie
    setting OOZIE_LOG=/var/log/oozie
    setting CATALINA_BASE=${OOZIE_DATA}/oozie-server
    setting CATALINA_TMPDIR=/var/tmp/oozie
    setting CATALINA_PID=/var/run/oozie/oozie.pid
    setting JAVA_HOME=/usr/java/default
    setting OOZIE_LOG=/var/log/oozie
    setting CATALINA_PID=/var/run/oozie/oozie.pid
    setting OOZIE_DATA=/var/db/oozie
    setting JAVA_LIBRARY_PATH=”/usr/lib/hadoop/lib/native/Linux-amd64-64″

    Injecting following Hadoop JARs

    /usr/lib/hadoop/hadoop-core-1.1.2.21.jar
    /usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar
    /usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar
    /usr/lib/hadoop/lib/commons-configuration-1.6.jar

    New Oozie WAR file with added ‘Hadoop JARs, ExtJS library, JARs’ at /var/lib/oozie/oozie-server/webapps/oozie.war

    #31573
    Marc A.
    Member

    Thanks for your support.

    Adding the library to the Oozie war file worked, as Sasha wrote.

    However, do you know, why it is not working when I set up the sharelib (that’s something I tried as well)
    http://blog.cloudera.com/blog/2012/12/how-to-use-the-sharelib-in-apache-oozie/

    I am wondering why I could not get it working with the sharelib…?
    Maybe one of you knows the reason for that?

    Thanks

    #31605
    Jayashankar VS
    Participant

    Not sure if I understand this question, the jar files shown below that you added were part of ozzie which is required for initializing oozie application.

    /usr/lib/hadoop/hadoop-core-1.1.2.21.jar
    /usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar
    /usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar
    /usr/lib/hadoop/lib/commons-configuration-1.6.jar

    Can you please elaborate on this statement which says “However, do you know, why it is not working when I set up the sharelib (that’s something I tried as well)”?

    #31772
    Marc A.
    Member

    Yes, let me elaborate on my previous statement.

    Without adding the library to Oozie war, I could start and stop Oozie but I could not see any errors when Oozie initializes.
    I could also execute the command “oozie admin -oozie http://sandbox:11000/oozie -status” but could not see any problems as well.

    Only when I started an Oozie job (no matter what kind of Oozie action in it), I got the ClassNotFound error message I mentioned.
    I was therefore thinking, that it is a “runtime problem” of my Oozie job which I could possibly solve when I add the lzo library to the Oozie sharedlib folders in HDFS – but that did not solve my problems.

    But after adding the lzo library to the Oozie war file, everything worked.

    #33695
    abdelrahman
    Moderator

    Hi Marc,

    From command line please run:

    ps aux | grep -i oozie

    And post the output in this forum.

    Thanks
    -Rahman

The forum ‘Oozie’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.