Home Forums Oozie Oozie and LZO problems

This topic contains 6 replies, has 4 voices, and was last updated by  abdelrahman 1 year, 1 month ago.

  • Creator
    Topic
  • #31091

    Marc A.
    Member

    Hello,
    I am running HDP 1.2 and used Oozie successfully to run Java MR, Pig and Hive jobs.

    Then I installed Lzo support which now seems to break my Oozie.

    I have the same situation as documented here:

    http://mapredit.blogspot.com/2013/02/lzo-compression-with-oozie.html

    Oozie can’t start any MR jobs and reports:
    ClassNotFound error (java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found)

    I tried several things that Oozie finds the library hadoop-lzo-0.5.0.jar:
    (1) cp /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar /var/lib/oozie
    (2) cp /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar /usr/lib/oozie/lib

    Can you help me how to fix this?

    Another question:
    Do you know what of purpose of /usr/lib/oozie/share/lib is?
    There are lots of jars in it.
    Additionally it is also in hdfs at /user/oozie/share/lib

    I appreciate any help.

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #33695

    abdelrahman
    Moderator

    Hi Marc,

    From command line please run:

    ps aux | grep -i oozie

    And post the output in this forum.

    Thanks
    -Rahman

    Collapse
    #31772

    Marc A.
    Member

    Yes, let me elaborate on my previous statement.

    Without adding the library to Oozie war, I could start and stop Oozie but I could not see any errors when Oozie initializes.
    I could also execute the command “oozie admin -oozie http://sandbox:11000/oozie -status” but could not see any problems as well.

    Only when I started an Oozie job (no matter what kind of Oozie action in it), I got the ClassNotFound error message I mentioned.
    I was therefore thinking, that it is a “runtime problem” of my Oozie job which I could possibly solve when I add the lzo library to the Oozie sharedlib folders in HDFS – but that did not solve my problems.

    But after adding the lzo library to the Oozie war file, everything worked.

    Collapse
    #31605

    Jayashankar VS
    Participant

    Not sure if I understand this question, the jar files shown below that you added were part of ozzie which is required for initializing oozie application.

    /usr/lib/hadoop/hadoop-core-1.1.2.21.jar
    /usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar
    /usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar
    /usr/lib/hadoop/lib/commons-configuration-1.6.jar

    Can you please elaborate on this statement which says “However, do you know, why it is not working when I set up the sharelib (that’s something I tried as well)”?

    Collapse
    #31573

    Marc A.
    Member

    Thanks for your support.

    Adding the library to the Oozie war file worked, as Sasha wrote.

    However, do you know, why it is not working when I set up the sharelib (that’s something I tried as well)

    http://blog.cloudera.com/blog/2012/12/how-to-use-the-sharelib-in-apache-oozie/

    I am wondering why I could not get it working with the sharelib…?
    Maybe one of you knows the reason for that?

    Thanks

    Collapse
    #31236

    Jayashankar VS
    Participant

    As pointed out, here is the exact command from the documentation to add hadop-lzo.jar to ozzie.war. There is also one other thing with this command and the tricky part is you need to set the oozie PID in ozzie-env.sh

    export CATALINA_PID=/var/run/oozie/oozie.pid

    /usr/lib/oozie/bin/oozie-setup.sh -hadoop 0.20.200 /usr/lib/hadoop -extjs /usr/share/HDP-oozie/ext-2.2.zip -jars /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar

    Executing the above command will bring the below output.

    /usr/lib/oozie/bin/oozie-setup.sh -hadoop 0.20.200 /usr/lib/hadoop -extjs /usr/share/HDP-oozie/ext-2.2.zip -jars /usr/lib/hadoop/lib/hadoop-lzo-0.5.0.jar
    setting OOZIE_CONFIG=/etc/oozie/conf
    setting OOZIE_DATA=/var/lib/oozie
    setting OOZIE_LOG=/var/log/oozie
    setting CATALINA_BASE=${OOZIE_DATA}/oozie-server
    setting CATALINA_TMPDIR=/var/tmp/oozie
    setting CATALINA_PID=/var/run/oozie/oozie.pid
    setting JAVA_HOME=/usr/java/default
    setting OOZIE_LOG=/var/log/oozie
    setting CATALINA_PID=/var/run/oozie/oozie.pid
    setting OOZIE_DATA=/var/db/oozie
    setting JAVA_LIBRARY_PATH=”/usr/lib/hadoop/lib/native/Linux-amd64-64″

    Injecting following Hadoop JARs

    /usr/lib/hadoop/hadoop-core-1.1.2.21.jar
    /usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar
    /usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar
    /usr/lib/hadoop/lib/commons-configuration-1.6.jar

    New Oozie WAR file with added ‘Hadoop JARs, ExtJS library, JARs’ at /var/lib/oozie/oozie-server/webapps/oozie.war

    Collapse
    #31171

    Sasha J
    Moderator

    You should add LZO codec to oozie .war file
    Use ooze-setupsh or addtowar.sh script for this.
    Take a look to the Hortoworks docs:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap8-1.html

    Thank you!
    Sasha

    Collapse
Viewing 6 replies - 1 through 6 (of 6 total)