Home Forums Pig pig -useHCatalog

Tagged: ,

This topic contains 2 replies, has 2 voices, and was last updated by  Arun Ganesh Jeganathan 1 year ago.

  • Creator
    Topic
  • #45308

    Hi,

    I am getting the following error when I run “pig -useHCatalog”.

    Exception in thread “main” java.lang.NoClassDefFoundError: /usr/lib/hive/lib/libthrift-0/9/1/jar:/usr/lib/hive/lib/hive-exec-0/12/0/2/0/6/0-76/jar:/usr/lib/hive/lib/libfb303-0/9/0/jar:/usr/lib/hive/lib/jdo-api-3/0/1/jar:/usr/lib/hive/lib/slf4j-api-1/7/2/jar
    Caused by: java.lang.ClassNotFoundException: .usr.lib.hive.lib.libthrift-0.9.1.jar:.usr.lib.hive.lib.hive-exec-0.12.0.2.0.6.0-76.jar:.usr.lib.hive.lib.libfb303-0.9.0.jar:.usr.lib.hive.lib.jdo-api-3.0.1.jar:.usr.lib.hive.lib.slf4j-api-1.7.2.jar
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    Could not find the main class: /usr/lib/hive/lib/libthrift-0.9.1.jar:/usr/lib/hive/lib/hive-exec-0.12.0.2.0.6.0-76.jar:/usr/lib/hive/lib/libfb303-0.9.0.jar:/usr/lib/hive/lib/jdo-api-3.0.1.jar:/usr/lib/hive/lib/slf4j-api-1.7.2.jar. Program will exit

    I have defined all the environement variables (including PIG_CLASSPATH) as below and also defined pig.additiona.jars in the pig.properties files

    export HADOOP_GROUP=hadoop ;
    export HADOOP_HOME=/usr/lib/hadoop
    export HCAT_HOME=/usr/lib/hcatalog
    export HIVE_HOME=/usr/lib/hive
    export templeton_host=$TEMPLETON_HOST
    export user_name=hcat
    export PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-*.jar:\
    $HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
    $HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
    $HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/conf:$HADOOP_CONF_DIR:\
    $HIVE_HOME/lib/slf4j-api-*.jar:/usr/lib/hive/lib/libthrift-0.9.1.jar:/usr/lib/hive/lib/hive-exec-0.12.0.2.0.6.0-76.jar:/usr/lib/hive/lib/libfb303-0.9.0.jar:/usr/lib/hive/lib/jdo-api-3.0.1.jar:/usr/lib/hive/lib/slf4j-api-1.7.2.jar

    export PIG_OPTS=-Dhive.metastore.uris=thrift://$HIVE_HOST:$HIVE_PORT

    In pig.properties file,

    pig.additional.jars=/usr/lib/hive/lib/libthrift-0.9.1.jar:/usr/lib/hive/lib/hive-exec-0.12.0.2.0.6.0-76.jar:/usr/lib/hive/lib/libfb303-0.9.0.jar:/usr/lib/hive/lib/jdo-api-3.0.1.jar:/usr/lib/hive/lib/slf4j-api-1.7.2.jar

    Please let me know how this could be resolved. Thanks!

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #45394

    Hi Rahman,

    Thanks for your reply. I tried this, but it did not work.

    The HDP version I was trying was HDP 2.0.2. I had another HDP installation HDP 2.0.6 and the command pig -useHCatalog worked fine there. I noticed that one of the jars /usr/lib/hive/lib/libthrift-0.9.1.jar in HDP 2.0.2 was a older version in HDP 2.0.6 (/usr/lib/hive/lib/libthrift-0.9.0.jar). So I copied all the jars from the /hive/lib library in 2.0.6 to 2.0.2 and the command worked fine.

    So I believe this is a jar issue in 2.0.2. Can you have a look or confirm?

    Thanks,
    Arun

    Collapse
    #45318

    abdelrahman
    Moderator

    Hi Arun,

    Most likely the issue with some of the Jars. Have you installed HCatalog? From command line please run:
    $ hcat

    Try to revert back all of the environment variables and only add the following:
    In pig.properties add:
    hcat.bin=/usr/bin/hcat

    In pig-env.sh add: ( JAVA_HOME should be set there)
    HADOOP_HOME=${HADOOP_HOME:-/usr}

    Hope this helps.

    Thanks
    -Rahman

    Collapse
Viewing 2 replies - 1 through 2 (of 2 total)