Home Forums HDP on Windows – Installation Grunt Shell Cannot Resolve HCatStorer

This topic contains 6 replies, has 5 voices, and was last updated by  S R 7 months, 4 weeks ago.

  • Creator
    Topic
  • #17713

    Vikas Dharia
    Member

    Hi,

    I was attempting to cross refer the metadata in HCat and I was attempting a Load Thru Pig. I am getting an Error

    grunt> Store A into ‘my_data’ using org.apache.hcatalog.pig.HCatStorer();
    2013-03-16 17:48:56,416 [main] ERROR org.apache.pig.tools.grunt.Grunt – ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
    Details at logfile: C:\HadoopBeta\logs\hadoop\pig_1363481326651.log
    grunt>

    I am sure I am missing a Environment Variable to point to the HCat Jar files. But need more inputs

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #50784

    S R
    Participant

    I encountered the same issue and creating HCAT_HOME ENVIRONMENT variable pointing to c:\hdp\hcatalog-0.12.0.2.0.6.0-0009 solved the issue. However there is another Env Variable called HCATALOG_HOME which seems the duplicate.

    I am Using HDP 2.0 in Windows 2012 STD R2 Server.

    Thanks, Satya Raju

    Collapse
    #18161

    Larry Liu
    Moderator

    Hi, Vikas

    Here are steps that works for me:

    1. set up HCAT_HOME in environment variable to point to C:\HadoopBeta\hdp\hcatalog-0.4.1
    2. start pig using following command:
    pig -useHCatalog

    Hope this helps

    Thanks
    Larry

    Collapse
    #17889

    Vikas Dharia
    Member

    Hi Seth,

    I created an Env Variable Called PIG_Classpath and added the Jar file as part of the this variable that allowed me to get past the first error
    “013-03-16 17:48:56,416 [main] ERROR org.apache.pig.tools.grunt.Grunt – ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]”

    but now my Job is now erroring out with Error
    Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:66)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
    at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.Child.main(Child.java:265)

    Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:6

    Collapse
    #17888

    Seth Lyubich
    Keymaster

    Hi Vikas,

    You can try to start pig with pig.additional.jars option. In your case I believe it should be something like this:

    pig -Dpig.additional.jars=C:\HadoopBeta\hdp\hcatalog-0.4.1\share\hcatalog\hcatalog-0.4.1.jar

    Please let us know if this helps.

    Thanks,
    Seth

    Collapse
    #17735

    Vikas Dharia
    Member

    The Fix of Adding PIG_CLASSPATH got the job started but it fails and error is not very descriptive. Still there is a problem

    Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:66)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
    at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.Child.main(Child.java:265)

    Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:66)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
    at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:35

    Collapse
    #17714

    Vikas Dharia
    Member

    I fixed the issue:

    Create a Class Path Env Variable for Pig “PIG_CLASSPATH” with Value
    C:\HadoopBeta\hdp\hcatalog-0.4.1\share\hcatalog\hcatalog-0.4.1.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\hive-metastore-0.9.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\libthrift-0.7.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\hive-exec-0.9.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\libfb303-0.7.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\jdo2-api-2.3-ec.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\httpclient-4.2.2.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\slf4j-api-1.6.1.jar;C:\HadoopBeta\hdp\hive-0.9.0\conf\;C:\HadoopBeta\hdp\hadoop-1.1.0-SNAPSHOT\conf\

    Collapse
Viewing 6 replies - 1 through 6 (of 6 total)