Grunt Shell Cannot Resolve HCatStorer

Hi,

I was attempting to cross refer the metadata in HCat and I was attempting a Load Thru Pig. I am getting an Error

grunt> Store A into ‘my_data’ using org.apache.hcatalog.pig.HCatStorer();
2013-03-16 17:48:56,416 [main] ERROR org.apache.pig.tools.grunt.Grunt – ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: C:\HadoopBeta\logs\hadoop\pig_1363481326651.log
grunt>

I am sure I am missing a Environment Variable to point to the HCat Jar files. But need more inputs

Topic

[Larry LiuModerator]

Hi, Vikas

Here are steps that works for me:

1. set up HCAT_HOME in environment variable to point to C:\HadoopBeta\hdp\hcatalog-0.4.1
2. start pig using following command:
pig -useHCatalog

Hope this helps

Thanks
Larry

[Vikas DhariaMember]

Hi Seth,

I created an Env Variable Called PIG_Classpath and added the Jar file as part of the this variable that allowed me to get past the first error
“013-03-16 17:48:56,416 [main] ERROR org.apache.pig.tools.grunt.Grunt – ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]”

but now my Job is now erroring out with Error
Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:66)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)

Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:6

[SefMember]

Hi Vikas,

You can try to start pig with pig.additional.jars option. In your case I believe it should be something like this:

pig -Dpig.additional.jars=C:\HadoopBeta\hdp\hcatalog-0.4.1\share\hcatalog\hcatalog-0.4.1.jar

Please let us know if this helps.

Thanks,
Seth

[Vikas DhariaMember]

The Fix of Adding PIG_CLASSPATH got the job started but it fails and error is not very descriptive. Still there is a problem

Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:66)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)

Error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.hcatalog.pig.HCatStorer.setStoreLocation(HCatStorer.java:89)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setLocation(PigOutputFormat.java:170)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.setUpContext(PigOutputCommitter.java:112)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:86)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.(PigOutputCommitter.java:66)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:35

[Vikas DhariaMember]

I fixed the issue:

Create a Class Path Env Variable for Pig “PIG_CLASSPATH” with Value
C:\HadoopBeta\hdp\hcatalog-0.4.1\share\hcatalog\hcatalog-0.4.1.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\hive-metastore-0.9.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\libthrift-0.7.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\hive-exec-0.9.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\libfb303-0.7.0.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\jdo2-api-2.3-ec.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\httpclient-4.2.2.jar;C:\HadoopBeta\hdp\hive-0.9.0\lib\slf4j-api-1.6.1.jar;C:\HadoopBeta\hdp\hive-0.9.0\conf\;C:\HadoopBeta\hdp\hadoop-1.1.0-SNAPSHOT\conf\

[Author]

Replies