Home Forums Pig pig udf classpath

This topic contains 10 replies, has 3 voices, and was last updated by  Sasha J 1 year ago.

  • Creator
    Topic
  • #29827

    Vinod K
    Participant

    Hello!
    I am trying to execute a java module from within a pig script and am having difficulty in locating details of an exception that seems to be occurring in the java code.

    None of the job log files seems to provide information about the actual exception.

    Here is what the log (just part) looks like from the pig UI

    2013-07-22 13:33:29,423 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1.1.3.0.0-107 (rexported) compiled May 20 2013, 03:04:35
    2013-07-22 13:33:29,423 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/mapred/taskTracker/hue/jobcache/job_201307221311_0002/attempt_201307221311_0002_m_000000_0/work/pig_1374525209421.log
    2013-07-22 13:33:29,667 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /usr/lib/hadoop/.pigbootup not found
    2013-07-22 13:33:29,829 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox:8020
    2013-07-22 13:33:29,974 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: sandbox:50300
    2013-07-22 13:33:30,335 [main] INFO org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
    2013-07-22 13:33:31,147 [main] WARN org.apache.hadoop.hive.conf.HiveConf - DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    2013-07-22 13:33:31,210 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox:9083
    2013-07-22 13:33:31,289 [main] INFO hive.metastore - Waiting 1 seconds before next connection attempt.
    2013-07-22 13:33:32,290 [main] INFO hive.metastore - Connected to metastore.
    2013-07-22 13:33:32,485 [main] WARN org.apache.hadoop.hive.conf.HiveConf - DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
    2013-07-22 13:33:32,521 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/uima/util/InvalidXMLException
    Details at logfile: /hadoop/mapred/taskTracker/hue/jobcache/job_201307221311_0002/attempt_201307221311_0002_m_000000_0/work/pig_1374525209421.log

    Any recommendation on where to look would be appreciated.

    Thanks.

Viewing 10 replies - 1 through 10 (of 10 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #30949

    Sasha J
    Moderator

    You should look at the job logs to find the reason.
    IN the listed file /hadoop/mapred/taskTracker/hue/jobcache/job_201307221311_0002/attempt_201307221311_0002_m_000000_0/work/pig_1374525209421.log
    or at the same location but for the different attempt.

    Sasha

    Collapse
    #30904

    Vinod K
    Participant

    Sasha, thanks for your response.

    Actually, I **incorrectly** said that I was using behemoth – in fact, I am *not* using behemoth for the error I am seeing. I am using just UIMA framework for Natural Language Processing.

    Thanks!

    Collapse
    #30834

    Sasha J
    Moderator

    Vinod,
    behemoth is not Hortonworks’s product, so it could not be supported in this forum.
    Please contact the authors and work with them on behemoth issues.

    Thank you!
    Sasha

    Collapse
    #30797

    Vinod K
    Participant

    I use behemoth – https://github.com/DigitalPebble/behemoth – to process data using UIMA framework. Behemoth supports UIMA annotators (in the form of a .pear file, which include resources, config files – also called descriptors, and java code also called annotators). Essentially, the framework heavily depends on XML config files (descriptors), looks like the runtime is complaining that one of the descriptors is invalid – it would be nice to get that stack trace – particularly because the module(s) I am trying to execute is/are complex.

    Thanks for your help.

    Collapse
    #30610

    Sasha J
    Moderator

    Error and stack trace should be in the mentioned log file:

    /hadoop/mapred/taskTracker/hue/jobcache/job_201307221311_0002/attempt_201307221311_0002_m_000000_0/work/pig_1374525209421.log

    If it not exist, check if there any other log files specific for the mentioned job.

    Could you give more details on jar file you built?

    Thank you!
    Sasha

    Collapse
    #30045

    Vinod K
    Participant

    I thought I should provide some more information on what I am trying to do -

    I have a java module that heavily uses XML files for configuration. I know the module works in regular environment (non-hadoop), however, once I try to invoke this from a pig script, something is going wrong (InvalidXMLException). More than likely the jar file that I have built has an issue, so, thought having access to the stack trace would be useful.

    Thanks.

    Collapse
    #30042

    Vinod K
    Participant

    Yeah, I did look there but there is no information about InvalidXMLException in that log when I grep for it.

    Thanks for your comments!

    Collapse
    #29970

    tedr
    Moderator

    Hi Vinod,

    Have you looked in the TaskTracker logs located at /var/log/hadoop/mapred/hadoop-mapred-tasktracker-sandbox.log?

    Thanks,
    Ted.

    Collapse
    #29918

    Vinod K
    Participant

    Thanks for your response Ted. I just noticed that although I had moved on from UDF classpath to the exception, I had forgotten to change the subject of the post.

    Anyway, yes, I understand that there is an issue with some XML file. I was wondering if there is a way to look at the stacktrace.

    Although the messages suggest I need to look in the log file with an absolute path, that path (Details at logfile: /hadoop/mapred/taskTracker/hue/jobcache/job_201307221311_0002/attempt_201307221311_0002_m_000000_0/work/pig_1374525209421.log) does not exist.

    Any suggestions?

    Collapse
    #29889

    tedr
    Moderator

    Hi Vinod,

    The InvalidXMLException points to the probability that one of your xml files either configs or input files is malformed. check over all of the configuration files and if you are using xml files for input check that they are all properly formatted.

    Thanks,
    Ted.

    Collapse
Viewing 10 replies - 1 through 10 (of 10 total)