Home Forums Sqoop Sqoop running by pig shell, is failing

Tagged: , ,

This topic contains 4 replies, has 2 voices, and was last updated by  tedr 1 year, 4 months ago.

  • Creator
    Topic
  • #28105

    Hi There,

    I have a pig script which includes an “sh” command which executes sqoop. The pig script works fine when executed using “pig -f script.pig” but
    fails when executed as part of an Oozie workflow.

    Versions of each component:

    Hadoop 1.2.0.1.3.0.0-107 (as part of HDP 1.2)
    Apache Pig version 0.11.1.1.3.0.0-107 (rexported)
    Sqoop 1.4.3.1.3.0.0-107
    Oozie client build version: 3.3.2.1.3.0.0-107

    We’ve been struglling with this issue and tried a few things (mostly with regards to classpaths of sqoop/pig/oozie) but none of them really helped.

    We also tried to play with log4j properties with no luck.

    The following exceptions are being logged to Oozie pig launcher map reduce job:

    The following exception is thrown by sqoop (sensitive data is obfuscated of course):
    Exception in thread “main” java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager
    at org.apache.log4j.Logger.getLogger(Logger.java:105)
    at org.apache.sqoop.util.LoggingUtils.setDebugLevel(LoggingUtils.java:50)
    at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:739)
    at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:722)
    at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:433)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:129)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

    Thanks in advance.

Viewing 4 replies - 1 through 4 (of 4 total)

The topic ‘Sqoop running by pig shell, is failing’ is closed to new replies.

  • Author
    Replies
  • #29567

    tedr
    Moderator

    Hi Moty,

    It looks like the log4j jars aren’t getting put on the classpath for sqoop when launched via oozie. But these jars are on the classpath of the system when launched as a pig script outside of oozie. We need to figure out how to make sure that they are on the classpath for pig in the oozie job config.

    thanks,
    Ted.

    Collapse
    #29470

    Hi Ted,

    The logs were part of Pig logs. Sqoop logs didn’t mention anything special (probably due to the fact it didn’t initialise its logger facade).

    Hope this helps.

    Best,
    Moty

    Collapse
    #29435

    tedr
    Moderator

    Hi Moty,

    Are these exceptions in the sqoop and pig logs, or are they in the console?

    Thanks,
    Ted.

    Collapse
    #28106

    And the following exception is thrown by Pig:
    java.io.IOException: java.io.IOException: sh command ‘sqoop import –connect jdbc:mysql://* –driver com.mysql.jdbc.Driver –username * –password * –as-textfile –fields-terminated-by , –enclosed-by “\”” –compress –table “events” –target-dir /user/tmp/sqoop-test.gz –input-null-string “null” –input-null-non-string “” –verbose –columns “id”‘ failed. Please check output logs for details
    at org.apache.pig.tools.grunt.GruntParser.processShCommand(GruntParser.java:1092)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:175)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:475)
    at org.apache.pig.PigRunner.run(PigRunner.java:49)
    at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
    at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
    at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
    Caused by: java.io.IOException: sh command ‘sqoop import –connect jdbc:mysql://* –driver com.mysql.jdbc.Driver –username * –password * –as-textfile –fields-terminated-by , –enclosed-by “\”” –compress –table “events” –target-dir /user/tmp/sqoop-test.gz –input-null-string “null” –input-null-non-string “” –verbose –columns “id”‘ failed. Please check output logs for details
    at org.apache.pig.tools.grunt.GruntParser.processShCommand(GruntParser.java:1088)
    … 23 more
    ================================================================================
    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)