Home Forums Hortonworks Sandbox Oozie-Sqoop job not working

Tagged: 

This topic contains 2 replies, has 1 voice, and was last updated by  Ajay Chitre 5 months, 2 weeks ago.

  • Creator
    Topic
  • #48673

    Ajay Chitre
    Participant

    When I run ‘Sqoop’ from command line it works fine, but when I run it via Oozie job it just keeps showing the following message:

    2014-02-12 19:04:47,854 DEBUG DefinitionXCommand:545 – USER[root] GROUP[-] TOKEN[] APP[sqoop] JOB[0000004-140211210850200-oozie-oozi-W] ACTION[-] Execute command [definition] key [0000004-140211210850200-oozie-oozi-W]

    My workflow.xml looks like this:

    <workflow-app name=”sqoop” xmlns=”uri:oozie:workflow:0.1″>
    <start to=”etl” />
    <action name=”etl”>
    <sqoop xmlns=”uri:oozie:sqoop-action:0.2″>
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
    <prepare>
    <delete path=”${outputBaseDir}/${tableName}/${period}” />
    </prepare>
    <configuration>
    <property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
    </property>
    </configuration>
    <arg>import</arg>
    <arg>–connect</arg>
    <arg>jdbc:mysql://${mysqlHost}/${mysqlDb}</arg>
    <arg>–username</arg>
    <arg>root</arg>
    <arg>–target-dir</arg>
    <arg>${outputBaseDir}/${tableName}/${period}</arg>
    <arg>–query</arg>
    <arg>${query}</arg>
    <arg>–split-by</arg>
    <arg>${splitBy}</arg>
    <arg>-m</arg>
    <arg>1</arg>
    </sqoop>
    <ok to=”end” />
    <error to=”fail” />
    </action>
    <kill name=”fail”>
    <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name=’end’ />
    </workflow-app>

    job.properties looks like this:

    nameNode=hdfs://10.0.2.15:8020
    jobTracker=10.0.2.15:8021
    queueName=default

    mysqlHost=10.0.2.15
    mysqlDb=test

    outputBaseDir=/user/ajay
    tableName=temperatures
    period=2014010200
    query=select *, DATE_FORMAT(created_time, ‘%Y%m%d%H’) as period from ${tableName} where DATE_FORMAT(created_time, ‘%Y%m%d%H’) = ${period} and \\$CONDITIONS
    splitBy=id

    oozie.wf.application.path=${nameNode}/user/ajay/workflow.xml
    outputDir=sqoop-output

    Started the job under Sandbox like this:
    oozie job -config ./job.properties -run

    The following works from command line:
    sqoop import –connect jdbc:mysql://10.0.2.15/test –username root –target-dir $OUTPUT_DIR –query “select *, DATE_FORMAT(created_time, ‘%Y%m%d%H’) as period from $TABLE_NAME where DATE_FORMAT(created_time, ‘%Y%m%d%H’) = $PERIOD and \$CONDITIONS” –split-by id

    What am I doing wrong? Please help. Thanks.

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #48738

    Ajay Chitre
    Participant

    Answering my question myself. May help someone in future. It worked when I did the following:

    1) ssh root@127.0.0.1 -p 2222 (Password: hadoop)
    2) ssh hue@sandbox (Password: hadoop)
    3) Used the following job.properties:

    user.name=hue
    nameNode=hdfs://sandbox.hortonworks.com:8020
    jobTracker=sandbox.hortonworks.com:8050
    queueName=default

    mysqlHost=10.0.2.15
    mysqlDb=test

    outputBaseDir=/user/hue/ajay
    tableName=temperatures
    period=2014010101
    query=select *, DATE_FORMAT(created_time, ‘%Y%m%d%H’) as period from ${tableName} where DATE_FORMAT(created_time, ‘%Y%m%d%H’) = ${period} and $CONDITIONS
    splitBy=id

    oozie.wf.application.path=${nameNode}/user/hue/ajay/workflows/workflow.xml
    oozie.use.system.libpath=true
    mapreduce.job.user.name=hue

    4) Added the following to workflow.xml
    <archive>/user/ajay/mysql-connector-java.jar#mysql-connector-java.jar</archive>

    So the workflow.xml looks like this:

    <workflow-app name=”sqoop” xmlns=”uri:oozie:workflow:0.1″>
    <start to=”etl” />
    <action name=”etl”>
    <sqoop xmlns=”uri:oozie:sqoop-action:0.2″>
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
    <prepare>
    <delete path=”${nameNode}/${outputBaseDir}/${tableName}/${period}” />
    </prepare>
    <configuration>
    <property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
    </property>
    </configuration>
    <arg>import</arg>
    <arg>–connect</arg>
    <arg>jdbc:mysql://${mysqlHost}/${mysqlDb}</arg>
    <arg>–username</arg>
    <arg>root</arg>
    <arg>–target-dir</arg>
    <arg>${nameNode}/${outputBaseDir}/${tableName}/${period}</arg>
    <arg>–query</arg>
    <arg>${query}</arg>
    <arg>–split-by</arg>
    <arg>${splitBy}</arg>
    <archive>/user/ajay/mysql-connector-java.jar#mysql-connector-java.jar</archive>
    </sqoop>
    <ok to=”end” />
    <error to=”fail” />
    </action>
    <kill name=”fail”>
    <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name=’end’ />
    </workflow-app>

    5) oozie -config job.properties -run

    Collapse
    #48674

    Ajay Chitre
    Participant

    Just to add some more info. The action remains in a ‘PREP’ status. Like this:

    Actions
    ————————————————————————————————————————————
    ID Status Ext ID Ext Status Err Code
    ————————————————————————————————————————————
    0000009-140211210850200-oozie-oozi-W@:start: OK – OK –
    ————————————————————————————————————————————
    0000009-140211210850200-oozie-oozi-W@etl PREP – – –
    ————————————————————————————————————————————

    Collapse
Viewing 2 replies - 1 through 2 (of 2 total)