Home Forums Oozie oozie configuration

This topic contains 21 replies, has 5 voices, and was last updated by  tedr 1 year, 4 months ago.

  • Creator
    Topic
  • #21764

    Evan Cutler
    Member

    greetings, I am using the sandbox.
    I am trying to start OOZIE jobs.. but they are failing…
    I’m noticing the HDP configuration for OOZIE’s connection to the jobtracker as sandbox:50300…
    I am not thinking that’s correct.

    where do I go and reset it to 50030? Thanks.

Viewing 21 replies - 1 through 21 (of 21 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #22120

    tedr
    Member

    Hi Evan,

    Thanks for continuing to work with the Sandbox.

    Have you tried to make sure this is an oozie problem and not a sqoop problem? Do workflows created for other types of jobs work?

    Thanks,
    Ted.

    Collapse
    #22090

    Evan Cutler
    Member

    HI…ok…
    When you work with the sandbox, you have about 8 example oozie workflows.
    The configuration is as the sandbox comes when you download it…I have not changed it.

    Here are the steps I take:
    1. I hit create
    2. I hit add step->sqoop
    3. I give sqoop a name, and enter the commands in the args positions
    4. I hit save. The screen comes back.
    5. I repeat step 2 and 3.
    6. I hit save, and I get the error. Then I notice the whole workflow disappears, along with the parent folder.
    7. I have to delete workflow and start again.

    I have tried also manually creating the XML in full, but there is no import option for creating workflows. Or at least that I’ve seen.

    Collapse
    #22058

    Seth Lyubich
    Keymaster

    Hi Evan,

    Can you please provide more details on what happened with the second issue? Can you please share configuration details?

    Thanks,
    Seth

    Collapse
    #22041

    Evan Cutler
    Member

    ok…
    I got the job to work.
    I broke the command into the args section as parts, and the first sqoop ran perfect.

    Now I have another problem.
    As soon as I add a second sqoop, I get a
    “IndexError at /oozie/edit_workflow/” list index out of range error.
    I don’t know why…
    I’ve added, deleted, copied, erased…tried everything to get more than one sqoop on this going.
    Sometimes, The first sqoop gives me this error…
    as soon as that happens the whole workflow disappears and I have to start over.

    Almost there.
    thanks so much.

    Collapse
    #22040

    Yi Zhang
    Moderator

    Hi Evan,

    Yes I meant user ooize’s home directory in HDFS. Make sure the jar there has the same permissions as other jars.
    The job tracker’s task logs would be helpful.

    http://sandbox-ip-addr:50030/ is the jobtracker web UI.

    Thanks,
    Yi

    Collapse
    #22038

    Evan Cutler
    Member

    ok Yi, figured out you meant to go on the folder in HDFS.

    Got them in there….jobs are still failing….
    Still looking.

    Collapse
    #22011

    Evan Cutler
    Member

    Sorry Yi…
    same result.
    Am I missing something else?
    Evan

    Collapse
    #22010

    Evan Cutler
    Member

    Hi Yi…
    I can’t find that directory….I have it right now in:
    /usr/lib/sqoop/lib/ojdbc6.jar

    I found these directories:
    /usr/lib/oozie/lib
    /usr/lib/oozie-server/lib

    I am placing it in those two spots…more to follow.

    Collapse
    #22007

    Yi Zhang
    Moderator

    Hi Evan,

    You need to put your odbc jar file in oozie’s share lib /user/oozie/share/lib/sqoop/. You probably want it in other sharelib directories for pig/hive as well.

    Thanks,
    Yi

    Collapse
    #21987

    Evan Cutler
    Member

    Thank you so much guys.
    I appreciate the time.
    I really want to get this to work.
    This is really cutting edge stuff.

    Thanks again,
    Evan

    Collapse
    #21910

    Yi Zhang
    Moderator

    Hi Evan,

    I see the same problem here. It looks like a oozie config issue. Will get back to you once figure this out.

    Thanks,
    Yi

    Collapse
    #21873

    Evan Cutler
    Member

    Thanks Larry, most appreciated.

    Collapse
    #21872

    Larry Liu
    Moderator

    Hi, Evan

    If you put sqoop in the command, it works. I am not sure if this is how it should work. Let me get back to you once I find the answer.

    larry

    Collapse
    #21871

    Evan Cutler
    Member

    and please forgive my spelling…new keyboard.

    Collapse
    #21869

    Evan Cutler
    Member

    oh, and I forgot to mention, I did download the oracle thin client jar and put into the sqoop libraries…it works that way in teh command line.

    Thanks fuch.

    Collapse
    #21868

    Evan Cutler
    Member

    I clicked on oozie on the top menu….got Workflow Editor
    I looked at the Sqoop example.

    I clicked on CREATE.
    I clicked on +Sqoop.
    gave it a name: QBC_Truncate
    put in command section:
    eval –connect jdbc:oracle:thin:Earthquake/rogue2@//10.10.137.189:1521/dataminion –query “truncate table quake_by_city”
    no parameters or any other values.
    Tried to run.
    Got error.
    What else do I do?

    Thanks very much.

    Collapse
    #21867

    Larry Liu
    Moderator

    Hi, Evan

    How did you run the sqoop command in oozie?

    Can you please provide all the information including job.properties, workflow.xml and script?

    Thanks
    Larry

    Collapse
    #21863

    Evan Cutler
    Member

    I wish I could….
    please forgive…I am a complete noob here..

    ok, I downloaded the VM for VMWARE.
    downloaded the player.
    opened the web front end
    started hue.
    opened a putty screen.
    loaded a file into hdfs
    created a pig job to do something.
    took pig output and registered it in HCATALOG.
    performed select against it in HIVE.

    All good.
    now comes oozie.

    I created a table in a local oracle database. downloaded odbc:thin jars and placed in sqoop lib.
    created sqoop string to load pig output into database. perfect.
    truncated table.
    moved command string into oozie into block that says paste complete command.
    tried to execute. failed.
    looked at example sqoop command. removed word “sqoop” from command. left with:
    eval –connect jdbc:oracle:thin:Earthquake/rogue2@//10.10.137.209:1521/dataminion –query “truncate table quake_by_city”

    if I added sqoop back and pasted line into putty, works perfect.
    pasted line without “sqoop” into oozie and hit run. fails with error: Action failed, error message[Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]]

    looked at logs…map, setup and cleanup says succeeded. I don’t know where to go from there.
    I also tried pig jobs. same issue. Am I missing something?

    Where do I go to troubleshoot from there.
    Thanks.

    Collapse
    #21861

    Larry Liu
    Moderator

    Hi, Evan

    What did you do to start oozie? Can you please provide more details?

    By the way, port 50300 is the job tracker RPC port. HDP uses this port for JobTracker.

    Larry

    Collapse
    #21859

    Evan Cutler
    Member

    hi yi,
    thanks for responding….
    I was responding to how oozie was showing reports.

    The reports show:
    oozie.use.system.libpath true
    nameNode hdfs://sandbox:8020
    hue-id-w 23
    jobTracker sandbox:50300
    oozie.wf.application.path hdfs://sandbox:8020/user/hue/oozie/workspaces/_sandbox_-oozie-23

    I didn’t think that was correct because the jobtracker has been sandbox:50030. I don’t know where to confirm the 50300 entry. I’m good iwth it if I can get oozie to start.

    oozie won’t execute anything. pig, sqoop, nothing.

    for example, these sqoop commands work in a putty screen:
    sqoop eval –connect jdbc:oracle:thin:Earthquake/rogue2@//10.10.137.209:1521/dataminion –query “truncate table quake_by_city”
    sqoop export –connect jdbc:oracle:thin:Earthquake/rogue2@//10.10.137.209:1521/dataminion –table quake_by_city –export-dir /apps/hive/warehouse/quake_by_city –input-fields-terminated-by ‘|’ –columns city,state,mag_avg,mag_cnt

    but for some reason they are not exporting in oozie. Am I doing this wrong or something?
    Thanks.

    Collapse
    #21807

    Yi Zhang
    Moderator

    Hi Evan,

    Oozie communicates with jobtracker through ipc calls. The proper port is what is configured for mapred.job.tracker. If that is set to 50300, then it is the right port.

    50030 is the http port by default and is not used by oozie.

    Any thing in oozie logs?

    Thanks,
    Yi.

    Collapse
Viewing 21 replies - 1 through 21 (of 21 total)