Home Forums Oozie Shell action through Oozie

This topic contains 8 replies, has 4 voices, and was last updated by  tedr 1 year, 6 months ago.

  • Creator
    Topic
  • #25027

    Tanzir
    Participant

    I’m here again!! :(

    I’m trying to execute a shell script through Oozie(sandbox) but I’m having some issues. I have a property file like this (import.properties):


    startIndex=2000
    chunkSize=2000

    The idea is, in every single execution the startIndex value will be updated by the chunk size. So if I execute it, it should have


    startIndex=4000
    chunkSize=2000

    I have tested the script separately and it works fine. Here are my other related files.

    job.properties


    nameNode=hdfs://192.168.56.101:8020
    jobTracker=192.168.56.101:50300
    wfeRoot=wfe
    queueName=default
    EXEC=script.sh
    propertyLoc=import.properties

    oozie.use.system.libpath=true
    oozie.wf.application.path=${nameNode}/user/${user.name}/${wfeRoot}/coordinator

    script.sh

    #!/bin/sh
    file=import.properties
    . $file

    SCRIPT=$(readlink -f $file)
    SCRIPTPATH=$(dirname $SCRIPT)
    echo $SCRIPTPATH

    newStartIndex=`expr $chunkSize + $startIndex`
    newStartIndexStr=startIndex=$newStartIndex

    oldStartIndexStr=startIndex=$startIndex
    chunkSizeStr=chunkSize=$chunkSize

    sed -i “s|$oldStartIndexStr|$newStartIndexStr|g” $file

    And I put all these files inside my HDFS working directory:


    [ambari_qa@sandbox coordinator]$ hadoop fs -lsr /user/ambari_qa/wfe/coordinator
    -rw-rw-rw- 1 ambari_qa hdfs 32 2013-05-09 00:12 /user/ambari_qa/wfe/coordinator/import.properties
    -rw-rw-rw- 1 ambari_qa hdfs 533 2013-05-09 01:19 /user/ambari_qa/wfe/coordinator/script.sh
    -rw------- 1 ambari_qa hdfs 852 2013-05-09 00:50 /user/ambari_qa/wfe/coordinator/workflow.xml

    I was expecting that the import.properties file will be changed after each execution. But I see it’s not changing even though the oozie job is successful. What I need to do so that it effects the working directory of HDFS? Thanks in advance.

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #25423

    tedr
    Member

    Hi Tanzir,

    Thanks for letting us know that you have solved the issue.

    Happy Hadooping!
    Ted.

    Collapse
    #25171

    Tanzir
    Participant

    Hi Larry,
    Thanks a lot for your response. I have resolved it. I didn’t know that when I place an oozie job from ambari client (ambari_qa as user), behind the scene a mapred user is executing that script. mapred user didn’t have write permissions to /user/ambari_qa folder (File did have write permissions to all users). After giving write permissions on that folder, it worked fine. Thanks again.

    Collapse
    #25141

    Larry Liu
    Moderator

    Hi, Tanzir

    What if you change the permission of the files?

    Larry

    Collapse
    #25137

    Tanzir
    Participant

    Hi Ted,
    Please look at post #25028, I mentioned it earlier that the script is invoking as expected. Looks like its changing a cache version of the file, not the exact file. And when I pass absolute path of the file which I want to change by script, then I’m getting permission error.

    rm: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode=”ambari_qa”:ambari_qa:hdfs:rwxrwx—
    put: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=EXECUTE, inode=”ambari_qa”:ambari_qa:hdfs:rwxrwx—

    Thanks again.

    Collapse
    #25063

    tedr
    Moderator

    Hi Tanzir,

    It may help to put some debugging lines in your script to make sure that it is being executes and which file it’s changing.

    Thanks,
    Ted.

    Collapse
    #25031

    Tanzir
    Participant

    Hi Ted,
    Thanks a lot for your response. I just tried with that but no changes. The files is not changing.

    From the syslog, it looks like right after I placed the job, corresponding files were moved to another location. So it’s changing there, not in the application path which I provided through oozie.

    syslog

    2013-05-09 12:11:56,822 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
    2013-05-09 12:11:57,353 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /hadoop/mapred/taskTracker/ambari_qa/distcache/2526702131827093121_-1976872463_168541270/sandbox/user/ambari_qa/wfe/coordinator/script.sh <- /hadoop/mapred/taskTracker/ambari_qa/jobcache/job_201305091117_0002/attempt_201305091117_0002_m_000000_0/work/script.sh
    2013-05-09 12:11:57,618 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /hadoop/mapred/taskTracker/ambari_qa/distcache/-6889195246335814631_889271025_125659265/sandbox/user/ambari_qa/wfe/coordinator/import.properties <- /hadoop/mapred/taskTracker/ambari_qa/jobcache/job_201305091117_0002/attempt_201305091117_0002_m_000000_0/work/import.properties
    2013-05-09 12:12:15,917 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
    2013-05-09 12:12:16,580 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@72f864fe
    2013-05-09 12:12:18,547 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
    2013-05-09 12:12:18,547 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
    2013-05-09 12:12:18,706 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
    2013-05-09 12:12:26,927 INFO org.apache.hadoop.mapred.Task: Task:attempt_201305091117_0002_m_000000_0 is done. And is in the process of commiting
    2013-05-09 12:12:28,588 INFO org.apache.hadoop.mapred.Task: Task attempt_201305091117_0002_m_000000_0 is allowed to commit now
    2013-05-09 12:12:30,063 INFO org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task 'attempt_201305091117_0002_m_000000_0' to hdfs://sandbox:8020/user/ambari_qa/oozie-oozi/0000001-130509111704292-oozie-oozi-W/shell1–shell/output
    2013-05-09 12:12:30,112 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201305091117_0002_m_000000_0' done.
    2013-05-09 12:12:30,177 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
    2013-05-09 12:12:30,799 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
    2013-05-09 12:12:31,088 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName mapred for UID 499 from the native implementation

    I thought that’s the temporary location and once the job is done the file will be copied to its original location.

    Collapse
    #25030

    tedr
    Moderator

    Hi Tanzir,

    Can you try it with these settings in job.properties:

    nameNode=hdfs://sandbox:8020
    jobTracker=sandbox:50300

    Thanks,
    Ted.

    Collapse
    #25028

    Tanzir
    Participant

    For debugging purpose, I printed out the location of the file during execution and found out that it copied to another location (from log):

    >>> Invoking Shell command line now >>

    Stdoutput /hadoop/mapred/taskTracker/ambari_qa/distcache/-5756672768810005023_889271025_125659265/192.168.56.101/user/ambari_qa/wfe/coordinator
    Stdoutput startIndex=4000
    Stdoutput startIndex=2000
    Exit code of the Shell command 0
    <<< Invocation of Shell command completed <<<

    Collapse
Viewing 8 replies - 1 through 8 (of 8 total)