The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Oozie Forum

Shell action through Oozie

  • #25027

    I’m here again!! :(

    I’m trying to execute a shell script through Oozie(sandbox) but I’m having some issues. I have a property file like this (


    The idea is, in every single execution the startIndex value will be updated by the chunk size. So if I execute it, it should have


    I have tested the script separately and it works fine. Here are my other related files.



    . $file

    SCRIPT=$(readlink -f $file)
    SCRIPTPATH=$(dirname $SCRIPT)
    echo $SCRIPTPATH

    newStartIndex=`expr $chunkSize + $startIndex`


    sed -i “s|$oldStartIndexStr|$newStartIndexStr|g” $file

    And I put all these files inside my HDFS working directory:

    [ambari_qa@sandbox coordinator]$ hadoop fs -lsr /user/ambari_qa/wfe/coordinator
    -rw-rw-rw- 1 ambari_qa hdfs 32 2013-05-09 00:12 /user/ambari_qa/wfe/coordinator/
    -rw-rw-rw- 1 ambari_qa hdfs 533 2013-05-09 01:19 /user/ambari_qa/wfe/coordinator/
    -rw------- 1 ambari_qa hdfs 852 2013-05-09 00:50 /user/ambari_qa/wfe/coordinator/workflow.xml

    I was expecting that the file will be changed after each execution. But I see it’s not changing even though the oozie job is successful. What I need to do so that it effects the working directory of HDFS? Thanks in advance.

  • Author
  • #25028

    For debugging purpose, I printed out the location of the file during execution and found out that it copied to another location (from log):

    >>> Invoking Shell command line now >>

    Stdoutput /hadoop/mapred/taskTracker/ambari_qa/distcache/-5756672768810005023_889271025_125659265/
    Stdoutput startIndex=4000
    Stdoutput startIndex=2000
    Exit code of the Shell command 0
    <<< Invocation of Shell command completed <<<


    Hi Tanzir,

    Can you try it with these settings in




    Hi Ted,
    Thanks a lot for your response. I just tried with that but no changes. The files is not changing.

    From the syslog, it looks like right after I placed the job, corresponding files were moved to another location. So it’s changing there, not in the application path which I provided through oozie.


    2013-05-09 12:11:56,822 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
    2013-05-09 12:11:57,353 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /hadoop/mapred/taskTracker/ambari_qa/distcache/2526702131827093121_-1976872463_168541270/sandbox/user/ambari_qa/wfe/coordinator/ <- /hadoop/mapred/taskTracker/ambari_qa/jobcache/job_201305091117_0002/attempt_201305091117_0002_m_000000_0/work/
    2013-05-09 12:11:57,618 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /hadoop/mapred/taskTracker/ambari_qa/distcache/-6889195246335814631_889271025_125659265/sandbox/user/ambari_qa/wfe/coordinator/ <- /hadoop/mapred/taskTracker/ambari_qa/jobcache/job_201305091117_0002/attempt_201305091117_0002_m_000000_0/work/
    2013-05-09 12:12:15,917 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
    2013-05-09 12:12:16,580 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@72f864fe
    2013-05-09 12:12:18,547 WARN Snappy native library is available
    2013-05-09 12:12:18,547 INFO Snappy native library loaded
    2013-05-09 12:12:18,706 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
    2013-05-09 12:12:26,927 INFO org.apache.hadoop.mapred.Task: Task:attempt_201305091117_0002_m_000000_0 is done. And is in the process of commiting
    2013-05-09 12:12:28,588 INFO org.apache.hadoop.mapred.Task: Task attempt_201305091117_0002_m_000000_0 is allowed to commit now
    2013-05-09 12:12:30,063 INFO org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task 'attempt_201305091117_0002_m_000000_0' to hdfs://sandbox:8020/user/ambari_qa/oozie-oozi/0000001-130509111704292-oozie-oozi-W/shell1–shell/output
    2013-05-09 12:12:30,112 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201305091117_0002_m_000000_0' done.
    2013-05-09 12:12:30,177 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
    2013-05-09 12:12:30,799 INFO Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
    2013-05-09 12:12:31,088 INFO Got UserName mapred for UID 499 from the native implementation

    I thought that’s the temporary location and once the job is done the file will be copied to its original location.


    Hi Tanzir,

    It may help to put some debugging lines in your script to make sure that it is being executes and which file it’s changing.



    Hi Ted,
    Please look at post #25028, I mentioned it earlier that the script is invoking as expected. Looks like its changing a cache version of the file, not the exact file. And when I pass absolute path of the file which I want to change by script, then I’m getting permission error.

    rm: Permission denied: user=mapred, access=EXECUTE, inode=”ambari_qa”:ambari_qa:hdfs:rwxrwx—
    put: Permission denied: user=mapred, access=EXECUTE, inode=”ambari_qa”:ambari_qa:hdfs:rwxrwx—

    Thanks again.

    Larry Liu

    Hi, Tanzir

    What if you change the permission of the files?



    Hi Larry,
    Thanks a lot for your response. I have resolved it. I didn’t know that when I place an oozie job from ambari client (ambari_qa as user), behind the scene a mapred user is executing that script. mapred user didn’t have write permissions to /user/ambari_qa folder (File did have write permissions to all users). After giving write permissions on that folder, it worked fine. Thanks again.


    Hi Tanzir,

    Thanks for letting us know that you have solved the issue.

    Happy Hadooping!

The forum ‘Oozie’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.