Home Forums HDP on Windows – Installation hadoop streaming error

This topic contains 1 reply, has 2 voices, and was last updated by  Seth Lyubich 1 year ago.

  • Creator
    Topic
  • #33751

    Taehyeon Oh
    Participant

    Counting the number of words in the text data is MapReduce.
    However, the following error appears.
    I do not know what the problem is.

    console
    C:\hdp\hadoop>
    hadoop jar C:\hdp\hadoop\hadoop-1.2.0.1.3.0.0-0380\contrib\streaming\hadoop-streaming-1.2.0.1.3.0.0-0380.jar -files “hdfs:///user/Administrator/abc/map.exe,hdfs:///user/Administrator/abc/reduce.exe” -input “/user/Administrator/abc/pic” -output “/user/Administrator/abc/result” -mapper “map.exe” -reducer “reduce.exe”
    packageJobJar: [] [/C:/hdp/hadoop/hadoop-1.2.0.1.3.0.0-0380/lib/hadoop-streaming.jar] C:\Users\ADMINI~1\AppData\Local\Temp\streamjob7824443013113408616.jar tmpDir=null
    .
    .
    .
    13/08/29 00:10:50 INFO streaming.StreamJob: Tracking URL: http://master1:50030/jobdetails.jsp?jobid=job_201308200211_0010
    13/08/29 00:10:51 INFO streaming.StreamJob: map 0% reduce 0%
    13/08/29 00:11:24 INFO streaming.StreamJob: map 100% reduce 100%
    13/08/29 00:11:24 INFO streaming.StreamJob: To kill this job, run:13/08/29 00:11:24 INFO streaming.StreamJob: C:\hdp\hadoop\hadoop-1.2.0.1.3.0.0-0
    380/bin/hadoop job -Dmapred.job.tracker=master1:50300 -kill job_201308200211_0010
    13/08/29 00:11:24 INFO streaming.StreamJob: Tracking URL: http://master1:50030/jobdetails.jsp?jobid=job_201308200211_0010
    13/08/29 00:11:24 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201308200211_0010_m_000000
    13/08/29 00:11:24 INFO streaming.StreamJob: killJob…
    Streaming Command Failed!

    syslog logs
    .
    .
    .
    2013-08-29 00:10:58,664 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [C:\hdp\data\hdfs\mapred\local\taskTracker\Administrator\jobcache\job_201308200211_0010\attempt_201308200211_0010_m_000000_0\work\.\map.exe]
    .
    .
    2013-08-29 00:10:58,773 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]
    2013-08-29 00:10:58,789 INFO org.apache.hadoop.streaming.PipeMapRed: Records R/W=1673/1
    2013-08-29 00:10:58,789 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done
    2013-08-29 00:10:58,789 INFO org.apache.hadoop.streaming.PipeMapRed: log:null
    R/W/S=1673/59/0 in:NA [rec/s] out:NA [rec/s]
    minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
    HOST=null
    USER=”Administrator”
    HADOOP_USER=null
    last Hadoop input: |null|
    last tool output: |involved 1 1 1|
    Date: Thu Aug 29 00:10:58 PDT 2013
    java.io.IOException: The pipe has been ended
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:282)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    .
    .
    .

    2013-08-29 00:10:58,789 WARN org.apache.hadoop.streaming.PipeMapRed: java.io.IOException: The pipe is being closed
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:282)
    .
    .

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #35049

    Seth Lyubich
    Keymaster

    Hi Taehyeon,

    Looks like the job failed due to:

    Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201308200211_0010_m_000000

    You can try checking in directory below to see if you can get more hints on what went wrong with the attempt.

    C:\hdp\data\hdfs\mapred\local\taskTracker\Administrator\jobcache\job_201308200211_0010\attempt_201308200211_0010_m_000000_0\work\

    Hope this helps,

    Thanks,
    Seth

    Collapse
Viewing 1 replies (of 1 total)