Home Forums HDP on Windows – Installation Error while running MR program in python

Tagged: 

This topic contains 2 replies, has 2 voices, and was last updated by  Seth Lyubich 8 months, 2 weeks ago.

  • Creator
    Topic
  • #43033

    Sorna Lingam
    Member

    HI all

    I have installed hadoop 1.3 msi on my machine and run the following MR program

    hadoop jar /HDP/hadoop-1.2.0.1.3.0.0-0380/contrib/streaming/hadoop-streaming-1.2.0.1.3.0.0-0380.jar \ -mapper “python C:\Python33\mapper.py” \ -reducer “python C:\Python33\redu.py” \ -input “C:\Python33\mm.txt” \ -output “C:\Python33\out.txt”

    but im getting the following error

    2013-11-06 11:13:12,933 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user sornalingam
    2013-11-06 11:13:13,177 INFO org.apache.hadoop.mapred.JobClient: Cleaning up the staging area hdfs://DEV144:8020/mapred/staging/sornalingam/.staging/job_201311061035_0010

    can any one help me what went wrong

    Thanks

    sornalingam

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #43592

    Seth Lyubich
    Keymaster

    Hi Sorna,

    Can you please make sure that you can execute other jobs as user sornalingam. Can you also try by specifying full hdfs path -hdfs://……//user/sornalingam/input/mm.txt, or testing on local file system?

    Hope this helps,

    Thanks,
    Seth

    Collapse
    #43166

    Sorna Lingam
    Member

    Hi

    I have updated the code

    hadoop jar /HDP/hadoop-1.2.0.1.3.0.0-0380/contrib/streaming/hadoop-streaming-1.2.0.1.3.0.0-0380.jar \ -mapper “python C:\Python33\map.cmd” \ -reducer “python C:\Python33\reducer.cmd” \ -input “/user/sornalingam/input/mm.txt” \ -output “/user/sornalingam/output”

    and my log containg

    2013-11-07 11:17:48,872 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
    2013-11-07 11:17:48,873 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
    2013-11-07 11:17:48,881 INFO org.apache.hadoop.mapred.FileInputFormat: Total input paths to process : 5
    2013-11-07 11:17:48,881 INFO org.apache.hadoop.mapred.JobClient: Cleaning up the staging area hdfs://DEV144:8020/mapred/staging/sornalingam/.staging/job_201311071036_0004
    2013-11-07 11:17:48,882 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:sornalingam cause:java.io.IOException: Not a file: hdfs://DEV144:8020/apps
    2013-11-07 11:17:48,883 ERROR org.apache.hadoop.streaming.StreamJob: Error Launching job : Not a file: hdfs://DEV144:8020/apps

    what went wrong

    Collapse
Viewing 2 replies - 1 through 2 (of 2 total)