HDP on Windows – Installation Forum

Error while running MR program in python

  • #43033
    Sorna Lingam
    Member

    HI all

    I have installed hadoop 1.3 msi on my machine and run the following MR program

    hadoop jar /HDP/hadoop-1.2.0.1.3.0.0-0380/contrib/streaming/hadoop-streaming-1.2.0.1.3.0.0-0380.jar \ -mapper “python C:\Python33\mapper.py” \ -reducer “python C:\Python33\redu.py” \ -input “C:\Python33\mm.txt” \ -output “C:\Python33\out.txt”

    but im getting the following error

    2013-11-06 11:13:12,933 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user sornalingam
    2013-11-06 11:13:13,177 INFO org.apache.hadoop.mapred.JobClient: Cleaning up the staging area hdfs://DEV144:8020/mapred/staging/sornalingam/.staging/job_201311061035_0010

    can any one help me what went wrong

    Thanks

    sornalingam

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #43166
    Sorna Lingam
    Member

    Hi

    I have updated the code

    hadoop jar /HDP/hadoop-1.2.0.1.3.0.0-0380/contrib/streaming/hadoop-streaming-1.2.0.1.3.0.0-0380.jar \ -mapper “python C:\Python33\map.cmd” \ -reducer “python C:\Python33\reducer.cmd” \ -input “/user/sornalingam/input/mm.txt” \ -output “/user/sornalingam/output”

    and my log containg

    2013-11-07 11:17:48,872 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
    2013-11-07 11:17:48,873 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
    2013-11-07 11:17:48,881 INFO org.apache.hadoop.mapred.FileInputFormat: Total input paths to process : 5
    2013-11-07 11:17:48,881 INFO org.apache.hadoop.mapred.JobClient: Cleaning up the staging area hdfs://DEV144:8020/mapred/staging/sornalingam/.staging/job_201311071036_0004
    2013-11-07 11:17:48,882 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:sornalingam cause:java.io.IOException: Not a file: hdfs://DEV144:8020/apps
    2013-11-07 11:17:48,883 ERROR org.apache.hadoop.streaming.StreamJob: Error Launching job : Not a file: hdfs://DEV144:8020/apps

    what went wrong

    #43592
    Seth Lyubich
    Moderator

    Hi Sorna,

    Can you please make sure that you can execute other jobs as user sornalingam. Can you also try by specifying full hdfs path -hdfs://……//user/sornalingam/input/mm.txt, or testing on local file system?

    Hope this helps,

    Thanks,
    Seth

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.