Home Forums HDP on Linux – Installation HDP2.1 – manual instalation – job stays unassigned

This topic contains 1 reply, has 1 voice, and was last updated by  Amin Amirali 1 month, 3 weeks ago.

  • Creator
    Topic
  • #57613

    Amin Amirali
    Participant

    Hi all,

    I am new to this forum and only recently did I start configuring my own cluster.

    I am following the manual installation instructions available here:

    http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_installing_manually_book/content/rpm-chap1.html

    I have 1 master and 2 slaves/data nodes running as CentOS virtual images, all of which have 100 GB disk, and 2 GB RAM.

    I am stuck in the smoke test of mapreduce (5.4). I start the command to generate some random data, the job gets created and accepted, but remains unassigned. I looked into the Namenode information and I see 2 live nodes.

    I have absolutely no clue where to start looking. I googled it up, some people seem to say that it could be related to configurations (if I configured some files to have too much memory, the task manager will keep aborting due to lack of resources), I added some lines to conf/mapred-site.xml, but no change:

    <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx512m</value>
    </property>

    <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx1024m</value>
    </property>

    Finally, here is what I see on my screen:
    [client@master41 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.4.0.2.1.3.0-563.jar teragen 10000 tmp/teragenout
    14/07/22 16:20:27 INFO client.RMProxy: Connecting to ResourceManager at master41/192.168.100.41:8050
    14/07/22 16:20:27 INFO terasort.TeraSort: Generating 10000 using 2
    14/07/22 16:20:27 INFO mapreduce.JobSubmitter: number of splits:2
    14/07/22 16:20:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406038152966_0001
    14/07/22 16:20:28 WARN mapred.YARNRunner: Usage of -Djava.library.path in mapreduce.admin.map.child.java.opts can cause programs to no longer function if hadoop native libraries are used. These values should be set as part of the LD_LIBRARY_PATH in the map JVM env using mapreduce.admin.user.env config settings.
    14/07/22 16:20:28 WARN mapred.YARNRunner: Usage of -Djava.library.path in mapreduce.admin.reduce.child.java.opts can cause programs to no longer function if hadoop native libraries are used. These values should be set as part of the LD_LIBRARY_PATH in the reduce JVM env using mapreduce.admin.user.env config settings.
    14/07/22 16:20:28 INFO impl.YarnClientImpl: Submitted application application_1406038152966_0001
    14/07/22 16:20:28 INFO mapreduce.Job: The url to track the job: http://master41:8088/proxy/application_1406038152966_0001/
    14/07/22 16:20:28 INFO mapreduce.Job: Running job: job_1406038152966_0001

    Any hints to where I should look at?

    Thanks!
    Amin

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

Viewing 1 replies (of 1 total)