MapReduce Forum

Simple MR job fails on HDP2.1 sandbox

  • #53742
    Param Dutta
    Participant

    Hi,

    I am unable to get simple word_count to run on HDP2.1 sandbox. I get the exception attached at the bottom. On looking at the container logs, I see:
    ” Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster ”

    I verified that the classpath is correct. “yarn.application.classpath” in the job configuration looks like this:
    yarn.application.classpath = /etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*

    I am not setting “yarn.application.classpath”, in my client application. It is automatically set and looks good to me. The class is in /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.4.0.2.1.1.0-385.jar.

    The stack in my client log:

    2014-05-15 10:28:15,121 INFO [Job] Job job_1400096640605_0002 failed with state FAILED due to: Application application_1400096640605_0002 failed 2 times due to AM Container for appattempt_1400096640605_0002_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
    org.apache.hadoop.util.Shell$ExitCodeException:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
    at org.apache.hadoop.util.Shell.run(Shell.java:418)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

    Thanks for any help with this.

    Best,
    Param

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #71539
    Jaffer Jamil
    Participant

    Param did you ever get to resolve this issue? I am experiencing this my Sandbox 2.2 when I do the tutorials.

    #71756
    Chris Schrader
    Participant

    I am also having the same error on an HDP 2.2 that was upgraded from 2.1 running in AWS (Redhat OS). MR, Hive, YARN, etc all worked fine before the upgrade.

    The exact error in the YARN logs is: Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

    I found this thread that mentioned the YARN classpath issue: http://hortonworks.com/community/forums/topic/strange-mapred-site-xml-after-installing-of-hdp2-2/

    And I did notice that my YARN classpath was missing the paths to map-reduce client libs. I added those in. It now looks like:

    /etc/hadoop/conf,$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/*,/usr/hdp/current/hadoop-mapreduce-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-app-2.6.0.2.2.0.0-2041.jar

    The last entry is actually an exact path to the class its looking for. Restarted MR, YARN, and Hive (I’m trying to run a hive query and getting this error), still no dice. Same error.

    Not sure if its related, but I had to fix another upgrade issue related to map-reduce classpath. I set the HADOOP_OPTS var to use the actual version as others have had issue with the variable. It is now like this in mapred_env.sh: export HADOOP_OPTS=”-Dhdp.version=2.2.0.0-2041 $HADOOP_OPTS”

    My mapreduce application classpath is now:

    $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,/usr/hdp/current/hadoop-client/mapreduce.tar.gz

    And my mapreduce application framework path is: /hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz

    I removed the #mr-framework because of errors saying that framework didn’t exist. This fixed that error.

    Any help is appreciated.

    #71791
    Chris Schrader
    Participant

    After re-checking several settings for mapred-site and yarn-site, and setting them according to the HWX documentation for a 2.1 to 2.2 upgrade (apparently we missed some steps), I am not getting a different error:

    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    … 13 more

    All searches still seems to point to a yarn.classpath problem. This is the most notable: http://www.srccodes.com/p/article/46/noclassdeffounderror-org-apache-hadoop-service-compositeservice-shell-exitcodeexception-classnotfoundexception

    But even after the map-reduce libs to the yarn classpath parameter, still getting same error.

    #71869
    Chris Schrader
    Participant

    One more interesting thing that I don’t know if it matters or not. The file hadoop-layout.sh references /usr/hdp/2.2.0.0-2041/ … while other variables get overwritten to /usr/hdp/current/…

    In some cases, I have env variables looking in different places, albeit with what appears to be correct configs from my ambari setup. IE, when I source hadoop-config.sh, I get the following:

    HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec
    HADOOP_PREFIX=/usr/hdp/2.2.0.0-2041/hadoop

    And then some class paths start to look like this:

    HADOOP_CLASSPATH=:/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.2.0.0-2041/tez/*:/usr/hdp/2.2.0.0-2041/tez/lib/*:/etc/tez/conf
    HADOOP_CONF_DIR=/usr/hdp/2.2.0.0-2041/hadoop/conf

    Those are all real directories with real libs and appear to be correct but not sure if that is contributing to the problem. I don’t see where/how ambari sets hadoop-layout.sh.

    #71993
    Chris Schrader
    Participant

    Ok I finally solved this problem.

    The issue was mapreduce.application.classpath. In the Ambari upgrade docs for going from 2.1 to 2.2, there’s a section for setting all the new classpaths. In particular:

    $PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: $PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: $PWD/mr-framework/hadoop/share/hadoop/common/*: $PWD/mr-framework/hadoop/share/hadoop/common/lib/*: $PWD/mr-framework/hadoop/share/hadoop/yarn/*: $PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: $PWD/mr-framework/hadoop/share/hadoop/hdfs/*: $PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure$

    I, like most people, copy and pasted that value into Ambari. All those paths are correct. The problem is on this site only, some of the classpaths are separate by a space after the colon. This was causing the error. Map-Reduce now runs correctly.

    This one was a HUGE pain in the ass and took a really long time to identify and fix. This page needs to be fixed so this property is correct (without the spaces). It would also be helpful if Hadoop had more meaningful error messages (instead of just generic Java class missing errors).

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.