The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

MapReduce Forum

Simple MR job fails on HDP2.1 sandbox

  • #53742
    Param Dutta


    I am unable to get simple word_count to run on HDP2.1 sandbox. I get the exception attached at the bottom. On looking at the container logs, I see:
    ” Error: Could not find or load main class ”

    I verified that the classpath is correct. “yarn.application.classpath” in the job configuration looks like this:
    yarn.application.classpath = /etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*

    I am not setting “yarn.application.classpath”, in my client application. It is automatically set and looks good to me. The class is in /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-

    The stack in my client log:

    2014-05-15 10:28:15,121 INFO [Job] Job job_1400096640605_0002 failed with state FAILED due to: Application application_1400096640605_0002 failed 2 times due to AM Container for appattempt_1400096640605_0002_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
    at org.apache.hadoop.util.Shell.runCommand(
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    at java.util.concurrent.ThreadPoolExecutor$

    Thanks for any help with this.


  • Author
  • #71539
    Jaffer Jamil

    Param did you ever get to resolve this issue? I am experiencing this my Sandbox 2.2 when I do the tutorials.

    Chris Schrader

    I am also having the same error on an HDP 2.2 that was upgraded from 2.1 running in AWS (Redhat OS). MR, Hive, YARN, etc all worked fine before the upgrade.

    The exact error in the YARN logs is: Error: Could not find or load main class

    I found this thread that mentioned the YARN classpath issue:

    And I did notice that my YARN classpath was missing the paths to map-reduce client libs. I added those in. It now looks like:


    The last entry is actually an exact path to the class its looking for. Restarted MR, YARN, and Hive (I’m trying to run a hive query and getting this error), still no dice. Same error.

    Not sure if its related, but I had to fix another upgrade issue related to map-reduce classpath. I set the HADOOP_OPTS var to use the actual version as others have had issue with the variable. It is now like this in export HADOOP_OPTS=”-Dhdp.version= $HADOOP_OPTS”

    My mapreduce application classpath is now:


    And my mapreduce application framework path is: /hdp/apps/

    I removed the #mr-framework because of errors saying that framework didn’t exist. This fixed that error.

    Any help is appreciated.

    Chris Schrader

    After re-checking several settings for mapred-site and yarn-site, and setting them according to the HWX documentation for a 2.1 to 2.2 upgrade (apparently we missed some steps), I am not getting a different error:

    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(
    at Method)
    at java.lang.ClassLoader.loadClass(
    at sun.misc.Launcher$AppClassLoader.loadClass(
    at java.lang.ClassLoader.loadClass(
    at sun.launcher.LauncherHelper.checkAndLoadMain(
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService
    at Method)
    at java.lang.ClassLoader.loadClass(
    at sun.misc.Launcher$AppClassLoader.loadClass(
    at java.lang.ClassLoader.loadClass(
    … 13 more

    All searches still seems to point to a yarn.classpath problem. This is the most notable:

    But even after the map-reduce libs to the yarn classpath parameter, still getting same error.

    Chris Schrader

    One more interesting thing that I don’t know if it matters or not. The file references /usr/hdp/ … while other variables get overwritten to /usr/hdp/current/…

    In some cases, I have env variables looking in different places, albeit with what appears to be correct configs from my ambari setup. IE, when I source, I get the following:


    And then some class paths start to look like this:


    Those are all real directories with real libs and appear to be correct but not sure if that is contributing to the problem. I don’t see where/how ambari sets

    Chris Schrader

    Ok I finally solved this problem.

    The issue was mapreduce.application.classpath. In the Ambari upgrade docs for going from 2.1 to 2.2, there’s a section for setting all the new classpaths. In particular:

    $PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: $PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: $PWD/mr-framework/hadoop/share/hadoop/common/*: $PWD/mr-framework/hadoop/share/hadoop/common/lib/*: $PWD/mr-framework/hadoop/share/hadoop/yarn/*: $PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: $PWD/mr-framework/hadoop/share/hadoop/hdfs/*: $PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure$

    I, like most people, copy and pasted that value into Ambari. All those paths are correct. The problem is on this site only, some of the classpaths are separate by a space after the colon. This was causing the error. Map-Reduce now runs correctly.

    This one was a HUGE pain in the ass and took a really long time to identify and fix. This page needs to be fixed so this property is correct (without the spaces). It would also be helpful if Hadoop had more meaningful error messages (instead of just generic Java class missing errors).

The forum ‘MapReduce’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.