The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Hue Forum

Issue in beeswax: NoClassDefFoundError: org/apache/hadoop/mapred/JobConf

  • #51865
    Very Test
    Participant

    Hello
    I have installed HDP based on the HDP provided PDF bk_installing_manually_book-20140210.pdf. Everything is Ok and checked with the various test explained in this PDF.
    But I do not success on making Hue working. I stil have the java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf when beeswax server tries to start.
    Any idea ?
    PS: I am using Hadoop 2.2.0.

  • Author
    Replies
  • #52214
    Dave
    Moderator

    Hi,

    Can you check the logs in the beeswax server in /var/log/hue and also check the beeswax configuration in the hue.ini.
    Can you run the hive CLI from the hue machine?

    Thanks

    Dave

    #52320
    Very Test
    Participant

    Hello Dave

    The only exception I found is in beeswax_server.out:

    Executing /usr/local/bin/hadoop/bin/hadoop jar /usr/lib/hue/apps/beeswax/src/beeswax/../../java-lib/BeeswaxServer.jar --beeswax 19805 --desktop-host 192.168.18.10 --desktop-port 19999 --query-lifetime 604800000
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
    	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:1136)
    	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:1103)
    	at com.cloudera.beeswax.Server.createDirectoriesAsNecessary(Server.java:224)
    	at com.cloudera.beeswax.Server.main(Server.java:191)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 9 more
    
    #52347
    Very Test
    Participant

    Hello

    Moreover, Hive CLI is working on the same machine than Hue:
    – /usr/lib/hive/bin/hive
    – /usr/lib/hive/bin/beeline with !connect jdbc:hive2://

    Regards

    #52348
    Very Test
    Participant

    Hello
    FYI, if I change port 19805 to something else (e.g. 19995), I got another error

    Executing /usr/local/bin/hadoop/bin/hadoop jar /usr/lib/hue/apps/beeswax/src/beeswax/../../java-lib/BeeswaxServer.jar --beeswax 19995 --desktop-host 192.168.18
    .10 --desktop-port 19999 --query-lifetime 604800000
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/TaskAttemptContext
    	at java.lang.Class.forName0(Native Method)
    	at java.lang.Class.forName(Class.java:190)
    	at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:120)
    	at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:115)
    	at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:80)
    	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:248)
    	at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:81)
    	at com.cloudera.beeswax.Server.createDirectoriesAsNecessary(Server.java:224)
    	at com.cloudera.beeswax.Server.main(Server.java:191)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.TaskAttemptContext
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 14 more
    
    #52357
    Very Test
    Participant

    Hello

    I ran manually the command
    /usr/local/bin/hadoop/bin/hadoop jar /usr/lib/hue/apps/beeswax/src/beeswax/../../java-lib/BeeswaxServer.jar –beeswax 19995 –desktop-host 192.168.18
    .10 –desktop-port 19999 –query-lifetime 604800000

    and I make it working only if I add the following JAR
    /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.2.0.2.0.6.0-102.jar
    in the HADOOP_CLASSPATH generated by Hue (the one displayed in beeswax_server.out)

    How to make it added automatically by Hue ?

    Regards

    #52417
    Dave
    Moderator

    Hi,

    What does the beeswax section of your hue.ini show?

    Thanks

    Dave

    #52429
    Very Test
    Participant

    Hi Dave,
    Please find below the beeswax section of hue.ini (stored in /usr/lib/hue/desktop/conf/hue.ini)
    Regards

    [beeswax]
    
      # Host where Beeswax server Thrift daemon is running.
      # If Kerberos security is enabled, the fully-qualified domain name (FQDN) is
      # required, even if the Thrift daemon is running on the same host as Hue.
      ## beeswax_server_host=<FQDN of Beeswax Server>
      beeswax_server_host=localhost
    
      # Port where Beeswax Thrift server runs on.
      ## beeswax_server_port=8002
      beeswax_server_port=19991
    
      # Host where internal metastore Thrift daemon is running.
      ## beeswax_meta_server_host=localhost
      beeswax_meta_server_host=localhost
    
      # Configure the port the internal metastore daemon runs on.
      # Used only if hive.metastore.local is true.
      ## beeswax_meta_server_port=8003
      beeswax_meta_server_port=19990
    
      # Hive home directory
      ## hive_home_dir=/usr/lib/hive
      hive_home_dir=/usr/lib/hive
    
      # Hive configuration directory, where hive-site.xml is located
      ## hive_conf_dir=/etc/hive/conf
      hive_conf_dir=/etc/hive/conf
    
      # Timeout in seconds for thrift calls to beeswax service
      ## beeswax_server_conn_timeout=120
    
      # Timeout in seconds for thrift calls to the hive metastore
      ## metastore_conn_timeout=10
    
      # Maximum Java heapsize (in megabytes) used by Beeswax Server.
      # Note that the setting of HADOOP_HEAPSIZE in $HADOOP_CONF_DIR/hadoop-env.sh
      # may override this setting.
      ## beeswax_server_heapsize=1000
    
      # Share saved queries with all users. If set to false, saved queries are
      # visible only to the owner and administrators.
      ## share_saved_queries=true
    
      # The backend to contact for queries/metadata requests
      # Choices are 'beeswax' (default), 'hiveserver2'.
      ## server_interface=beeswax
    
    #52548
    Very Test
    Participant

    Hello Dave

    I was so upset than I reinstall everything 😉 and beeswax can start now !

    The difference of these 2 installations:
    – I compiled hadoop 2.2 from source -> I got the error specified below
    – I installed hadoop package from HDP repo -> beeswax starts correctly

    Strange ?

    #52667
    Dave
    Moderator

    Hi Very,

    Ah, I thought you were using HDP and getting the error below.
    It was probably settings in Hive or hive-site.xml or even JAR files.

    Also You had 2 parts in beeswax uncommented. I usually leave it all commented out to see how it goes initially as you only need the first 2 settings uncommented.

    Thanks

    Dave

The forum ‘Hue’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.