Home Forums Hue Issue in beeswax: NoClassDefFoundError: org/apache/hadoop/mapred/JobConf

This topic contains 9 replies, has 2 voices, and was last updated by  Dave 5 months, 3 weeks ago.

  • Creator
    Topic
  • #51865

    Very Test
    Participant

    Hello
    I have installed HDP based on the HDP provided PDF bk_installing_manually_book-20140210.pdf. Everything is Ok and checked with the various test explained in this PDF.
    But I do not success on making Hue working. I stil have the java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf when beeswax server tries to start.
    Any idea ?
    PS: I am using Hadoop 2.2.0.

Viewing 9 replies - 1 through 9 (of 9 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #52667

    Dave
    Moderator

    Hi Very,

    Ah, I thought you were using HDP and getting the error below.
    It was probably settings in Hive or hive-site.xml or even JAR files.

    Also You had 2 parts in beeswax uncommented. I usually leave it all commented out to see how it goes initially as you only need the first 2 settings uncommented.

    Thanks

    Dave

    Collapse
    #52548

    Very Test
    Participant

    Hello Dave

    I was so upset than I reinstall everything ;) and beeswax can start now !

    The difference of these 2 installations:
    – I compiled hadoop 2.2 from source -> I got the error specified below
    – I installed hadoop package from HDP repo -> beeswax starts correctly

    Strange ?

    Collapse
    #52429

    Very Test
    Participant

    Hi Dave,
    Please find below the beeswax section of hue.ini (stored in /usr/lib/hue/desktop/conf/hue.ini)
    Regards

    [beeswax]
    
      # Host where Beeswax server Thrift daemon is running.
      # If Kerberos security is enabled, the fully-qualified domain name (FQDN) is
      # required, even if the Thrift daemon is running on the same host as Hue.
      ## beeswax_server_host=<FQDN of Beeswax Server>
      beeswax_server_host=localhost
    
      # Port where Beeswax Thrift server runs on.
      ## beeswax_server_port=8002
      beeswax_server_port=19991
    
      # Host where internal metastore Thrift daemon is running.
      ## beeswax_meta_server_host=localhost
      beeswax_meta_server_host=localhost
    
      # Configure the port the internal metastore daemon runs on.
      # Used only if hive.metastore.local is true.
      ## beeswax_meta_server_port=8003
      beeswax_meta_server_port=19990
    
      # Hive home directory
      ## hive_home_dir=/usr/lib/hive
      hive_home_dir=/usr/lib/hive
    
      # Hive configuration directory, where hive-site.xml is located
      ## hive_conf_dir=/etc/hive/conf
      hive_conf_dir=/etc/hive/conf
    
      # Timeout in seconds for thrift calls to beeswax service
      ## beeswax_server_conn_timeout=120
    
      # Timeout in seconds for thrift calls to the hive metastore
      ## metastore_conn_timeout=10
    
      # Maximum Java heapsize (in megabytes) used by Beeswax Server.
      # Note that the setting of HADOOP_HEAPSIZE in $HADOOP_CONF_DIR/hadoop-env.sh
      # may override this setting.
      ## beeswax_server_heapsize=1000
    
      # Share saved queries with all users. If set to false, saved queries are
      # visible only to the owner and administrators.
      ## share_saved_queries=true
    
      # The backend to contact for queries/metadata requests
      # Choices are 'beeswax' (default), 'hiveserver2'.
      ## server_interface=beeswax
    
    Collapse
    #52417

    Dave
    Moderator

    Hi,

    What does the beeswax section of your hue.ini show?

    Thanks

    Dave

    Collapse
    #52357

    Very Test
    Participant

    Hello

    I ran manually the command
    /usr/local/bin/hadoop/bin/hadoop jar /usr/lib/hue/apps/beeswax/src/beeswax/../../java-lib/BeeswaxServer.jar –beeswax 19995 –desktop-host 192.168.18
    .10 –desktop-port 19999 –query-lifetime 604800000

    and I make it working only if I add the following JAR
    /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.2.0.2.0.6.0-102.jar
    in the HADOOP_CLASSPATH generated by Hue (the one displayed in beeswax_server.out)

    How to make it added automatically by Hue ?

    Regards

    Collapse
    #52348

    Very Test
    Participant

    Hello
    FYI, if I change port 19805 to something else (e.g. 19995), I got another error

    Executing /usr/local/bin/hadoop/bin/hadoop jar /usr/lib/hue/apps/beeswax/src/beeswax/../../java-lib/BeeswaxServer.jar --beeswax 19995 --desktop-host 192.168.18
    .10 --desktop-port 19999 --query-lifetime 604800000
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/TaskAttemptContext
    	at java.lang.Class.forName0(Native Method)
    	at java.lang.Class.forName(Class.java:190)
    	at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:120)
    	at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:115)
    	at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:80)
    	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:248)
    	at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:81)
    	at com.cloudera.beeswax.Server.createDirectoriesAsNecessary(Server.java:224)
    	at com.cloudera.beeswax.Server.main(Server.java:191)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.TaskAttemptContext
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 14 more
    
    Collapse
    #52347

    Very Test
    Participant

    Hello

    Moreover, Hive CLI is working on the same machine than Hue:
    – /usr/lib/hive/bin/hive
    – /usr/lib/hive/bin/beeline with !connect jdbc:hive2://

    Regards

    Collapse
    #52320

    Very Test
    Participant

    Hello Dave

    The only exception I found is in beeswax_server.out:

    Executing /usr/local/bin/hadoop/bin/hadoop jar /usr/lib/hue/apps/beeswax/src/beeswax/../../java-lib/BeeswaxServer.jar --beeswax 19805 --desktop-host 192.168.18.10 --desktop-port 19999 --query-lifetime 604800000
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
    	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:1136)
    	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:1103)
    	at com.cloudera.beeswax.Server.createDirectoriesAsNecessary(Server.java:224)
    	at com.cloudera.beeswax.Server.main(Server.java:191)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:606)
    	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 9 more
    
    Collapse
    #52214

    Dave
    Moderator

    Hi,

    Can you check the logs in the beeswax server in /var/log/hue and also check the beeswax configuration in the hue.ini.
    Can you run the hive CLI from the hue machine?

    Thanks

    Dave

    Collapse
Viewing 9 replies - 1 through 9 (of 9 total)