The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Hive / HCatalog Forum

Hive_11 java.lang.NoSuchMethodError: o.a.h.h.serde2.ColumnProjectionUtils.set…

  • #28061
    Subroto Sanyal
    Participant

    Hi,

    I can see two different flavors of class:org.apache.hadoop.hive.serde2.ColumnProjectionUtils. One is available in hive-serde-0.11.0.1.3.0.0-107.jar and the other in hive-exec-0.11.0.1.3.0.0-107.jar. I have both the jars in my client clasppath and end up with the problem of: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.setReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/ArrayList;)V

    Though I couldn’t find this problem in Apache Hive-11 release(Though both the jars have duplicate copy of class).

    It will helpful to know whether this change is intentional or a bug in hdp release?
    I can find a bug in open-source related to this:
    https://issues.apache.org/jira/browse/HIVE-3017

    but, this bug isn’t responsible for this problem…

    Cheers,
    Subroto

  • Author
    Replies
  • #28607
    tedr
    Moderator

    Hi Subroto,

    If you look at the code for these two copies are they actually two different classes? If so, is there a way you could try swapping the order of the jars on the classpath? Since java will use the version that it finds first.

    Thanks,
    Ted.

    #28744
    Subroto Sanyal
    Participant

    Hi Ted,

    The content of the classes are not same but, they have same name and are in same package. Swapping the order might be a workaround.
    Can you confirm whether this is bug from Distribution or not ?? If it is when can we expect it to be resolved?

    Cheers,
    Subroto Sanyal

    #28940
    Carter Shanklin
    Participant

    Subroto,

    Between Hive 9 and Hive 10 the signature of this method changed to use a List instead of an ArrayList. Based on the traceback you have something is trying to use an ArrayList rather than a List. Do you maybe have some old UDFs or custom JARs that might be trying to use an old interface? I don’t think there was any intentional change by HWX that would cause this.

    carter$ find ./ -name ColumnProjectionUtils.java | xargs grep setReadColumnIDs

    .//branch-0.9/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configuration conf, ArrayList ids) {
    .//branch-0.10/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configuration conf, List ids) {
    .//branch-0.11/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configuration conf, List ids) {

    #28978
    Subroto Sanyal
    Participant

    Thanks Carter….

    #37868
    Zeke Huang
    Member

    Hey guys,

    I’m having similar issue which seems happening during mapreduce. Pasted call stack below.
    How can I change the order of jars in hadoop classpath?

    2013-09-24 20:45:03,205 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;)V
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:427)
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
    at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:190)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:411)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

    #38361
    Yi
    Member

    Hi Zeke,

    The org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;) is in hive 0.10 and later’s hive-serde jar. Do you have it in your class path? or you have an earlier version of hive-serde?

    Thanks,
    Yi

The forum ‘Hive / HCatalog’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.