Home Forums Hive / HCatalog Hive_11 java.lang.NoSuchMethodError: o.a.h.h.serde2.ColumnProjectionUtils.set…

This topic contains 6 replies, has 5 voices, and was last updated by  Yi 11 months, 3 weeks ago.

  • Creator
    Topic
  • #28061

    Subroto Sanyal
    Participant

    Hi,

    I can see two different flavors of class:org.apache.hadoop.hive.serde2.ColumnProjectionUtils. One is available in hive-serde-0.11.0.1.3.0.0-107.jar and the other in hive-exec-0.11.0.1.3.0.0-107.jar. I have both the jars in my client clasppath and end up with the problem of: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.setReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/ArrayList;)V

    Though I couldn’t find this problem in Apache Hive-11 release(Though both the jars have duplicate copy of class).

    It will helpful to know whether this change is intentional or a bug in hdp release?
    I can find a bug in open-source related to this:

    https://issues.apache.org/jira/browse/HIVE-3017

    but, this bug isn’t responsible for this problem…

    Cheers,
    Subroto

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #38361

    Yi
    Member

    Hi Zeke,

    The org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;) is in hive 0.10 and later’s hive-serde jar. Do you have it in your class path? or you have an earlier version of hive-serde?

    Thanks,
    Yi

    Collapse
    #37868

    Zeke Huang
    Member

    Hey guys,

    I’m having similar issue which seems happening during mapreduce. Pasted call stack below.
    How can I change the order of jars in hadoop classpath?

    2013-09-24 20:45:03,205 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnIDs(Lorg/apache/hadoop/conf/Configuration;Ljava/util/List;)V
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:427)
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
    at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:190)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:411)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

    Collapse
    #28978

    Subroto Sanyal
    Participant

    Thanks Carter….

    Collapse
    #28940

    Carter Shanklin
    Participant

    Subroto,

    Between Hive 9 and Hive 10 the signature of this method changed to use a List instead of an ArrayList. Based on the traceback you have something is trying to use an ArrayList rather than a List. Do you maybe have some old UDFs or custom JARs that might be trying to use an old interface? I don’t think there was any intentional change by HWX that would cause this.

    carter$ find ./ -name ColumnProjectionUtils.java | xargs grep setReadColumnIDs

    .//branch-0.9/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configuration conf, ArrayList ids) {
    .//branch-0.10/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configuration conf, List ids) {
    .//branch-0.11/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: public static void setReadColumnIDs(Configuration conf, List ids) {

    Collapse
    #28744

    Subroto Sanyal
    Participant

    Hi Ted,

    The content of the classes are not same but, they have same name and are in same package. Swapping the order might be a workaround.
    Can you confirm whether this is bug from Distribution or not ?? If it is when can we expect it to be resolved?

    Cheers,
    Subroto Sanyal

    Collapse
    #28607

    tedr
    Moderator

    Hi Subroto,

    If you look at the code for these two copies are they actually two different classes? If so, is there a way you could try swapping the order of the jars on the classpath? Since java will use the version that it finds first.

    Thanks,
    Ted.

    Collapse
Viewing 6 replies - 1 through 6 (of 6 total)