Home Forums Hive / HCatalog Adding Hive custom serde jars permanently

Tagged: 

This topic contains 0 replies, has 1 voice, and was last updated by  Hari Sekhon 3 months, 3 weeks ago.

  • Creator
    Topic
  • #58218

    Hari Sekhon
    Participant

    Hive custom jar addition isn’t documented and doesn’t work by default – I know the procedure of adding to hive.aux.jars.path but this doesn’t work in Ambari as it gets overridden by HIVE_AUX_JARS_PATH from hive-env.sh – which itself is not manageable by Ambari (AMBARI-5802) and changes to hive-env.sh get overwritten at each restart. Setting it per session is not an option users would tolerate.

    Also, you have to be careful with hive-env.sh as adding a jar to HIVE_AUX_JARS_PATH causes the /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-*.jar to not evaluate properly and breaks Tez pre-run checks:

    Exception in thread “main” java.lang.RuntimeException: java.io.IOException: Previous writer likely failed to write hdfs://nameservice1/tmp/hive-root/_tez_session_dir/f9ecb2b2-c034-43b1-ac11-1e0aac1962b6/hive-hcatalog-core-*.jar. Failing because I am unlikely to write too.
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:358)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
    Caused by: java.io.IOException: Previous writer likely failed to write hdfs://nameservice1/tmp/hive-root/_tez_session_dir/f9ecb2b2-c034-43b1-ac11-1e0aac1962b6/hive-hcatalog-core-*.jar. Failing because I am unlikely to write too.
    at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:865)
    at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempFiles(DagUtils.java:741)
    at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:690)
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:149)
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:123)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:356)
    … 7 more

    I’ve worked around by editing the hive-env.sh.j2 templates in Ambari server and agent caches for the 2.0.6 stack (which it seems to use even though I’m using HDP 2.1) and pre-evaluating the hcatalog jar to work around above exception:

    export HIVE_AUX_JARS_PATH=$(ls /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-*.jar|tr ‘\n’ ‘:’)/path/to/my/custom-serde.jar

    Yes there is no colon between the ) and next jar otherwise it gives :: – ie a blank which results in a “Can not create a Path from an empty string” error.

    A bunch of ugly hacks to get this working. Does anybody have a better way?

    Regards

    Hari Sekhon
    Big Data Specialist Contractor

    http://www.linkedin.com/in/harisekhon

You must be logged in to reply to this topic.