Shark 0.9.1 on HDP 2.1


This topic contains 4 replies, has 4 voices, and was last updated by  Vinay Shukla 8 months, 3 weeks ago.

  • Creator
  • #52313

    Raj Kk

    Trying to install Shark 0.9.1 on HDP 2.1 (TP).
    Get this message when I start the shell “No HADOOP_HOME specified. Shark will run in local-mode”
    I cannot seem to be initiate the sharkserver2 . Is there a config or so missing?

Viewing 4 replies - 1 through 4 (of 4 total)

The forum ‘HDP 2.1 Technical Preview’ is closed to new topics and replies.

  • Author
  • #57217

    Vinay Shukla

    Thanks for the update Hari.

    On Spark’s protobuf, there is a detailed thread!topic/apache-spark-user-mirror/BT10oU4Z4aM that should give you more insight into the issue.

    I believe with Spark 1.0.0 the issue is addressed with making the dependency explicit.

    Will take note of priority.


    Hari Sekhon

    I did the same as Craig suggested to get Shark working on HDP 2.1, but had to downgrade my Spark back to 0.9.1 otherwise the serial version mismatch in Spark was preventing job submission. Dissappointing to have to strip out the Google protobuf classes from the jar… was that a build flaw?

    Also, my other cluster has metadata authentication so the lack of authentication to the metastore layer is still an issue there.

    Shark is clearly not ready for prime time yet, but the desire for it is significant.

    Fast SQL should be the highest priority for Hortonworks (along with security).


    Craig Lukasik

    To get Shark 0.9.1 running on HDP 2.1 I did the following:
    0) Installed shark at /opt/shark-0.9.1-bin-hadoop2
    1) Removed the protobuf classes from Shark’s hive-exec jar at lib_managed/jars/edu.berkeley.cs.shark/hive-exec/hive-exec-0.11.0-shark-0.9.1.jar
    2) Copied hive-site.xml (to /opt/shark-0.9.1-bin-hadoop2/bin/hive-conf) and
    a) Removed hive.exec.*.hooks properties
    b) Changed the value of “” to “”
    3) Set env variables:
    4) Stopped HiveServer2
    5) Started shark:
    /opt/shark-0.9.1-bin-hadoop2/bin/shark –service sharkserver2

    So far so good.



    Vinay Shukla

    We just announced Spark Tech Preview on YARN with HDP 2.1

    It support in memory compute & machine learning. The Spark Tech Preview does not support Shark.

    Please try out the instructions at

Viewing 4 replies - 1 through 4 (of 4 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.