Home Forums HDP 2.1 Technical Preview Shark 0.9.1 on HDP 2.1

Tagged: 

This topic contains 4 replies, has 4 voices, and was last updated by  Vinay Shukla 4 months, 1 week ago.

  • Creator
    Topic
  • #52313

    Raj Kk
    Participant

    Trying to install Shark 0.9.1 on HDP 2.1 (TP).
    Get this message when I start the shell “No HADOOP_HOME specified. Shark will run in local-mode”
    I cannot seem to be initiate the sharkserver2 . Is there a config or so missing?

Viewing 4 replies - 1 through 4 (of 4 total)

The forum ‘HDP 2.1 Technical Preview’ is closed to new topics and replies.

  • Author
    Replies
  • #57217

    Vinay Shukla
    Participant

    Thanks for the update Hari.

    On Spark’s protobuf, there is a detailed thread https://groups.google.com/forum/#!topic/apache-spark-user-mirror/BT10oU4Z4aM that should give you more insight into the issue.

    I believe with Spark 1.0.0 the issue is addressed with making the dependency explicit.

    Will take note of priority.

    Collapse
    #57214

    Hari Sekhon
    Participant

    I did the same as Craig suggested to get Shark working on HDP 2.1, but had to downgrade my Spark back to 0.9.1 otherwise the serial version mismatch in Spark was preventing job submission. Dissappointing to have to strip out the Google protobuf classes from the jar… was that a build flaw?

    Also, my other cluster has metadata authentication so the lack of authentication to the metastore layer is still an issue there.

    Shark is clearly not ready for prime time yet, but the desire for it is significant.

    Fast SQL should be the highest priority for Hortonworks (along with security).

    Collapse
    #56842

    Craig Lukasik
    Participant

    To get Shark 0.9.1 running on HDP 2.1 I did the following:
    0) Installed shark at /opt/shark-0.9.1-bin-hadoop2
    1) Removed the protobuf classes from Shark’s hive-exec jar at lib_managed/jars/edu.berkeley.cs.shark/hive-exec/hive-exec-0.11.0-shark-0.9.1.jar
    2) Copied hive-site.xml (to /opt/shark-0.9.1-bin-hadoop2/bin/hive-conf) and
    a) Removed hive.exec.*.hooks properties
    b) Changed the value of “hive.security.authenticator.manager” to “org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator”
    3) Set env variables:
    HADOOP_HOME=/usr/lib/hadoop/
    HIVE_CONF_DIR=/opt/shark-0.9.1-bin-hadoop2/bin/hive-conf
    4) Stopped HiveServer2
    5) Started shark:
    /opt/shark-0.9.1-bin-hadoop2/bin/shark –service sharkserver2

    So far so good.

    Cheers,
    clukasik@zaloni.com

    Collapse
    #52675

    Vinay Shukla
    Participant

    We just announced Spark Tech Preview on YARN with HDP 2.1 http://hortonworks.com/labs/spark/

    It support in memory compute & machine learning. The Spark Tech Preview does not support Shark.

    Please try out the instructions at http://hortonworks.com/wp-content/uploads/2014/05/SparkTechnicalPreview.pdf

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)