Home Forums Hive / HCatalog Hive/Hbase integration

Tagged: ,

This topic contains 1 reply, has 2 voices, and was last updated by  Enis Soztutar 1 year, 4 months ago.

  • Creator
    Topic
  • #31241

    Hi,

    I am attempting to extract data from our HDFS placed there using thrift/hbase into a Hive table, but I am struggling to find a solution to the following error message;

    ERROR client.HConnectionManager$HConnectionImplementation (HConnectionManager.java:checkIfBaseNodeAvailable(751)) – Check the value configured in ‘zookeeper.znode.parent’. There could be a mismatch with the one configured in the master.

    followed by;

    FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException

    These errors appear in the hive log after running the following query from a hive terminal;

    create external table d_table (id bigint, date string) stored by ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’ with serdeproperties (“hbase.columns.mapping” = “:key,a:date”) TBLPROPERTIES (“hbase.table.name” = “demo_table”);

    The table exists, is populated and can be found from within a hbase shell, and I have checked that the HMASTER process is running. The ‘zookeeper.znode.parent’ is set to the default value, and hbase accepts this (if it is changed, hbase will complain, so I am fairly confident that it is correct). I am able to establish a ZK client with no problems.

    I have reached the point in researching this problem where I am told the solution is to recompile Hive against the version of HBase I am using, which sort of defies the point of using a distribution. ZK and HBase masters are on the same machine that I am running the HiveQL command from.

    If I run the query from another machine, I see the following;

    ERROR client.HConnectionManager$HConnectionImplementation (HConnectionManager.java:checkIfBaseNodeAvailable(751)) – Check the value configured in ‘zookeeper.znode.parent’. There could be a mismatch with the one configured in the master.
    WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1089)) – Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
    java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
    WARN zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:retryOrThrow(219)) – Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

    If I add a property to the hive-site.xml file for zookeeper.quorum, I get the same behaviour as the first machine. I have taken the values for the zookeeper.quorum from Ambari.

    I am using the release 1.3 distribution. We have successfully run Hive with HCatalog to create tables etc, and we have run a variety of other scenarios. If further information is needed to help, let me know, I wanted to keep this post clean.

    Thanks in advance,

    Andy Forrest
    CERN

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #32797

    Enis Soztutar
    Participant

    HBase clients discover the running HBase cluster using two properties, namely hbase.zookeeper.quorum and zookeeper.znode.parent. The former is used to connect to the zookeeper cluster, while the latter shows which znode keeps the data (and address for HMaster) for the cluster.

    Ambari and HDP installations use different zookeeper.znode.parent values than the default (/hbase). On secure setup, the value is configured as /hbase-secure, and in unsecure mode it is set as /hbase-unsecure.

    For Hive-HBase integration to work, thus Hive should be configured with these settings with either supplying these options via HIVE_OPTS=”-hiveconf hbase.zookeeper.quorum=server1,server2,server3 zookeeper.znode.parent=/hbase-unsecure” or inside the hive script with
    set hbase.zookeeper.quorum=server1,server2,server3
    set zookeeper.znode.parent=/hbase-unsecure

    Hope this helps.

    Collapse
Viewing 1 replies (of 1 total)