Home Forums HBase zookeeper node failure causes regionserver shutdown?

This topic contains 1 reply, has 2 voices, and was last updated by  abdelrahman 10 months, 2 weeks ago.

  • Creator
    Topic
  • #45572

    David Nguyen
    Member

    I have a regionserver that is running hdfs datanode + hbase regionserver only and the regionserver shutdown right when one of the zookeeper nodes in the cluster went offline. This is what is in the regionserver log around that time:

    2013-12-16 02:44:16,268 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to sac-prod1-ops-hbase-06.unix.newokl.com/10.140.96.56:2181, initiating session
    2013-12-16 02:44:16,274 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x442d9dd48ef0002 has expired, closing socket connection
    2013-12-16 02:44:16,276 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server sac-prod1-ops-hbase-02.unix.newokl.com,60020,1386634675805: regionserver:60020-0x442d9dd48ef0002 regionserver:60020-0x442d9dd48ef0002 received expired from ZooKeeper, aborting
    org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:363)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:282)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
    2013-12-16 02:44:16,278 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
    2013-12-16 02:44:16,280 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, ……..
    2013-12-16 02:44:16,284 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: regionserver:60020-0x442d9dd48ef0002 regionserver:60020-0x442d9dd48ef0002 received expired from ZooKeeper, aborting

    I’m running HDP 1.3.3 (hbase-0.94.6.1.3.3.0-58.el6.noarch) on centos6. Has anybody seen this one before?

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #46529

    abdelrahman
    Moderator

    Hi David,

    Any errors you see in the Zookeeper logs?

    Thanks
    -Rahman

    Collapse
Viewing 1 replies (of 1 total)