HBase Forum

zookeeper node failure causes regionserver shutdown?

  • #45572
    David Nguyen

    I have a regionserver that is running hdfs datanode + hbase regionserver only and the regionserver shutdown right when one of the zookeeper nodes in the cluster went offline. This is what is in the regionserver log around that time:

    2013-12-16 02:44:16,268 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to sac-prod1-ops-hbase-06.unix.newokl.com/, initiating session
    2013-12-16 02:44:16,274 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x442d9dd48ef0002 has expired, closing socket connection
    2013-12-16 02:44:16,276 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server sac-prod1-ops-hbase-02.unix.newokl.com,60020,1386634675805: regionserver:60020-0x442d9dd48ef0002 regionserver:60020-0x442d9dd48ef0002 received expired from ZooKeeper, aborting
    org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:363)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:282)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
    2013-12-16 02:44:16,278 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
    2013-12-16 02:44:16,280 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requestsPerSecond=0, ……..
    2013-12-16 02:44:16,284 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: regionserver:60020-0x442d9dd48ef0002 regionserver:60020-0x442d9dd48ef0002 received expired from ZooKeeper, aborting

    I’m running HDP 1.3.3 (hbase- on centos6. Has anybody seen this one before?

to create new topics or reply. | New User Registration

  • Author
  • #46529

    Hi David,

    Any errors you see in the Zookeeper logs?


    Sanjeev T

    how many zookeeper nodes are there in quorum.

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.