HDFS HA from client side

to create new topics or reply. | New User Registration

This topic contains 2 replies, has 2 voices, and was last updated by  Pavel Hladik 1 year, 6 months ago.

  • Creator
  • #47154

    Pavel Hladik


    We have fully working HDP 2.0.9 stack on our test environment with NameNode HA. If first of NN is in active state and will fall the other will take his place and will change state from standby to active.

    My question is regarding to HA of NN from client side. I mean if we have client host with client libraries and accessing first of NN, we have to detect first NN has fall and switch to second NN manually? Is there any proxy or something like that? Is there automatic process that will let client host know about stop using fallen NN and start using the second one?

    Thanks for answer.


Viewing 2 replies - 1 through 2 (of 2 total)

You must be to reply to this topic. | Create Account

  • Author
  • #47157

    Pavel Hladik

    Thanks for quick answer.


    Jing Zhao

    Currently the DFSClient in Hadoop has this functionality. Basically a DFSClient instance will load the host/port information of both NN from the configuration. When it sends a RPC request, it will first try one of the NNs (specifically, the first one specified in the configuration). Then if the NN is actually in standby state, a StandbyException will be sent from the NN to the client. The client will then automatically failover to the other NN and sends the request there. This client failover process can also happen when the first NN fails and the client receives exceptions like ConnectException.

    This client failover mechanism has also been implemented in webhdfs.

Viewing 2 replies - 1 through 2 (of 2 total)
Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.