RPC Improvements and Wire Compatibility in Apache Hadoop
Hadoop RPC is the primary communication mechanism between the nodes in an Apache Hadoop cluster. Maintaining wire compatibility, as new features are added to Apache Hadoop, has been a significant challenge with the current RPC architecture. In this blog, I highlight the architectural improvement in Hadoop RPC and how it enables wire compatibility and rolling upgrades.
Challenges for Wire Compatibility
Earlier Hadoop RPC used Writable serialization that made it difficult to evolve the protocols while maintaining wire compatibility. Hadoop RPC also did not distinguish between data types that are exchanged over the wire, and the types used on the client side or the server. This made it more complicated to maintain compatibility as new features that required changes to the common data types were added.
New Approach and Benefits
The new approach separates wire data types from the data types used in the service implementation or client code. This allows complete flexibility in making enhancements on a server or client. Wire compatibility can be maintained independently of the changes in the server or client.
The new architecture allows using different serialization mechanisms for the on-wire data types in a pluggable manner. The default serialization chosen for on-wire data types in Hadoop is protocol buffers.
Protocol buffers are a proven technology and have several strong benefits such as efficiency, extensibility, compactness and cross language support. Protocol buffers allow certain modifications to a protocol in a compatible way, such as adding new optional fields. Protocol buffers are widely supported across many popular languages, and pave the way for non-java Hadoop clients. Early tests have shown significant performance gains with protocol buffers compared to Writables.
Another significant benefit of the new design is to enable rolling upgrades. This is very critical in the context of high availability of Namenode, where active and standby Namenodes can be upgraded independently without disrupting the service.
Acknowledgements & Conclusion
This effort was a significant overhaul in the Hadoop RPC code. This will be available in 0.23.2 release of Apache Hadoop as per the current plan.
It was a fun project with my fellow Hortonworkers, Sanjay Radia and Suresh Srinivas.