With Apache Hadoop YARN as its architectural center, Apache Hadoop continues to attract new engines to run within the data platform, as organizations want to efficiently store their data in a single repository and interact with it for batch, interactive and real-time streaming use cases. More and more independent software vendors (ISVs) are developing applications to run in Hadoop via YARN. This increases the number of users and processing engines that operate simultaneously across a Hadoop cluster, on the same data, at the same time.
The Apache Knox Gateway (“Knox”) provides perimeter security so that the enterprise can confidently extend Hadoop access to more of those new users while also maintaining compliance with enterprise security policies.
This blog gives an overview of these new features and how they integrate with other Hadoop services. We’ll also touch on additional innovation we plan for upcoming releases.
High availability of HDFS resources through Apache Knox and WebHDFS routing services with HA support allow Knox users to take advantage of high-availability provided by HDFS transparently. HDFS HA provides failover in case of NameNode failure, shielding Knox users if this occurs.
As part of the recent release of Ambari 1.7.0, Knox is now integrated so that Ambari can install and manage Knox’s configuration and process lifecycle. This simplifies Knox install for someone familiar with Ambari and allows Knox to be managed by Ambari.
This improvement empowers Apache Ranger to centrally manage the service-level authorizations enforced by Knox. With this integration, all benefits of Ranger-based centralized policy management are also exposed to Knox users.
Now the Apache Hadoop YARN’s rich web service REST APIs are accessible through a new Knox routing service for the resource manager. This provides access to the metadata, monitoring and management capabilities of YARN’s application resources and should be useful for many upcoming innovations in the community of Knox developers and users.
The Apache Knox team and contributors have done a great job understanding the needs of the user community and meeting those in this release. We see the quality and number of features growing with each successive Apache Knox release.
We will continue to focus on improvements in three primary areas: