With the release of Apache Hadoop YARN in October of last year, organizations are moving from single-application Hadoop clusters to a versatile, integrated Hadoop 2 data platform hosting multiple applications — eliminating silos, maximizing resources and bringing true multi-workload capabilities to Hadoop. Many enterprises have adopted YARN as the architectural center of a set of integrated technologies and capabilities that form the blueprint for enterprise Hadoop.
Hortonworks is making it easier to develop YARN applications through a number of technologies. HDP2.0 introduced the native YARN interface, and made MapReduce a YARN engine, making MapReduce applications YARN-enabled. HDP 2.1 introduced Apache Tez, a YARN engine that accelerates next-generation batch and interactive capabilities. And yesterday, Hortonworks announced availability of the Apache Slider Technical Preview, a set of platform services that enables other applications to quickly run in the YARN environment. These foundational technologies can be used to build YARN-based applications:
These approaches are summarized in the diagram below
The beauty of this approach is flexibility to the application developer. It also allows applications that process data through higher-level APIs (such as Hive or Pig) to be YARN Ready.
Customers are telling us loud and clear: they want solutions that run on YARN because it enables them to run multiple workloads on the same common data pool. At the same time, partners are telling us they want to provide assurance that their YARN-based applications work well in Hadoop. To meet these needs, Hortonworks has created the YARN Ready program.
The YARN Ready program assures customers and partners that their applications have been integrated with the Hortonworks Data Platform through YARN. It is a higher-level of integration than simply reading/writing HDFS data or running tasks without regard for what else is on the system. YARN Ready applications are verified as “good citizens” of Hadoop, ready to work cooperatively in a multi-workload environment.
Partners can get the new YARN Ready badge as part of their HDP 2.1 certification. Once your application uses YARN, either natively or through a YARN engine, submit your application’s information to the Hortonworks certification team. Instructions are included in the certification kit available to members of the Hortonworks Certified Technology Program.
Hortonworks Certified Technology Partners gain a number of benefits:
Our web site allows you to find YARN Ready applications. From the Hortonworks Partners Page, select the “YARN Ready” filter button to see a list of partners that have qualified as YARN Ready. These solutions are certified to run on HDP 2.1 under YARN. We’ll be adding more all the time, so check back often.
In addition to partner solutions, several components of the Hortonworks Data Platform are YARN Ready, like Apache Hive and Apache Spark. Others are in the process of becoming YARN Ready, like Apache HBase, Apache Accumulo, and Apache Storm. You can try these applications by downloading the Apache Slider technical preview.
The YARN Ready program underscores Hortonworks’ commitment to expanding the Hadoop ecosystem with YARN as the architectural center of Hadoop. It makes YARN-based solutions easier for customers to identify and adopt, while providing partners with valuable technical resources and visibility for their YARN-based technology.