First-Class Support for Long Running Services on Apache Hadoop YARN

Introduction

Apache Hadoop YARN is well known as the general resource-management platform for big-data applications such as MapReduce, Hive / Tez and Spark. It abstracts the complicated cluster resource management and scheduling from higher level applications and enables them to focus solely on their own application specific logic.

In addition to big-data apps, another broad spectrum of workloads we see today are long running services such as HBase, Hive/LLAP and container (e.g. Docker) based services. In the past year, the YARN community has been working hard to build first-class support for long running services on YARN.  

YARN Service framework coming in Apache Hadoop 3.1!

This feature, what we call as YARN service framework, has been just merged to trunk back in Nov 2017.  In total, it has 108 commits with 33539 lines of code changes. It is expected to become available in the Apache Hadoop 3.1 release. This effort primarily includes the following:

1. A core framework (ApplicationMaster) running on YARN serving as a container orchestrator and responsible for all service lifecycle managements.
2. A RESTful API-service for users to deploy and manage their services on YARN using a simple JSON spec.
3. A YARN DNS server backed by YARN service registry to enable discovering services on YARN with standard DNS lookup.
4. Advanced container placement scheduling such as affinity and anti-affinity for each application, container resizing, and node labels.
5. Rolling upgrades for containers and the service as a whole.

And YARN Service framework goes hand-in-hand with a few other features in YARN:

1. First class support for Docker on YARN.
2. A native YARN Timeline Service based on HBase to record lifecycle events and metrics and provide rich analytics APIs for users to fetch/query application details.
3. Services UI in YARN UI2.

Example

The bulk of the complexity of managing a Service on YARN is all hidden from the users. Users only deal with a JSON specification to deploy and manage the services running on YARN through CLIs or REST APIs. Below is an example JSON specification for deploying a httpd container on YARN. Users can simply post this JSON spec through REST API or using CLI, and the system will handle the rest – launch and monitor the containers or any other actions required to keep the application running such as auto-restart of a container if it fails. For example:

1. To launch a service, run below command with the supplied JSON

yarn app -launch my-httpd /path/to/httpd.json

{

"name": "httpd-service",

"lifetime": "3600",

"components": [

 {

   "name": "httpd",

   "number_of_containers": 2,

   "artifact": {

     "id": "centos/httpd-24-centos7:latest",

     "type": "DOCKER"

   },

   "launch_command": "/usr/bin/run-httpd",

   "resource": {

     "cpus": 1,

     "memory": "1024"

   }

 }]

}

2. To get the status of app
yarn app -status my-httpd

3. To flex the number of containers to 3:
yarn app -flex my-httpd -component httpd 3

4. To stop the service:
yarn app -stop my-httpd

5. To restart the stopped service:
yarn app -start my-httpd

Under the hood

The diagram below illustrates the main components involved in a full-fledged YARN cluster to support long running services – under the hood.

A typical workflow is:

1. User posts a JSON request that describes the specification of the service such as the container memory size, number of CPU cores, Docker image ID, etc. to the YARN Service REST API. Similarly, user can submit their service creation request using the YARN CLI also.
2. RM, after accepting the request, launches an ApplicationMaster (i.e. the container orchestration framework).
3. The orchestration framework requests resources from RM adhering to the user’s resource requirements and then, when a container has been allocated, launches the container on NodeManager.
4. NodeManager in turn launches the container process (where user code lives) or uses the Docker container runtime to launch the Docker container.
5. The orchestration framework monitors containers’ health and readiness and acts upon container failures or unhealthiness. It writes the service lifecycle events and metrics into the YARN Timeline Service (backed by HBase). It also writes the additional service meta info (such as container IP and host) into the YARN service registry backed by a ZooKeeper quorum.
6. The Registry DNS server listens on znode creation or deletion in ZooKeeper and creates all sorts of DNS records such as the A record and Service Record to serve DNS queries.
7. Each Docker container is given a user-friendly hostname based on information provided in the JSON spec and the YARN configuration. Client can then lookup the container IP by the container hostname using standard DNS lookup.

What’s good

Which area is YARN service framework good at?

1. Hadoop has been in the industry for over 10 years. YARN arrived a little later in Hadoop 2 but also a very mature project and has been endorsed by many companies who run it in large scale in production. Additionally, the YARN Container Orchestration Framework leverages all the existing stable functionalities in Hadoop.
2. YARN has been proven to support batch processing workloads like MapReduce and Spark very well. This feature opens the possibility of bringing existing container based services into YARN as well. The result is that users can now use a single cluster to run both batch processing jobs and long running services. This also enables resource sharing between batch jobs and services which could have very different resource utilization patterns. For example, services typically run with high resource utilizations during the day whereas batch jobs are typically run in the night. Therefore, resource sharing can greatly improve overall cluster utilization.
3. It supports kerberos security and fits in well with the standard kerberized Hadoop cluster.
4. It also supports standard tar packaged apps, in addition to docker containerized apps.

What’s next

This blog post gives a high level overview on how Apache Hadoop YARN supports container based services. In the next blog post, we’ll talk about more details on how simple it is to run more complicated services such as HIVE LLAP on YARN using this framework. Stay tuned!