Developing Custom YARN Applications

Description

This 2-day hands-on training course teaches students how to develop custom YARN applications for Apache Hadoop. Students will learn the details of the YARN architecture, the steps involved in writing a YARN application, the details of writing a YARN client and ApplicationMaster, and how to launch Containers.  Applications are developed using Eclipse and Gradle connected to a 7-node HDP 2.1 cluster running in a virtual machine that the students can keep for use after the training.

Duration

2 days

Target Audience

Software engineers familiar with Java who need to develop YARN applications on Hadoop 2.x by writing custom YARN clients and ApplicationMasters in Java.

Prerequisites

Experienced Java developers who have: 1) attended the Developing Applications with the Hortonworks Data Platform using Java course; or 2) attended the Data Analysis with the Hortonworks Data Platform using Pig and Hive course; or 3) possess similar Hadoop development knowledge and understand HDFS and the MapReduce framework.

Format

Instructor-led lecture/discussion, with approximately 50% of the course being devoted to hands-on labs.

Course Objectives

At the completion of the course students will be able to:

  • Explain the architecture of YARN
  • Explain the lifecycle of a YARN application
  • Write a YARN client application
  • Run a YARN application on a Hadoop 2.x cluster
  • Monitor the status of a running YARN application
  • View the aggregated logs of a YARN application
  • Configure a ContainerLaunchContext
  • Define a LocalResource for sharing application files across the cluster
  • Write a YARN ApplicationMaster
  • Explain the differences between synchronous and asynchronous ApplicationMasters
  • Allocate Containers in a cluster
  • Launch Containers on NodeManagers
  • Write a custom Container to perform specific business logic
  • Explain the job schedulers of the ResourceManager
  • Define queues for the Capacity Scheduler

Agenda

Day 1

  • Unit 1: The YARN Architecture
  • Unit 2: Overview of a YARN Application
  • Unit 3: Writing a YARN Client

Day 2

  • Unit 4: Writing a YARN ApplicationMaster
  • Unit 5: Containers
  • Unit 6: Job Scheduling

Lab Content

Students will work through the following lab exercises using the Hortonworks Data Platform 2.1:

  • Running a YARN Application
  • Setup a YARN Development Environment
  • Writing a YARN Client
  • Submitting an ApplicationMaster
  • Writing an ApplicationMaster
  • Requesting Containers
  • Running Containers
  • Writing Custom Containers

Additional Information

Upcoming Classes

No Classes Scheduled