This is the fourth post in a series that explores the theme of enabling diverse workloads in YARN. See the introductory post to understand the context around all the new features for diverse workloads as part of YARN in HDP 2.2.
When it comes to managing resources in YARN, there are two aspects that we, the YARN platform developers, are primarily concerned with:
Resource allocation: Application containers should be allocated on the best possible nodes that have the required resources and
Enforcement and isolation of Resource usage: On any node, don’t let containers exceed their promised/reserved resource-allocation
From its beginning in Hadoop 1, all the way to Hadoop 2 today, the compute platform has always supported memory based allocation and isolation.…