Apache Hadoop YARN is the modern distributed operating system for big data applications. It morphed the Hadoop compute layer to be a common resource management platform that can host a wide variety of applications. Many organizations leverage YARN in building their applications on top of Hadoop without themselves repeatedly worrying about resource management, isolation, multi-tenancy issues, etc.
In this talk, we’ll start with the current status of Apache Hadoop YARN – how it is used today in deployments large and small. We’ll then move on to the exciting present and future of YARN – features that are further strengthening YARN as the first class resource management platform for data centers running enterprise Hadoop.
Speakers: Vinod Vavilapalli / Wangda Tan
HDFS’s evolution and Ozone Introduction
HDFS has several strengths: horizontal scaling of IO bandwidth over petabytes of storage. Further it provides very low latency metadata operations and scales to over 60K concurrent clients. Apache Hadoop 3.0 recently added Erasure Coding, Multiple NameNode support and HDFS federation improvements. We will talk about the latest HDFS enhancements in Apache Hadoop 3.0 and what the road ahead might look like…
One of HDFS’s limitations is scaling number of files and blocks in the system. We describe a radical change to Hadoop’s storage infrastructure with the upcoming Ozone filesystem. It allows Hadoop to scale to tens of billions of files. Ozone fundamentally separates the namespace layer and the block layers. Further, the use of RAFT protocol has allowed the storage layer to be self-consistent. We will provide a high-level overview of the Ozone architecture.
Speakers: Jitendra Pandey / Arpit Agarwal