Meet the Committer: 3 Minutes on Apache Hadoop YARN with Arun Murthy
We’re continuing our series of quick interviews with Apache Hadoop project committers at Hortonworks.
This week – as Hadoop 2 goes GA – Arun Murthy discusses his journey with Hadoop. The journey has taken Arun from developing Hadoop, to founding Hortonworks, to this week’s release of Hadoop 2, with its Yarn-based architecture.
YARN turns Hadoop from a single use system for batch data processing into a multi-use platform for storing and processing data in many ways other than batch.
MapReduce used to do two things at once: data processing and resource management. Now YARN does resource management, and MapReduce is just another application that runs natively in Hadoop. With the launch of Hadoop 2, YARN is the Hadoop operating system.
Now other applications can run simultaneously IN Hadoop as peers to MapReduce:
- Tez can do interactive query
- Storm can handle streaming data
- Giraph can handle graphs
- And so on…