If when we started building an Apache Hadoop team at Yahoo!, someone had told me that in the future we would partner with Microsoft to improve Hadoop’s performance on Windows, I would have found the prediction hard to believe. The first time a Microsoft executive suggested that they would like to work with us to improve Apache Hadoop, I told them I found their proposal “mind-bending”. I also told them that if we could do it the right way, I liked the idea. Our core mission is to bring Apache Hadoop to the widest possible user base and Windows and SQL Server have a very large user bases.
Why is adding a fraction of the Microsoft Windows, Azure and SQL Server user bases to the Hadoop community a good thing for Apache Hadoop? Microsoft technology is used broadly across enterprises today. Ultimately, open source is all about community building. A growing user community feeds a virtuous circle. More users means more visibility for the project. Their successes fuel the adoption of the project by more users. More users mean more folks who will ultimately become contributors or committers. This makes the code evolve more quickly, which allows it to satisfy more use cases and hence attract more users, which further drives the project forward. As the number of users and developers grow, more companies will decide that they can build hardware, tools, applications and services for Apache Hadoop users. Growth of the ecosystem allows more users to solve more problems with Apache Hadoop, driving further growth, etc. Feeding this virtuous cycle is what Hortonworks is all about.
So what does “doing it right” mean to me? Doing it right involves really understanding the dynamics of open source and striving to work with the community to make Apache Hadoop better for everyone, while also meeting your business goals. When done right, there is a kind of magic to the process, like Stone Soup. Our partners have to accept the constraints of the process if they want the magical outcome. Hortonworks is an open source company. We will work with any company committed to using Apache Hadoop. Hadoop is owned by the committers, PMC and the Apache Foundation and not by any company. No one can push anything into Apache Hadoop unless it makes sense to its community.
So why did Hortonworks partner with Microsoft? Because we see our interests as aligned. Microsoft did their homework before coming to us. They are perfectly capable of making any software work better on Windows by themselves. What they want to do is engage the community in a dialog about how to make Apache Hadoop better for everyone in a way that also makes it work a lot better on Windows. We’re signed up to work with them on that because we see this as a win-win for Microsoft’s users and for the wider Apache Hadoop community.