We’ve been looking for the elephant in the room for some time. We knew he was there, but we just couldn’t find him. It’s clear that he is now here and his name is Hortonworks. As such, we are very excited to announce today that Index Ventures has made an investment in Hortonworks.

The elephant toy – Hadoop – has become a household name in the Big Data sector these days and we’ve been tracking it for some time at Index. The Big Data world is complex and there are many components of it, but at the core of it all is Apache Hadoop’s revolutionary compute and storage architecture for data. We think that this might be one the most significant trend in data architecture in a decade.

To understand Apache Hadoop’s impact on compute and storage architectures, it’s important to understand where it came from and why it was needed in the first place. While applications have always produced a lot of data, we only ever stored the “valuable” data. This data was labeled and tagged and ultimately stored in neatly architected databases. Meanwhile, we simply ignored data like application logs and website logs, which offered the potential for in depth user insights, but were too voluminous and lacked the structured necessary for data systems at the time. It’s also important to realize that this unstructured data is increasing at exponential rates. As IT infrastructures across the economy get instrumented with Internet-scale solutions, the amount of data product increases exponentially – which further exacerbates the problem and expands the opportunity.

Read More