Small companies, big data.
Big data is sometimes at odds with the business-savvy entrepreneur who wants to exploit its full potential. In essence, the business potential of big data is the massive (but promising) elephant in the room that remains invisible because the available talent necessary to take full advantage of the technology is difficult to obtain.
Inventing new technology for the platform is critical, but so too is making it easier to use.
The future of big data may not be a technological breakthrough by a select core of contributing engineers, but rather a platform that allows common, non-PhD holding entrepreneurs and developers to innovate. Some incredible progress has been made in Apache Hadoop with Hortonworks’ HDP (Hortonworks Data Platform) in minimizing the installation process required for full implementation. Further, the improved MapReduce v2 framework also greatly lowers the risk of adoption for businesses by expressly creating features designed to increase efficiency and usability (e.g. backward and forward compatibility). Finally, with HCatalog, the platform is opened up to integrate with new and existing enterprise applications.
What kinds of opportunities lie ahead when more barriers are eliminated?
The current situation is similar to data processing servers before Cloud-based solutions like Amazon’s S3 and Elastic MapReduce (EMR). In the early 2000s, entrepreneurs had to spend a great deal of time running and maintaining servers in-house that ran their business. When cloud-based solutions entered, it allowed developers to focus on using servers to enhance their business rather than be bogged down by its limitations. This revolution allowed a small 10-person startup and focus 100% of their attention on innovation and bringing value to their customers rather than on the limitations of the technology. Making the platform simpler and easy-to-use will have the same effect for big data.
Greater Adoption through Innovation
Buoyed by the efforts of the Apache Hadoop community, key enterprise software players have improved access to the platform. Hadoop platforms like HDP democratizes big data by providing easy-to-use and wide spread access for the greater community. Efforts like these help to push the technology past the early adopters to mass adoption markets. However, companies at this level focus on the invention of the platform. Sustainable technological growth arises only when companies use that invention in new, unexpected ways.
Business-to-Business (B2B) Applications
Beyond the large players like Yahoo!, Netflix, smaller (often non-Hadoop) operations have sprung up all across the country around the idea of big data. One well-known example is Splunk, which created its own propriety platform to process and analyze big data on a large scale for companies that need it. The benefit of companies like Splunk is their ability to identify desired elements from a variety of sources – machine data, cloud architectures, visual dashboards, and Hadoop – and package their offerings into a single product.
Another more recent entry is Durham, NC based company named EvoApp. The company has built a big data platform called Bermuda specializing in customer and market intelligence. Continuing the trend begun by Splunk, they focus primarily on analytics, though betting its speedy and accurate runtimes will be a significant differentiator in the market place.
Startups are also working toward using big data to solve difficult problems for the everyday consumer.
One innovative use of big data is with a mobile app called Parker by Streetline. In major cities, locating empty parking spaces can be a major concern for commuters. City governments and app developers alike are using big data to help car drivers locate available parking spaces more effectively by having modified parking meters broadcast their availability to the targeted servers that are paired with a notification system.
Another, The Climate Corporation, tailors its insurance policies based on weather-related risk factors that could negatively affect or potentially destroy entire crop yields. The company uses big data to make weather and soil predictions to more intelligently bet against crop failure and issue policies accordingly. The customer may not know (nor care) how the system works, but recognizes the value in being issued tailored insurance policies based on their personal risk factors.
Limits to Widespread Adoption
Imagine the possibilities of every high school student dreaming of the software possible with Hadoop in much the same way they now do for smartphone apps. While technology champions are necessary to invent and evangelize young technologies, the real technological boom occurs when mainstream developers get involved and begin to push the limits of the platform. As more startups innovate using big data technologies, we can look forward to seeing a new majority.