How To Perform Spatial Analytics with Hive and Hadoop
One of the big opportunities that Hadoop provides is the processing power to unlock value in big datasets of varying types from the ‘old’ such as web clickstream and server logs, to the new such as sensor data and geolocation data.
The explosion of smart phones in the consumer space (and smart devices of all kinds more generally) has continued to accelerate the next generation of apps such as Foursquare and Uber which depend on the processing of and insight from huge volumes of incoming data.
In the slides below we look at a sample, anonymized data set from Uber that is available on Infochimps. We step through basics of analyzing the data in Hive and learn how a new using spatial analysis decide whether a new product offering is viable or not.
Specifically, we look at:
- The Uber dataset itself, which contains more than 1.1 million GPS readings covering 25,000 Uber trips.
- New SQL windowing features in Hive 11 that make slicing and dicing datasets simple.
- The Spatial Framework for Hadoop from ESRI, and how it makes analyzing geospatial data including GPS signals simple.
- Apply spatial analytics to understand basic facts about the Uber data, including average trip length.
- Use more sophisticated spatial analytics to determine the viability of a possible new product.
You can test out the whole tutorial using your friendly neighborhood Hadoop-in-a-box: Hortonworks Sandbox.
Try it with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.