Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
August 17, 2012
prev slideNext slide

Apache Hadoop, the Energy Softgrid and my Imaginary Tesla

This week, I spent some time and enjoyed speaking at the Softgrid 2012 conference in San Francisco. It was a great collection of speakers and attendees and opened my eyes to some Hadoop driven possibilities that not only differentiate utilities companies but will also transform our day-to-day lives.

The conference focused on software (in this case intelligent analytics) as a competitive advantage to enable value and growth for utilities.  These often large and historically conservative organizations have moved beyond the notion that their sole business is to distribute electric power efficiently, reliably, and cost-effectively to consumers. They now rely on analysis of massive amounts of data they already collect from smart meters and existing networks about distribution and consumption, and are taking progressive action on that data.

As we have seen in other markets, such as Financial Services and Retail, data is becoming the currency for an energy market transformation.

While I am not a Prius, Volt or Tesla (unfortunately) driver, I am sensitive to eco-friendly causes that have a large and immediate impact on the way we consume our natural resources.  I feel I am like many consumers in that saving five to ten or twenty dollars on my monthly bill is important but honestly I am more interested in knowledge and insight into usage and just how green I am. Call me an armchair activist I guess.

This conference opened my eyes to a broad range of possibilities for the utilities to really change the way we live and increase their bottom line through green tech. Here are two possible uses of big data in Energy.

Generation vs. consumption

My friend and Hortonworker, Rikin Shah, walked me through one potential use case of Hadoop in energy before I even left for San Francisco.  There is no such thing as a big battery that will hold any excess energy that is generated by the utility companies.  That means if we burn the coal or split the atoms we have to use all the energy produced or it gets wasted.  The challenge is that the consumption curve is erratic and this leads to waste, as we have to produce more than necessary to avoid a brownout when consumption extends beyond generation.  It is difficult at best to predict consumption.  However we can get a lot better through data.

In some companies they use smart meter technology that can automatically read meters at any desired interval. For many organizations this is once or twice a month, however they are moving to collect readings every four hours. That’s 6/day x 30 – 180x growth in data points collected per month per house! Why shouldn’t this be eve more frequent?  Well the amount of data is massive.  What if we could extend this to near continuous meter reads and analyze in near real time.  It could get us to better predictability of spikes and reduce the padding between production and consumption.  Further, new technologies (such as Nest thermostats) bring this direct touch to the point of consumption.  As we evolve, certainly smart light switches and wall outlets could all be tied into the grid to provide real touch with real consumption.  Perhaps we combine usage data with detailed weather data that drills down to a square meter.  The profound analysis could revolutionize help us conserve through near real time production.

Individually provisioned consumers

My phone goes everywhere I go.  I use it… a lot and I am often found borrowing a charger or asking someone if I could plug in and give it a charge.  They pay the bill.  This is ok when it is just a few kilowatts but what if I was out of electricity and I was at your house with my (pretend) Tesla?  I will presume I would consume much more than a few kilowatts to give my car a jump.  Currently, there is no way to track and provision usage per device and through to the owner.    Why not?  In part it is a data problem.

A more intelligent provisioning system would require a massive registry of devices and locations and the ratings engine…  well, that would be heck of an algorithm and would require some pretty heavy computation.  If you think the telecom call data record analysis is complex, this would be insane.  Devices come and go and there are a magnitude more options for plugging in.

Enter Hadoop

Apache Hadoop with massively parallel process and widespread storage.  Many utilities companies are already enjoying the benefits of this open source data platform.  There are probably a few innovations and some fairly substantial capital expense necessary to make a fully connected grid a reality but it is not that far off and definitively a possibility.  When I was a kid, my dad used to yell at me when I left lights on in the house.  Maybe if the light switch registered my presence, I could ask my pop to take the cost out of my allowance.

Back to our green Earth

IT was an interesting day full of great speakers and roomful of people very interested in how technology and software in general can aid in creating amore efficient grid.  Sure, a more intelligent grid will reduce costs through more intelligent production, but it can also change the way we all think about consumption.

I’m still waiting to get that Tesla!


Leave a Reply

Your email address will not be published. Required fields are marked *