Kiss the Weatherman

Weather Hurts

Catastrophic weather events like the historic 2011 floods in Pakistan or prolonged droughts in the horn of Africa make living conditions unspeakably harsh for tens of millions of families living in these affected areas.  In the US, the winter storms of 2009-2010 and 2010-2011 brought record-setting snowfall, forcing mighty metropolises into an icy standstill. Extreme weather can profoundly impact the human kind.

The effects of extreme weather can send terrible ripples throughout an entire community.  Unexpected cold snaps or overly hot summers can devastate crop yields and forcing producers to raise prices. When food prices rise, it becomes more difficult for some people to earn enough money to provide for their families, creating even larger problems for societies as a whole.

The central problem is the inability of current forecasting models to more accurately predict large-scale weather patterns.  Weathermen are good at predicting weather but poor at predicting climate.  Weather occurs over a shorter period of time and can be reliability predicted within a 3-day timeframe.  Climate stretches many months, years, or even centuries.  Matching historical climate data with current weather data to make future weather and climate is a major challenge for scientists.

Big Wet (or dry) Data

Current practices in predicting extreme weather and climate only go so far.  The World Meteorology Organization uses a combination of readings from 15 satellites, 10,000 land stations, 7300 sea vessels, 3000 aircrafts and a host of other sources for their analysis.  The organization standardizes measurements across the globe, which helps a network of meteorologists to track global temperatures, land/air pressures, precipitation, humidity, wind speeds, and a myriad of other parameters.

The measurements produce an incredible amount of data.  The National Center for Atmosphere Research has over 250 terabytes of weather data for analysis freely available for anyone to download going back 40 years.  Using these methods allows them to forecast roughly 21 days into the future but are notoriously inaccurate.

A good example is this past winter from 2011-2012, when most weather reports predicted a normal-to-cold winter.  By the end of the season, we saw record high temperatures all across the US.  Things can get better.

Better Climate Predictions

But what would happen if big data could be used to more accurately predict destructive weather?  Several companies and organizations are working toward harnessing the power of big data to enable them to better prepare for an uncertain future.

Earth Risk, a startup based out of San Diego, is solving this issue using historical weather data going back 60+ years as a basis for their numerical prediction models in an attempt to more accurately predict weather patterns better than ever before.

Knowledge of multi-month weather trends can provide enormous benefits.   Dams and levies can be reinforced.  Market expectations can be modified to match actual crop yields.  Governments can better prepare their cities and towns with winter equipment to get ready for major storms.

Using big data analytics, they are able to run 4 million calculations a day that allows the company to create a 40-day forecast with around 26% greater accuracy than current methods – predicting the unseasonably warm winter sooner than almost any other source.

NASA uses Big Data

Going into the government sector, NASA’s Center for Climate Simulation Center (NCCS) has begun to make moves toward better utilizing its big data.  CSC stores 32 petabytes of data across 1,200 nodes eight million observations a day and then turning them into visualizations scientists can use create better forecasts.  Called the “Visualization Wall”, 16 Linux-based servers project weathers patterns on a 17×6 foot wall giving them a high resolution image of all the accumulated data.

The Discover cluster within CSC has already begun to produce success stories using the Hadoop and MapReduce architectures.  The cluster was introduced in 2009 with the goal of providing qualitative and quantitative assessment of the Hadoop and MapReduce’s benefits.  The product of the effort was a 30-year comprehensive analysis of global weather data and multi-century climate analysis for a panel on climate change.   Using this information they were able to create a high–fidelity global simulation of global cloud and hurricane patterns using predictive analytics.

And Yes, Weather Insurance

Beyond predicting the weather, The Climate Corporation is also practicing new ways to use big data analytics for insurance policies to farmers on their crops.

Insurance policies work when a company bets that the odds of a paying out to a policyholder for particular catastrophic event is less than total amount of revenue it receives monthly from policyholders.

The Climate Corporation custom tailors its insurance policies based on weather-related risk factors that could negatively affect or potentially destroy entire crop yields.  The company developed a system for reviewing measurements from 2.5 million locations with 150 billion soil observations and generating 10 trillion data points the company uses for its insurance pricing.  This allows them to use weather and soil predictions to more intelligently bet against crop failure and issue policies accordingly.  Payouts are then dispersed automatically, based on measured weather conditions and the terms of the policy.

The granularity of the company’s system is impressive.  They are able to know the difference in temperature between two points 2.5 miles away.  For a single farm, they run simulations of weather for the next 730 days – 10,000 times.  This allows them to generate policies with an incredible level of precision, made possible using big data technologies.

The effect of this is profound.  Farmers how have a way to more adequately protect themselves against extreme weather in a more effective, data driven way, unlike never before.

Big Data for the Weather

The utilization of big data frameworks in climate science is still in its infancy.  However, a lot of growth is possible and we are only just beginning to see how big data can help have a positive impact on our lives.

Categorized by :
Big Data Business Analytics Hadoop Ecosystem Other

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
HDP 2.1 Webinar Series
Join us for a series of talks on some of the new enterprise functionality available in HDP 2.1 including data governance, security, operations and data access :
Contact Us
Hortonworks provides enterprise-grade support, services and training. Discuss how to leverage Hadoop in your business with our sales team.