Over multiple conversations and espressos, Steven Witt, Senior Director of Industry Solutions at Hortonworks, and I have been exploring the diverse challenges associated with collecting, conducting and curating data flows from the well site.
Steven recently joined Hortonworks when we acquired Onyara. Steven was the Onyara CEO and co-founder. This is the first in a series summarizing our conversations, focused on how Hortonworks DataFlow collects data from the field in upstream oil and gas operations, then conducts that through to the data center and back in order to make critical decisions related to drilling and production.
This summer McKinsey & Company released an industry report entitled “The Internet of Things: Mapping the Value Beyond the Hype”. It described their findings on sensors on rigs around the world. McKinsey found that on a typical rig with 30,000 sensors:
An article in CNBC—“Oil Firms Are Swimming in Data They Don’t Use”—describes the McKinsey report and other findings. The article makes the point that the data is underutilized for both real-time decisions and long-range historical analysis:
The problem is that while oilfield sensors offer real-time data on operations, the information is usually used to make immediate, binary decisions—either do this, or do that—rather than being stored, filtered and analyzed to inform future decision-making.
Hortonworks DataFlow, powered by Apache NiFi, works in powerful combination with Hortonworks Data Platform, allowing oil & gas firms to make full use of their sensor data.
Hortonworks DataFlow can analyze data as it is created and generate alerts for immediate action, and it can also dynamically decide which data to conduct back to onshore servers for long-term retention and analysis.
Today, real-time data flows in the upstream oil and gas industry involve geographically dispersed, disparate business units that conduct operations in a variety of legal and network domains.
This “jagged edge” in upstream oil and gas is very complex and includes:
The following graphic depicts a typical oilfield data flow with its inherent complexities:
Operators across North America ask me a consistent set of questions about sensor data. Whether in the Permian, EagleFord, or other unconventional plays, as well as across upstream assets worldwide, companies want to know about:
In the past, these upstream firms focused on optimizing communications and moving data quickly between the field devices and a centralized location. But we know that this is just a small part of the challenge.
Flexibility and configurability are equally important. An operator must be able to recalibrate and optimize drilling or production operations in real time. This ability can reduce improve both gross margins and corporate cash flow.
This is of paramount importance in today’s highly challenged commodity price climate.
Hortonworks’ recent acquisition of the company developing open source Apache NiFi promises to truly transform the Oil and Gas industry by bringing new solutions for our data flow challenges.
Hortonworks DataFlow (HDF) powered by Apache NiFi enables a modern approach for upstream companies that require end-to-end real time data flow and also an intelligent closed loop solution for long-term optimization of drilling and production. Specifically HDF will:
If you are an Oil & Gas leader that wants to make decisions on more than 1% of your sensor data, join us for a webinar on September 23rd at 12:00pm Central/10:00am Pacific. Joe Witt and Tim Hall, Hortonworks vice president for product management will discuss how Hortonworks DataFlow and Hortonworks Data Platform are transforming our industry.