Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
September 15, 2015
prev slideNext slide

Solving the Data Flow Challenge from the Well Site’s Jagged Edge

Over multiple conversations and espressos, Steven Witt, Senior Director of Industry Solutions at Hortonworks, and I have been exploring the diverse challenges associated with collecting, conducting and curating data flows from the well site.

Steven recently joined Hortonworks when we acquired Onyara. Steven was the Onyara CEO and co-founder. This is the first in a series summarizing our conversations, focused on how Hortonworks DataFlow collects data from the field in upstream oil and gas operations, then conducts that through to the data center and back in order to make critical decisions related to drilling and production.

The Current State of Data Flows in the Industry

This summer McKinsey & Company released an industry report entitled “The Internet of Things: Mapping the Value Beyond the Hype”. It described their findings on sensors on rigs around the world. McKinsey found that on a typical rig with 30,000 sensors:

  • Only about 1% of data can be streamed onshore for daily use, and
  • Less than 1% of the data is used for decision making


An article in CNBC—“Oil Firms Are Swimming in Data They Don’t Use”—describes the McKinsey report and other findings. The article makes the point that the data is underutilized for both real-time decisions and long-range historical analysis:

The problem is that while oilfield sensors offer real-time data on operations, the information is usually used to make immediate, binary decisions—either do this, or do that—rather than being stored, filtered and analyzed to inform future decision-making.

Data Flow Challenges in the Industry

Hortonworks DataFlow, powered by Apache NiFi, works in powerful combination with Hortonworks Data Platform, allowing oil & gas firms to make full use of their sensor data.

Hortonworks DataFlow can analyze data as it is created and generate alerts for immediate action, and it can also dynamically decide which data to conduct back to onshore servers for long-term retention and analysis.

Today, real-time data flows in the upstream oil and gas industry involve geographically dispersed, disparate business units that conduct operations in a variety of legal and network domains.

This “jagged edge” in upstream oil and gas is very complex and includes:

  • A constantly growing array of sensors (on pumps, actuators, and valves, measuring temperature, pressure, and flow),
  • Broad changes in well site infrastructure (involving thousands of data collection devices and remote communication of the data they generate),
  • Numerous SCADA data collection points (a large basin could have 10-20 OPC servers)
  • Challenged by limited data communication bandwidth through radio, satellite, and cellular channels.

The following graphic depicts a typical oilfield data flow with its inherent complexities:


The Upstream Oil and Gas Industry Is Seeking Data Flow Solutions

Operators across North America ask me a consistent set of questions about sensor data. Whether in the Permian, EagleFord, or other unconventional plays, as well as across upstream assets worldwide, companies want to know about:

  1. Collection – How can I collect real-time data when I have intermittent communications, limited computing power, and low wattage devices? How can I access my data reliably when my bandwidth is expensive and latency is high?
  2. Security – What can I do to secure my data across its entire journey from the well site, through to the data center and back?
  3. Lineage – How can I trace the digital chain of custody for real-time data generated at the well site? How can I use that lineage data to evaluate my possible infrastructure investments?
  4. Prioritization – How can I make interactive changes to prioritize data that I collect and conduct to my datacenter without needing to involve my backlogged IT team?

In the past, these upstream firms focused on optimizing communications and moving data quickly between the field devices and a centralized location. But we know that this is just a small part of the challenge.

Flexibility and configurability are equally important. An operator must be able to recalibrate and optimize drilling or production operations in real time. This ability can reduce improve both gross margins and corporate cash flow.

This is of paramount importance in today’s highly challenged commodity price climate.

Delivering Data Flow and Analytics from the Well Site to the Data Center (and back) 

Hortonworks’ recent acquisition of the company developing open source Apache NiFi promises to truly transform the Oil and Gas industry by bringing new solutions for our data flow challenges.

Hortonworks DataFlow (HDF) powered by Apache NiFi enables a modern approach for upstream companies that require end-to-end real time data flow and also an intelligent closed loop solution for long-term optimization of drilling and production. Specifically HDF will:

  1. Guarantee high fidelity data delivery and eliminate data loss even with intermittent communications and limited field infrastructure. You can now make decisions based on credible and reliable data.
  2. Provide secure data collection and movement from data generation, on data in motion to data at rest, using 21 encryption algorithms.
  3. Enable comprehensive provenance and lineage with a complete digital chain of custody for real-time multidirectional data streams, flow specific quality of service, and visibility into valuation of your field data capture infrastructure.
  4. Allow for real time prioritization of data and interactive changes to the data flow command and control tools that help you visualize the data.


Learn More

If you are an Oil & Gas leader that wants to make decisions on more than 1% of your sensor data, join us for a webinar on September 23rd at 12:00pm Central/10:00am Pacific. Joe Witt and Tim Hall, Hortonworks vice president for product management will discuss how Hortonworks DataFlow and Hortonworks Data Platform are transforming our industry.

Register for the Webinar


Leave a Reply

Your email address will not be published. Required fields are marked *