Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
March 31, 2016
prev slideNext slide

Big Data on a Budget in Oil & Gas

The recent decline in oil prices put pressure on the IT budgets of exploration and production (E&P) companies across the globe. However, these macro-economic challenges also represent an opportunity to rethink legacy IT platforms for storing and processing data.

At Hortonworks, we partner with our E&P customers to renovate their data systems by adding Hortonworks Data Platform (powered by Apache™ Hadoop®) and Hortonworks DataFlow (powered by Apache NiFi).

But this post is not only about cost savings for the sake of cost savings. We also help Hortonworks subscribers in oil and gas find use cases that redirect those savings towards uses that impact the top line.

Renovation with Connected Data Platforms from Hortonworks, allows E&P companies to supplement legacy platforms such as an enterprise data warehouse (EDW) without losing functionality. This in turn frees up economic resources that can be redirected towards innovation.

That innovation delivers actionable intelligence via new modern data applications to enable E&P companies to address the challenges they’re facing in the current economic and technological climate. These include:

  • Historic commodity pricing pressures
  • Aging assets
  • An aging workforce that will soon retire with a wealth of industry knowledge
  • Skyrocketing amount and variance in the data that needs to be collected, processed, stored, integrated, and analyzed
  • The desire to “democratize” data science, by breaking legacy system data siloes to enable analytic initiatives

 The Challenge Renovating Legacy Data Storage Platforms

Every E&P company that I’ve ever worked with has some sort of old, outdated hardware gathering dust in the corner. It sits alongside the newer, shinier servers. It still costs money to maintain, but it fills some need that forces the IT team to keep it around—whether it’s for compliance reasons, concern about downtime or because of the high cost to migrate data.

Hortonworks Data Platform (HDP) helps you free up those data storage resources that are holding part of your IT budget hostage, allowing you to use those resources to fund innovation where you need it most.

Most of our customers begin renovation by optimizing their data architecture with an Active Archive, ETL Onboard or Data Enrichment use case:

  • Active Archive relocates cold data from higher-cost storage platforms into HDP, without harming data availability
  • ETL Onboard moves ETL processing into HDP and frees processing capacity in higher-cost systems of record
  • Data Enrichment makes data already under management more valuable by combining it with additional data sources. For example, data scientists can join well log data with data on production and historical project costs

E&P firms also face a challenge bringing data from the edge of their data footprint into storage, and Hortonworks DataFlow (HDF) solves that challenge. HDF captures and conducts data to HDP where it becomes available for real-time decisions and historical analysis. Because the core technology behind HDF was originally developed at the US National Security Agency, it is extremely secure and delivers data with a fine-grain audit trail on how each bit of data arrived at its destination.

Converting Renovation into Innovation: The LAS Files Use Case

Before we jump into a detailed description of LAS Analytics with Connected Data Platforms, watch a 5-minute demo on the subject.
Watch the LAS Analytics Demo

Screen Shot 2016-03-30 at 1.14.34 PM
Reference Architecture for Hortonworks LAS Analytics Solution

Hortonworks E&P customers are already using HDP for well log analysis, as portrayed in the video. As far as HDP is concerned, a log is a log is a log. But of course analysts do data discovery on semi-structured LAS Files for a very specific reason: they want to analyze well logs to find more oil and gas.

A well log contains two basic types of information:

  1. The metadata on each log, such as the sensors that it contains, the sensor mnemonics and their units of measurement partitioned by well ID
  2. The actual readings of the well curves

Hortonworks DataFlow is the ideal tool to ingest this raw well log data-in-motion, along with its relevant metadata. In fact, as it transmits those logs to HDP for long-term storage, HDF can add more metadata describing any in-flight transformation or combination that it does.

Once the data is resting in HDP, the platform’s multi-tenant processing capabilities help engineers query the huge LAS dataset, with all of the log information from all of the wells. Data scientists can then overlay schema on the data, as it suits their needs.

Moreover, those data scientists can create a master table that includes LAS log data, along with other data from disparate datasets describing a well in formats like HTML, XML and CSV. When combined in HDP, these diverse sources build a universal history of the well that can include auction history, production data, rock formations, geolocation data and perforations.

 LAS Data Visualization Powered by LAS Log Data Stored in Hortonworks Data Platform
LAS Data Visualization Powered by LAS Log Data Stored in Hortonworks Data Platform

Data scientists and analysts then use visualization tools already on their desktops to turn queries on that data lake into compelling, intuitive graphs. Here’s an example of that using technology from Hortonworks partner Tableau:

LAS Graphing Using Tableau Over Data Stored in HDP
LAS Graphing Using Tableau Over Data Stored in HDP

Oil and gas data scientists may begin their LAS log data exploration with a specific research agenda. They can also discover new insights by taking a “random walk” through a comprehensive data lake storing years and years of detailed well logs.

Connected Data Platforms from Hortonworks power that new type of well analysis that has never before been possible at scale.

Additional Information on Hortonworks Solutions for Oil and Gas



Ridhima says:

Thanks for the good words! Really appreciated. Great post. I’ve been commenting a lot on a few blogs recently, but I hadn’t thought about my approach until you brought it up.

SEO training in Adyar

Ruby says:

Hi Sir
well aquainted with DPTS (Well Man) LOG card(Hampton Data) And Neura Log Digitizing Software(TGS) with 12 years experience. please providing well log Data. I am looking forward to provide me work from your end. If you arrange me from any other company, We are a Team .being involved in well-log Data Digitizing. Providing the services for various companies. We have an experience team well-versed with the software for well log digitizing. . We are capable to produce the output any million curve feet per month. All digitizing is performed under rigorous, proprietary quality control procedures. We always deliver accurate digital well log data in standard NLG,RPT, LAS or LIS LAS2, LAS1.2 or Ascll as your preferred format. Besides the output can be given in any desire formats .if you have your own Neura Log (NLG)digitizing software then we are also ready.Any sent /100 feet Data volume please providing data Looking forward for well log data.

Dennis says:

It’s amazing the impact that big data is having. When you combine that with a major industry like Gas and Oil, it can effect currencies around the world like 1 aud to usd.

ravi prakash vishwakarma says:
Your comment is awaiting moderation.

Dear Sir
I am RAVI , NDS8 Software NLG, RPT, LAS Report Well Log Digitization
Experience : 6 Years
Company Name: PAN India Consultant India Pvt Ltd
I Want to DATA and Process of system in Joint How?

Leave a Reply

Your email address will not be published. Required fields are marked *