Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
March 16, 2018
prev slideNext slide

Manufacturing Industry Use Cases, Challenges, and Strategies for Dealing with Huge Data Volumes

Last month, we held the most recent Manufacturing and Transportation Customer Community call. These calls occur a couple times per quarter and act as an opportunity for leaders in both the manufacturing and transportation industries to have a roundtable discussion regarding use cases, business challenges, and best practices.

Industry communities are collaborative, industry-focused communities that are organized and curated by the Hortonworks Industry Solutions Group. The agenda and discussion is driven by the customers.

Currently, the Manufacturing Customer Community consists of 104 members across 34 companies, while the Transportation Customer Community consists of 26 members across 7 companies. These community calls started less than a year ago and membership continues to grow steadily.

The topic for this call centered around connected products and assets in the field.

Connected Products & Assets Use Cases

For one major heavy equipment manufacturer, the connected assets and products are mainly tractors and mining equipment spanning the major industries of construction, mining, agriculture, and others. There are different requirements and standards for each industry, which presents a unique set of challenges. The initial use cases involved simple machine tracking (location, fuel level, hours, etc.), and have since progressed towards predictive analytics. As the company moves to predictive analytics use cases, it has realized the need to increase frequency of data sampling. This frequency will need to increase from 1 Hz over several channels to 100 Hz over many channels for advanced use cases.

A large auto manufacturer also shared their experience on the call. The company has data coming off many vehicles which include future use cases for predictive analytics, monetizing connected vehicle data, and incorporating manufacturing data.

Challenges Being Experienced

With these use cases, each company shared the various challenges they have been experiencing along their Big Data journey.

One major heavy equipment manufacturer has massive amount of time series data that won’t stream, instead the data comes in via batch. Writing this data into data stores has been a challenge. Historically, it had been placed into relational databases, but the limit has been reached. Instead, the data is now being placed into a Hortonworks Phoenix database.

A large auto manufacturer has encountered the challenge in policy management to define access and ownership of data. This has led to trying to manage company-wide consent of data, which has proven to be a tremendous challenge. The company wants to manage who can access the data, how the data can be utilized by users, among other factors. Another challenge while supporting multiple use cases has been the excessive replication of data.

Strategies for Dealing with Huge Data Volumes

To combat the issue of IoT data volumes being excessive for relational databases, one heavy equipment manufacturer has been placing data into a Hortonworks Phoenix Database. There have also been various data ingestion challenges for the company, both technical and business level. On the technical side, ingesting data in different formats per device type has proven difficult because each device has different formats and fields. To combat this, it setup Apache Storm topologies for each device type and generation. Next, it needed to standardize data definition into the proper internal standards so that the Storm topologies mapped to those definitions. This lets consumers subscribe to the data they need.

On the business side, the focus has been on securing buy-in from people across construction, mining, agriculture, and other areas across the company. The goal has been to create data governance board and data stewards. Additionally, the company has been leveraging both cloud and on-prem environments (mostly Azure with HD Insights).

More to Come

The Manufacturing and Transportation Customer Community continues to grow, and these insights are simply a small part of what the roundtable discussions offer. Stayed tuned for details about future calls.

Leave a Reply

Your email address will not be published. Required fields are marked *