Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
April 02, 2018
prev slideNext slide

Transportation Industry Use Cases, Challenges, and Strategies for Dealing with Huge Data Volumes

At a recent Manufacturing and Transportation Customer Community call we discussed connected products and assets in the field. These calls occur a couple times per quarter and act as an opportunity for leaders in both the manufacturing and transportation industries to have a roundtable discussion regarding use cases, business challenges, and best practices.

Industry communities are collaborative, industry focused-communities that are organized and curated by the Hortonworks Industry Solutions Group. The agenda and discussion is driven by the customers.

Currently, the Manufacturing Customer Community consists of 104 members across 34 companies, while the Transportation Customer Community consists of 26 members across 7 companies. These community calls started less than a year ago and membership continues to grow steadily.

The topic for this call centered around connected products and assets in the field.

Connected Products & Assets Use Cases

Some of the top companies in the transportation industry discussed how connected products and assets play a big role in their businesses. One large railway company shared that Positive Train Control (PTC) is a major driver for their business. PTC is a system designed to automatically stop a train before accidents occur. Currently they have regional implementations underway, and there are regulatory requirements to reach milestones for PTC Failsafe. The goal is to upgrade their networks to the point where everything in the US operations has Internet of Things (IoT) devices along their tacks. These IoT devices help monitor train movement, safety systems, and predictive maintenance.

For this railway company, the initial use cases that sparked its journey with Big Data were centered around the Federal Rail Administration (FRA) regulatory reporting requirements. The FRA requires reporting for all emergency events, such as emergency stopping, including the information surrounding the events and the reasons why the events occurred.

Another driver was real-time operational dashboards for monitoring trains and events from a support point of view. The company is starting to get into near real-time use cases involving data from locomotives and infrastructure, including wayside devices, switches, signaling, and lights. These devices produce sensor data and high volumes of unstructured data from test cars on the track, which allows for track predictive maintenance analytics.

Challenges Being Experienced

Surrounding these use cases, each company shared the various challenges they have been experiencing along their journey. One top railway company discussed how its biggest challenges have been ingesting varying types of real-time and batch data. Its expectation is that the volume of this data will grow to be in the Petabyte range within the next 18 months. This data growth is driven largely by unstructured video and photo data.

The company is also moving towards near real-time use cases, which involve data from locomotives and infrastructure. This brings about the challenges of aligning all data based on time. The data must synchronize down to the 100th of a second. This level of detail enables the organization to know exactly what’s occurring via operational dashboards. Other challenges for transportation companies have included trying to incite a culture change within its organization for adaption to Big Data, and the struggle to bring in additional talent to assist.

Strategies for Dealing with Huge Data Volumes

Each of these transportation companies is dealing with huge volumes of data, which requires specific strategies to properly store, manage, and analyze. As previously mentioned, a top railway company has the challenge of addressing FRA emergency stop reporting requirements. To do this, it ingests logs using Nifi and Kafka topics to organize the information. By getting the data into coherent structures it is possible to fun monthly reports.

Data in the cloud and on-prem are both components of this strategy. It started with Microsoft Azure in development environments, however with the data volumes at hand and the desire for quick throughput, the decision was made to move to on-prem for production instances. The production moved from VMWare to a “Bare Metal” implementation of Hortonworks.

Throughout its Big Data journey, it was important for the railway company to utilize open source technology as much as possible, to avoid vendor lock-in and have more rapid innovation. This commitment to open source, along with premier support and mentorship, helped make the partnership with Hortonworks an obvious match.

More to Come

The Manufacturing and Transportation Customer Community continues to grow, and these insights are simply a small part of what the roundtable discussions offer. Stayed tuned for details about future calls.


Apoorva Verma says:

By investing less than hundred thousand dollars a month, IoT implementors in transportation industry are realizing business benefits of millions of USD within 4-6 months. From loading cargo in the trailer to the last mile delivery, the Industrial Internet of Things has taken the transportation industry to the next level with features like: Geofencing, Fleet Tracking, Real-time weight measurement, Remote area monitoring etc. which help in optimum resource allocation and management.

Leave a Reply

Your email address will not be published. Required fields are marked *