Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
February 25, 2015
prev slideNext slide

HDP for Telecommunications: Common Data Challenges

Hortonworks provides enterprise Hadoop for the telecommunications service provider, and Hortonworks Data Platform (HDP) is architected from the ground up with the centralized YARN-based architecture and core enterprise services for data governance, security and cluster operations that can revolutionize your telecommunications business.

As the originators of Hadoop, leaders in the developer community, and partners for your success, nobody is better to help you become a data-centric telecommunications enterprise.

Hortonworks supports most of the largest North American carriers. They subscribed with Hortonworks and adopted HDP to address three key challenges they faced because of explosive growth in their industry and the data flowing through their networks and devices:

  1. They wanted new ways to build a single view of their subscribers in order to acquire, grow and retain customers;
  2. They sought ways to predict their constantly fluctuating network traffic—using a combination of real-time, interactive and batch processes; and
  3. They needed ways to discover then monetize the immense amount of data they managed by offering new analytic services to consumers and corporate clients.

Our customers in retail, healthcare, insurance and manufacturing partner with Hortonworks for the same root reasons: to build advanced analytic applications for a single view of their business, predictive analytics and data discovery. We bring these and our direct telecom experiences to each additional provider that subscribes with Hortonworks.

Challenges Building a Single View of the Customer

All of our telco and cable clients compete with other providers to acquire, grow and retain subscriber relationships. HDP gives those service providers advantages in that highly competitive market because it powers a single view of their customers, helping them deliver high-quality, personalized customer experiences. HDP and a modern telecommunications architecture helps them acquire new customers, grow existing relationships and retain valuable customers.

Why has this single view of customers been challenging with legacy data platforms—despite the hundreds of millions of dollars already invested? There are two interrelated reasons:

  1. Customers interact with multiple devices across many channels; and
  2. Telcos store that data in multiple data repositories managed by different business groups

Before Apache Hadoop, each data capture and storage system was architected with one specific type of data in mind. Billing systems expected billing data. CRM systems expected data on the customer wireless plan and interaction with the support organization. The web team captured data about logins and clicks on the website.

For each business use, the data “schema” had to be predefined according to the expected source and structure of the data. But your customers do not behave according to the schema you create for them—they choose when, where and how they communicate with your business and with each other. And those options are continuously expanding.

The result is that as new methods of interaction arose—the Internet, text messaging, online chat, or social media—each one further fragmented your understanding of your customers.

Before you knew it, your company had multiple communication channels, isolated data siloes, and millions of customers who felt like “you just don’t know who I am.”

Challenges with Predictive Analytics

Just as your customers don’t understand (or much care) how you schema their data, your networks are mechanically indifferent to your data architecture—but architecture matters.

Telecommunications networks were architected for data as it existed years ago. Network data was typically considered “exhaust” that you might sample and store for a month or two for forensic purposes, to diagnose the root cause of a security breach or a service disruption that had already occurred. Storing that exhaust data for years was far too expensive on relational data platforms—the cost outweighed the perceived benefit.

But Apache Hadoop dramatically reduced the cost to store that data in the aggregate, and now the potential of using that data for predictive analytics on network traffic far outweighs the cost of keeping the data around longer.

Lisa Hook, CEO at Neustar (a Hortonworks customer) challenged her team to continue delivering trusted, neutral network services while simultaneously creating new businesses providing data analytics and real-time information services using authoritative, accurate, permissible-use data.

At the time, Neustar’s data architecture was insufficient to meet Hook’s challenge. Because of storage costs and capacity limitations, the company was storing only 20 terabytes of its network data (about 1% of the total data available). They were only retaining that for 45 days, on a rolling basis. The Neustar team took on the challenge of capturing 100% of the data and storing that for at least one year.

Now Neustar offers the services that Hook envisioned. For example, Neustar SiteProtect prevents malicious traffic from affecting its clients’ Web infrastructures and defuses the largest, most complex DDoS attacks. Before HDP, Neustar was storing only a fraction of incoming DNS data and the company was unable to leverage all available attack signature data to offer the most compelling service.

Challenges Discovering Value in All the New Types of Data

Some telecommunication providers process millions of phone calls per second. Now add in: services for web browsing, videos, television, streaming music and movies, text messages and email. That all adds up to a rate of data growth that can be very expensive to analyze for value.

HDP enables exploration of those new data sources and large datasets. This exploration opens insight across more data and combined disparate datasets—for rolling out new analytic services fed with data from HDP.

Juergen Urbanksi, prior CTO of Deutsche Telekom, wrote about this opportunity in his June 2014 post for Wired, “Monetizing Consumers’ Digital Exhaust with Hadoop”:

With Hadoop, telcos and cable companies can tap new revenue streams by selling intelligence on what customers are doing. Whether it’s user-generated data like call detail records and clickstreams or media content, communications and entertainment providers are storing, processing and carrying vast amounts of bits. But rarely do they turn this data into new revenue streams. Which is a pity because a telco, even in a medium-sized country, can easily generate between $50-100 million in additional revenue annually just by tapping into consumers’ digital exhaust.

Enterprise Hadoop makes this possible because it can store any data for any application that needs clickstream, sensor, social, geo-location, voice, text or server log data—alone, or in combination with existing legacy datasets. As with a single view of the customer or predictive analytics, HDP’s schema-on-read architecture makes it easy to combine data in multiple formats and at a lower cost than in RDBMS platforms, for a complete picture of your business that you can offer to your customers as a value-added service.

Hear how Sean Thorne, Director of Business Intelligence IT at T-Mobile, describe how their HDP data lake powers data discovery, “By bringing all that data into one place, we can glean insights that you couldn’t see with small sets of data.”

From Challenges to Solutions

Although telecommunications and cable providers face the same three general challenges as they build advanced analytic applications for their big data assets, we’ve seen our customers meet those challenges with many different solutions enabled by their single customer views, predictive analytics and data discovery capabilities.

In my next post in the series, I’ll discuss business impacts at carriers using HDP.

Learn More About HDP for Telecommunications

Other Hortonworks telco blog posts

About the author

Sanjay Kumar currently leads the global telecom practice at Hortonworks, helping providers leverage Hadoop to transform their data into a force of business growth and competitive differentiation. He is a telecom industry veteran with extensive experience in the strategy and execution of next generation data-centric industry solutions for enhancing customer experience, optimizing network operations and increasing revenue generation and data monetization for service providers and network operators.


Leave a Reply

Your email address will not be published. Required fields are marked *