Luminar Extracts Insight with Hadoop
Luminar is the leading big data analytics and modeling provider uniquely focused on delivering actionable insights on U.S. Latino consumer to advertisers. The company uses Hortonworks Data Platform to store and process the data that feeds its targeting models. HDP delivered the ability to capture eight times more data and also reduced processing time from hours to minutes.
Moving Beyond Random Samples of Data
Typical market research studies involve three steps:
- Identify a target segment for the study
- Create a sample panel that matches that segment
- Study the sample panel and apply findings to programs for the entire population
Luminar did not want to take samples of Latinos living in the United States; they wanted data on all of the Latinos. And they did not want to store only some transactions from one or two sources; they wanted to acquire and save as many transactions as possible from as many different sources as possible.
Luminar wanted to build models based on empirical insights, not sample insights.
Luminar’s Growing Latino Data Set
So Luminar set about building the largest U.S. Latino customer datastore it could obtain, with as many transactions as it could capture for adult U.S. Latinos. With its data set consisting of nearly 15 million adult U.S. Latinos, Luminar was in a position to investigate some of its clients’ most challenging hypotheses about what those customers wanted.
Those clients could answer both the questions they had already formulated and also discover unexpected trends. All of this would help Luminar’s clients isolate the unique, actionable marketing signals that they wanted to understand.
Luminar gathered about 300 data sources such as credit card transactions, set top boxes, voter records, and social data. This data came in structured formats like CRM records.
There was also a massive amount of data that came in less structured forms like transaction logs, geolocation records, social media messages and web clickstream data.
Luminar’s Need for a Big Data Storage and Processing Solution
As Luminar’s data set grew in size and scope, the company was able to offer more reliable market research data. The Luminar IT team also needed a storage platform that could handle the size and complexity of the company’s growing data set.
As Luminar grew, the company pulled in more and more diverse data (such as transaction logs, geolocation records, social comments and clickstream data). These formats were not easily parsed into columns and rows with pre-defined attributes.
Luminar needed to assure that the company could continue to pull in diverse sources of data and generate reports quickly and efficiently to best serve its clients.
In addition, Luminar needed sufficient data storage capacity to meet its current growth and future needs.
Luminar began evaluating Apache Hadoop as a solution to meet its cost, efficiency, and flexibility needs.
The Hortonworks Differences: Transparency, Partnership & Technical Leadership
Luminar compared Hortonworks and other solutions for their ability to support their growing data workloads, without the risk of getting locked in with something that might limit them in the future. They appreciated the Hortonworks commitment to nurturing Hadoop in the open source community.
The evaluation team downloaded Hortonworks Sandbox, a one-node Apache Hadoop cluster, and tested the Sandbox tutorials against Luminar use cases. This was an important step for Luminar before committing to a particular Apache Hadoop distribution.
Finally, Luminar Insights appreciated the Hortonworks team’s commitment to partnering with them to deliver real-world benefits over the long run. Throughout the sales process, Hortonworks responded promptly to Luminar’s technical questions and showed enthusiasm to educate them about Hadoop. This gave Luminar confidence that as their data set grew, Hortonworks would always be there to help them grow their Hadoop environment in lock step with the needs of their business.
After evaluating these factors and the company’s needs, Luminar chose Hortonworks Data Platform and Hortonworks enterprise support.
The Results: Data on More Than Half of Adult U.S. Latinos, Stored at a Lower Cost per Terabyte
The Luminar team believes its business is stronger for adopting HDP and partnering with the Hortonworks support team to make the most of their new, modern data architecture. During its first year with Hortonworks, Luminar enjoyed the benefits of easy integration, improved storage capacity and a quantum leap in processing speed.
Easy Integration: Amazon Cloud, R, Talend, and Tableau
One of Luminar’s key reasons for choosing Hortonworks was HDP’s interoperability with other pieces of their storage and business intelligence framework. HDP integrated easily with their data store and Amazon Cloud platform.
Luminar used Talend to interconnect the various pieces of its infrastructure, on premise and in the cloud. Talend is a Hortonworks certified technology partner for data integration.
Tableau is Luminar’s data visualization portal. Tableau’s attractive visualization capabilities helped Luminar’s customers see and understand their data, stored on HDP. Tableau Online provides a cloud-based layer for login credentials for each customer’s proprietary data sets. Tableau is a Hortonworks certified partner for data visualization.
Because of these low barriers to adopting HDP, Luminar immediately felt its value to the business.
Better Predictions: Increased Sources of Raw Data from 300 to 2,000
Before switching workloads to HDP, Luminar was ingesting 2 terabytes of new transactional data per month, from 300 raw sources. Now, Luminar appends new transactional data every day from 2,000 raw sources, which adds up to 15 terabytes per month. With Hadoop, Luminar increased the amount of data it captured by a factor of eight.
Agile Analytics: From 3 Days to 3 Hours
Luminar updates its data with constant ingest of new transactional data. Before Hadoop, it could take Luminar several days to ingest and join data to refresh one of their models. Now it takes only hours.
This allows Luminar to tackle a greater number of projects and to process data for more clients in parallel. The company avoids an analytics bottleneck.
Oscar Padilla is Luminar Insight’s Vice President of Strategy, who frequently meets with their customers.
I need it to work flawlessly for our customers, and it does.
“The biggest compliment from the front end,” says Oscar, “is that everything is seamless. We have a true big data environment. I need it to work flawlessly for our customers, and it does.”
Moving from 1:1 Client Solutions to Mass Market Products
Luminar grew its business by delivering its data analytics services on a client-by-client basis where they gathered information in batch and then incorporated that into executive presentations.
The company is now building custom portals for select high-value clients, through a Tableau web interface. This will give those clients near real-time (and constantly updated) access to their most important data. Luminar sees this innovation as an important next step to scaling their business.
Luminar Insight’s President, Franklin Rios, sees a bright future in the coming year:
“We are growing our business and building our client portfolio.
…to listen for what U.S. Latino consumers want and to communicate that voice to more clients through innovative applications running in Hadoop.
We are the leading data analytics company applying big data to the U.S. Latino consumer market.
In the coming year, we are going to be improving our ability to listen for what U.S. Latino consumers want and to communicate that voice to more clients through innovative applications running in Hadoop.”
Based on the success of their first year with Hortonworks Data Platform, Luminar Insights recently renewed their support relationship at three times the level of their first year with Hortonworks.
Luminar, a unit of Entravision Communications Corporation, is the first analytics and modeling provider that uniquely serves the U.S. Latino market. The company uses its proprietary technology to transform Latino consumer data into insights and business intelligence. Luminar unlocks big data and develops predictable models that reflect the behaviors and attitudes of today’s Latino customers across a wide range of vertical categories.
The Luminar team is comprised of data analysts and scientists with expertise in Latino marketing. Through its proprietary cultural filter algorithms, Luminar accelerates its clients’ business growth with improved targeting and management of the entire marketing lifecycle. These clients can then win more business with adult U.S. Latinos, representing $1.5 trillion in present purchasing power.
- Integrated seamlessly with Amazon Cloud, R, Talend and Tableau
- Captured 8X more data than prior platform
- Cut processing time from days to hours