Symantec helps consumers and organizations secure and manage their information-driven world by protecting digital information and online transactions.
The Symantec Cloud Platform team turned to Hortonworks to ingest an enormous volume of security logs, analyze that security metadata and then use that insight to protect its customers. Symantec now analyzes threat data much more quickly because it optimized its data architecture using the storage and processing power of HDP—for both historical and real-time analysis.
Security metadata streams into Symantec from a huge range of end points and devices, in many different formats, and the Symantec Cloud Platform team is always looking for ways to minimize the amount of time that it takes to analyze that highly variable metadata, detect threats and implement protection.
On a typical day, the Symantec team ingests 500,000 security log messages per second (around 40 billion messages daily).
Before Hadoop, the team could face processing latencies of four hours to move from analysis to detection to protection. With HDP in the cloud, the team shrunk its processing latency from four hours to an average of about two seconds. By accelerating its time to analysis, Symantec strengthens its customers’ security.
Apache Storm plays a very important role in delivering that instantaneous analysis. And because Storm works within Hortonworks Data Platform’s YARN-based architecture, its real-time streaming analysis runs alongside other tools for historical data analysis, as well as HDP’s enterprise-ready components for operations, security and data governance.
Real-time analysis accelerates time to protection, and historical analysis of all previous metadata can help Symantec strengthen its real-time algorithms—creating a virtuous cycle that strengthens protection further as more data accumulates in HDP.
The Symantec Cloud platform team particularly appreciates HDP’s cloud-based operational flexibility with Apache Ambari and Cloudbreak. David “DTL” Lin is Senior Director of Engineering for the Symantec Cloud Platform. Here’s how he describes the exciting opportunities opened up by HDP with Cloudbreak:
Cloudbreak for us is extremely valuable. When we can deploy into OpenStack and deploy into containers, and deploy onto metal, and deploy onto other partners that are out there, it gives us the flexibility that we simply didn’t have before.
Symantec’s customers benefit from that flexibility, but so do Symantec developers and data scientists. Lin sums it up like this:
Whether it’s machine learning. Whether it’s advanced analytics. Whether it’s new and different data lakes and data ponds and data tributaries and the data ecosystem. I’m really excited about seeing the kinds of security that we can create with these capabilities, once they are in the hands of all the developers.