Hello from the Metron PM and Engineering Team
Over the last few months, you may have read a series of blogs written by the Metron Product Management and Engineering teams on CyberSecurity and Analytics and the role that Big Data / Hadoop / HDP plays in this space. In December 2015, James Sirota, Director of Security Solutions at Hortonworks, authored “Leveraging Big Data For Security Analytics,” which describes how the Cisco OpenSOC project evolved into Apache Metron, a Big Data security analytics framework. Apache Metron integrates a variety of open source big data technologies in order to offer a centralized tool for cyber security monitoring and analysis.
Michael Schiebel, CyberSecurity Strategist at Hortonworks, introduced himself via his blog series, “echo ‘Hello, world‘” providing a glimpse into today’s challenges faced by a Security Operations Center (SOC) analyst. He explains why looking at alerts generated by rules engines in point security solutions and security information event management (SIEM) tools are the wrong approach. Rather, the need for next generation security tools to get to the right data quickly is vital for SOC Analysts to monitor, analyze and perform front-line investigations of cyber security risks. Next in the series, “CyberSecurity: The end of rules are Nigh” and “Why Context Matters”, Michael Shiebel describes how to find the few alerts that matter and the importance of a single platform that stores telemetry data (logs, network, packet capture, etc.), as well as provide data analysis tools. These tools help analysts to “look where the bullet holes aren’t” as described by Schiebel.
Today, the Hortonworks product management and engineering teams are kicking off a multi-part blog series on Apache Metron, a next gen security analytics application built by the Apache Metron Community led by Hortonworks. Over the course of the next few weeks, the team will release articles covering key Apache Metron topics:
Each of these blogs will provide an intro to their respective topics and the deeper level details will be continued in HCC articles in the Hortonworks Community Connection in the new CyberSecurity Track.
Roots of Apache Metron
To understand Apache Metron, we have to first start with the origins of the project which emerged from the Cisco Project called OpenSOC. The below diagram highlights some of the key events in the history of Apache Metron.
2005 to 2008
The Problem – Cyber crime spiked significantly and a severe shortage of security talent arose. The first set of companies alerted to this issue are high profile banks and large organizations with interesting proprietary information to state sponsored agents. All of the best investigators and analysts were gobbled up by multinational banking and financial services firms, large hospitals, telcos, and defense contractors.
The Rise of a New Industry, the Managed SOC – Those who could not acquire security talent were still in need of a team. Cisco was sitting on a gold mine of security talent that they had accumulated over the years. Utilizing this talent, they produced a managed service offering around managed security operations centers.
The Age of Big Data Changed Everything – The Age of Big Data arrived, bringing more streaming data, virtualized infrastructure, data centers emitting machine exhaust from VMs, and Bring Your Own Device programs. The amount of data exploded and so did the cost of the required tools like traditional SIEMs. These tools became cost prohibitive as they changed to data driven licensing structures. Cisco’s ability to operate the managed SOC with these tools was in jeopardy and security appliance vendors took control of the market.
OpenSOC is Born and Hadoop Matures – Cisco decided to build a toolset of their own. They didn’t just want to replace these tools but they wanted to improve and modernize them, taking advantage of open source. Cisco released its managed SOC service to the community as Hadoop matured and Storm became available. It was a perfect combination of a use case need and technology. OpenSOC was the first project to take advantage of Storm, Hadoop, and Kafka, as well as migrate the legacy ways into a forward thinking future type paradigm.
September 2013 thru April 2015
The Origins of Apache Metron – For about 24 months, a Cisco team, led by their chief data scientist James Sirota, with the help of a Hortonworks team, led by platform architect Sheetal Dolas, worked to create a next generation managed SOC service built on top of open source big data technologies. The Cisco OpenSOC managed SOC offering went into production for a number of customers in April of 2015. A short time after, Cisco made a couple of acquisitions that brought in third party technologies transforming OpenSOC into a closed source, hardware based version.
OpenSOC Chief Data Scientist Joins Hortonworks – James Sirota, the chief data scientist and lead of the Cisco OpenSOC initiative, leaves Cisco to join Hortonworks. Over the course of the next 4 months, James starts to build a rock star engineering team at Hortonworks with the focus of building an open-source CyberSecurity application.
Metron Accepted into Apache Incubation – Hortonworks, with the help and support of key Apache community partners, including ManTech, B23 and others, submit Metron (renamed from OpenSOC) as an Apache incubator project. In December of 2015, the project is accepted into Apache incubation. Hortonworks and the community innovate at impressive speeds to add new features to Apache Metron and harden the platform. The Metron team builds an extensible, open architecture to account for the variety of tools used in customer environments (thousands of firewalls, thousands of domains and a multitude of Intrusion Detection Systems). Metron’s open approach makes it much easier to tailor to the community’s use cases.
First official Release of Apache Metron 0.1 – After 4 months of hard work and rapid innovation by the Metron community, Apache Metron’s first release Metron 0.1 is cut.
Given Hortonworks proven commitment to the Apache Software Foundation process and our track record for creating and leading robust communities, we feel uniquely qualified to bring this important technology and its capabilities to the broader open source community. Without Hortonworks, the Apache Metron project would not exist today!
Understanding Apache Metron Deeper
To get a deeper level understanding of Apache Metron, continue this blog in the following article in the Hortonworks Community Connection: Apache Metron Explained!
About the Authors
Bio: George Vetticaden is a Principal Architect at Hortonworks, Senior Product Owner/Manager for Metron/CyberSecurity, and committer on the Apache Metron project. Over the last 4 years at Hortonworks, George has spent time in the field with enterprise customers helping them build big data solutions on top of Hadoop. In his previous role at Hortonworks, George was the Director of Solutions Engineering where he led a team of 15 Big Data Senior Solution Architects helping large enterprise customers with use case inception, design, architecture, to implementation of use cases monetizing data with Hadoop. George graduated from Trinity University with a BA in Computer Science.
(LinkedIn Profile: https://www.linkedin.com/in/georgevetticaden)
Bio: James Sirota is Director of Security Solutions at Hortonworks and committer on the Apache Metron project. Previously James was the Chief Data Scientist at Cisco focused on Big Data security analytics, and spearheaded OpenSOC. His primary expertise is in the design and implementation of Big Data platforms on top of Hadoop, MapReduce, Yarn, Storm, Kafka, Elastic Search and Flume. James holds a Data Science degree, a Master’s in Computer Engineering and is a licensed information security professional.
(LinkedIn Profile: https://www.linkedin.com/in/jsirota )