newsletter

Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
December 17, 2015
prev slideNext slide

Leveraging Big Data for Security Analytics

In September, Hortonworks partnered with ManTech and B23 to foster a vibrant open community to accelerate the development of OpenSOC. In December we additionally partnered with Rackspace Managed Security and submitted OpenSOC to the Apache Incubator as a podling under the name of Apache Metron. A decision to rename the project was made to represent the new direction and the new community. Now the process of graduating Metron to a top-level project (TLP) has begun. Given our proven commitment to the Apache Software Foundation process, we feel uniquely qualified to bring this important technology and its capabilities to the broader open source community.

Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat-intelligence information to security telemetry within a single platform.

Metron can be divided into 4 areas:

  1. A mechanism to capture, store, and normalize any type of security telemetry at extremely high rates: Because security telemetry is constantly being generated, it requires a method for ingesting the data at high speeds and pushing it to various processing units for advanced computation and analytics.
  2. Real time processing and application of enrichments: Such as threat intelligence, geolocation, and DNS information to telemetry being collected. The immediate application of this information to incoming telemetry provides the context and situational awareness, as well as the “who” and “where” information that is critical for investigation.
  3. Efficient information storage: Based on how the information will be used:
    1. Logs and telemetry are stored such that they can be efficiently mined and analyzed for concise security visibility
    2. The ability to extract and reconstruct full packets helps an analyst answer questions such as who the true attacker was, what data was leaked, and where that data was sent
    3. Long-term storage not only increases visibility over time, but also enables advanced analytics such as machine learning techniques to be used to create models on the information. Incoming data can then be scored against these stored models for advanced anomaly detection.
  4. An interface that gives a security investigator a centralized view of data and alerts passed through the system: Metron’s interface presents alert summaries with threat intelligence and enrichment data specific to that alert on one single page. Furthermore, advanced search capabilities and full packet extraction tools are presented to the analyst for investigation without the need to pivot into additional tools.

Big data is a natural fit for powerful security analytics. The Metron framework integrates a number of elements from the Hadoop ecosystem to provide a scalable platform for security analytics, incorporating such functionality as full-packet capture, stream processing, batch processing, real-time search, and telemetry aggregation. With Metron, our goal is to tie big data into security analytics and drive towards an extensible centralized platform to effectively enable rapid detection and rapid response for advanced security threats.

jamessirota

Bio: James Sirota is Director of Security Solutions at Hortonworks. Previously James was the Chief Data Scientist at Cisco focused on Big Data security analytics, and spearheaded OpenSOC. His primary expertise is in the design and implementation of Big Data platforms on top of Hadoop, MapReduce, Yarn, Storm, Kafka, Elastic Search and Flume. James holds a Data Science degree, a Master’s in Computer Engineering and is a licensed information security professional.

(LinkedIn Profile: https://www.linkedin.com/in/jsirota )

Tags:

Comments

vishnu says:

Why is spark streaming not used. Why was storm preferred

Ideals says:

Interesting article. I think storm is better in this variant and it suits better for this task.

Priyanka Rai says:

LTS Secure Access Management is part of the LTS Secure IDM & CASB Suite™, the only commercial CASB offering for access management, identity management, user-managed access, directory services and an identity gateway design and built as single, uniform platform. The solution is built on a highly salable, modular, extensible, and easy-to-deploy architecture. Context-aware capabilities enable your employees, customers, or citizens a personalized experience on any digital channel, whether a mobile device, connected car, home appliance, or whatever the next connected innovation might be.
For more information about this content >> Click Here

Priyanka Rai says:

LTS Secure Access Management is part of the LTS Secure IDM & CASB Suite™, the only commercial CASB offering for access management, identity management, user-managed access, directory services and an identity gateway design and built as single, uniform platform. The solution is built on a highly salable, modular, extensible, and easy-to-deploy architecture. Context-aware capabilities enable your employees, customers, or citizens a personalized experience on any digital channel, whether a mobile device, connected car, home appliance, or whatever the next connected innovation might be.
For more information about this content >> http://bit.ly/lts_training

Leave a Reply

Your email address will not be published. Required fields are marked *