Hortonworks Cybersecurity Platform (HCP) is powered by Apache Metron and other open-source big data technologies. At the prime intersection of Big Data and Machine Learning, HCP employs a data-science-based approach to visualize diverse, streaming security data at scale to aid Security Operations Centers (SOC) in real-time detection and response to threats. This open source platform is built on top of the unmatched scalability and governance of data in Hortonworks Data Platform (HDP) and the real-time ingest and processing capability in Hortonworks DataFlow (HDF). Core features³ of HCP include:
Apache Metron is a real-time security solution focused heavily on streaming data sources and fast data processing. It consists of modules of parsing, normalising and enriching data with internal and third-party threat intelligence including STIX feeds. The smart modules includes the behaviour profiler which provides a number of algorithms for modeling typical behaviour and anomaly detection, and Model as a Service which allows machine learning models to plugged directly into the real-time pipelines.
There are also user interface and presentation modules, focusing of different users, including a front line alert triage dashboard, and highly flexible investigation notebook interface, which allows experienced users to deploy the full power of components in the big data stack, like Apache Spark for everything from large scale SQL to advances machine learning.
The Metron architecture is based on a real-time streaming platform but abstracts that platform from the end user with a simple extensible configuration language. The Metron project focuses significant effort on optimizing the streaming pipeline as much as possible. We also rely on Apache Kafka for resilience of input, output and intermediate staging, which ensures effective buffering and prevents data loss from equipment failure.
Another key element is Metron’s ability to push configuration changes to the pipeline in real-time, so we don’t need to restart streaming applications to change behaviours. This makes it easy for operators to deal with changes in things like thresholds and alerting rules, without compromising the throughput, which can be a real help in a DDoS emergency attack situation.
Metron is an application built on top of the Hortonworks Data Platform and Hortonworks Data Flow. The application itself uses many of the highly scalable components of the platforms. Metron can also be deployed on a cloud based platform using Cloudbreak to allow for rapid scaling to meet changing demand. This can be particularly important in the case of high volume attacks, or environments with a very cyclical day, with higher capacity and demand for real-time processing during office hours.
All the provisioning and configuration for Metron is handled via Apache Ambari which provides a single interface of cluster install, management, and configuration tool.
Metron uses Apache Knox to front authentication, and so has the ability to integrate with a wide range of single sign on and enterprise authentication methods, including platforms like Active Directory, Kerberos, and modern web authentication methods like OAuth.
The Metron infrastructure mainly uses a combination of managed config through Apache Ambari and Apache Zookeeper service discovery to do things like discover instances of Machine Learning based models spread across a cluster.
Metron accepts threat intelligence from a variety of sources from simple flat file blacklists to STIX formatted indicators of compromise. The platform provides a high-performance engine to match threat intel against incoming data. Unsupervised machine learning algorithms can be particularly useful for this kind of automatic correlation of events to intelligence.
Machine learning can benefit from threat intel in a number of ways. Metron for example uses clustering and similarity techniques to find zero-day events which might look like other events threat intel managed to catch. The threat intelligence and enrichment feeds also create good features to boost the power of machine learning algorithms, as well as labelled examples of known bad events which can feed into supervised algorithms.
Metron is purpose-built for cyber security at scale. The following are common use cases:
Like any open source project Metron is always growing. With production ready releases and deployments across the globe, it is definitely starting to grow into a strong platform. As this continues, and the community around the project continues to grow from strength to strength, we expect to see more complex use cases emerging, a sharing platform for behaviour profiles and machine learning models, and data structure emerging from real use-cases instead of hypothetical standards.
Metron’s strength is in sheer scale and performance. It is designed for medium to large enterprise use cases and teams with a SOC and Security Data Science capability. To date is has also appealed strongly to Managed Security Service Providers who run multi-tenant versions of the Metron platform, often bringing their own models, extensions and service expertise to the platform. This MSSP sector makes the platform far more accessible to the small and medium scale enterprise, who can also benefit from the a kind of herd immunity thanks to the massive scale of data and machine learning that these multi-tenant Metron platforms present.
Metron supports a number of means of sharing threat intelligence. The primary means is the industry standard STIX format, though numerous other sources can be supported with pluggable parsers and a broad range of ingest methods available in the underlying big data platform.
IoT opens up a huge range of opportunities for businesses to make sense of the physical world. However, with the range of sensors, and extension of networks to broader environments, comes the unfortunate danger of a large attack surface. Many IoT devices are, by definition, low powered devices intended to run for extended periods on batteries with partial connectivity. They are slimmed down for speed and cost, so traditional endpoint agents and protection running on the devices just isn’t in the battery budget. Metron takes a more network centric approach, and in combination with intelligent edge collection tools Apache NiFi allows security people to tap into IoT networks without disrupting the devices and ensure the environment they operate in is managed and secure. Detection of the kind of botnets emerging from IoT environments and the spread of infections is also a key strength of Metron, allowing administrators to catch and contain infections before they become epidemics.