Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
April 21, 2016
prev slideNext slide

Apache Metron Use Case: Finding the Needle in the Haystack

In Part 3 of the Apache Metron announcement series, Apache Metron Tech Preview 1 – Come and Get it! , we outlined Apache Metron 0.1 release’s new features and enhancements. Then we demonstrated how to deploy on a single node VM using vagrant and a cloud-based install for a complete 10 node Metron cluster using Ambari blueprints and AWS APIs.

Now that you have a full application for cyber security monitoring, analysis, and threat detection, how do you proceed? Read on!

Metron UI

After running the installer, you should have access to the Metron UI. The UI looks like this:


Metron provides a Kibana-based UI that is designed to be a single pane of glass. It utilizes a big data approach (having all the data available at the same time) to filter through the noise and display the information, alerts, and context that an analyst or investigator views on just one panel. The Metron UI has several advantages over conventional SIEM tools. The difference is in it’s flexibility and extensibility.

Apache Metron helps you find the needle in the haystack (which is what forecasting a breach is like), as well as the context around it. The data is presented cohesively with all the metadata on the same screen, eliminating the need for jumping around from console to console in an attempt to intelligently piece together the right information.

More details on the Metron UI can be found in the following HCC article: Metron UI – Explained

Finding the Needle in the Haystack Use Case

What are the “needle” and the “haystack”? The haystack represents the millions of telemetry events that Metron has ingested, parsed, enriched and analyzed. These events are coming from a variety of telemetry sources and emitting hundreds of alerts from security devices and Metron. The needle represents a handful of critical alerts and related telemetries worthy of an immediate investigation.

The following is an example of a “finding the needle in the haystack” Apache Metron use case:

  1. A SOC investigator navigates to the Metron UI.
  2. He views a few alerts in the Alerts Panel and the corresponding metadata. There is something unusual that intrigues him.
  3. He down-selects / filters the datasets with pinned queries and filters.
  4. He configures additional fields to display enriched metadata for detailed message tables.
  5. In the PCAP panel, he downloads the PCAP files from the PCAP panel associated with the filtered dataset. He opens the PCAP files in Wireshark for further investigation.
  6. A SOC investigator analyzes the data and determines that there is a possible threat or breach that needs further attention. He has found the “needle”.
  7. At this point, the SOC investigator has everything he needs in the filtered Metron UI dashboard to create a Metron Ticket for follow-up.
  8. He exports the Kibana dashboard with filters in place. The exported artifact has a set of alerts across different data sources, a set of metadata events that are really telemetry event details enriched from different sources, and the PCAP exported data.
  9. The export artifact is then used to create a Metron Ticket in the users’ workflow engine (Remedy, JIRA, etc.).
  10. The Metron Ticket is assigned to an analyst with all the context from Metron. The Workflow Engine is outside the scope of Metron TP1.

More Details on HCC

To get more details on the Metron UI and use cases, continue to the following article in the Hortonworks Community Connection: Finding a Needle in the Haystack.

Apache Metron and the Metron logo are trademarks of the Apache Software Foundation. All other trademarks are the property of their respective owners.

Apache Metron is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. For more information, please visit

About the Authors

Bio: George Vetticaden is a Principal Architect at Hortonworks, Senior Product Owner/Manager for Metron/CyberSecurity, and committer on the Apache Metron project. Over the last 4 years at Hortonworks, George has spent time in the field with enterprise customers helping them build big data solutions on top of Hadoop. In his previous role at Hortonworks, George was the Director of Solutions Engineering where he led a team of 15 Big Data Senior Solution Architects helping large enterprise customers with use case inception, design, architecture, to implementation of use cases monetizing data with Hadoop. George graduated from Trinity University with a BA in Computer Science.

(LinkedIn Profile:


Bio: James Sirota is Director of Security Solutions at Hortonworks and committer on the Apache Metron project. Previously James was the Chief Data Scientist at Cisco focused on Big Data security analytics, and spearheaded OpenSOC. His primary expertise is in the design and implementation of Big Data platforms on top of Hadoop, MapReduce, Yarn, Storm, Kafka, Elastic Search and Flume. James holds a Data Science degree, a Master’s in Computer Engineering and is a licensed information security professional.

(LinkedIn Profile: )

Leave a Reply

Your email address will not be published. Required fields are marked *

If you have specific technical questions, please post them in the Forums