Modern information security encompass broader data sets than in the past, in order to create context and generate a complete picture of network data, user behaviour pattern and business data – all combined together so that a trendline of normal operations can be created. Then from that, it is possible to determine deviations from the norm (Reference “To detect the abnormal I need to predict the normal” by Michael Schiebel.
As the hyperconnected digital world produces cybersecurity data at a volume and rate that fits the definition of big data, the next step of processing that data needs to be addressed. Hackers have the power of nation states and are constantly attacking enterprises for the information they contain – credit cards, passwords, healthcare data, and more. These threats are no longer isolated point attacks in time, but are carefully choreographed and orchestrated, sometimes laying stealthy traps of advanced persistent threats that are not simple to uncover and mitigate through existing point solutions and manual processes.
It is impractical for security personnel to manually piece together all the relevant security data to detect threats, and modern cybersecurity solutions need to lean on the automation of manual tasks of processing very large sample sets made possible through big data and machine learning. And as the hackers constantly evolve their game, security teams need to adapt at the same time, at the same rate, to efficiently detect and interpret the signs of the most relevant threats which require further investigation, and to quickly respond to evolving threats.
There is incredible energy and innovation targeted at applying big data and machine learning to the realm of security right now. However, consuming that innovation is difficult to scale as innovators are solving the same data engineering problems over and over. After all, cybersecurity isn’t the only industry that has a deluge of data to be processed – businesses in a range of industries face similar changes in processing large volumes of streaming data in real-time. Thus the open source community initiated Apache Metron , a big data cybersecurity platform that enables a single view of diverse, streaming security data at scale to aid security operations centers in rapidly detecting and responding to threats.
Apache Metron is a faster path to innovation, encapsulating a decade of big data science and streaming analytics experience into a curated package of technology for security teams to utilize. It gives security personnel a platform to quickly leverage machine learning for real time profiling and statistical analysis, without the need to expend resources on the repeatable data engineering problems involved in creating the baseline infrastructure for an enterprise ready machine learning cybersecurity platform. In the end, this means accelerated times to respond to the ever-evolving threats generated by nation states intent on attacking and breaching enterprises assets.
To learn more check out Hortonworks cybersecurity solution or join our upcoming webinar with Michael Schiebel, General Manager, Cybersecurity, and James Sirota, Apache Metron committer and Director of Security Solutions on March 14, 2017 to learn more on why cybersecurity needs big data.