Tackle the Data Deluge Problem for Better Cybersecurity

Chris Braden

We’re now in a machine-scale world, where the scale, complexity and dynamism of data exceeds human capacity to keep up with it. Traditional cybersecurity strategies aren’t fluid enough to keep pace with the adoption of emerging technology and platforms. The amount of data organizations take in continues to grow exponentially while groups are faced with a shortage of available cybersecurity talent that’s needed to process and protect this data. Every year in the United States, there are roughly 350,000 open roles for IT talent with security skills and experience.

The simple laws of supply and demand make it clear that designing and executing an effective security strategy is becoming more and more difficult. Emerging technologies such as artificial intelligence (AI) and machine learning (ML) present one potential way to make grappling with this mass amount of data easier, but they aren’t a cure-all. To be effective, these technologies must be coupled with a company culture centered on taking a proactive approach to security and threat detection.

Dealing with the Data

The average midmarket team spins up as many as 10,000 pieces of information a day, from an expanding number of sources, but an average IT team can only handle about 5-10% of the information coming in. That means there is a huge mass of information being left unexamined. You’re essentially looking for a needle in a haystack, but you’re not even looking through all the hay.

It’s unrealistic to think that you’re going to be able to update your software, update the known threat list and keep your next-gen firewall fine-tuned and up to date. These are all good and necessary practices that you should be doing, but there’s just too much to keep up with. There’s already a lack of skilled individuals and, when coupled with the huge amount of data that continues to grow, it’s obvious that a new approach must be taken.

Zero-Trust Approach

Here’s a visual: If the electromagnetic spectrum were the Brooklyn Bridge, the part humans could see would be only about 20 feet long. This is roughly the same ratio that exists in the world of cybersecurity and threat-hunting. What you see is only a small part of the picture, and it’s the things you can’t see that pose the greatest risk.

Even for a midmarket company, a properly tuned and adjusted alerting process will generate roughly 10,000 alerts a day. Most IT teams, staffed with two or three dedicated threat hunters at best, can only process 500-1,000 alerts a day at best. Now, most people would agree that 5-10% isn’t security, and it may not even be compliance.

Digitization and the drive to the cloud have exacerbated this problem by creating more threat vectors, more opportunities for criminals to penetrate a company’s security strategy. As the complexity of your environment grows, so too do the number of alerts generated. That figure of 10,000 alerts will only get larger. However, the number of alerts a team of three people can process will not improve significantly. And so, the gap continues to grow.

Scalable Solution

Given the vast amount of information coming into most IT teams, it’s unrealistic to think that humans alone will ever be able to tackle it. You have to use a zero-trust approach, and this data is borderline unusable unless you have a strategy for processing all of it.

One answer is to use machine learning and AI to help sort through the mountains of data, filter it and make it accessible for threat hunters to take action. Applying machine learning to the network’s security and IT logs at a massive scale properly categorizes false positives and benign alerts while surfacing and correlating signals that will expose a threat actor. This empowers security analysts to disrupt and contain threats. Combined with machine speed, your IT team will be able to detect threats as they happen and respond in minutes rather than hours. Whether or not you’re in a position to use AI, you simply have to have …