9 Tips on Using Big Data Analytics to Improve Security
Page Contents
With the widening adoption of Hadoop and in-memory databases, companies are starting to realize that Big Data analytics has uses well beyond traditional business intelligence. Some are now harnessing it to detect potential security threats and improve their defenses and responses.
The application of Big Data analytics to this area is relatively immature and users are still going through a learning curve on it, however. Here are nine tips that will help you get the most out of using Big Data to improve security.
Look for the Unusual
In marketing, you use Big Data to identify trends or unmet needs that you can leverage to retain existing customers and acquire new ones. You are not looking for that one customer in a million who didn’t like your product. Rather, you want to make sure you service those million people well, not change what you are doing because of a non-representative complaint.
With security, it is the opposite. It is the unusual that matters.
“Security use cases must find irregular activities, aberrant behaviors and anomalies that represent high risk events — which creates challenges creating queries into the data,” says Seth Goldhammer, director of product management at LogRhythm. “This is different from business intelligence, IT operations and other Big Data applications that search for known activities/trends or error conditions.”
Look Beyond the Logs
Logs and alerts are set up based on previously known threats. They won’t necessarily contain the data you need to discover tailored or emerging threats.
“Do not restrict yourself to just using logs and events to provide the needed visibility for security detection,” says Matthew Gardiner, senior manager, RSA, the security division of EMC. “Logs and events just do not provide the necessary visibility, in particular for the more advanced or targeted attacks that often don’t generate a noticeable log or event sequence.”
Use Broad Data Sets
While analytic tools have the capacity for conducting deep dives into a data set, it is more important that you have all the relevant data to begin with. The most sophisticated tools still depend on the data being provided.
“Relatively simple analytics applied against a comprehensive data set — no matter how large — is much better at finding important security anomalies in need of investigation than using sophisticated analytics across a limited set of telemetry data,” says Gardiner.
Integrate Analytics with Rapid Response
Most Big Data analytics applications provide information that is used to guide planning, strategy and decision making. With security analytics, you need to act, not think.
“Whereas business intelligence may take months of data to understand trends and patterns, the security use case must recognize these concerning activities within seconds or minutes in order to initiate immediate remediation and must be accurate, or security analysts will spend their limited time pursing false positives,” Goldhammer says. “The longer a breach is undetected, the remediation efforts become larger in scope and more costly.”
Don’t Just Look for Known Threats
Your firewalls, antivirus and other tools should be up to the task of protecting against any previously identified threats. Security analytics is needed to find those that haven’t been identified yet, which requires a continuous learning process.
“Let the data speak,” says Idan Tendler, CEO and co-founder of Fortscale. “Leverage advanced machine learning algorithms in order to pinpoint unknown threats and rogue users.”
Start Simple
As with any new tool, it is tempting to immediately explore its full range of capabilities. But it is also likely that there are low-hanging fruit that will deliver an immediate benefit. Focus on those first, and use what you learn to tackle more advanced threats.
“Use your Big Data security analytics tool to defend against known tools, techniques and attackers, not primarily as an excuse to conduct unbounded security explorations,” says Gardiner. “As you mature your people, processes and use of technology, you can shift some of your efforts to hunting for pure security unknowns, but start with a laser-focused program to accelerate your detection of the unknowns.”
Consider a Purpose-Built Security Solution
Security analytics tools may be built on top of other common platforms, but just because one is familiar with Hadoop or R doesn’t mean that you should try to adapt those for security analysis. In many cases, a purpose-built security solution makes it easier to add the necessary security context to all the monitored traffic and offers a portfolio of analytics that are able to detect attack patterns.
“Given the shortage of skills in security analysis, this will prevent an organization from having to take on the challenge of enumerating potential attack patterns themselves,” says Vijay Dheap, global product manager, IBM Big Data Security Intelligence & Mobile Security. “If custom solutions need to be developed, a purpose-built security intelligence solution provides a foundation upon which other capabilities can be introduced.”
Don’t Expect Analytics to Replace Experience
A security analytics package will have the capacity to automatically discover linkages across disparate data elements, qualify those relationships and perform deep and possibly customized analyses. It should also offer a methodology for security analysts to extend the knowledge base of the system about new threats or attacks. But it is a tool to assist security personnel; it doesn’t substitute for their knowledge and experience.
“To convert machine IQ into actionable insights, a simple but effective user experience is paramount,” says Dheap. “This will lower the barrier to deployment and utilization within the organization, as well as help the organization optimally capitalize its skilled security resources to address qualified security incidents rather than false positives.”
Help Improve Security Analytics Technology
Finally, there is still a lot of work to be done to move analytics platforms to the point where they provide simple and complete security analytics.
“Use the technology available today as an opportunity to learn, to build skills and to identify gaps that are potential cyber or physical security threats,” says Stephen Harris, North American Big Data lead at Capgemini in the San Francisco area. “In that learning phase, work in partnership with the product vendors to mature their applications and the technology to help close those gaps that exist today.”
Drew Robb is a freelance writer specializing in technology and engineering. Currently living in California, he is originally from Scotland, where he received a degree in geology and geography from the University of Strathclyde. He is the author of Server Disk Management in a Windows Environment (CRC Press).
Drew Robb is a writer who has been writing about IT, engineering, and other topics. Originating from Scotland, he currently resides in Florida. Highly skilled in rapid prototyping innovative and reliable systems. He has been an editor and professional writer full-time for more than 20 years. He works as a freelancer at Enterprise Apps Today, CIO Insight and other IT publications. He is also an editor-in chief of an international engineering journal. He enjoys solving data problems and learning abstractions that will allow for better infrastructure.
