SOAR for Success: How to properly measure KPIs for security operations

Nowadays, virtually every organization has to establish a set of KPIs (or Key Performance Indicators) in order to have a tangible perception of their progress toward reaching their desired targets, and the same applies to cyber security operations. By establishing the right KPIs, SOCs (Security Operations Centers) can determine the effectiveness of their cyber security strategy. And the very first step of establishing KPIs starts with identifying the most crucial goals of the security operations program.

But before you learn how to properly set and measure KPIs, you need to learn how to define your optimal KPIs, how many KPIs you should set, and how to make sure the KPIs you’ve chosen are directly related to the goal of your cyber security program.

What is a KPI for security operations?

Most security operations teams don’t incorporate traditional KPIs with the objective of achieving a goal or specific target. What they do, instead, is that they continuously measure their performance in accordance with those KPIs in order to properly perceive positive and negative trends and recognize unwanted patterns. For instance, some SOCs use KPIs in the following way:

• Follow recurring patterns to recognize potential attacks and malicious activity

• Assessment of employee workload and overall productivity analysis

• Analyze how long it takes for the organization to detect and remedy cyber attacks

• Analyze how accurately false positives and false negatives are assessed

Furthermore, it needs to be pointed out that key performance indicators for security operations vary depending on what those SOCs are trying to achieve. In other words, each security organization has a different way of measuring their success depending on their priorities; ergo, they have different KPIs. Nonetheless, quality KPIs serve as program enablers, reinforcing the continuity of the security operational programs at the highest level.

How to define your KPIs?

When you and your organization know what you’re trying to achieve by implementing a security operations program, you can quickly determine the very core of your KPIs. Whether you’re trying to:

• Protect sensitive data

• Reduce false positive alerts that disturb the productivity of your SecOps team

• Increase the productivity capacity of your workflow process

• Or optimize the average time it takes for your organization to detect and remedy actual threats

You’ll need to set benchmarks for how well you’re doing in achieving those goals, and that’s how you best define your KPIs. But it is crucial that each KPI is appropriate for an individual organization, and that should be determined through a detailed assessment of the organization’s security operations program.

Examples of KPIs that security operations analysts focus on

Even though every organization has different definitions of success, most SOCs and security analysts track the following KPIs in the cyber security world:

• Authentication errors

• Policy violations

• Time needed to resolve errors

• Cost per incident

• Malware events

• Phishing events

• Vulnerability management

• Third-party risk management

• How long it takes an analyst to investigate an incident

• Incident management

• Phish fail percentage

• Number of false positive alerts detected

• Number of devices monitored

• Total number of events

Keep in mind that the order and the number of KPIs listed above don’t necessarily mean they’re the most important ones or the ones that your organization should focus on. In fact, there are so many KPIs in the cyber security world that it seems like a new one is being invented each week.

Nonetheless, the increasing number of KPIs in cyber security shouldn’t detach your organization from the original purpose of setting KPIs in the first place - to monitor, measure, and improve cyber security performance.

How many KPIs should I set?

Determining how many KPIs to set for security operations and incident response is directly correlated to the targets your organization wants to achieve. As we mentioned above, by defining your goals, you automatically define the KPIs, but you need to determine priority and relative KPIs in order to properly benchmark and assess your success toward achieving them:

• Priority KPIs: Key performance indicators that are vital for the prosperity of your security program.

• Relative KPIs: Key performance indicators that matter but do not play a crucial role in your security program.

Determining the number of KPIs is not something that someone else can answer for you. Yet, this is something that should be done internally by your organization. KPIs only serve to inform you of the results regarding a critical operation in your system. So, you alone need to determine how many critical operations or goals your organization needs to monitor in order to ensure optimal success.

What makes a KPI effective? How to properly measure KPIs?

Think of KPIs as measurable metrics that you can accurately benchmark. It’s very important to note that KPIs need to be actionable metrics that will allow you to visually measure your performance. In this regard, KPIs are best measured if they are in accordance with the “SMART” criteria:

• Simple: KPIs should be easy to measure with a clear understanding of how they affect the security program.

• Measurable: KPIs should be measurable in a quantitative or qualitative manner. Either way, each KPI should be measurable clearly, concisely, and consistently.

• Actionable: The purpose of a KPI is to generate actionable decisions based on measurable results.

• Relevant: KPIs should be relevant to the functioning of the security program. The KPIs should be directly related to the performance of the SOCs.

• Time-based: KPIs should be used to show how performance is changed over time.

The KPIs you set should accurately communicate relevant information regarding your cyber security performance. The key performance indicators you choose should be the heart and soul of your security operations program, and they depend 100% on the precise nature of your security program. That’s why, by following the SMART criteria, you can obtain a good grasp on how to shape and set your KPIs.

How to make sure you’ve chosen the right KPIs to follow

As long as the KPIs you’ve chosen reveal valuable information regarding a critical component of your security program, then those KPIs are solid. That’s basically the only thing that KPIs are good for - allowing you to keep track of the important elements of your security program’s success. In order to make sure that the KPIs you’ve chosen are right for you, think about the following:

• Do the KPIs track valuable information that is integral for my security operations?

• Do the KPIs reveal menial information that doesn’t play a crucial role in my security operations?

• Is the KPI relatively easy to calculate, comprehend, and report?

• How much extra work do those KPIs require in order to be created and tracked?

This is the best way to determine whether the key performance indicators you’re tracking are quality. KPIs need to keep you up to date with your most important operations and shouldn’t draw your attention toward something trivial. If the KPIs you’ve set don’t provide valuable information about your security organization’s progress, then those KPIs are no good and should be disregarded.

How to align KPIs with SOAR?

Integrating KPIs with your SOAR solution is important. SOAR allows you to drastically enhance your security operations by automating and orchestrating a big chunk of your everyday security operations procedures. Plus, SOAR allows you to measure security information relevant for making tactical and strategic security decisions. For instance, our Cloud SOAR solution provides:

• Real-time situational awareness of the actual state of the security operations

• Benchmarking and optimizing security operations and incident response activities

• Analyzing over 140 customizable KPIs via a customizable dashboard

• Measuring every individual phase of the incident response workflow to allow analysts to optimize current performance

By providing real-time data that can help you assess and optimize security operations, SOAR provides an even better way to keep track of your most relevant KPIs.

Can my security program function properly without KPIs?

In theory, yes. But in practice, that would be like sailing a ship without a helm. KPIs allow you to set the direction of your security operations. By establishing KPIs, you also establish the components that matter the most to your security program, and that allow everyone in your organization to have a clear perception of how you define success.

You could operate your security program without setting KPIs, but that would be counter-productive. Not knowing how to measure success means that you have no clear understanding of which path your organization should take, which in return would mean that your organization wouldn’t know which areas you strive to improve. And that is not the recipe security organizations should follow if they are determined to continuously enhance their cyber security performance.