Threat Intelligence: Don’t Fall in Love with the Wrong KPIs

Posted: August 12, 2019
Category: Uncategorized

You’ve probably heard of the myth of Narcissus, the Greek hunter who was known for his extreme beauty… and vanity. Many of the old stories talk about how many suitors’ hearts were broken by Narcissus. But in an ironic twist, Narcissus eventually came to his own doom when he saw his reflection in a pool of water and fell in love with himself. He realized he couldn’t have what he most desired, and he committed suicide.

I know that’s a dark way to start a blog post, but I do have a point relevant to threat intelligence: if you fall in love with the wrong metrics – often called “vanity metrics” – when trying to build KPIs (key performance indicators) around your use of threat intel, you could be causing your organization great harm. Commonly used in marketing and other areas to make nice-looking graphs (downloads, registered users, raw pageviews, etc.), vanity metrics are ultimately for show rather than substance.

Flashpoint’s CEO, Josh Lefkowitz, gives some examples in a blog post at Security Week of threat intel vanity metrics that look like good KPIs but aren’t really indicating the success of the threat intelligence program:

  • Number of intelligence reports written
  • Number of indicators of compromise (IoCs) processed
  • Number of data points collected
  • Number of keyword alerts received

What should we be measuring? Travis Farral, Chief Security Officer at LEO Cyber Security, helps answer this in a blog post he wrote when he was with Anomali. In this post, Travis describes some reports from Anomali Enterpise that give focus on the effectiveness of the threat intelligence:

…one section displays the number of matches found and how many high confidence matches were found for each threat feed. It even highlights how many internal assets were affected by those matches. (emphasis added)

In other words, your metrics and KPIs should focus on your environment and how the threat intelligence helped thwart fraud in your organization. How you pull those numbers together can differ, but emphasis should be on effectiveness within your organization, not simply the number of actions taken.

It all comes back to what you value as a metric or KPI: form or substance. If you value form, you will fall in love with pretty graphs and measurements and ultimately cause harm to your organization. If you value substance, your organization will benefit from the use of threat intelligence.

***

And since we’re talking threat intelligence, be sure to check out our ThreatDay DFW 2019 event that is coming up in Dallas on August 27th. Set Solutions is teaming up with passionate cyber security experts from the FBI, Anomali, Flashpoint and others to provide a day of education around threat intelligence, information sharing, metrics, DOS defense, and other stuff. If you’re in the DFW area, you shouldn’t miss out on the great talks by some really smart folks.

***

This blog was written by Michael Farnum, Director of Solutions Architects – South, at Set Solutions.