Good Security Metrics are a Work in Progress

Tenable recently sponsored the publication of an ebook, Using Security Metrics to Drive Action. This ebook is a compilation of thoughtful essays from 33 CISOs and other experts, who all share their strategies for communicating security program effectiveness to business executives and the board. In this article, excerpted from the ebook, Gary Hayslip, Deputy Director/CISO City of San Diego, CA, shares his thoughts on using security metrics to drive action.

Gary Hayslip found himself sitting next to the mayor of San Diego, California, one evening over dinner. The mayor turned to San Diego’s chief information security officer (CISO) and asked, “Just how secure are our networks?”

“They are a work in progress,” Hayslip responded.

It wasn’t what the mayor wanted to hear, but it started the two and a half-hour conversation. In it, CISO Hayslip helped the mayor understand that cybersecurity is a life cycle, not an event. “And part of that life cycle,” Hayslip explains, “is breaches. You never get 100 percent secure.”

That’s one reason why metrics are so important, Hayslip says. “When you collect metrics, you’re collecting them to tell a story,” he states. “They have to be able to tell the story of your business.” To that end, Hayslip keeps a sharp eye on three measurements:

• Time to detect. San Diego’s networks average 66,000 attacks per day—22 million a year—that are successfully blocked, Hayslip indicates. It’s inevitable that some attacks get through, he says. “My concern is, when they get in, how fast do I get alerts on them? How quickly do my firewalls and sensors detect that we’ve got an incident?”
• Time to contain. This metric allows Hayslip to know how quickly attacks are contained and cleaned up. Those numbers need to be examined carefully, however, he says. If incidents are contained in 20 minutes on average, that might seem fine, but if within that average some departments take as long as an hour, it might mean that some brainstorming is in order to find new security layers to protect remote or mobile assets.
• Number of compromised systems. San Diego hosts 14,000 desktop and laptop computers in its 40 departments, Hayslip notes. “So I have about 14,000 different doorways into my network.” On average, 45 machines are infected per month. By monitoring the number of compromises, he can gauge whether the city is staying within the acceptable exposure rate—for Hayslip, that’s about 1 percent of 10,000 machines per month. It also tells him whether he’s closing in on his personal goal of 10 machines per month. “That would be kind of phenomenal, when you look at the size of my network,” he adds.

These and other metrics—such as what types of attacks are getting through—tell Hayslip whether he’s succeeding in his overarching goal. “I want to be proactive,” he says. “I want to be able to see an attack before it infects the machine and to be able to stop it and kill it.” Metrics, in short, tell him how much work is yet to be done.

As it turns out, there’s still a fair amount of work to do, though much has been accomplished. Intrusions have fallen dramatically since Hayslip came on the scene, from a high of 160 intrusions per month down to 40. Phishing email attacks and infection from flash drives and websites are all down. Recently adapted cybersecurity technologies, including the Tenable Nessus agent scanner suite, have clearly been a big help, Hayslip asserts.

Not all metrics are created equal, of course. Hayslip used to monitor the number of help desk tickets that employees filed. That proved not terribly useful. “They could be submitting requests to my team’s email box that don’t even apply to us, just hoping someone is going to help them,” he explains.

In the end, Hayslip counsels CISOs to choose which metrics to track based not on their personal curiosity but on their business’ bottom line. “The metrics you collect need to mean something to the organization,” he says.

If possible, he concludes, tie metrics to hard dollars. He did that recently, showing city leaders that by replacing some vulnerable legacy technologies, the city could reduce direct financial risk by $4.5 million and associated legal exposures by a whopping $75 million. “That room was quiet,” Hayslip recalls. “Everyone was looking at us like, ‘Wow!’”

More information

• Get your copy of the ebook, Using Security Metrics to Drive Action.
• Watch the Tenable Blog for weekly excerpts from Using Security Metrics to Drive Action. You can subscribe to the blog by clicking Blog email updates on the Blog Home Page.

About the author

As CISO for the City of San Diego, California, Gary Hayslip advises the city’s executive leadership, departments, and agencies on protecting city information and network resources. Gary oversees citywide cybersecurity strategy, the enterprise cybersecurity program, and compliance and risk assessment services. His mission includes creating a risk-aware culture that places high value on securing city information resources and protecting personal information entrusted to the City of San Diego.