Telling Stories with Security Metrics

We know that storytelling is a key to communicating risk to the business. How do we integrate metrics to help us tell those stories?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Joining us is our sponsored guest, Shirley Salzman, CEO and co-founder, SeeMetrics.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, SeeMetrics

Full Transcript

Intro

0:00.000

[David Spark] We know that storytelling is a key to communicating risk to the business. So, how can we integrate metrics to help us tell these very stories?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark. I’m the producer of the CISO Series and joining me as my co-host for this very episode, it’s Steve Zalewski. Steve, say hello to the audience.

[Steve Zalewski] Hello, audience.

[David Spark] He’s talking to you, people. Our sponsor for today’s episode is SeeMetrics. See the metrics that matter. They’re our sponsor for the show and they’re responsible for our guest who I’ll introduce in just a moment. But first, Steve, our topic. Metrics are the lifeblood of CISO decision making, but they can be technical and hard to understand when relayed directly to the rest of the business. So, creating narratives around security initiatives is a great way to get buy-in from decision makers and essentially have them understand what the heck’s going on. But how do we connect these cold metrics over to storytelling, Steve?

[Steve Zalewski] Yeah, and at RSA, this was a theme I heard a lot, which was everybody knows we need to do it. Nobody knows how to do it well. Because we can’t agree yet on whether we’re doing measurement or metrics, whether we’re doing storytelling or vision telling, and we don’t have a common executive team that’s agreed on how they want to measure success. And so because you’ve got all of those variables in the equation, I think this is going to be a great episode.

[David Spark] Yeah, and let me also just throw out this one thing. We have done lots of shows talking about metrics, but often the question is, so what metrics do you measure, or what’s most important? And it’s a very sort of like single-shot discussion. It is not looking at the amalgamation of the metrics because on their own, they don’t tell you much. It’s when you start connecting them that you get the stories, and that’s essentially what we’re after, right?

[Steve Zalewski] Right. And that’s the difference between am I here to tell you how I’m being efficient in using my resources, or am I here to show you how I’m being effective at stopping the attack? And so that difference, that nuance is, for the most part, we worry about efficiency and the specific metrics to determine how we are using the maximum capability of a tool versus how well are we at enabling the business.

[David Spark] Well, guess what? We have an expert on this very subject. And in fact, their business is making sense of this very issue we’re talking about. It is the CEO and co-founder over at SeeMetrics, our sponsored guest, none other than Shirley Salzman. Shirley, thank you so much for joining us.

[Shirley Salzman] David, Steve, it’s great to be here.

What’s the optimal approach?

3:05.561

[David Spark] Andrew Wilder, who’s the CISO over at Community Veterinary Partners, said, “First, I look at our most critical business processes and how we are focusing on those. Second, I look at where those things are, i.e., what systems are supporting them. Third, I look at how are we securing all the aspects of these systems that are supporting the critical business processes. Sounds simple, but it’s easier said than done.” Oh yeah, that’s true.

Let me also read this. This is a long quote from Damian Leger over at Acadiana Security Plus, but hold on, listen to this. “First off, ask ‘Why are we even talking about this?’ and ‘What are your concerns?’ Then you need a standard to measure to, example, NIST RMF, for example. Something to measure with, think along the lines of something like Qualys. And a good gap analysis, where are we now? And what does success look like from the board perspective? And how do we get there? Coupled with serious, dedicated, and funded senior management buy-in sponsorship. Otherwise, it’s going to go nowhere. And above all else, you need to tie cyber risk directly to business risk using compelling business language that the board easily understands, preferably in dollars and cents, risk and liabilities, and most importantly, profits. We cyber folk are in the business of business development.” I think that quote from Damian explains the entire path, doesn’t it, Steve?

[Steve Zalewski] So, I would say what Damian has said is, if I have unlimited resources and I get my way, so wave my magic wand for how I can get perfect security, this is all I need to do. So, just get out of my way and let me do it. The problem is it’s business enablement. And the business, okay, is no longer prepared to have me tell them how to do business. We have to negotiate what’s good enough security for the business. And I think that’s what the optimal approach is the challenge that we’re up against today.

[David Spark] This rings true with you, Shirley, doesn’t it? Yes?

[Shirley Salzman] Yeah, absolutely. I think we just heard two different approaches on how to measure and how to report to the board. Those two approaches are eventually showing that the market has different views on what needs to be measured. We’re seeing from the same organization different measurement needs. And it’s interesting to see that sometimes you want to use that data to show how you progressed, and sometimes you want to show that data to show, no, we still have to catch up if you speak with the team. And sometimes you just want to see that in a specific view that is the most one who’s lagging behind the organization.

In order to see all of that, you need to have something to allow you to blend all the information into one layer of data. Qualys may be a great tool, but it’s only a set, a very limited set of the controls you have in your organization. How do I take the data in Qualys, blend it with my asset manager to help me to understand the business structure to be used in the company? And how do I couple that along with the data from the endpoints and the threat intelligence I may have? So, really for the system to measure, you really need to be able to blend all of this together.

[David Spark] And this blend that you’re talking about, which is what I referenced at the very beginning of the show, saying when we’ve done metric stories, it’s been very single metric discussion. This blend allows for a story to be told. Can you just give me one example of when you have this, this, and this, now you can tell this story?

[Shirley Salzman] Yeah, absolutely. We recently had a customer that did an external exercise of here are the top risks we would like to look at and to measure across the next few months. Each risk statement is an assembly of several operational metrics. Coverage is one. SLA of critical vulnerabilities in specific critical assets is two. Vulnerabilities of, or how we mitigate end of life assets that we know it’s hard for us to patch and reach out to, it’s another one. So, eventually to be able to narrate the risk story, I’d like to be able to compile building blocks of metrics that would narrate the story and tell me this risk is trending A to B, or 20% more, or two points less, doesn’t matter how you actually measure that. But how do you get all those very small building blocks to be able to tell a risk story, a threat story?

Steve earlier mentioned, how am I utilizing the tool? I’d like to measure that as well. I may have a framework I’d like to measure. There’s so many things a security organization needs to measure, that eventually to make it viable, you have to have those building blocks to be able to aggregate the story. And this story can be one for this quarter and the next quarter, I may want to measure and focus on something else. But without that, we are stuck with static Excels that we see our customers coming to us, “I built that Excel. I’ve heard from all my colleagues; this is what I need to measure. How am I going to do that?” I might have gone to the Big Four. I paid them a lot of money. They came back with a list of 100 metrics. How am I going to do that? And this piece is really why metrics remains in theory rather than in practice.

What are the elements that make a great solution?

8:44.470

[David Spark] Jorge Lopez of Zoom said, “You can answer this specific question by conducting a third-party ransomware readiness assessment and sharing the results.” And let me, let me qualify that. In your question, because you posted this question on LinkedIn, Steve, you asked the question to the audience or the security audience on LinkedIn, “How do you answer the question how ready are we to deal with ransomware?” And so that’s why Jorge is making this reference to ransomware readiness assessment. And he goes on to say, “Much like program maturity is assessed today – example, NIST CSF – a good assessment will look at things like most common actor TTPs and your business’s crown jewels. From that assessment, just like with NIST CSF, analyze and prioritize findings for action.”

Jared Pfost of the Kalles Group, who also is kind of an expert on this because he dealt with metrics when he was working over at Disney, he says this, “The largest challenge I see is ‘how’ to build an automated story-driven dashboard, measuring control effectiveness organized by different threat scenarios. Integrating and visualizing multiple control sources is expensive. I think investing in measurement should be an explicit part of the strategy.” Oh, I mean, I’m going to go to you first, Steve, on this one. That last line is the line we hear, like, “If you’re not measuring, what the heck are you doing?” Right?

[Steve Zalewski] And here, I’ll be curious to hear what Shirley has to say, which was his quote comes from the mindset of a Fortune 500 CISO, where you have audit and compliance, right? And you have a lot of standards, and you’re having to have the auditors tell you, right, how well you’re providing evidence of your policies against your standards. That is what his mindset is, and that’s where many people are coming from, which is here’s how to build a mature security program. But for many of us, right, SaaS, small to medium enterprise, we don’t have that luxury. And so therefore, what we’re trying to figure out is how do we measure and generate metrics that reflect what our business needs from a business impact analysis. And so what I really like about Jared is he’s calling it out for the Fortune 500, the right way to do it, which is how much money it’s going to take, how many resources, and how do I build a really good organization that can measure anything? The challenge I have is, while that is kind of like nirvana, for many of us, it’s just unattainable.

[Shirley Salzman] Yeah, absolutely, Steve. I think it’s when we started SeeMetrics or started discussing SeeMetrics in the midst of COVID, everybody told us or started telling us, “Ah, I’m complied. I got my ISO, I got my SOC 2 Type 2, I maybe got my HIPAA. I have all those nice logos on my website. Doesn’t this mean I’m secured?” And the answer is, it means you’re probably been secured on the point of time where you did the compliance process, but it’s December now and you did it in August. What happened to your controls? You’re spending so much money and funds and resources to guard your castle. Shouldn’t we make sure those controls are actually not only on the same level they’ve been in August but actually improving?

And we got to be honest, the threat landscape these days is much more dynamic than a point in time. That’s why we would like to be able to see metrics and the controls and their performance, in the way that we can see that in a continuous way that will not just give us the visibility but will actually help us to see that we are enforcing the policies that we have probably set during the risk assessment. And does these policies actually match the threats landscape of our company, the risk assessment we’ve done? And can we go to the board, not just once a quarter or once a year, but on a daily basis, not just to the board, let’s leave the board aside, to ourselves and tell ourselves we are doing a good job. We know our risk. We know where we are weak. We know which views are lagging behind. This is where I have a gap. I need to maybe do some compensating mechanisms, but you’re not doing it on a data from a few months back or from some Excel that’s been aggregated from different data points. You’re doing it back-to-back from the data, from your operational stack. And I believe this is what Jared has been alluding to, how you do this.

Sponsor – SeeMetrics

13:37.372

[David Spark] Who’s our sponsor this week? Why, it is SeeMetrics. You’ve heard their name already on the show but let me tell you how absolutely spectacular they are. SeeMetrics, and you should know that it’s spelled the way it sounds. It’s the word See and then Metrics. SeeMetrics automates metrics programs, eliminating tedious manual work and offline spreadsheets. SeeMetrics’ data fabric is a response to the industry need to measure pretty much everything around your organization – policies, risks, frameworks, threats, and projects. SeeMetrics transforms all siloed security data into one cohesive source and provides ready-to-use metrics assembled from data correlated from multiple tools. Once connected, security leaders and teams mix and match metrics to tell different stories while completely eliminating manual effort.

Now, why is this important? Well, because on any given day, security organizations are required to respond to endless questions like the one we were saying, how ready are you for ransomware? Where are we not adhering to our policies? Which exposure is most critical and in which business unit? How much will this risk impact the business? Simply plug, measure, and gain continuous responses and trends. Enough of static spreadsheets and outdated data. It’s time to align all security and risk teams on the same baseline, reuse metrics for multiple purposes while continuously measuring different risks, threats, and security programs. All right. So, you’re ready to automate your metrics program? Sounds good. Yes? Sounds like something you want to do? Here’s what you need to do. Just go to their site. It’s SeeMetrics, sounds just like I say it, SeeMetrics.co. Not .com, but .co. Go to SeeMetrics.co and get your own free trial environment.

Who owns this issue?

15:36.323

[David Spark] Michael Calderin, who’s the CISO over at the YAGEO Group, said, “What are the business processes and data most in need of protection? How well have we defined RTO, that’s our real-time objective, and RPO, recovery point objective? Have we tested those scenarios, both in terms of preventing and detection/response/recovery? In answering the board, what percentage of those scenarios are we meeting our goals, and what are the specific scenarios where we’re not, especially the highest risk, and what are we doing about it?” Wow. I mean, I will just say having the answers to those questions would be highly desirable by any executive. And let me close here with Neda Pitt, who’s the CISO over at Belk, said, “Providing effective proof points isn’t just about knowing the statistics. It’s also about choosing which metrics are most valuable for your audience and presenting them in a way that is most helpful for that audience.”

So, Shirley, I want to start with you. Michael and Neda make really, really good points here. Michael is saying, like, we have to answer these questions about our readiness, and Neda is saying, to get to the point where we’re answering the questions about readiness, are we measuring the right things? Are we measuring vanity statistics? And I want to quickly quote something Wendy Nather once said, and something that aggravates the crap out of other CISOs is when security professionals goes “Oh, we’ve blocked X millions of threat attacks,” which Wendy Nather said, “I don’t care how many raindrops your umbrella protected. I only care about what comes to me.” So, I want to know about these two issues of answering the readiness question and are we selecting the right metrics to be able to answer that question?

[Shirley Salzman] Absolutely. And it’s a great distinction. As far as I see it, the CISO responsibility is on the prevention side. How do I make sure that I’m focusing on the right assets, on the right criticalities, on the right threats, on the right risks that my organization is most vulnerable? The recovery and the readiness piece, as a CISO, they have responsibility, but they are not alone in the teams and the discussion to resolve, let’s say, a ransomware attack. That’s where you need to have a political discussion with the board even. Are we going to pay this or we’re not going to pay this? Is it leaked to the news or not leaked to the news? So, this is way above the CISO’s head to be able to measure that. What we are focusing, or at least the CISO as an office, as an organization, they need to be able to come and say, “I’ve done everything I can. I’ve seen where I’m lagging behind. I put the prevention plan. I’ve asked for more budget. I’m on top of things. I know where my weak links are. That’s why we’ve put some compensating mechanisms. I was able to communicate and engage with you, my peers, the board, the auditors, the risk team, the compliance, the governance of the organization, financial as well. We were all transparent and you knew exactly where our weak spots are.” And for CISO as an office to be able to do that today, it’s nearly impossible. And the data they are basing it on is eventually sort of a guesstimation, stitching, Excels. And we need to have much more confidence once we are going and telling those stories across other peers.

[David Spark] Steve, I throw this to you. This whole conversation comes from the choice of metrics. We can’t get to our readiness answer if we’re not choosing the right combination and things to measure.

[Steve Zalewski] So, you just actually hit on it in the way you asked the question, which is you confuse measurement with metrics. And it’s something I keep hammering on, which was the ability to measure something, okay, is great. And SecOps is a great place to start. Where I want to measure, are my controls, right, being effectively deployed? But what I’m really saying is, am I efficiently deploying them? Metrics are the things you bet your job on. Measurement is the way that you show you’re being responsible with the money. And so, what Shirley is doing, right, and when she talked about this is we can’t agree on the metrics because there is no one right answer. And oftentimes we’re using measurement as justification and we’re not actually driving the metrics problem. How am I betting my job?

And prevention, again, as Shirley said, is where we focus. Yet ransomware and why I positioned it with ransomware is that’s resilience. That, in spite of everything I did, it still happened. So, how do I tell the story with metrics that shows how I’m preventing, detecting, containing, and moving on? And so, what I really like about the passion, and for people that are listening, Shirley is ready to jump out of this mic for every one of you to solve this problem. The passion in her voice and in her demeanor, you got to know it’s there, is that she wants to give us the tool to be able to do a business impact analysis, to do security operations measurement, and really to drive ultimately the metrics conversation, which is the one or two things that each particular executive team wants to focus on, and she wants to give us the tool to do that.

What do most people think it is, and what’s the reality?

21:35.718

[David Spark] Patrick Lee of Safe Security said, “It starts with risk tolerance and risk acceptance. How much of a chance percentagewise of likelihood is the business/board willing to accept? How much losses/costs are they okay with if it inevitably does happen? If they’re not okay, here’s the various ways we can reduce likelihood and/or impact to get it within the acceptable levels. And John Scrimsher, who’s CISO over at Kontoor Brands, said, “When the board asks, what are you doing to protect me from XYZ?” In our case, we were asking ransomware, but it could be anything. And he goes on and said, “They’re really trying to determine if you understand the business, have done risk analysis of the threats and criticality of systems, and then taking steps to mitigate those threats. They want to know if you can give them confidence that their investments are cared for.” Man, John hits it on that quote right there, doesn’t he, Steve? He’s like, “I’m not asking to know, give me the cyber answer.” It’s like do you understand what we’re doing? Have you created a system that protects what I’m doing? Isn’t that the point here, Steve?

[Steve Zalewski] Yes. When we talked earlier, I said for the Fortune 500, and I use that as kind of a class, I think what John is saying is exactly right because the board is asking a risk assessment question because they have an audit committee. And their understanding, their job is to manage the risk, and they know how they want to have a business conversation around it. Yet on the other hand, you’ve got all the other types of businesses who CISO is in the position of, “It needs to be good enough security for the state of the maturity of my organization because I only have so much money. Okay? And it’s not that I don’t agree that security’s important, but I cannot put 20% of my development dollars into security at this point when I’m still trying to grow the company. And so give me a balanced approach.”

That’s a very different problem. And that’s what I’m saying is you’ve got this crawl, walk, run. You’ve got protect the company, I’m sorry, secure the company, protect the business, or sell more jeans. Okay? And then you’ve got boards and executives and other leadership types that have in their own mind how they’re growing and building their company. And that’s why I say is this is awesome because what John is saying is that is Nirvana, but you have to figure out as a CISO what is appropriate for your organization, and you have to be able to stretch that way, and it isn’t one-size-fits-all.

[David Spark] So, Shirley, I want to set you up here and say that what John says – by the way, I love what both of them say, Patrick and John – but your tool, while SeeMetrics is a great tool, it can’t do everything. It is up to the user to be able to communicate, “All right, I have put this grouping together. This is the grouping that tells my readiness-for-ransomware story.” It’s up to them to put that grouping together to then show and demonstrate the business, “Hey, we understand what you’re doing. We understand this. We also understand the value that you have in the business, and this is what we’re doing to protect it.” I mean, that’s what you’re trying to offer. It is up to the user to do this, but I do want to qualify because SeeMetrics, one of kind of the nice features of it that you showed me was you actually have templates. So, you could actually support [Distortion 00:25:25]. You don’t have to completely guess on your own. Right, Shirley?

[Shirley Salzman] Absolutely. It’s really part of the security organization Maslow needs. A, with SeeMetrics, it’s really about eliminating all the process of how do you take dozens of different tools and just shifting them into metrics. Connect your API, get metrics. B, how do you measure your security programs, your whole organization? This is something you’re getting out of the box. The next phase we have is the templates. Measure risks, measure threats, or create your own board based on those measurements. Now where I would need the security organization feedback is the level of maturity. Just as Steve, he said something that really resonates with me. It’s good enough security. Why is that? Because the level of maturity of one organization is different than the other. So, maybe in the next quarter, I can push specific areas because I know I got more teams in place for a specific tool to be handled, so I can make sure they are improving, but I have clear visibility of what’s happening across my organization. And that’s the beauty of the flexibility of having reusable metrics. You can use them for different purposes in the same place.

Closing

26:39.778

[David Spark] Excellent. Well, that brings us to the very end of this episode. But first, I want to ask you, Shirley, lots of great quotes here, which quote was your favorite and why?

[Shirley Salzman] I’m all in with Jared Pfost. There is a lot of theory around the what. How you’re streamlining metrics, this is a whole different, major challenge security organization’s faced with. This is what I think security organization needs to focus to make sure their teams are busy with security rather than data ingestion.

[David Spark] All right, Steve, very good.

[Steve Zalewski] Oh, that’s a good one. I’m struggling because there were several good quotes. But ultimately, I’m going to go with Patrick Lee of Safe Security, where he says, “It starts with risk tolerance and risk acceptance. How much of chance, percentagewise of likelihood, is the business and board willing to accept?” The reason why I’m picking this one is because we’ve all leapt to this conversation around wanting to do risk tolerance and risk acceptance. And that is ultimately what we’re trying to do now as leadership, as CISOs for many companies. But if we can’t get there, measurement is fine. You start with things like SecOps because you realize that that conversation may be too premature for your organization or your executive team. So, why I really like Patrick is that ultimately is the conversation you want to have. But if you can’t have it, that doesn’t mean you can’t do it.

[David Spark] Excellent. Well, that brings us to the tail end of this wonderful show. I want to thank our guest, Shirley Salzman, who is the CEO and co-founder of SeeMetrics. I’m going to let you have the very last word, Shirley. But first, I do want to mention your awesome company. And if you haven’t gone there already while listening to the show, go now. Now, if you’re driving and you’re listening to the show, please do not go to their website at that point. Please wait till you stop driving and then go to SeeMetrics.co. Check out what they’re doing. You can set up for a free trial right away. Shirley, you get the closing comment right here. First, obviously, people can reach out to you. We’ll have a link to your profile on LinkedIn from the blog post episode. But tell us, I mean, this has been a long journey for you, this whole metrics effort.

[Shirley Salzman] Absolutely. We are extremely excited about it. We are happy to bring back all the accumulated knowledge we’ve been gathering in the past years to the community with the templates and out-of-the-box metrics our users get. Feel free to reach out to us. We are happy to chat. We love to chat about your metrics challenges and your measurement challenges and how you narrate stories to your peers and boards.

[David Spark] Excellent. And by the way, I have actually seen this, and it does look pretty darn cool. So, check it out yourself – SeeMetrics.co. Thank you to our audience. We greatly appreciate your contributions and listening to Defense in Depth.

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.