How can we bridge the disconnect between compliance and building meaningful resilience? The business can understand compliance, so the focus is at least understandable. But how can CISOs better communicate that there’s a much wider range of risks they need to account for?

This week’s episode is hosted by me, David Spark, producer of CISO Series and Andy Ellis (@csoandy), principal of Duha. Joining us is Peter Clay, CISO, Aireon.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Safe Security

Full Transcript

Intro

0:00.000

[Voiceover] Biggest mistake I ever made in security. Go!

[Peter Clay] First of all, I’m typically pretty happy if I can get through an entire day without having made a mistake in information security in the first place. I think the biggest one, and the one that keeps me awake at night, was when I decided that I was really going to test the limits and boundaries of the enterprise firewall. By the time I got done entering 23,000 rules into the enterprise firewall, it turned out that the enterprise firewall didn’t work anymore and had collapsed. So instead of blocking anything, it was basically doing nothing. Second only to the time that I turned an enterprise firewall completely around, so I was protecting the internet from us, but allowing everything from the internet in. So those two, really, I just try to avoid internet firewalls at this point and have somebody else do that.

[Voiceover] It’s time to begin the CISO Series Podcast.

[David Spark] Welcome to the CISO Series Podcast. My name is David Spark. I am the producer of the CISO series, and joining me for this very episode, it is none other than Andy Ellis, partner over at YL Ventures. Andy, say hello to the audience. 

[Andy Ellis] Good afternoon, folks and depending on when you are in the world. Good morning, good evening, or good night. 

[David Spark] We’re going back to English. Yup.

[Andy Ellis] I realized I hadn’t done it in English in months, so maybe I should do that.

[David Spark] Have you cycled any other languages, or no?

[Andy Ellis] I’ve considered it, but I’ve not yet. But I might. I’ll probably cycle in Hebrew and Spanish. 

[David Spark] Hold it. Have you done it in Spanish or French yet, or no?

[Andy Ellis] So I’ve done it in Spanish and in French. The challenge is in Spanish. Good afternoon and good evening are both Buenos tardes, and so the flow doesn’t work for me because once you take out one of them, then it’s only two things at the end, and I’m not a big fan of it.

[David Spark] It’s like singing the song, Hello, Goodbye by the Beatles in Hebrew. Shalom, shalom.

[Andy Ellis] There you go. Shalom, shalom.

[David Spark] There you go. Or same thing with Hawaiian, Aloha, Aloha.

[Andy Ellis] Aloha.

[David Spark] There you go. We’re available at CISOseries.com. We have four other shows on our network. If you haven’t checked out all of them, what are you doing? Well, you’re listening to this one, and we’re glad you’re here. Our sponsor for today’s episode, a phenomenal sponsor of the CISO series. It’s SAFE! Continuous cyber risk management. You know what we’re all doing, right? It’s reinvented with Agentic AI. We’re going to talk about that a little bit later in the show. All right. So, Andy, exciting news. We have won an award, the CISO series. We have won an award.

[Andy Ellis] Yes.

[David Spark] I actually am not going to be able to be present when the award is being given out. I don’t know if I’m going to get a physical award, too, for that matter.

[Andy Ellis] Tell them we need three copies, because I need a copy, you need a copy, and I bet Mike would like a copy.

[David Spark] I think it’s for the whole… I don’t think it’s specifically for the show, I think it’s for the whole site, for CISOseries.com.

[Andy Ellis] Duh, the show gets it.

[David Spark] Okay.

[Andy Ellis] I need a physical copy. I don’t even know what the award is, and I want a physical copy.

[David Spark] But here’s my question. I know you have won awards. I see them behind you, in fact, in this video.

[Andy Ellis] Yup.

[David Spark] I sent a video acceptance speech, which was very tongue-in-cheek. I was being kind of silly, and I was essentially mocking the fact that they took so long to recognize us. But my question is, what are the kinds of acceptance speeches that you have given in the past, and is there any one that’s memorable? 

[Andy Ellis] So actually for the very large award behind me, that’s the… 

[David Spark] The phallic-looking one I’m looking at.

[Andy Ellis] Oh, if you think it is phallic when it comes to a point at the top, I do not want to talk about your love life.

[David Spark] Go see a doctor.

[Andy Ellis] Go see a doctor. This should be inducted into the hall of fame for CISOs.

[David Spark] Yeah, not bad.

[Andy Ellis] So that was a very meaningful one, but it happened in COVID. So I’m part of the class that came in during COVID. So I did a video acceptance for award. It was actually one of the first things I recorded in my no-longer green screen studio that was next to me. Yeah, that was a pretty meaningful one because it’s like, “This actually did matter. It’s a lifetime achievement.”

That said, I’ve gotten a lot of other awards, some of which I did not even know I was up for, and I’ll admit that I have to restrain my tendency to want to make a little fun of the award because, at that point, if you’re going to accept the award, then you have to have the courtesy and grace to treat it with some seriousness, even if it’s not an award you’re super proud of. I’ll be honest, I hate all the LinkedIn, “I am proud to receive this recognition that I didn’t even know existed until I got added to it.”

[David Spark] Well, the phrase “I’m humbled” actually doesn’t really mean what people think it means.

[Andy Ellis] Yeah, no. No, you can be humbled when you get into a Hall of Fame and you’re like, “I am humbled to stand among these other people.” The people who are in my Hall of Fame class, I’m like, “That’s humbling to be like, ‘Wait…'”

[David Spark] But it seems it seems like every time one wins it, “Oh, I have to use that very specific word, because if I don’t, I sound like a jackass.” 

[Andy Ellis] No, no, no, no. That’s not where the word came from. Somebody in marketing, usually at the awards, whoever’s giving out the award, wrote that word. That tells me that you did not write this post. This post was written by either your marketing team or, more likely, the people who issued the award gave you a template, and they all either start, “I am humbled” or “I am proud.” If you write that, it tells me you did not write this post, and you actually don’t care about the award.

[David Spark] Now, you winced when I said that I’m… I didn’t mock it. I did something very tongue-in-cheek, and it was played very silly and fun.

[Andy Ellis] You can probably get away with it. You’re a stand-up comedian at heart.

[David Spark] Yes.

[Andy Ellis] But there are people who would hear that and say, “Oh, I’ll be like David and I’ll make fun of this award.”

[David Spark] I wasn’t making fun of the award. I wasn’t making fun of the award. I was poking really fun at ourselves for that matter.

[Andy Ellis] “I can’t believe it took you this long to notice me.” Yeah, that’s it.

[David Spark] Yeah. It was more us. All right. Let’s get into our show, and let’s introduce our guest. Actually, first time ever having them on the show, thrilled to have them on. A CISO. Who’da thunk we’d get a CISO on the CISO series?

[Andy Ellis] Whoa.

[David Spark] We’re taking a turn here, Andy. A CISO over at Aireon, none other than Pete Clay. Pete, thank you so much for joining us.

[Peter Clay] Thank you so much for having me. I really am excited to be here.

Could this possibly work?

6:29.721

[David Spark] “Blue and Red has outlived Function at this point.” Now we’re referring to blue teams and red teams when I say that. “Now, rather, we should be embracing a whole of cyber defense mentality,” argued Joe White on LinkedIn. “Purple teaming…” and this is what he’s leaning to, “…isn’t anything new,” but he argues it’s critical given that sophisticated adversaries are increasingly laying the long game with compromises, some lasting years rather than days or weeks. But does that reflect the reality most organizations deal with day-to-day? Aren’t we still dealing with straight-up smash-and-grab attacks, as well as that they could still benefit from good old-fashion red teaming? Andy, what do you think?

[Andy Ellis] So this is one of those takes that it sounds nice. It’s like the little sign you put on your lawn that says, “We believe in science.” Sounds great, doesn’t mean a whole lot. No offense to Joe, who I’m sure is an amazing red teamer, pen tester, but when you say, “Oh, blue teams need to have a red team around them, but not red teams need to have a blue team on them,” I’m going to take your advice with a grain of salt. Here’s the challenge which is that, blue functionality is a heavily operational task, and yes, a lot of the design needs to be informed by a red team mentality.

So you definitely want to have your architects, your systems designers, the people who are doing your threat modeling should absolutely include pen testers and people with that level of experience. But if you’re saying that the person who is in the SOC doing real-time response to things, they’re functioning in a box. Much as we don’t like to admit that, and that’s why there’s a lot of startups talking about the autonomous SOC and how can you use Agentic AI to replace the work inside that box, this is not somebody who’s fungible, who’s turning and looking to somebody next to them, and if you put a red teamer there, I’m pretty sure they would go crazy.

[David Spark] All right. I don’t know how you feel about that, Pete, but your thoughts on essentially what Joe White said here, it’s the blue and red teaming, the classic configuration has outlived its point, and whether you agree with Andy. What’s your thoughts?

[Peter Clay] I think it’s moving actually well beyond that. I think we need a new rubric to think about it, because if you look at cybersecurity jobs that are going to get replaced the fastest by AI, it’s going to be red teaming. Red teaming is going to be a dying art very, very quickly. But number two, it’s actually much more complicated to be a blue team member. Yes, you’re following playbooks and everything else, but the minute you see…

You know, the real value of a blue teamer when they run into a zero-day. What do they do when they’re off book? How do they get help? It’s not just tier one, tier two, tier three, you’ve got to have speed kills problems, right? That’s the first rubric of everything else. Whether you’re talking about kill chain or whatever, the faster you detect it, the faster you fix it, the better the outcome for your organization. End of the day, you’re going to see a real decline in red team skills that are out there because machines honestly are almost equal or can do it better in many cases today.

The blue team approach though, has got to be infused with a lot more knowledge, and a lot faster way of making decisions than just following a playbook or following a checklist. So I think you’re going to see really dynamic shifts in both of them that have already started.

[David Spark] Maybe the fact that AI is going to replace red team faster than blue team, as Pete says, which I think I agree with, maybe drives home the point of the initial poster of Joe, “The red teamers better find roles on the blue team because otherwise they’re going to be out of work.”

If you haven’t made this mistake, you’re not in security.

10:18.034

[David Spark] “Every failure is a rehearsal for success.” Oh, this is speaking to your opening comment, Pete. This is great. Now that comment came from a cybersecurity subreddit discussion talking about dealing with the realities of getting popped. Rather than focusing on not being as, “good” as you thought you were, other commenters tried to keep it positive, offering that, “lessons learned” should be a core part of your post remediation process.

We know we’re going to have to deal with failures in cybersecurity. Pete, you admitted to many of them. So let me ask, one of the things you’ve learned is get someone else to do it, which I think is, by the way, a good solution. Andy’s fully on board with that. But let’s talk more in general. What do you need in the postmortem process to learn the most from a failure? 

[Peter Clay] You’ve got to understand one rule from the very beginning. You cannot stop stupid. Stupid is the most powerful force in the universe. It beats gravity, it beats everything else.

[David Spark] Hold on. Wait, stupid can beat gravity?

[Peter Clay] Oh, absolutely. Absolutely.

[Andy Ellis] How else did Arthur Dent learn to fly?

[Peter Clay] Interesting question. Well, we could riff on that for a while, as the Wright brothers learned to fly. That was stupid.

[Andy Ellis] It was stupid. 

[Peter Clay] Fundamentally, within cybersecurity, when we talk about failures, a lot of time we’re talking about people failures, we’re talking about tool failures, we’re talking about failures of imagination. “I couldn’t even imagine that an attacker would try to do this,” right? It all comes down to stupid. You cannot stop stupid. That’s why you have to build in resilience, that’s why you have to really think about what you’re doing, but you also have to own it.

What I’ve told every boss that I’ve had for the last 25 years is, “I’m going to get beaten. Somebody will find something that I didn’t think about, somebody will find something I didn’t get. The only thing you can really count on me is, I won’t get beaten the same way twice. I will find a new way to fail tomorrow than I did today.” Really driving it forward that way has got to be the way that you approach cybersecurity.

[David Spark] All right. Now, I’ll throw one thing about this. I used to think, and Andy, you’ve been in security a lot longer than I’ve been following it. There was a time, I believe, that you could just be smart and sort of protect yourself. But I think that stopped about seven, eight years ago. Yes. No.

[Peter Clay] No.

[Andy Ellis] That’s never been true.

[David Spark] Never been true.

[Andy Ellis] That’s never been true.

[Peter Clay] No, no? 

[Andy Ellis] No. No, I’ve got a blog post that I wrote probably 20 years ago about the Sledgehammer argument, which is, “Imagine you have a computer with sensitive data on it, and you want to keep it safe.” This is around 20-year-old argument, and I’d heard it 10 years prior, I was just here stealing it. All right, how do you keep it safe? Well, first of all, you should put this machine in a locked room so that nobody can get into the room.

You should probably unplug it from the network so that nobody can connect it over the network. You should probably just unplug it from the wall while you’re at it, so that nobody can read the electromagnetic emissions. The last thing you do before you leave it in there, you should just take a sledgehammer to the hard drive to destroy the data. There is no way to keep data safe, is functionally the answer. You can be smart and minimize your risk, but you’re never actually getting to safe.

[David Spark] All right. So let’s then go back to the original question here. You’ve dealt with postmortems before. Is there a playbook you run for a postmortem?

[Andy Ellis] Oh, absolutely. Multiples. So first of all, the one thing, really important, I want to reinforce the thing Pete said, and it happens to be Chapter 5 of my lovely leadership book, 1% Leadership.

[David Spark] Which I have read. It is excellent. I highly recommend it.

[Andy Ellis] Even if you don’t succeed, a story about failure can be a reward. The important thing here is, do not bury your failures because then you can’t learn from them. It’s important to tell the story of how we failed so that we can do better next time. If you hide the failure… So first thing walking into a postmortem to do lessons learned is to say, “We failed.” That’s okay. We’re not trying to blame people. We’re not trying to figure out who to blame, to say, “We’re going to fire you because you did this wrong.” What we’re trying to do is say, “We failed. What went wrong?”

Here’s the other thing, which is, unless you find 17 things that went wrong, keep looking. The human brain wants to tell a simple story. “This thing happened. Oh, this user didn’t check the inputs. They put it in, and a bad thing happened.” Okay, well, why were those inputs allowed to be consumed by a system if the bad thing was going to happen? We clearly didn’t have input validation. Where was the human getting these inputs from?

What was the output to the human that might suggest this was something… There’s 17 things that went wrong here, until you list them all, you don’t have the opportunity to start to learn, because what you want to learn is… 

[David Spark] But let me pause you there. That’s a really good point to make because whenever you hear the report of an incident, it is always boiled down to, “An intern did this, someone did that,” it never said, “And these 17 things happened,” because, by the way, that’s hard to read in a report or put in a headline.

[Andy Ellis] Really hard to… It doesn’t have to go into the report, everybody sees, but I hate the term “root cause analysis”. Incidents don’t have root causes. They have proximate triggers.

[David Spark] Pete is giving thumbs up to that.

[Andy Ellis] Right. There’s a trigger that hit this. An intern did X. Okay. That’s not your root cause. That’s your trigger. You handed a loaded gun to an intern, and asked them to pull a trigger, and didn’t tell them where to point it. How many failures do you have in that story that are now brutally obvious to everybody? You have to have that same level of brutal obvious. You gave an intern admin permissions to do an X with no controls. These are your problems, not that the intern pushed a button.

Sponsor

16:27.518

[David Spark] Before I go any further, let’s tell you a little bit about SAFE and risk management, because if those are concerns of yours, and if you’re listening to this show, it is, then you should be paying attention. So this episode is sponsored by SAFE, the cyber risk management company flipping the script on third-party risk management with Agentic AI, very, very hot lately. So let’s face it. Traditional TPRM, third-party risk management, is broken. Manual questionnaires, static assessments, and siloed tools can’t keep up with the speed and complexity of today’s vendor ecosystems. Yes, I’m sure we’re all connected to many vendors.

So this is where SAFE comes in. SAFE leads away with Agentic AI, powering a zero-effort TPRM platform that automates the entire third-party risk lifecycle from onboarding and due diligence to continuous monitoring. SAFE’s growing fleet of specialized AI agents does the heavy lifting, so your team can focus on what really matters, making smart, risk-informed decisions. That means 100% automated vendor risk assessments and real-time risk prioritized visibility without the manual grind. Now it’s no surprise SAFE was recently named a leader in Liminal’s 2025 TPRM report, top in the charts for product capability.

SAFE is also the fastest-growing vendor in the TPRM space, hitting 10 million in annual recurring revenue from its TPRM platform alone in just 10 months. If you’re looking to scale your TPRM program without adding headcount, visit safe.security, and see why autonomous TPRM is the future with SAFE leading the charge. Remember, it’s safe.security. Just go to safe.security.

It’s time to play What’s Worse.

18:15.379

[David Spark] All right, Andy, here we go. This is a little bit longer, so just hang tight.

[Andy Ellis] Ooh, how long? Do I have to take notes again? You did this to me in San Francisco, where we literally had to take notes on the What’s Worse.

[David Spark] No, no, you won’t have to take notes. You won’t have to take notes. It’s going to be involved, but I’ll summarize it at the end too. All right. 

[Andy Ellis] Yeah. By the way, Alexandra, if you’re listening, thank you for having taken the notes so that I could play that What’s Worse?

[David Spark] All right, this comes from Jim Nitterauer of Graylog, and here we go. Two scenarios. Scenario number one, pay attention, Pete. Remember, you’re going to answer second, Andy answers first. The IT team mistakenly downloads a version of a well-known and often-used software package from a website hosting a compromised version of the installer. The team builds the Intune package and deploys the malicious version to all 600 devices in your organization.

The malicious installer deploys a version that has been compromised with an auto-starting reverse shell only after your outsourced SOC declares an incident based on the sudden surge in alerts. Does the IT team realize what they have done? Here’s the key. Users of the impacted devices are reporting they were locked out of their devices as well. Okay. 

[Andy Ellis] Okay. The attack…

[David Spark] You’ll see. Essentially, they both have the same end result, but how we get there is a little different.

[Andy Ellis] Okay.

[David Spark] All right. So that’s scenario number one. Scenario number two, a member of the support team with admin rights to your corporate customer support site gets phished and inadvertently discloses their credentials. MFA has not yet been implemented as the executive team heard too many complaints from support agents when it announced that MFA would be activated. Well, within 15 minutes of the phish, your entire support team is locked out of your support, and all users with admin rights can no longer access the platform.

Within 20 minutes, customers are calling the support line and reporting that they are also locked out of their access to the portal, and have begun receiving strange emails generated by the support portal, requesting that customers immediately upgrade their installations due to a serious bug. The download link in the email points to an unknown site that is later found to host a malicious version of your software. So in summary, the first scenario, every device is now running a malicious software version that generates a reverse shell from each device to the malicious actor, and all the result is that users are locked out.

And the second scenario, your support team is locked out of the support platform, and the malicious actor is pushing malicious versions of your product to your customers. Which one is worse, Andy?

[Andy Ellis] Oh, so the second one is way worse just on impact. The scenario that led to the impact, I actually would prefer the first scenario but the outcomes… If they actually had the same outcome, I would have preferred the second scenario, because I could use this as sort of an object lesson very gently of, “Oh, we need to get MFA on to our admin users.” Boom, really nice when you have a light incident that supports a project you’ve been wanting to do.

This is not a light incident. The downside of the first one, look, my user’s got locked out, but we can recover from that because my admins are not locked out of everything. Let’s deploy a new thing, wipe all the systems, not the end of the world. My customers are getting compromised because they’re being told, they think, by me, to install compromised software on their end machine. That’s a business disaster. So that is way worse than the internal challenge we have of rebuilding our systems, which we have to do in the second case anyway.

So I think, and maybe Jim didn’t intend for the second one to have such a worse outcome, but the moment you’re affecting my end-users directly because of my breach, that’s always going to trump.

[David Spark] All right. Good point. All right. It’s both situation of lockouts is what I was…finding similarity.

[Andy Ellis] No, okay, both are bad. I don’t want to be in either one of these companies.

[David Spark] They’re both bad. Yes. Yes. We both agree. Pete, do you agree with Andy here that the second scenario is worse? Maybe you do agree for a different reason, or you don’t agree. What is it?

[Peter Clay] I absolutely agree.

[Andy Ellis] My favorite words. 

[Peter Clay] The minute that it escapes your four walls, you’ve also just run into regulatory help. A lot of times, it’s not even the technical recovery or the technical communication, crisis communication, or whatever else. If you are a regulated entity, it’s not a lot of fun to have the regulator sitting in your conference room for days, weeks, and months helping you with that ubiquitous root cause analysis to determine the level of fines and the level of everything else.

In scenario one, you can fix the problem, get on with your life. In scenario two, you’ve now been publicly shamed, whipped, and you’re about to have a really bad day.

[David Spark] All right. Yeah. I assumed we would go in that direction, but I did like the sort of the creative scenarios Jim brought here of essentially two environments being locked out. But you make a good point. Once it’s out of your four walls, it’s like… 

[Andy Ellis] Yeah, that changes things. So for future folks, I love the idea of two different incidents that have the same result, and which one would you hate more? So I want to see more like that, but try to make the end results exactly the same from an impact, and so we can talk about how we got there. Those would be some fun ones.

It’s time to measure the risk.

23:49.368

[David Spark] “Risk management, as practiced in most organizations today is trying to take your jacket off in a crowded train. Awkwardly executed, fatiguing, and wildly misunderstood by everyone around you.” I like that visual. Richard Cromwell of RiskQuanTech was the one reviewing the problems with legacy risk management solution, but one of his key points stood out. He recommended hiring a deeply experienced risk expert as part of modernizing. But can anyone be a risk expert? If so, what would you expect that person to be able to do? Let’s say someone is a risk expert. How would they help your business, they know nothing about? Andy, you’re smiling.

[Andy Ellis] Sorry, I’m laughing. What exactly is a risk expert?

[David Spark] Well, that was my question. That’s why I’m asking it to you, too, as well.

[Andy Ellis] I knew you’d get to your part of the read, and I’m like, “What is this?” I did go check, and I’m like, “Oh, this person is a risk expert.” That’s why they’re saying, “Hire a risk expert.” 

[David Spark] But hold on. But let’s… I mean, security people should be risk experts sometime, but when someone touts that I’m trying to understand what they can help you with. 

[Andy Ellis] So if I was going to hire somebody who’s focused on risk, I want to understand like, “What do you think risk is? Do you have a methodology?’ As a sneak preview, my second volume in How to CISO, which is not yet published, is on risk measurement and management. I’m at 35 pages, and I’m not even remotely done yet, so I’m not even sure. I don’t even know how to distill that down to what is a risk expert because what you really want is somebody who is an expert in telling fairy tales. That’s going to sound really trite, but what is the job of a security team? It’s to enable the company to make wiser risk decisions. Risk experts think that they make wiser risk decisions. No, you don’t. You don’t actually own the business. You don’t own the risk. Our job is to make stories about the risk, compelling in a way that enables the rest of the business to choose to do things that are more profitable and have less downside. That’s it.

[David Spark] Let me ask you, Pete. Have you ever hired a risk expert?

[Peter Clay] I actually built a whole company around quantifying cybersecurity risk, and found out a couple of different things. Found out a while ago that cyber insurance isn’t based on actuary, it’s actually based on loss. So cyber insurance is priced the exact same way that casinos price their games. “How much money do we want to make at the end of the year, and how much money do we want to pay out?” So we launched an entire company around the idea of building actuary information, and it provided some really interesting feedback, provided a lot of feedback on what’s important from a risk management standpoint, but truly cuts your risk down, as well as really provided some interesting feedback on what actual expected losses are, given scenarios.

Ultimately, what we found was, the methodologies that you choose to quantify and determine your risk are absolutely critical, but without the context of understanding what cybersecurity is, and what you can do with it, and how to manage that risk, it’s absolutely valueless. The way we all used to report risk as the CISO is, we would go into boardrooms and we would say, “We had 42,597 attacks on the firewall last year,” and the board would go, “Uh-huh.” “We stopped 8,700 different pieces of malware.” “Okay.”

[David Spark] Wendy Nather of Cisco used to have a great line of this. He goes, “I don’t care how many raindrops your umbrella can protect me from.”

[Peter Clay] At the end of the day, though, being able to go in and talk about risks in terms of dollars and cents and expected loss can change the way that the board responds to you, can change the way that you’re focused. But your methodology, if you walk in there and your CFO hasn’t signed off on the methodology, you’re done. You have no right to even be saying anything.

[David Spark] All right, so Andy was making a little bit… Hold it. Before Andy, you talk. 

[Andy Ellis] So everybody who loves fair needs to hear those words. No, no, I know, Dave, you want to go somewhere. But those words are the most important words. If you want to talk dollars and cents, if your CFO is not going to stand behind your numbers, it’s career suicide.

[David Spark] Okay, so that’s a good point. So that’s an interesting theme because we talk a lot about different ways to manage your risk, and it’s really… What I keep hearing is, it’s about what matters to you, and for most companies, it is about money and attaching it to money. But talk to other organizations like healthcare or anything that deals with human safety, and that actually takes precedence over money, and so you have to tie it off into that as well. But going back to what Pete was saying, Andy, you were… I was just watching you go through an emotional roller coaster of reactions of, “Ah, oh, ah.” Like that. I’m doing the verbal versions of it. So take us through your reactions to Pete’s response. 

[Andy Ellis] So as Pete was talking about the folks who would talk about how many firewall scans they blocked. I will tell you that in the early days of Akamai, I was basically the second or third person on a sales call, that if the security team, if the customer started to object, they would bring me in to handle the objections. You have this number one objection that I heard about Akamai in the early days when we were selling just basic CDN services, “Put our CDN in front of your website, we take all of the requests, and then we’ll only come back to you for content we need,” the biggest security objection was not that like, “We now control your content. We could change it if we wanted to.” People weren’t asking me that.

They said, “But now I won’t see the port scans against my website because you’ll just block them, and how will I show my value?” I’m like, literally, this is what I had to deal with. So like, “Well, can you give us a report of how many port scans you block?” and I’m like, “No, I don’t look at that data. I don’t care. Why would I keep track of somebody scanning a port I don’t have active on a machine?” Like, really? So I was laughing because I have lived that. I spent three years where that was the number one objection from security teams about why not to buy a CDN. Had nothing to do with the risk of another vendor, the loss of visibility of a metric that doesn’t produce any value.

[David Spark] But it sure is pretty. Can we just agree on that?

[Peter Clay] Well, but then you had the executive about a year ago that came out and was talking about it. It weren’t a cybersecurity person. He’d obviously been fed this number, and they came out and said, “We have 42 billion attacks on our firewall,” at a conference, and you just saw the whole conference go…

[David Spark] So what?

[Peter Clay] No. No. Being able to talk about risk in a way, back to your point, that the business understands is the critical driver here, right? Whether you’re using a Bayesian analytic or you want to talk fair, or you want to talk something else, there’s some really good tools out there to really help you understand it. But at the end of the day, you’ve got to be able to talk to the business in a way that they understand.

[David Spark] Right.

[Peter Clay] If you can’t do that, you might as well just be available to industry and move on to the next one because you’re not going to get there.

[David Spark] This goes back to Andy’s comment is, if the CFO is not signed off, or whomever cares about the thing, if it’s not directly money-related, because it could be, you know, physical security related or both, you have to get both parties signed off.

[Andy Ellis] Yup, right.

How can we align different departments’ objectives?

32:00.373

[David Spark] “Risk registers win audits, resilience wins a crisis.” That’s how Levi Gundert, of Recorded Future, frames the disconnect between compliance-focused security and actual cyber resilience. While CISOs spend most of their time managing audit committees and regulatory frameworks, they’re only getting 15 minutes a year to brief boards on the four other major cyber risk impacts beyond legal compliance failure. Now operational disruption, financial fraud, brand impairment, and competitive intelligence. Those are the four we’re talking about. The problem isn’t that compliance is bad, but that organizations are treating point-in-time audit checklists as if they represent actual security posture.

As one industry observer noted, boards see “traffic light audit check marks” but miss “the real color of cyber risk”. This raises a critical question about how different risk domains can work together rather than operate in silos. I’m going to start with you, Pete, here. Can operational risk teams help compliance teams understand real-world threat scenarios, and can compliance frameworks actually inform better incident response, or are we stuck in a world where passing on audits has nothing to do with surviving actual attacks?

[Peter Clay] That’s a very loaded question for me because I also work in a space where aircraft safety, we have an entire safety team.

[David Spark] Right. This was my reference in the last segment.

[Peter Clay] Right. So the answer is, we measure risk differently, we understand it differently, but even the regulators have caught on to the fact now, after a lot of struggle, that cyber has a seat at the table when it comes to safety, and to aircraft safety. When you look at the GPS jamming pieces that are going on, 41,000 incidents a year, those are also cyberattacks. So that means something different when you’ve got a lens aircraft safety, “Absolutely, let’s look at that,” but you also have the cyber piece that you also have to honor and think through.

What it comes down to, at the end of the day is, you’ve got to be willing to have the conversation as to the give and take in terms of risk and understand my safety risk may be higher or lower than my cybersecurity risk, given all different factors. But you have to have the conversation, in some cases, you have to have the argument. I’m not much of a fan anymore, and I think just like with red teaming, we’re also moving into a post-audit environment as we’ve understood them. Somebody’s sitting there with a checklist and going, “You complied with this control or you didn’t.”

Most of that information, I can gather in a very automated fashion today, and get a lot closer to what I would consider to be continuous compliance. So those two things are really starting to drive very different behaviors, right? Because when you’re in continuous compliance, and suddenly the control didn’t fail three months ago, and you’re not aware of it, you’re now aware of it almost… You’re not waiting for the auditor to find something, you’re responding to it, and you’re moving forward with it, that really starts to tie into, and put you ahead of the audit processes.

At the same time, it gets really complicated when you’ve got different risk frameworks that you’re trying to manage and balance against. So conversation is key, having a same set of facts is absolutely a must, and auditors typically don’t understand your business well enough to provide really useful insights into what your program should be or how it’s accomplishing its processes. 

[David Spark] Don’t rely on the otters. 

[Peter Clay] Ever. 

[David Spark] Andy, again, you went through a series of emotions there. Walk me through them. 

[Andy Ellis] I’m very emotional today. Not really sure what’s going on there. 

[David Spark] You are. I’m thinking we should capture just a video clip of you. Not even the audio of what you’re hearing, just to watch the different winces you make when the guest talks about something, and it’s pretty clear you agree or disagree with it by just your facial gestures. My guess, like my wife, you have a pretty bad poker face. Am I right, Andy?

[Andy Ellis] When I want to have a poker face, I have a poker face. This was giving Pete the opportunity to have a live audience reacting with him. So I’m in somewhat agreement with Pete here, which is, first of all, I love the fact that we’ll get to a world where compliance is no longer about looking at the past to see if it’s still true, but looking at the present. That’s the whole goal of continuous compliance. The reality is, if you don’t know your control failed until an auditor found it, you did not have a control. I just want to put that out there, that control systems don’t fail silently.

If yours can fail silently, it was not a control system, and therefore you were never actually compliant, and you’ve been lying to yourself and your auditors. That’s just a completely separate soapbox. Somebody can buy me some wine at a conference, and I’ll expound on that one. The reality is that compliance is about past-proofing your systems. In the past, we knew some things you shouldn’t do, and so you’re just not going to do those. It’s like 90% of building codes. They’re designed for building buildings that we already know how to build. So like, “Don’t do certain things,” is baked into the architecture.

They’re not about future-proofing buildings. They don’t talk about technologies that didn’t exist when the building code got written, and that’s the big challenge that we’re really addressing here, which is, if you’re spending all of your time saying, “Are you compliant with the rules of the past?” you’re not talking about, “Are you dealing with the specifics of your problem?” Whether or not your building is safe to inhabit says nothing about whether the business you’re going to run inside your building is a safe business to run inside that building. That’s the big challenge that we have is, “How do we shift that conversation?”

In my universe, if you’re a security person and you’re talking to the board about compliance, you are wasting your time. Your compliance conversation should be really fast. “Because we’re in the following 7 businesses, we have to comply with these 85 things. That’s a product requirement.” We pass all of those audits. That says nothing about our security. That says a lot about how well we tell a story to our auditors. Boom, great, done, move on.

The board-level conversation should be about all of the rest of risk. So if you’re going to get 15 minutes to talk about the rest of risk, and you’re spending across the year, two hours talking about compliance, that’s the wrong waiting, unless you’re really bad at security, and the compliance controls are showing that to your board.

[Peter Clay] So having spent five years as an IT auditor, and unfortunately having come out of the pen test world before I was an IT auditor, favorite trick, first page of the system security plan, “Is this what your network looks today?” The minute they start studying their shoes really vigorously, because you know that’s a hub and spoke network, and they’re fully cloud deployed, the rest of the document’s pretty much nonsense.

[Andy Ellis] Yup.

[Peter Clay] Right. It always does come down to resource management. You’re never going to have enough time, you’re never going to have enough people, and you don’t have enough fingers and toes to plug all the holes in the dyke ever. So a lot of audit, particularly as it’s been practiced today, you know, a SOC2 is a fully retrospective look at how your controls perform over time. A lot of that stuff has really got to give way to a much more modern interpretation of risk and risk management, and what’s important and what isn’t? Back to your point, if I’m sitting there telling my board anything else, but, “Yes, we comply.

The only checklist I’m going to show you is, ‘Yes, I’ve checked all these off. We do comply. Thank you very much.’ You’re losing time.” But the dependence upon documentation versus performance, the dependence upon not understanding the outcomes of a well-executed control, which most auditors don’t, the focus on process versus outcome, are all things that are incredibly limiting when it comes to the value of IT auditing.

Closing.

40:32.604

[David Spark] All right. Well, that brings us to the very end of this episode. Thank you very much, Pete. I appreciate your conversations about failures, which was a theme, and risk. I think, failure and risk was the two big themes that came through today’s show. So thank you very much for both of those. I want to thank our sponsor. That would be SAFE security, continuous cyber risk management. In fact, they have a lot to do with risk and also specifically connecting it to finances.

Check out what they’re doing. They’ve reinvented with Agentic AI. Go to their website, safe.security, and you could go check it out. Andy, as always, thank you so much. Greatly appreciate your great emotions and your phallic awards that are right behind your head.

[Andy Ellis] Okay, for those of you who are listening, it is not a phallic award. It looks the Washington Monument.

[David Spark] Let me describe what it looks like. It is a blue glass obelisk that stands straight up and comes to a point. How would you describe it?

[Andy Ellis] Like the Washington Monument, only smaller. 

[David Spark] Also phallic. 

[Andy Ellis] It is square, it comes to a point. That is not phallic. I cannot believe, on this show, that this is the argument that we’re having.

[David Spark] Pete, by the way, you could be the decider here. What is that award behind him? 

[Peter Clay] Having worked in DC most of my life, and understanding that the Washington Monument is known as George’s last erection, I think I’m with you.

[Andy Ellis] Okay, I guess I lose on that one. But look at the rest of my awards.

[David Spark] They’re not phallic. They’re dope. 

[Andy Ellis] I mean, come on. Things that are spherical, things that are rounded, those are not phallic shaped, things with holes in them. Come on.

[Peter Clay] As Freud said, sometimes.

[Andy Ellis] I guess it’s all in what the beholder is brain… where their brain is.

[Peter Clay] Exactly.

[Andy Ellis] I think this says more about David than me.

[David Spark] There you go. Sometimes a cigar is just a cigar. Thank you very much, Andy, thank you very much, Pete, and thank you to our audience. We greatly appreciate your contributions and for listening to the CISO Series Podcast.

[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows, Super Cyber Friday, our virtual meetup, and Cyber Security Headlines Week in Review. This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at david@CISOseries.com. Thank you for listening to the CISO Series Podcast.