Security spending on AI grew 51% over the last two years. But many CISOs think adopting new LLM-based tools will make breaches more likely. So why the rush to throw money at them?

This week’s episode is hosted by me, David Spark (@dspark), producer of CISO Series and Andy Ellis (@csoandy), operating partner, YL Ventures. Joining me is our guest, Richard Ford, CTO, Praetorian.

Got feedback? Join the conversation on LinkedIn.

Thanks to our sponsor, Praetorian

Full Transcript

Intro 

0:00.000

[Voiceover] What’s a great approach from a security vendor? Go! 

[Richard Ford] I love it when a security vendor comes in and they lose all the marketing fluff, and they just tell me what this thing really does, and we get straight to the chase. It’s refreshing, saves a lot of time, it’s fun, and bluntly, it’s better for everyone.

[Voiceover] It’s time to begin the CISO Series Podcast. 

[David Spark] Welcome to the CISO Series Podcast. My name is David Spark, I am the producer of the CISO Series. And joining me as my very co-host for this episode is Andy Ellis, the operating partner over at YL Ventures. Say hello to the nice audience, Andy.

[Andy Ellis] Happy Hanukkah, everyone. We’ll be in the middle of Hanukkah when this drops. 

[David Spark] Excellent. Happy Hanukkah, everybody. We’re available at CISOseries.com where you will find your eight-night Advent calendar for Hanukkah. No, you will not find that there at all, [Laughter] but I wish we did. 

[Andy Ellis] It’s a great idea! We should do that. 

[David Spark] We should do that. I don’t know what I would put into an Advent calendar, a Hanukkah Advent calendar. Probably gelt is what you would throw in there. 

[Andy Ellis] Gelt. 

[David Spark] Yeah, there you go. Hey, our sponsor for today’s episode is Praetorian. We’re very excited to have Praetorian onboard. They were an early sponsor of the CISO Series, and they have come back many years later and they’re joining us again and very excited about it.

In fact, they’ve also brought our guest. But before I introduce our guest, I do want to mention that although we are recording this about a month early, the two of us at this very moment are in Austin, Texas, for the Cybersecurity Marketing Society’s Conference of which I am sponsoring, the CISO Series is sponsoring, very excited about it.

And you’re there and I’m there and we’re both speaking and I’m very excited about it. So, what’s your excitement about attending this? Have you ever attended like a marketing conference like this before, Andy? 

[Andy Ellis] I have never, but I work so much in marketing, and I always find myself I’m talking to marketers. And it’s really fun because Gianna reached out to me and said, “Hey, do you want to come to a book signing?” I said, “Sure, and I’ll do a leadership talk if you want.” And then when we were talking, I said, “Oh, here’s this thing I always pitch to marketers I work with.” And she’s like, “That’s a talk.

You should give that talk also.” So, I’ll actually be doing two talks. 

[David Spark] So, what is the talk that you pitched to her? 

[Andy Ellis] So, it’s the nine questions that a CISO needs to have answered before they will buy a product. 

[David Spark] That is an excellent talk. Oh, you’ll have a packed house for that.

[Andy Ellis] And how it drives your content strategy as a result.

[David Spark] Well, I am going to have a workshop there that has nothing to do with cybersecurity, it has to do with in-person networking, and it’s called business networking pick-up lines.

[Andy Ellis] Ooh.

[David Spark] And what it is, it’s a series of workshops. Nobody’s sitting down, you’re all standing. And it’s a series, there’s 30- to 60-second exercises flexing different in-person networking muscles.

[Andy Ellis] I love it.

[David Spark] And each one shows you how to do one unique tactic differently, and even the most introverted and extroverted people who think their total pro salespeople will get something out of this. And by the way, I’ve done this for sales kick-offs.

Speaking of that, if you’re a security vendor listening, if you go to my other business site sparkmediasolutions.com, go to Solutions, you’ll see a button and it’ll show some videos from the business networking pick-up lines. But hopefully, we have done such a great job, in fact, the 12th, I will have just done the presentation and hopefully you can talk to your co-workers who were at this event and they’ll tell you how wonderful it was.

That’s my hope. Let’s get on with the show, why don’t we? I’m very excited to bring our guest on to talk all wonderful cybersecurity topics because that’s what we do on this show. It is the CTO of Praetorian, our sponsor, and our guest, it’s Richard Ford.

Richard, thank you so much for joining us.

[Richard Ford] Well, thanks for having me on. 

Are we making this situation better or worse? 

3:58.863

[David Spark] When it comes to cybersecurity staff, we all want people who can take initiative and step up when needed. But relying on this type of IT heroism in your SOC – security operations center – is actually a really bad practice, as outlined by Anton Chuvakin, who is the host of the Cloud Security Podcast, and he talked about this in a recent Medium post.

So, often the need to rely on heroism is a sign you haven’t scaled process to allow for unexpected incidents. For Anton, the first step in fixing these processes is to let them break to uncover systematic issues. So, I’m going to ask you, Andy, first.

From there, how do you go about building a security program that doesn’t depend on individuals’ heroism? And by the way, do you recommend his tactic of let things break and let’s see what happens?

[Andy Ellis] Well, I think his tactic is a great conversation starter, and it should not leave the conversation, it’s an awful idea, “Oh, let’s just let things break that we saw breaking, and we could have fixed.” Seems like a great way for the CISO to no longer have a job.

Not going to solve your hero culture problem. That said, look, I firmly believe in this, and in fact, I have a chapter about it. It’s been a while since I’ve pimped the book. 

[David Spark] Which you’ll be signing at the Cybersecurity Marketing Study. 

[Andy Ellis] I’ll be signing it at Cybersecurity Marketing. 

[David Spark] And that reminds me I should bring my copy, which I’ve already read, to be signed. 

[Andy Ellis] Yeah, so I will sign it for you. 

[David Spark] I’m not buying it a second time. 

[Andy Ellis] Well, no, no. I think Gianna’s bought them and is going to give them away. 

[David Spark] Ah! And I already bought it! What was I thinking? 

[Andy Ellis] But it’s only going to be like 100 copies, so it’s like first come, first serve. 

[David Spark] Oh, yeah, there’s going to be more than four times the people there. 

[Andy Ellis] So, it’s on you as a security leader or a leader in your organization to recognize this problem, recognize that you have individuals that are picking it up, and work to create process that is scalable. Like, if you’re saying, “I can see that the process doesn’t work, but I need it to break to get buy-in,” you have some serious organizational dysfunction.

What you should do is say, “Look, Richard’s the one who always picks up this process. So, Richard, next time you see it fail, David is going to fix it, you’re just going to tell him how to do it.” And you’re going to keep doing that until you have a whole organization that knows how to deal with the problem, you fix the process, and you’re not relying on the heroes.

But don’t just, like, get rid of your heroes. 

[David Spark] That is a really good point is that when a hero does shine, don’t make them have to do it a second, third, and fourth time. Which, by the way, I think heroes would like to do that because it makes them look as invaluable. So, how do you prevent someone wanting to do the thing to make themselves look invaluable?

Like, “Hey, it’ll raise my value in the company if I do this.”

[Andy Ellis] Well, some of them do, but many of them when they realize the personal cost of that value… Like I had a guy who worked for me, he was one of these people that, if you had an unsolvable problem, you’d say, “Hey, go solve this.” And he would fix it, he’d build a system, he’d be the one stuck maintaining the system because nobody wanted to take it.

Not because it wasn’t good, but because it didn’t come with funding. And so he just sort of would accumulate these over time. And then one day he comes in and he says, “I’m engaged.” And we’re like, “Great, we want you to take a honeymoon. So, all the time between now and your honeymoon is going to be taking practice vacations so we can make sure that other people have picked up your work.” 

[David Spark] Very good point. All right, Richard, do you agree with Andy’s strategy? Do you have another strategy to not always rely on the heroes? 

[Richard Ford] Well, I think going back a little bit, I do think sometimes there are some organizations where you do have to let things break, Andy. I think that’s very contextual. In a healthy organization, you absolutely would never do that. But I’ve definitely seen organizations where if you don’t actually let the sort of blood hit the walls, nobody pays any attention to it.

It’s not healthy, but sometimes it’s in the best interest of the organization. I think the other thing I’d say is that hero syndrome that you touched on, David, is a thing where somebody goes, “Yeah, it feels good to be a hero. So, I’m going to let this thing run a little bit and I’ll be a hero.” And that’s a bad thing.

[David Spark] Well, and also people have this sense of, you do it enough, [Inaudible 00:08:15] “they couldn’t live without me” kind of attitude. 

[Richard Ford] That’s right. And that comes from either insecurity of the person or, again, an unhealthy culture. I mean, one of the things that I’ll tell my team all the time is I want them to be hugely winning everywhere because I don’t need to do anything, right?

Their success is my success. So, again, a lot of the time, these things go wrong when there’s a sort of lack of health culturally within the organization. It’s not a technical thing. 

How can we secure new technology without creating new risks? 

8:45.312

[David Spark] Every conference I’ve attended has a session about cybersecurity and AI. In fact, I think it’s now the law. If you’re going to have a cybersecurity conference, it’s by law you have to have an AI session. 

[Andy Ellis] I was at a conference that wasn’t cybersecurity, but half of the talks were on cybersecurity and AI. It was the CIO conference.

[David Spark] It’s amazing how it squeezes in there. [Laughter] 

[Andy Ellis] And I was on a cybersecurity talk that the panelists kept trying to pivot it to AI, and I’m like, “No, there’s like four other people talking about that. Let’s stop.” 

[David Spark] By the way, I’ve yet to see an AI session that doesn’t just have more questions at the end than answers during the session. But nearly half of executives fear that adopting AI tools will lead to security pitfalls. Now, this is as reported on CSO Online about a recent survey from the IBM Institute for Business Value.

“Bad actors are using the technology to improve the speed, precision, and scale of their attempted intrusions,” this is what Jon Gold who’s the author of the article said. We’ve heard this many times before. But as a result, AI security spending has risen 51% over the last two years.

Virtually all respondents said adopting LLMs – large language models – will make a breach more likely. That’s also questionable. I will start with you, Richard, on this. Is this just a reflection of organizations not fully understanding these tools, or do businesses adopting LLMs truly present new targets for threat actors?

[Richard Ford] I think realistically, anytime you add any new technology stack to your attack surface, you’re adding risk, right? The question is, how much risk? 

[David Spark] And Andy gave you the point on that one. 

[Richard Ford] Is that a good thing? I think that’s a good thing, right? 

[David Spark] Yes, yes, it is. 

[Andy Ellis] No, no, it’s a good thing. Everybody needs to understand that. You always add risk when you add something. Then the question is, can you use that to remove some other risk? 

[Richard Ford] That’s right. And technology exists to solve business problems, and it’s our job to figure out how to adopt that technology effectively, right? We need to be the department of yes, not the department of no. With that said, I don’t think it’s the fact that organizations don’t understand these things well enough.

I don’t think any of us really understand these things well enough, and we certainly haven’t worried about the second-order effects. The attack chains are very different for these things. And there’s some really sort of sneaky attacks, like data poisoning in advance that you use for training data, especially when these things are connected to the internet.

[David Spark] So, you pointed out when Richard just said, “Yeah, introducing something new, a new technology,” and you, Andy, just said, “Introduce anything, something, it adds new risk.”

[Andy Ellis] Yep.

[David Spark] But this something is so confusing to everybody, and it’s coming on strong. The growth in just a year, I don’t think I’ve ever seen a something or a technology grow at this rate in one year ever. What do you think, Andy? 

[Andy Ellis] So, I think when we talk about LLMs in general, generative AI, I think there’s two completely different ends of the spectrum, and people keep confusing it. There’s what I think of as consumer GenAI, right? Think about ChatGPT. For most people, it’s consumer GenAI.

I’ve got this great thing I use on a plugin on YouTube that I can literally take any YouTube video, click a button, and it hands me a transcript and a summary of what was talked about in that video. Like, that’s fantastic. Like, I use that, that’s consumer GenAI.

Then there’s like the enterprise, and I don’t mean just, “Oh, well, all of your people are using consumer grade.” No, I mean like, oh, you have LLMs trained on your data set and on your corpus. And there’s companies in that space that’ll do things like, “We’ll answer security questionnaires for you by training them on your security documentation so you can write them.” So, two different problems here.

And even at the extreme end of enterprise, there’s some really crazy stuff you can do. 

Most of the adoption of AI is at the consumer level within the enterprise. So, it’s marketing teams that are using it to write content. And let’s just explore the risk. Let’s just say that I’m going to have David write content for me. So, I have to make sure that I do not hand David confidential stuff, and that I sanity check everything David produces.

And you’ve probably heard some of David’s content. What most of you don’t realize is that there’s layers and layers of editors and fact-checkers that make David and I sound good. And the rate at which we produce content is actually not very high, right?

Because it’s humans who are doing it. And so if you’re trying to outsource this, you’re like, “Oh, I get one piece of content a month from my David or from my whomever.” Well, ChatGPT and other LLMs have basically completely upended that model and as fast as you can ask for content, you get it, but it doesn’t have any of those layers of editors or fact checkers.

[David Spark] But I would also throw this out, and I’m going to toss this to you, Richard, is that really what you need to do is go down the rabbit hole, which you were starting to do just then, Andy, and just think how could this actually happen? 

[Andy Ellis] Right.

[David Spark] Like it’s really, really difficult. And when you start to realize how unbelievably difficult it is to pull any sensitive information out, you realize, “Oh, my God, I’m trying to protect something for which the risk is very low actually because of how hard it is.” In fact, I don’t even know how to get it out.

Then if I don’t know how to get it out, I don’t know how some random other attacker might get it out, but I knew how this random other attacker is doing other things to get at me. But I think, per one of the quotes that was said, and this is where I toss to you, is the real concern is not so much our use but the attackers’ usage of it to facilitate the efforts that they make.

Is that really the concern?

[Andy Ellis] That is a huge concern, but I don’t think people are really focused as much on that outside of the pundits. But the pundits are like, “Oh, yeah.” They can use AI to become more efficient to implement their processes, and yes, there’s some huge value there, but most of the AI and security conversations I see, that’s not what people are actually talking about.

[David Spark] Richard?

[Richard Ford] So, I think the interesting point here, Andy, is I’m not that interested in attackers being faster, better, cheaper, right? I mean, yeah, it’s going to be a pain, it’s going to be hard to defend because it’s a little bit more work. Faster, better, cheaper – who cares?

But when AI enables new attack models or new dark business models, that’s when life gets interesting for us, right?

[David Spark] Well, that’s what we did talk about, is that there are ways to create zero-day malware. You can’t just say it in one step, “Create a zero-day piece of malware for me,” and I can’t remember who it was, but if you piecemeal it together, there is a way to do it, as we saw from an earlier report.

And that’s where I would think your greatest concern is. Yes, Richard?

[Richard Ford] Malware or anything where the business model fundamentally changes, right? Because if you think about attackers, they have a fitness function, they have some sort of ROI on what they’re doing. And when the AI can make something that was not profitable for them profitable…

[Andy Ellis] Yes.

[Richard Ford] …that’s when we see the sudden uptick, and that’s where it’s going to be a problem for defenders. 

Sponsor – Praetorian

16:00.885

[David Spark] Before I go on any further, I do want to mention Richard’s awesome company, Praetorian. So, here’s what you need to know about them. Listen. Praetorian, which by the way, let me spell them for you – Praetorian. So, Praetorian helps security teams go back on the offense and shift cyber programs from a reactive, assume-breach mentality to a preventative first strategy.

Praetorian’s adversarial experts protect your business from evolving cyber threats by continuously emulating attackers to help you identify and mitigate your material risk before attackers have the opportunity to exploit them. Good old-fashioned red teaming here.

So, with Praetorian as your offensive security partner, stay several steps ahead of attackers and avoid the high cost of a breach. That’s what we all want to avoid, right? Best defense is a good offense. For more, go to their website. That’s Praetorian.com.

It’s time to play “What’s Worse?” 

17:08.474

[David Spark] Richard, you know how this game is played, correct? 

[Richard Ford] I do. 

[David Spark] I’m going to make Andy answer first. You will respond whether you agree or disagree with Andy. This comes from our listener, Sherman Homan, who is with HCL/Fresenius, and he throws this scenario. The bad guys broke in and exfiltrated several gigabytes of data.

You’re sure it’s encrypted…

[Andy Ellis] Only gigabytes? Yes! 

[David Spark] Hold tight. Could be 999 gigabytes. [Laughter] 

[Andy Ellis] Still, we’re not talking exabytes or petabytes. I’m okay. 

[David Spark] You’re sure it’s encrypted and you’re pretty sure it’s useless, but you’re not that confident, okay? So, you’re just staying on the, “Pretty sure it’s encrypted, I’m pretty sure it’s not valuable at all.” All right, number one – public disclosure, fearing reputational damage, and the industry questioning you as to how safe they are, that you said it’s encrypted.

Or two – you keep the breach under wraps, hoping that it is never revealed, and you face the risk that it won’t be revealed, nor the encryption will be broken. Which one’s worse? 

[Andy Ellis] Who am I answering as, the CISO or the CEO? 

[David Spark] You’re answering as the CISO here. 

[Andy Ellis] Oh, the second one is absolutely the worst in that case. 

[David Spark] [Laughter] The second one is the absolute worst. 

[Andy Ellis] This is literally we’re trading off a nebulous business risk, but also a very strong professional risk. 

[David Spark] Oh, well, this is coming on the heels of the FCC ruling.

[Andy Ellis] That’s what I’m thinking. I better disclose or I’m going to jail at this point. But seriously, let’s say you have this breach, right? And fine, maybe it’s a system compromise, not quite the data breach because what went was encrypted. Like, you give a good message, and the trick on the message is you don’t say we can guarantee the attackers can’t ever get to it because then you’re totally screwed.

But you say like what was stolen, we believe was encrypted, we had the controls to do it, we don’t believe the adversaries are going to be able to monetize this, but we might be wrong. So, out of an abundance of caution, here’s what we recommend you do, is you treat this data as if it’s stolen if it was your data.
So, go get an identity lock, go rotate passwords, whatever it takes. You control the messaging by choosing to do it early. If you do not, you lose messaging. 

[David Spark] Isn’t that the answer for pretty much every breach? 

[Andy Ellis] Yes.

[David Spark] Even if you’re 100% secure, you still kind of have to say that. 

[Andy Ellis] Absolutely. 

[David Spark] All right. Richard, do you agree or disagree on this one? 

[Richard Ford] Well, Andy can come and be my CISO anytime because I think he shot that one straight up the middle of the lane. I couldn’t agree with him more. Yeah. I mean, what’s worse, right? Having to come out on the back foot of, “Here’s something we didn’t disclose, and things have gone badly wrong,” or just take your licks up front and use it as an opportunity to build some trust, not lose some trust.

[David Spark] So, you see this as a trust-building exercise, not a trust-losing exercise. 

[Richard Ford] I think if you look at the expected value, you’re going to lose an awful lot of trust if it all comes out and you tried to hide it as opposed to being straight. 

[David Spark] But it seems in scenario two that the risk is pretty low the encryption is going to be broken and that there’s any valuable data there. 

[Andy Ellis] But even if the encryption is never broken, how many people know that we chose not to disclose? And at some point, that story comes out, even if it’s just back channel. Because here’s what’ll happen, right? So, Richard works for me, I make this call because he’s very cranky because he got overruled because he thought we should tell people and I said, “No, we’re not going to tell people.” Richard leaves and goes to another organization.

He tells another CISO. He says, “Oh, when I was working for Andy, let me tell you,” right? And that will just go. CISOs talk to one another. And so your reputation… And if you’re the CEO and you overrule the CISO and say, “You can’t tell anyone,” you should expect the CISO is actually going to tell some of their friends, be like, “Hey, just so you know, there was no breach notification, we’re pretty sure your data didn’t get lost, but here’s what I think you should do about it.” And that reputation follows you around.

[David Spark] Very good point. 

[Richard Ford] Secrets take energy. Secrets take a whole bunch of energy out of you. 

[Andy Ellis] Oh, secrets take so much energy. I just started watching the Big Bang Theory. I’d never watched it before. I’m in season one and there’s this whole episode about making up a lie where one of them doesn’t want to go to the performance that the neighbor is doing, and so just makes up a stupid lie, like, “Oh, yeah.

We’re going to a seminar series on some molecular physics thing.” And the roommate’s like, “That’s not a believable lie,” and makes this awful crazy believable lie, but has to hire an actor to come in to be the deranged cousin as, “This is the reason we couldn’t make it.” And you’re just looking here and you’re going, “How much work are you putting in here to cover up because you couldn’t tell the truth?” 

[David Spark] I will say this, by the way. If someone needs me to hold a secret, I am the person. So, quick story. 

[Andy Ellis] Anybody, you have a microphone, don’t ever give a secret to someone with a microphone. 

[David Spark] No, no, no. Well, actually, it’s a time I was on air. So, I was working at a television network, and there was an in-office romance that nobody knew about, and I stumbled onto it, not in a compromising position, but the two people were holding hands.

And they were very, very fearful that I was going to release this information to the rest of the office, and I said, “Don’t worry about it. I won’t say anything.” And I did keep it a secret and nobody knew about it. And later, they got married, had kids.

It all went in the best direction it possibly could. But I told that guy, “If I ever need a reference that I can keep a secret, you’re the guy.” 

[Andy Ellis] Yeah, I’ll have to break your secret so that people will know that you’re my reference. 

[David Spark] Well, the secret’s out. Married and kids, I think we all know at this point. 

[Andy Ellis] Yeah. Oh, I’ve been doing security investigations and discovered, like in email, people making plans for which conference room they were going to have sex in. Wasn’t relevant to my investigation, kept the secret, although did avoid certain conference rooms from that point forward.

Please, enough! No more! 

23:31.952

[David Spark] So, today’s topic is offensive cybersecurity. And this is not cybersecurity that smells and should use deodorant, but this is offensive cybersecurity that one should essentially take the proactive stance. And by the way, another thing we should state because people misuse this term offensive cybersecurity – it does not mean you attack back.

[Andy Ellis] Yeah, it does. 

[David Spark] That is clear. We are all against that. That is for law enforcement, I don’t even want to go down that road too. But I’m going to start with you, Andy, knowing that it is not that it smells, or we’re not attacking back. What have you heard enough about with regards to offensive security, and what would you like to hear a lot more?

[Andy Ellis] So, first of all, let’s just call it red teaming, it’s not offense. Like, you’re the scout team. That’s what I want to hear. If you want to talk about offensive cybersecurity, you’re like the scout team in football, right? These are the people who pretend to be the adversary but are not the adversary.

That’s what I want to hear, that you should recognize that your whole job is the scout team for your actual defenders to do their job. 

[David Spark] I like that, the scout team. All right, we’re going to throw this one to you, Richard. Offensive cybersecurity, I know you guys embrace it over there at Praetorian. Why do you embrace it? Why? And I want you to convince Andy how wrong he is.

[Richard Ford] Sure. I think the way we typically think about red teaming, Andy, is it’s like this capstone. You’ve got all your defenses up and then you do a red team to see how well or how badly you built it. And to me, that’s completely backwards.

I think you start with attacking yourself and then build your defenses based on what you learn by attacking. So, offensive cybersecurity, we don’t call it red teaming for a reason, actually. Red teaming is very tactical. It’s I’m measuring how well I did.

If you do offensive right, it’s strategic. It tells you where you should be building your defenses, it tells you where your walls should be, it tells you where you’re missing things. There’s a lot of nonsense about, “We will find every vuln.” It’s not really important to find every vuln.

Who cares? You can’t patch them anyway. Do you know how many 9+ CVEs were released in 2023 so far by Halloween? 

[Andy Ellis] The answer is none. Although there are numbers that have that score, but you can’t actually have a 9+ until you do the environmental score for the environment it applies to, and so no CVE that’s actually released can have a CVSS score. What it has is the numbers that will go in before you actually get to the environmental impacts.

I’m just going to be pedantic because I can. But I assume it’s some large, stupid number. 

[David Spark] Richard, you say offensive security is not red teaming and it’s more strategic. I want you to drill down more a level because I want to understand that a little bit more. 

[Andy Ellis] I mean, because you’re describing what I think real red teaming is. Just to be very clear, what you’ve just described is what I thought red teaming was. 

[David Spark] Well, same here, but hold it. I want to hear from Richard here. So, I get a red teaming report versus an offensive security report from Praetorian. How does that look different? What is it I’m doing that allows me to behave differently from a Praetorian report versus anybody else’s red team report?

[Richard Ford] Right. So, I think one of the important things for a start is this whole point-in-time assessment isn’t very useful, right? So, point-in-time assessments just tell you how something was as opposed to how it is and how it’s going to be, right?

So, one of the things that we would argue very strongly is doing this continuously is really, really important, right? Not a single sort of, “I took a run at your stuff.” This sort of punctuated snapshots of your attack surface isn’t very useful. 

And let’s face it, Andy, David, if I ran a red team against any organization, do you think we’d get in? Yeah, of course we would. It’s like going up against a black belt and getting kicked in the head. It didn’t really teach me anything except that it hurts when I get kicked in the head.

The idea of continuous – it’s like having a sparring partner who is at whatever level you’re at, maybe just 5% better, and they’re going to help you evolve and get better and better in this continuous process. As opposed to this punctuated, “I bring in the Praetorian boys, they beat me up a little bit, I don’t feel good about it,” and then we do it again in 12 months.

Did you really get better? 

So, one of the things that we even think about is we don’t count vulnerabilities found. I don’t care how many vulns my team finds. What I care about is how many vulnerabilities do we get the customer to fix? That’s the only metric that matters, right?

Coming up with 10,000 vulns – waste of time. 

[David Spark] Hold it. Then, so what is your secret? Because this is a classic complaint we’ve heard, in that a report comes out, they do nothing. Next year they do a report, it’s the same report as last year’s. How do you get customers to fix things?

Do you actually get in there and help them, there’s different guidance? Like, what do you do? 

[Richard Ford] Yeah, so I think it’s all of the above, right? It’s about relationship at the end of the day. It’s hard to have a relationship with somebody you see every 12 months who’s going to kick you in the head. That’s point one. Whereas if I’m talking to you every day on Slack and I can say, “Hey, David, this thing’s really dangerous.

Would you like to see a directory listing of the machine that we can now get in?” And so we work together. You’ve got to get out of this adversarial game, right? So, when you say “red team,” immediately you’re in this zero-sum winner-and-loser adversarial game.

[David Spark] Ah. So, this is maybe more of the purple teaming area.

[Andy Ellis] So, I actually think that what I’m hearing… And which I agree, just to be very clear. I think that Richard and I are mostly arguing over words rather than over concepts, right? Is the semantic is actually not about what the adversary is doing for you, whether it’s simulated or not.

It’s about how you’re engaging with them, right? It’s the difference between adversarially informed remediation, what we would think of as traditional red teaming, like, “Let me scan and find all the open things and give you a list of things to go patch,” and adversarially informed defensive design that says, “Oh.

Here’s what’s going on.” It’s not enough to just patch this system because next month I’ll get back in when there’s a new vulnerability that comes out. The problem is you have a design that is weak, and we have to understand how adversaries would exploit it.

And so this really comes back to the safety engineering, like the Nancy Leveson school of complex system safety, really is, “Here and this is…” And here’s why I don’t like the term “offensive security” – because offensive means something, at least to those of us who are in the old school that we did offensive cybersecurity that meant we won’t [Phonetic 00:30:10] go and attack your computers.

That this is really making sure that your design model has adversarial input rather than just your resilience model. Which most people do, they say, “Well, I have some adversarial input, it’s going to tell me what’s broken,” and that’s an ops problem to react to it rather than saying that’s a developer problem to react to it.

[Richard Ford] That’s exactly right. 

[Andy Ellis] And so what you call it doesn’t actually matter if you’re not using it the right way. 

[Richard Ford] Yes. It’s getting from that reactive mindset, where I poke a hole and you patch it, to a proactive mindset. So, for example, when we’re looking at somebody’s attack surface, if we see something that we don’t like the look of, even if we don’t have a vuln, there’s no CVE, we’ll talk to that customer and say, “Hey, Mr.

Customer or Mrs. Customer, this thing really shouldn’t be out on your attack surface.” And usually we have enough of a relationship to convince that. We also have a team that does VR. If we really can’t convince somebody, we’ll demonstrate why it shouldn’t be out there.

[David Spark] Isn’t a lot of red teaming, blue teaming, purple teaming, whatever it is you’re doing, isn’t a lot of it not just poking holes with these specific classic CVEs, but more of a situation of what just looks like you don’t have the basics set up?

And you can see that through a red teaming exercise. 

[Andy Ellis] Yeah, if you don’t have the basics set up, obviously the red team will tell you. 

[David Spark] Right, right. But the thing is everyone’s in a journey of building out their security program. 

[Andy Ellis] Yeah. 

[David Spark] So, when I say the basics, I mean, I should say the fundamentals at the level that you’re at. 

[Andy Ellis] Right. Look, if I can pop you with Metasploit in 30 minutes, you probably don’t need to be hiring somebody from outside to help you. Like, this is go do some basics. But let’s think about this for a minute, let’s use the analogy, red and blue actually have an origin.

What do we use? Where do those come from? Those come from military exercises. So, let’s take Red Flag at Nellis, which is red flag, for those who are not aware, is basically this opportunity for Air Force pilots to go and fly against a literal red team who are flying actual planes from the other side.

Like expert combat pilots, Americans who’ve trained on these, and you have to fly up against them and practice fighting them. There’s a similar thing that the Army does with tanks. 

[David Spark] We’ve all seen Top Gun, Andy. 

[Andy Ellis] No, no, this is not Top Gun. This is something very different and everybody does it. But that’s an operational task, right? You’re taking an operator and you’re putting the operator up against an adversary to see how the operator functions.

What most people don’t realize what that also feeds into is on the acquisition side. When we build new weapon systems, we take all of the adversary capabilities that we have learned and say, “We need to design a weapon system to counter that.” So, we write what’s called the System Threat Assessment Report, which is sort of your offensive security.

It says, “Here’s what the adversary can do.” We write a mission need statement, “Here’s what we need to do.” We design a set of operational requirements; we design a system to meet those. Sorry, I’m an acquisitions geek, I used to do this for the Air Force a long, long time ago.

And then we go build a system to give to operators. 

And I think the distinction we’re arguing over is, should you be investing your energy in the operational activities or in the engineering activities to build new systems? The answer is both. You can’t just do one. But if you think that you can drive everything from the operations side and somehow magically your systems will become architecturally safer, you’re deluded.

Like, you have to be doing adversarially informed design. 

[Richard Ford] Yeah, I couldn’t put it better myself, right? The models that we use today for defense are just not working very effectively because they’re sort of all over here on the reactive operational side. And what we’re looking to do is help you take that tactic and become a strategy.

Use the attacking yourself as a strategy. And again, Andy, you said it. It’s not that what’s done is so different. It’s your relationship with it and how you take that input to create better fences, better fire suppression systems. That’s the big difference, and we found it to be a game changer for the people that use it.

What’s the best way to handle this? 

34:23.865

[David Spark] Data is the lifeblood of any organization. Yeah, we’ve been hearing it’s the new oil for a decade now, if not longer. We get it, data is great, wonderful. So, why does it feel we’re drowning in it, for that matter? And I’m not just talking about just what comes through your SIEM.

What I’m talking about here is I’m looking at this ESET’s Q2-Q3 Activity Report, and there’s a ton of information that all seems really important. There’s lists of APT activity, there’s a ton of critical CVEs, and important context connected to this, and this is just one report.

CISOs are also seeing new threat intelligence reports from internal and external sources. It’s part of the unending cavalcade of data that a CISO seems to see every day. So, what I’m getting at here is how do you make sense of all this wonderful input that’s coming in?

How do you do this, and how do you actually turn that into something actionable? And I’m going to start with you, Richard, on this. 

[Richard Ford] So, let me take the counterpoint, right? I mean, I think we’ve probably all read Gladwell’s Blink – more data isn’t more. And so for a lot of this data, I would come back to you with the question of, does it change what you actually need to do?

Because if it doesn’t, why are you spending time on it? So often, this data comes over us in waves, and we’re still not doing the basics. I hate to say it, but as somebody who’s broken into hundreds and hundreds of networks, it’s very rare I get somebody with something totally esoteric.

I get them because they’re not doing the basics. And all that clever reading and data analysis they did didn’t stop them from leaving a single factor auth setting out there. Get the basics right and stop worrying about every sort of bit of data that’s floating out there.

[David Spark] The best line I heard quoted, and I wish I could remember the name of the person, is the thing that you were worried about before you went to the Black Hat Conference, guess what? That’s going to be the same thing you’re going to be worried about after the Black Hat Conference.

Andy, how do you make sense? Is it like you’re already there and all this stuff is just validating what you think? Or can we actually use this stuff to make better decisions? 

[Andy Ellis] Yeah, 95% of it is just validating what we already know, right? And maybe it’s giving you a better argument, giving you deeper insight, somebody else has gone and done the hard work of really assessing it. And I’m a big fan of, like, something new will pop out, and I’m just like, “Oh, I should learn about that, but I’m going to wait 24 hours.” I’m going to let seven other CISOs write up blog posts on it, and I’ll get their takes before I dig in deep, right?

And that sometimes is really helpful, but you have to be open for that like 5% of the time where you’re like, “Oh, this is truly something novel.” 

Now, most of the time when something is novel, a lot of other people are going to notice it’s novel, and you’ll see the buzz, and that will point you at what’s novel. But sometimes you’re going to stumble and be like, “Oh, most people won’t think this is interesting, but to my environment, this is fascinating.” And so you kind of have to pay attention to that, but it means you have to have a very deep, complex understanding of your environment.

Because it’s how does this affect you versus how does this affect some random generic enterprise? 

And in fact, 90% of the time that you have to talk to your board, that’s actually what your conversation is, is you’re saying, in different words, you’re saying, “Look, you heard about this thing, somebody told you it was bad, and for 90% of the people out there, it is bad, but I’m in the 10%.

It’s not, and here’s why.” Or worse, you don’t think it’s bad and everybody told you it would be fine, but for us, oh, my God, this is nuclear, and we’ve got to go deal with it. 

[David Spark] Very, very good point. 

Closing 

38:12.244

[David Spark] Excellent points, both of you, and I want to bring this wonderful show to a close. Thank you very much, Richard, for being our guest today. I’m going to let you have the very last word here. I want to thank your company, Praetorian. Again, I want to remind everybody, if they want better strategy for their offensive security, build a better security program, please check out what they’re doing at praetorian.com, and Richard will give more information in just a second.

Again, they are praetorian.com. Andy, any last words you have for today’s show? 

[Andy Ellis] I’m really excited that we’ve made it to December. Stay warm.

[David Spark] It’s warm where I live in Southern California. 

[Andy Ellis] It’s always warm where you live. You don’t have weather, you have climate. 

[David Spark] I’m happy as a clam. Trust me, everywhere I’ve lived – Chicago, St. Louis, Boston – we had it all, I’m happy here. Richard, any last words how you’d like people to get in touch with you? Anything you want to mention about Praetorian? Let us know.

[Richard Ford] Yeah, absolutely. Thanks, David. I mean, just go to the website, and then engage with us, right? I mean, I’m very easy to find. If you think that I’m wrong, I’d love to hear from people. If you think I’m right, I’d love to help you make your security stronger and better and cheaper, right?

It’s all about actionability. I’ll leave you with this – never forget that no matter who you’re spending your money with or what you’re doing, we’re on the side of good. We’re on the right side of this fight, right? And so what we do stops a lot of bad in the world.

Let’s go out and do that. However you do it, do it. 

[David Spark] Awesome. I appreciate it. Thank you very much, Richard. Thank you very much, Andy. Thank you to Praetorian as well and thank you to our audience. We greatly appreciate your contributions, give me some more “What’s Worse?” scenarios, and thank you for listening to the CISO Series Podcast.

[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows – Super Cyber Friday, our virtual meetup, and Cybersecurity Headlines Week in Review.

This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com.

Thank you for listening to the CISO Series Podcast.