In cybersecurity, we know what controls work well. Think of MFA. But beyond the basics, it’s often hard to tell what is actually effective. If we don’t know what’s working, how do we decide what tools to invest in?
This week’s episode is hosted by me, David Spark, producer of CISO Series and Andy Ellis (@csoandy), principal of Duha. Joining us is Sara Madden, CISO, Convera.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, ThreatLocker
open the door to ransomware or data loss. With ThreatLocker, unauthorized apps,
scripts, and devices are blocked before they can ever run. See how ThreatLocker can
help you gain more control over your environment.
AI-infused security operations tip of the week – Anvilogic
Jump to the full tip here.
To learn more about saving costs and optimizing analysts’ capacity with a hybrid SIEM and data lake, go to anvilogic.com.
Full Transcript
Intro
0:00.000
[Voiceover] Ten-second security tip. Go!
[Sara Madden] Our world is driven by urgency, and this causes mistakes that create security incidents all the time. So, my tip is it’s never too urgent to not fully think through. Calm down. Think before you act.
[Voiceover] It’s time to begin the CISO Series Podcast.
[David Spark] Welcome to the CISO Series Podcast. My name is David Spark, I am the producer of the CISO Series. Joining me as my co-host, because he’s here all the time, you can’t get rid of him. We’ve tried. He’s like a cockroach. It’s Andy Ellis, [Laughter] principal and legendary CISO of Duha.
I’ve now called you a legendary CISO and a cockroach.
[Andy Ellis] Yeah. Now I’m a cockroach too.
[Laughter]
[Andy Ellis] I’ve sometimes been likened to a plague you can’t get rid of, but a cockroach is a new one, David.
[David Spark] There you go. Well, cockroaches are tough to get rid of.
[Andy Ellis] Yep.
[David Spark] We’re available at CISOseries.com. We have lots of wonderful programming over there. Why don’t you go check it out? Our sponsor for today’s episode, if you’ve been listening to this programming, you’ve heard them before because they’re a phenomenal sponsor of ours.
It’s ThreatLocker, Zero Trust Protection Platform. You will be interested in what we have to say about them later in the show. But first, Andy, we were talking about earthquakes off the air, and you keep saying you’ve missed them. So, have you never felt or experienced an earthquake?
[Andy Ellis] Oh, no, no. I have definitely felt and experienced earthquakes. I grew up in LA. I was a senior in high school in ’89 when we had all of the pre-shocks, but I headed off to college right before the Loma Prieta earthquake.
[David Spark] Okay.
[Andy Ellis] So, I was at college during that one. I came back for a while and then headed back out east again, just in time for the Northridge earthquake. I was born just after the San Francisco earthquake. So, none of the big ones have I actually been in California for, but I certainly have woken up many times to an earthquake, but never anything that was big.
[David Spark] Now, what was your first reaction to an earthquake? Because mine wasn’t, “That was an earthquake.” I didn’t know what was happening the first time.
[Andy Ellis] I mean, when you grow up in California, like they’re just a fact of life.
[David Spark] Mm-hmm.
[Andy Ellis] At some point, you’re just like, “Oh, some things shook, must’ve been an earthquake.” You learn, I don’t know if you’ve ever assembled IKEA furniture, and you had little tabs to nail things to the wall.
[David Spark] Yes.
[Andy Ellis] And everybody here in New England is like, “Why would I do this other than the fact that my floors aren’t level?” Because it will tip over in an earthquake. You’re not worried about like pulling it over. You’re worried about the whole thing swaying and dropping everything onto you.
[David Spark] My first reaction to an earthquake was, “What are the neighbors doing?”
[Andy Ellis] [Laughter]
[David Spark] Because I didn’t think it was an earthquake. I thought there was just some loud party going next door and they shook my room. And it took me a little while for me to do the basic math to go, “No, that was an earthquake.”
[Andy Ellis] Yeah. That’s an earthquake. Especially the difference between like there’s the S waves and the P waves, and after a while you start to tell a little bit of difference between them. And then there was the driving into school every day and half the time I missed the earthquake.
I’d get to school and people were like, “Did you feel the earthquake?” I’m like, “I have shock absorbers on my car, so no, I didn’t.”
[David Spark] We have another Californian who is our guest today. She has felt a few earthquakes in her time.
[Sara Madden] I have. I actually have a SoCal tip for how to tell how big an earthquake is. You look at your pool if you have one and you can judge it by the waves.
[Andy Ellis] There you go.
[Sara Madden] You’re also giving me SoCal PTSD talking about cockroaches because I live in LA and we’ve got them everywhere. But hi, gentlemen. [Laughter] Nice to talk to you today.
[David Spark] That, by the way, everyone, is the voice of the CISO of Convera, none other than Sara Madden. Sara, thank you so much for joining us today.
[Sara Madden] Yeah, thanks guys. Great to talk to you.
Where does a CISO begin?
3:47.070
[David Spark] For our very first segment, let’s take a deep breath and let’s think about our dreams for building a greenfield IT infrastructure. Now these dream makers recently popped up on the cybersecurity subreddit and the responses they had for doing this range from idealistic, like cloud first with zero trust, everything containerized, and hardware keys mandatory.
And one commenter just wanted accurate configuration management database, CMDB, where every team actually logs their infrastructure instead of building fiefdoms. Now several admitted the real challenge isn’t the tech stack, but that, “If you’re reliant on the end user doing the right thing, it’s already too late.” Now one of my favorite takes was, “As a security expert, it will never work,” because security people building IT infrastructure ignore business needs.
Andy, I’m going to start with you. So, when you’re designing a security architecture – again, thinking greenfield, you got a clean playing field – are you optimizing for the world you want or the one that currently exists?
[Andy Ellis] So, I think the answer is both, and I love the comment in there about if you’re reliant on the end user doing the right thing, it’s already too late, and nothing captures the blind spot of security professionals more than that sentence.
[David Spark] Well, it’s the feeling that it is the user’s fault.
[Andy Ellis] Right. It’s the user’s fault. You need them to do the right thing, and if they don’t do that, they’re doing the wrong thing is the implication. When the reality is you have to design for humans. Humans actually are the people who provide value in your business.
Humans operate with some pretty well-understood rules. Like you can model how most humans will interact with things. And if you say, “Hey, I’m going to put a security control in your way and you have to take an extra 30 seconds every time you do a task,” they’re going to at some point be like, “This is stupid.
Why am I wasting my time with this? Let me go around the task.” That’s not the human doing the wrong thing. That’s the human doing the right thing from the business perspective. So, always design with humans in mind, human-centric design that deals with the infrastructure you have today, but is aimed at the future.
I love zero trust, and I know it’s a big buzzword, but for me, the core of zero trust is actually taking a laptop and associating it with a human and authenticating the pair of them.
[David Spark] Mm-hmm.
[Andy Ellis] And everybody gets this wrong. They try to authenticate the human and build a barrier between them and the laptop. If you say, “Look, the laptop is yours, we authenticate it with you attached to it,” all of a sudden, you can do things that are very user friendly.
Like the user logs in once, proves their identity once, and then the laptop carries on for them, and it’s not, oh, I have to do MFA 17 times a day. No, I proved to my laptop that I’m me, and now I and my laptop can go get stuff done, seamless authentication continuously.
[David Spark] That is a very good example right there. Sara, I’m going to put you to the test. I do like Andy’s setup there. You got a greenfield, you’re building an IT environment. What do you want that is friendly to this IT environment that you’re building, but also thinking about the humans?
[Sara Madden] Yes. And I have some good perspectives here because I was able to build the ground field security system at Convera, which is the reason I came here. I do want to address something, Andy, that you said real quick, though. Trust is not a control.
This is something I say all the time in my role. Our job is to design security controls that limit user impact to create an actual massive security incident, right? So, I have to have enough controls in place where users, they do something too quick, something too urgent, something stupid, it doesn’t cause an incident, right?
[Andy Ellis] Absolutely.
[Sara Madden] At the end of the day, they’re trying to do the right thing for the business. They’re trying to be productive. And with this growth of a bunch of SaaS tools available, it’s very easy for users to go rogue and use their corporate credit card, come up with some new SaaS solution for something, and they’re doing it with best intent in mind.
So, control is the name of the game and it’s productivity versus control, right? So, bringing me back to the greenfield scenario, one of the things that I think is the biggest bang for the buck from a greenfield perspective is we don’t have toolset bloat.
We have clarity on what we need to manage. We were able to deploy in one cloud, so we don’t have multi-cloud bloat. We don’t have toolset bloat. We know what things go where. And we had the opportunity to say, “We are going to use this control for this thing.
We’re going to use this tool for this type of business use case.” And everybody is clear on what we use for what.
And when you come into a scenario where a user doesn’t have what they need, they go through this intake process and they say, “Hey, I want to do this thing. What’s the tool for this?” And you centralize everything, right? One of the challenges you always have in other environments is it’s just a business that’s grown over 5, 10, 15, 20 years, and you’ve got all this toolset bloat, and it’s very hard to get out of, “I’m used to using this thing for this thing, and now I have to go do this?
We didn’t have that problem,” right?
[David Spark] That actually minimizes problems dramatically.
[Sara Madden] Yep.
[David Spark] The funneling of the requests that they don’t go outside.
[Andy Ellis] Yes.
[Sara Madden] Yeah.
[Andy Ellis] Well, especially the not carrying forward the legacy.
[Sara Madden] Mm-hmm.
[Andy Ellis] One of the best things I ever did at Akamai, and it will sound very, very silly because I fought so hard during M&As, said whenever we acquire a company, we’re throwing out their entire IT stack, everything. Day one, they get brand new laptops that are pre-provisioned as if they’re new employees here.
And there’s a lifespan for their old infrastructure. We’re throwing it all away. You have to choose what to migrate. There’s no presumption that we keep anything, and every CIO forward from that, when they came in the door, they’re like, “Wait, I don’t have stacks of things I have to support from the 30 acquisitions we did?
This is amazing.”
[Sara Madden] Wow. So, you just ripped the Band-Aid from the jump instead of trying to unravel things over time.
[Andy Ellis] Right. Because you’ll never unravel.
[Sara Madden] Yeah. So, we had to do a balance, right? So, we acquired a division of Western Union, and we lifted and shifted their assets into a new data center environment that we created, and that was the greenfield that we got to create. But anytime you buy old legacy systems, and you put it into a brand new network, it’s like you’re lifting a Pinto into a Ferrari engine, right?
Or Pinto parts into a Ferrari engine. And so, there’s a part of it that you have to unravel, but I think one of the benefits of modern tooling in the cloud is you have levels of visibility that you never had before. So, even when you bring in old stuff, the amount of control you have and the amount of visibility you have gives me comfort that we can manage even old clunky stuff in a new shiny system.
So, I love your idea from an acquisition standpoint where you just say, “Hey, you are now coming into this world. This is the new system. Ways of working have changed X, Y, and Z. Like, that’s a really easy way to integrate the user experience, and that’s a different approach that a CISO can take versus you have to bring the systems in no matter what, right?
[Andy Ellis] Right. You got to bring the production systems over, but you don’t have to bring the IT.
[Sara Madden] Delineating the difference between the user experience and the IT. Yeah.
[Andy Ellis] Yeah.
[Sara Madden] Yeah. Versus the systems that you have to manage from a production standpoint. Yeah. It’s a good way to look at it. I like it. I might steal that for the next acquisition.
[Andy Ellis] Go for it.
[Sara Madden] [Laughter] Thanks. Isn’t this what we’re here for? Sharing tips for making our lives easier and being better at our jobs, right?
[Andy Ellis] Yeah.
How is AI going to solve this problem?
10:47.381
[David Spark] What is the new GRC archetype? All GRC players are AI-infusing their products, which calls out for staffers to engineer AI-enabled governance rather than just monitor policies, Proposed Nikhil Sarnot of Accenture. Assuming we get to the point where we can automate routine compliance work, humans will shift into areas requiring ethical reasoning, foresight, and creating escalation protocols.
Is that where GRC is shifting? And if so, how does this change how we hire for GRC roles like this sort of AI-enabled, or we were talking about the bionic hacker, so it could be the bionic GRC person. And if you’re already working in the field, how do you stay ahead of the sea change?
So, if I’m in GRC, how do I become this bionic GRC person, Sara?
[Sara Madden] Mmm. So, we’ve been on a path of automation for quite a few years, right? And that’s the impetus for AI, right? So, I think every security person’s mindset should be in, if it’s not already, how do I automate this? And then eventually AI’ is going to pick that up, right?
We’re at a stage now from an efficiency and scale perspective where we’re looking at GRC engineering, where we’re looking at policy enforcement and config and code. AWS re:Inforce in Philly a couple months ago had tons of workshops on DRC engineering that several of my team members went to and got really, really excited about.
We are shifting the role of somebody on my team that used to be in a primarily audit role, in a security audit role where they’re doing all the audit and compliance and testing controls, they’re shifting their day-to-day work into the UI of AWS and the native toolsets that exist there for automating policy and config and starting to actually click automation buttons.
Like if this is our set of controls, I can automate it in AWS, and I can prevent somebody from ever violating this control. Instead of me having automated audit tool suites that tell me somebody violated a change control policy or an access policy, we’re just preventing it in the config.
I am really excited about this evolution in the way that we work. So is my team members. They’re looking at it from a sense of excitement in their day-to-day jobs versus kind of doing the same old audit nuance every single day. It is the future of how we will manage compliance in an automated way, and it gets us out of this mindset of writing people up or slapping people on the wrist for doing silly things and causing issues, and it’s just all automated in code.
We’re getting out of the firefight. This is the part of security that I’m actually really excited about right now. Beyond broader capabilities of AI, I’m really excited about GRC engineering now.
[David Spark] All right. Do you have the same excitement? Looks like you’re totally on target here, Sara. What about you, Andy?
[Andy Ellis] So, I love Sara’s approach, and I’ve got a slightly different one, but I’m more and more convinced that people should just listen to what Sara said and ignore me for the moment.
[David Spark] By the way, oh God, that is a soundbite. I am taking it. [Laughter]
[Andy Ellis] There you go, right there. Like, she nailed it. Because the challenge is, like, I actually hate the term GRC because let’s be honest, 90% of the profession is the C. It’s just compliance. And governance and risk is what the CISO does. And oftentimes we have people whose job is enforce descriptive compliance that we said we had rules.
They’re not actually controls. We say, “Oh, we don’t do X.” “No, of course you do X.” You have compliance people who hunt for people who did X, and then some security ops person comes in and tells you to undo it. And we need to move to a world that is control systems-based, where when you say you have a control, what that actually means is it is impossible for it to happen in a different way.
That’s literally what control means. And I think that’s going to be the evolution that AI, and really, better systems automation is going to give us. And we say, “Oh, look, we bought a bunch of tools. We actually have to verify implementation success that we turn these tools on, we set them up in a controlling fashion, and that’s what our compliance becomes.”
Now, if today you work in the GRC field, hopefully you’re one of the 10%, if you’re listening to this, it isn’t just doing the compliance validation, but you already know what you need to be doing, which is understanding how do we implement control systems?
How do I identify the risks that are being created by bad controls, and the risks that are being created by not having controls? That’s an architectural function. That’s not a traditional compliance analyst function. That’s where the humans are going to need to be.
They’re not going to be the ones who are validating that we have all of our paperwork, because that’s automatable. Humans need to be the ones saying, “Do we actually have the right story? Do these controls fit what our customers need us to do and how our business needs to operate and what are the gaps that we’re going to go forward with?”
[Sara Madden] Mm-hmm. Is it designed correctly and can it automate the enforcement?
[Andy Ellis] Yep.
[Sara Madden] Back to trust is not a control. People don’t want to fail controls, right? They do it because of mistakes or misunderstandings or just lack of knowledge and awareness. And the more you can automate them out of making mistakes, the happier they are, right?
[Andy Ellis] Right. If I don’t have to type numbers in, I’m happy.
[Sara Madden] Yeah, totally. Totally. And I think outside of the compliance part of the governance, risk, and compliance, the risk part is really important to me too because we’ve been playing whack-a-mole with vulnerability management for our entire careers, and that’s the biggest opportunity with AI is automating vulnerability management.
That part I’m equally excited about, if not more probably, right?
[Andy Ellis] Yep.
[David Spark] Automate, automate, automate.
[Sara Madden] Mm-hmm.
Sponsor – Threatlocker
16:13.813
Who’s our sponsor this week? It is ThreatLocker, I told you about this, and we have something new you may not have heard before. Even the most reliable employees make mistakes. We were just talking about this. An unauthorized USB device or accidental click can expose sensitive data and create serious risk.
Traditional user-based access controls rely on trust, and trust alone isn’t security. You remember what Andy was talking about, securing the device and securing the person, like the two together. So, ThreatLocker takes a different approach. By enforcing program-based policies, it ensures only approved applications can access, read, or copy data.
Sensitive files stay locked down while approved software continues to run without disruption. And when exceptions are necessary, it does happen, administrators can jump in, and they can approve them in seconds, keeping productivity high without sacrificing protection.
Also, with ThreatLocker, every action is logged in a detailed audit to capture the exact user, file, application, and device serial number. Ah, we’re getting into compliance issues and watching behavior. So, this is all zero trust action in precise, enforceable, and it’s simple to manage.
Discover how ThreatLocker can help you gain more control over your environment. Go to their website. It’s ThreatLocker.com. And I’m going to ask you a small favor. When you go to ThreatLocker.com, do this. Just add a /CISO. That lets them know that you heard about them through us, the CISO Series.
It’s a small thing. They just want to know that people are coming because they heard about them from us. ThreatLocker.com/CISO.
It’s time to play “What’s Worse?”
17:56.004
[David Spark] Sara, are you familiar with this game?
[Sara Madden] Yep.
[David Spark] All right.
[Sara Madden] I’m ready.
[David Spark] I’m going to set you up. This is pretty darn goofy. All right? This comes from Howard Holton, who is now the CEO over at GigaOm, and he’s a good friend of the show. And here we go. I make Andy answer first. You can agree or disagree. Andy, this is very different than what you’ve had before.
[Andy Ellis] Okay.
[David Spark] You’re outsourcing your security awareness training to a random Russian chatbot. That’s option one.
[Andy Ellis] Okay.
[David Spark] Pretty bad. Or you hire a group of clowns who jump out at people every time they click a link in their email. Now that’s any link, phishing link or anything. Which one is worse?
[Andy Ellis] Wow. Okay. This one definitely wins so far for being the silliest.
[David Spark] Yes.
[Andy Ellis] And honestly, I don’t even care about my answer. It’s so silly.
[David Spark] It is pretty silly. These are both ludicrous environments.
[Andy Ellis] So, I think I’m going to go with the clowns is worse, and that is strictly because it is so disruptive and disorienting in the workplace that I would rather have somebody catch me having used a Russian chatbot than have clown…the running around the environment and it’s my fault.
[David Spark] But the Russian chatbot is doing the, quote, “training.” The Russian chatbot is probably “training you,” as I put it in air quotes, to do the wrong thing.
[Andy Ellis] Oh, sure. But you’ve heard my opinions about security awareness training that’s outsourced anyway. Like 99% of it is pretty awful. There’s a couple of decent vendors, but primarily most of the security awareness training is pretty bad. So, like fine, I will take that badness over I have clowns jumping out at my CEO.
[David Spark] By the way, hold it. I think there’s a net benefit here. I want to get Sara’s answer and then I’m going to get your take on something else. All right. So, Andy thinks the worst of these two scenarios is the clowns jumping out. Sara, what do you think?
[Sara Madden] I definitely agree with Andy. I think clowns jumping out is worse because you didn’t say anytime they clicked on a malicious link, you said any link.
[David Spark] Any link, yes. Exactly.
[Sara Madden] And so, now you’re going to have users in a state of panic. They’re going to make mistakes constantly, constantly worried, they’re going to have PTSD. Whereas I am going to have appropriate defenses, so even if the Russian chatbot tells my employees to do the wrong thing, I have controls in place that I mitigate bad ideas and bad logic and psychological manipulation.
Because that’s the spirit of phishing in general, right?
[David Spark] Okay. I want to just say all of these are good arguments but let me throw out this, couple things.
[Andy Ellis] I will point out that Sara agreed with me. So, just for keeping score.
[David Spark] Sara agreed with you.
[Sara Madden] You’re right.
[David Spark] But I’m going to throw out this, that may change your opinion here. If you truly have the clowns jumping out, scaring the crap out of everybody in your office, and everyone’s having a panic about it, let’s just think about how incredibly viral these videos would be.
If you had video of this, of the clowns jumping out, people would go, “Oh, my God!” like that. This would do an amazing branding of, “Look at what we put our employees through and they’re still working for you.” I think this would be an amazing viral video moment that would put the business on the map and be huge for the business.
Andy, your thoughts.
[Andy Ellis] So, do you remember the company that went viral when their security awareness training sent out links that employees had been laid off and they had to click here? That’s in the same category.
[David Spark] I do remember this, I can’t remember who… No, it’s not! It is not because this is visually very amusing.
[Andy Ellis] Oh, [Sinister laughter] yes, it is.
[David Spark] That was just cruel.
[Andy Ellis] But how’s this? Let me… I’ll put Sara on the spot. Sara, let’s just imagine for a moment that you authorized this, and tomorrow a fleet of clowns was in the Convera offices or apparent tents in people’s houses, jumping out at folks, do you believe you will be employed at the end of the day?
[Sara Madden] Absolutely not.
[Andy Ellis] There we go.
[Laughter]
[Andy Ellis] I think the drive for viral content…
[David Spark] No, hold on! No, wait, wait! She was authorized to do this, obviously from the business, so of course, she’s going to stay employed.
[Andy Ellis] Doesn’t matter. Even if you went and told the CEO you were going to do this, and the CEO said, “This sounds like a great idea. Go do it.” You do it. Everybody complaints to HR, it goes viral. Do you still think you’re going to be employed by the end of the day?
Even though the CEO had said, “Go for it”?
[Sara Madden] I…
[David Spark] I’m telling you, the employees are giggling watching the videos of their colleagues jumping out of their seat because they’ve scared the crap by a clown.
[Sara Madden] Would we all laugh at it? Yes. Is it a useful thing to do? Absolutely not. I argue with regulators all the time that I don’t do simulated phishing tests for a reason because the data doesn’t prove that it’s effective. For a lot of different reasons, right?
[Andy Ellis] Mm-hmm.
[Sara Madden] There’s so many things that factor into whether or not somebody is susceptible to a spear phishing attack or a general phishing attack, and even security professionals fail them all the time, right? Like, there’s so many different things you could do – the age of the employee, like…
It doesn’t work. It’s better to do customized training.
[Andy Ellis] Yep.
[Sara Madden] Do not do out-of-box security awareness training. We don’t buy that from anybody else. We make it internally and we consistently…
[David Spark] So, if I’m a Russian chatbot, I should not be approaching you?
[Laughter]
[Sara Madden] Try if you want to.
[David Spark] All right. I try to argue that this could be a huge boon for the business, with the viral videos.
[Sara Madden] It would be hilarious. It would be hilarious.
[David Spark] It would be hilarious. It could put the business on the map. This would be a great way that security could be an enabler for the business.
[Andy Ellis] Yeah. [Laughter]
[Sara Madden] I mean, we’ve used embarrassing tactics for different things in the past, too. Remember when people used to leave their keys unlocked, and you’d go change their desktop background to like getting Hasselhoffed or something?
[Andy Ellis] Yeah. I have gone a little far in that one. Like I’ve got stories from 30 years ago of sending a breakup note from somebody to their…
[Sara Madden] Oh, yeah.
[Andy Ellis] …significant other who was trying to come up with a reason to break up with them and was like, “Oh, thank you.”
[Sara Madden] So, if we’ve evolved in our jokes and our enjoyment, I could get behind this being one of them.
[Andy Ellis] Right.
[David Spark] Or cattle prods, clouds.
[Andy Ellis] There are things that are funny in small groups that do not scale organizationally.
[Sara Madden] Yeah, yes.
[David Spark] That is a good point.
[Sara Madden] Yeah. It’s not going to be effective.
[Andy Ellis] Yeah.
Is this benefitting the company or just making my life easier?
24:08.674
[David Spark] “These AI engineers find the wildest of bugs and they just keep finding them after every run.” That’s security engineer Joshua Rogers describing AI-native security scanners that recently found hundreds of real vulnerabilities in critical open-source software that didn’t show in traditional scanners.
Rather than pattern matching, these tools can tease out business logic and spot mismatches between developer intent and code. I’ve previously said automated tools lack human creativity and fall into familiar patterns, but these tools might make me rethink that.
I will ask you, Sara, what does a red team do that can’t be automated? What do you think?
[Sara Madden] I still think there’s so much of the business logic that a red team has that can’t be automated yet, but I don’t think we’re far off.
[David Spark] Can you give me an idea of what can’t be automated?
[Sara Madden] I would want to run simulations here. We’re not using it yet.
[David Spark] Mm-hmm.
[Sara Madden] I think we need to test this out a little bit more and see where we have gaps, and I think only trying it will tell us, and we’re not there yet. I think I’m very excited about the evolution here, and there’s a lot of buzz in the industry and excitement around it.
And we are going to start using different AI models in the testing that we do, in the tabletops that we do. I think there’s a potential for replacing some of our pentesting with this in the future, but we haven’t tried it yet. I want to, I’m interested in it.
But I suspect we’re going to end up with kind of awareness gaps from a business logic perspective that it might not see, but that’s assuming we don’t also feed it code bases to figure these things out.
[David Spark] So, we’re kind of going to wait and see because I know I met with a bunch of automated pentesting companies at Black Hat. They would like us to believe that you just need their solution. Andy, do you think that there is something that just, even with how good AI is getting, that these automated solutions will never be able to really fully understand business logic, that we’ll also need the human pentesters?
[Andy Ellis] So, absolutely there is things that an AI will never be able to do, but mostly, people don’t pay for pentesters to do that. They might say that’s what they’re paying for, but honestly, if you took every pentesting contract on the market today, I’d be willing to bet AI can do 95% of that.
Because honestly, most people aren’t even having pentesters read code and do static source analysis, which is what this blog post is about. Like, most people are hiring pentesters to run a vuln scanner, figure out what they could break into, break in, now look to see where they could go next.
This is all automatable. This doesn’t require complex AI. This is literally just iterative automation.
On the AI side, I’m very excited about the source code analysis capabilities to be able to look at code and say, “Look. Here’s how it’s being called. Here’s where the gaps are.” I’m looking forward to MCP almost being inverted and people saying, “Let’s take an MCP-based analysis of an API, and say what else could it do that isn’t advertised?” But that represents a risk.
That’s the things that humans sometimes have a hard time finding unless they’re really good, and that AI is more likely to find for you. It’s like I have bloated code that has a thousand capabilities, but I’m only calling 20 of them. It means 980 of them are potential vulnerabilities that you need to understand.
This is never going to be called, why do you have it available there? So, one day it can be the reason why you’re the headlines of The New York Times.
[Sara Madden] I think the near-term opportunity is just using it internally for vulnerability management within your software development lifecycle, SDKs, refactoring old code, vulnerabilities, things like that. Like we’re going that route. And then externally, red teaming, I think will be next.
Right? I can get more out of some out-of-box pentest that you buy just from those third-party sites that do your security scanning. [Laughter] Like I could get more out of that than paying a pentester, right?
[Andy Ellis] Right.
[Sara Madden] And we do both, right? We do the pentest that we have to do for compliance that produce reports, and then we pay real people that are really good at their jobs to find things that we can’t, right? And that’s the part that I think would be hard to replace in the next year or so with AI.
[Andy Ellis] I actually think there’s a business model here, which is just external washing your internal pentesters because actually most of the value of having an outside pentester is you get to say, “Oh, an outside pentester found this; therefore, it’s real.”
[Sara Madden] Mm-hmm.
[Andy Ellis] Even if your team knew about it. So, you can almost like take your own internal pentest reports, like upload them.
[David Spark] Oh, by the way, that didn’t start with external pentesting. Just any external consultant, their value is greater than anybody telling you inside.
[Andy Ellis] Right. But imagine a world where you have your own AI-based tool, you get all the results, and then you have it washed so it looks like it came from an outside company.
[Sara Madden] [Laughter]
[Andy Ellis] Show that to your management, like, “Here’s all of our exposures found by a third party. They can see them.”
[Sara Madden] A crisis-tunity.
[Andy Ellis] There you go.
[David Spark] A crisis-tunity, I like that.
[Andy Ellis] Yep.
[Sara Madden] Sometimes you have to create them.
This week’s AI-infused security operations tip is sponsored by Anvilogic
29:22.863
[David Spark] AI can’t defend what it doesn’t understand. Too often, SOCs deploy machine-learning models that have been trained on generic datasets, which means it has been tuned for someone else’s risk environment, or even no one’s risk environment.
The real power of AI defense is realized when you feed your organization’s unique context into the system, from business-critical assets to common adversary tactics, data flows, and compliance priorities. This contextual knowledge teaches the model what your normal really looks like in your world and helps it recognize those faint signals that indicate when something’s off, from a user logging in from an unexpected region, or a workload suddenly communicating with a new cloud service.
These can sometimes be subtle signs that might otherwise get buried in the noise. You could think of it as like training a guard dog. A well-trained dog doesn’t react to every sound or scent. It learns and then knows what belongs and what doesn’t. Having AI as part of your defense means letting it understand your specific landscape, which means every detection, correlation, and risk score becomes sharper, faster, and more meaningful.
Rather than having more AI, think about having your AI.
[Voiceover] To learn more about saving costs and optimizing analyst capacity with a hybrid SIEM and data lake, go to anvilogic.com.
What works? What’s not working?
31:07.828
[David Spark] “It seems it’s really hard to prove that something doesn’t work statistically. Definitely sounds like cybersecurity.” Now that’s Jeremiah Grossman, who’s the CEO over at Root Evidence, after talking to cyber insurance folks. While they can point to a handful of controls that measurably work, like MFA and EDR, they struggle to say what doesn’t work at all.
Are we just terrible at measuring security effectiveness? And if the insurance companies who have every financial incentive to understand what can prevent a cyber attack and what can’t, they can’t tell what’s useful and what’s worthless, what does that say about how we’re making security investment decisions, Andy?
[Andy Ellis] I love the question because this is, “We really don’t understand what we’re doing.” We don’t ask the right questions.
[David Spark] So, people should stop hiring cybersecurity professionals because they don’t have a clue, right?
[Andy Ellis] That is one outcome one could take from my statement. I don’t know if it’s the right one.
[David Spark] Okay. [Laughter]
[Andy Ellis] But here’s the challenge. How many security professionals look at all their controls and say, what would be different if I didn’t have this control? Like third-party risk management is one of my favorite ones. What changed in your company in the last year as a result of your TPRM program?
Are there vendors that the business was going to buy from that now they didn’t because you identified a problem? Are there vendors that actually made substantial changes to their security roadmap because you identified a problem? If you can’t identify a place in which TPRM helped you, then functionally TPRM is completely and utterly ineffective, except you have to check the box that you have TPRM because your insurance carrier will get cranky if you don’t.
Like, this is one of those funny ones is we do things to satisfy our insurance carriers and our regulators, that they themselves can’t point at how those are being effective, and that’s an interesting challenge. But I think the core reason is insurance is actually, and I’m going to get hate mail for this one, it’s a real simple problem because it mostly deals with simple systems.
Like, building houses is functionally the same technology for the last 3,000 years. We stack boxes on top of one another.
[David Spark] Hold on, so you’re comparing cyber insurance to home insurance?
[Andy Ellis] Yes! Think about, all the insurance markets that work relatively well are all relatively simple systems.
[David Spark] I always argue that if you looked at an actuarial table of car insurance incidents and home insurance, you would see patterns. Do you think you still see the same patterns in cyber?
[Andy Ellis] No, you don’t see the same patterns in cyber. Because when we look at homes and automobiles, we look very carefully at things like what were the building materials? Where were the houses built? Like, there’s a lot of norms that don’t have huge variation in them.
[David Spark] Right, right. But that’s what I’m saying. I think cyber’s kind of all over the map, isn’t it?
[Andy Ellis] Because cyber isn’t replicable.
[David Spark] Right. Don’t you think this is a more difficult problem?!
[Andy Ellis] It is a very difficult problem. That’s why we don’t have data is because we don’t have this large corpus of identical entities doing the same thing every day. Everybody does something different, and they all think they’re special snowflakes.
[David Spark] Special snowflakes, Sara, what’s your opinion here?
[Sara Madden] [Laughter] Hey. I am not going to own that one at all, I am not a special snowflake. I grew up in Alaska, my dad was a fisherman, I can be tough.
[David Spark] All right.
[Sara Madden] If I take the cyber insurance side of it, no knock on the dozens and dozens of global regulators that knock on my door.
[Laughter]
[Sara Madden] But cyber insurers ask better questions than regulators and auditors, and they have to because there’s a vested interest, and they have to pay out money if something fails, right? They have been hyper focused on ransomware because that’s where they’ve been paying out the most of their premiums over the last couple of years, and so I think they’re overly focused in the things you need to do to prevent ransomware and phishing and some of the conversations we were having about how to test employees before.
I think they’ve been optimized over that for the last couple years. But I will give them credit that they are better at asking important questions as it pertains to the threat landscape, as it pertains to what will cause a breach that will cause them to pay out a premium, right?
And I think we need them there because we need policies in order to land customers, and so I think this is a good driver for security, even better than regulators and audits, from my opinion, over the last couple of years.
I think it’s important to note that regulation and even cyber insurers are years behind the attack landscape, and I think what we don’t do a good job about – because that was one of the questions – was we don’t do a good job at root cause analysis. We don’t focus on understanding the actual root of the problem and then using that as our narrative of what needs to get fixed.
I think there’s a lot of really great security people that are good behind a computer and they’re good communicating with a computer and they’re good at analyzing things, but when it comes to articulating risk – to get budget, to make changes that impact productivity – it requires good communication, a solid way to communicate risk, and a clear understanding of root cause of a problem.
And I think that’s the hard part, right? And it’s too easy sometimes to go from one incident to the next because we’re defenders at the end of the day. It’s hard to go back to that longer-tail process of root cause analysis, risk treatment plans, budgets, arguing with people, changing controls, communicating with employees.
That’s the harder part. The change management part of security’s the harder part.
[David Spark] I would agree with that.
Closing
36:35.808
[David Spark] We’re going to leave everyone on that cliffhanger – the change management part is the harder part. We are not going to answer that question. You’ll have to tune in next week when we may or may not answer it. [Laughter] We’ll see. This show is not serialized.
Sara, you were excellent. Thank you so much for coming. Let me just make a few mentions, and I want to hear your last word on today’s show. Our sponsor for today’s episode is ThreatLocker. Remember, go to threatlocker.com, add the /CISO, it’s a great way to let them know that you heard about them from us, from the CISO series.
Or just tell them, “Hey, we heard about you through the CISO series.” But they really have a very impressive entire platform of zero-trust tools. Take a look at it as you’re building out your zero-trust environment as well. Sara, any last thoughts on today’s episode?
And why should everyone be checking out Convera?
[Sara Madden] Well, if you want to move money around the world, check us out. That’s what we do. We move money around the world fast. So, you can move money, say, for instance, from somewhere in Africa to the US in less than a couple of days, or China within a day.
We move money around fast. Last thought for this podcast – I thoroughly enjoyed this conversation. Thank you, guys. I think sometimes it’s hard to step out of your bubble and have conversations with colleagues, and there’s so much we can learn from each other, if not just a therapy session.
I think we’re all fighting the same fight every day and just taking a step back and talking to each other and helpful tips and tricks that we learn from each other, or just, again, having a therapy session. These conversations are really valuable. So, I thank you for inviting me today, and I would just encourage everybody in the defender space to just keep talking to each other.
[David Spark] So, are you saying that I’m your therapist?
[Sara Madden] You did help me a little bit today. [Laughter]
[Andy Ellis] Awesome.
[David Spark] But everyone heard this therapy session today.
[Sara Madden] Yeah, we all need it.
[Andy Ellis] So, it’s a group therapy with the whole industry.
[David Spark] Well, I’m not bound to any ethical guideline of not disclosing this because we’re recording this and it’s going to go out to everybody. So, everyone’s going to hear it. Hear all your dirty laundry, Sara. And yours as well, Andy.
[Andy Ellis] There goes your social worker license.
[Laughter]
[David Spark] Thank you very much, Sara. Thank you very much, Andy. Thank you to our sponsor, ThreatLocker. And thank you to our audience. We greatly, and I do not mean this lightly, appreciate your contributions and for listening to the CISO Series Podcast.
[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows – Super Cyber Friday, our virtual meetup, and Cyber Security Headlines Week in Review.
This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com.
Thank you for listening to the CISO Series Podcast.