Just because metrics are derived from cold hard facts doesn’t mean we should present them as such. If CISOs don’t frame their metrics to tell a story about their security program, how can they expect the business to care?
This week’s episode is hosted by me, David Spark, producer of CISO Series and Andy Ellis (@csoandy), principal of Duha. Joining us is our sponsored guest, Nathan Hunstad, director, security, Vanta.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, Vanta
AI-infused security operations tip of the week – Anvilogic
Jump to the full tip here.
To learn more about saving costs and optimizing analysts’ capacity with a hybrid SIEM and data lake, go to anvilogic.com.
Full Transcript
Intro
0:00.000
[Voiceover] Biggest mistake I ever made in security. Go!
[Nathan Hunstad] Early in my career, thinking that the security team was the de facto security expert in any situation, and when there was disagreement between developers and the security team, clearly the security team was right, and that was obviously not always the case.
And I learned very quickly that security teams have different incentives and motivations and maybe being the best at security is not one of them.
[Voiceover] It’s time to begin the CISO Series Podcast.
[David Spark] Welcome to the CISO Series Podcast. My name is David Spark, I’m the producer of the CISO Series, and joining me, it’s the principal of Duha, none other than Andy Ellis, a legendary CISO.
[Andy Ellis] Andy, say hello to the audience.
[Andy Ellis] Good afternoon, folks, or depending on when you are in the world, good morning, good evening, or good night.
[David Spark] It’s never catching on, Andy, keep going. [Laughter]
[Andy Ellis] You know something? Fifty percent of hosts on this show use that introduction.
[David Spark] No, more like 25%. No, 33%, I take it… There’s two hosts and then we have guest co-hosts every now and then.
[Andy Ellis] Yeah, 33%, because, yeah, Steve doesn’t count.
[David Spark] [Laughter] Thirty-three percent of the regular co-hosts.
[Andy Ellis] Of the regular hosts, yeah.
[David Spark] We are available at CISOseries.com. If you have not spent at least two to three hours a day on our site, I would definitely seek medical attention, definitely consider that. Our sponsor for today’s episode, a phenomenal sponsor of the CISO Series, that would be Vanta.
Automate compliance, manage risk, and accelerate trust with AI. You do that with Vanta. More about that a little bit later in the show. And also, they’re responsible for our guest today. That in just a moment. But first, Andy, we are one week away from probably the biggest travel period, Thanksgiving.
Always a giant pain in the butt to travel during this time.
[Andy Ellis] It is a giant pain in the butt to travel during Thanksgiving, and I feel a special sympathy for college students and their families who might be traveling around this. Partly because when I dropped one of my kids off for college, it was the Friday before Labor Day, and I had to drive from New York City to Boston.
[David Spark] Ooh.
[Andy Ellis] A three and a half hour drive.
[David Spark] That was…?
[Andy Ellis] That was just over five and a half hours.
[David Spark] Yeah, that’s not good.
[Andy Ellis] Yeah.
[David Spark] Yeah. So, here’s my question for you. Do you have Thanksgiving traditions?
[Andy Ellis] Yes.
[David Spark] What are they?
[Andy Ellis] So, Thanksgiving is done at our house.
[David Spark] Mm-hmm. What’s the biggest crowd you’ve ever had?
[Andy Ellis] Ah. Biggest crowd, I think, was 24.
[David Spark] That’s a good sized crowd.
[Andy Ellis] All at one table.
[David Spark] One table, wow.
[Andy Ellis] See, here’s our little lesson. If you think your table isn’t big enough, you’re probably right. But most tables aren’t wide enough so we have sheets of plywood, four foot by five foot, that are then wrapped in sort of the vinyl coverings.
[David Spark] Mm-hmm.
[Andy Ellis] So, we just put those on top of our existing tables.
[David Spark] To extend it out.
[Andy Ellis] And so, we get a 15-foot long table, four foot wide, and they have space down the middle for all the food.
[David Spark] Mm-hmm.
[Andy Ellis] Right? Because that’s what you usually don’t actually have is space for food and so people need more elbow room. So, you squeeze them in if the food’s in front of them rather than having to fight with them.
[David Spark] How long do people stay at the table? Or is it like the kids eat quickly and they run?
[Andy Ellis] Mmm, we’ve never had so many kids that they’ve sort of eaten and run, but that does happen a little bit.
[David Spark] Yeah.
[Andy Ellis] People are at the table for a little while. Although we have gone in more recent years to a buffet style of you just put the food somewhere else and have people just go make their plates and then come back.
[David Spark] That usually works better instead of all the endless passing, and if you want more food, you just get up.
[Andy Ellis] Right, you just get up and go get it. So, I’m in charge of the turkey.
[David Spark] Do you do a turkey trot, like a run?
[Andy Ellis] No, no, no, I actually just cook a turkey.
[David Spark] You do?
[Andy Ellis] I try not to give people the turkey trots, which are the runs after Thanksgiving instead of the run before Thanksgiving.
[David Spark] We will, by the way – oh, this is a good time to announce this. We will actually be doing a CISO Series Meetup in Boston, Andy, that’ll be on November 24th from 4:30 to 7:00 p.m. at, get ready for this, the City Taphouse, Boston. That’s at 10 Boston Wharf Road.
But what you should do is just go to the Events page on CISOseries.com, and there’ll be a link to register for it, and you can just register there, and we would love to see you. Just get the chance to network with your other cybersecurity professionals in the Boston area.
I will be there and I will also, on that Thursday morning, be doing a turkey trot at the Franklin Park.
[Andy Ellis] Ooh.
[David Spark] Where the Franklin Park is and the Franklin Park Zoo and all that stuff.
[Andy Ellis] That’s a fun zoo to go walking around.
[David Spark] It’s not like the San Diego Zoo.
[Andy Ellis] No, no, no. It’s not like one of these big, amazing, over-the-top zoos. It’s like very much a classical city zoo.
[David Spark] Yes, it is. All right, let’s get our guest in on the show. We’ve had this guest on before, thrilled that he’s back again. He was joking that he didn’t screw up so he’s been asked [Laughter] back again. That’s a glass half-empty way of looking at it.
[Andy Ellis] I mean, I have never been asked back. I’ve only guested once on the CISO Series.
[David Spark] Well, that’s true. We didn’t ask you back. We asked you to host [Laughter] so we wouldn’t have to invite you as a guest.
[Andy Ellis] Yeah, [Laughter] exactly.
[David Spark] Pleased to bring him back again, our sponsor guest with Vanta, the director of security, none other than Nathan Hunstad. Nathan, thank you so much for joining us.
[Nathan Hunstad] Thank you for having me back.
It’s time to measure the risk.
5:39.462
[David Spark] “Metrics don’t show progress, they show impact. Use metrics to shape a dialogue around material impact, not simply activity.” That’s how Nick Ryan frames his approach to CISO reporting in a recent LinkedIn post. His list of 10 C-suite metrics includes expected items like MTTD and MTTR, which is mean time to detection and mean time to response, and vulnerability management, but also pushes CISOs to think differently.
Are you tracking security initiatives tied to business objectives or employees who repeatedly fail phishing tests over six months? That last one is a little contentious. Now, is it just security theater or does it reveal something deeper about whether your security culture is actually changing behavior?
And I’m going to ask you, Andy. Which metrics are best at telling the story of your security program’s growth? I must say that MTTR is the one that we hear a lot.
[Andy Ellis] We do, and I don’t want to not dunk on it, so I’m going to dunk on the phishing failure rate. Let’s just be very honest. You get to pick what phishing failure rate your company has, and then you tell your vendor to hit it, and the complexity of the phishes they send is going to drive to the rate.
So, think of the rate as something you’re sort of measuring. Like you want this to be an awareness activity, not actually a measure of how you’re doing. The real thing you want to be doing is making sure people can safely click stuff. So, let me just get off that soapbox.
[David Spark] Sure.
[Andy Ellis] I went and I looked through these 10 metrics and there were some good, some bad. I’m not a huge fan of MTTR/MTTD.
[David Spark] Well, many others in the industry are. Why aren’t you a fan?
[Andy Ellis] It’s a good interim metric. That’s an operational metric for your organization. The challenge is that you really have this weird parsing of, are you going to include attacks that you stopped in this metric? And the answer is probably no, everyone would laugh if you said, “Oh, look.
Somebody sent me packet floods or TCP storm and I blocked it, but I’m going to count that as a whole bunch of things with zero-second MTTD,” right? So, now imagine you’ve got a hundred threats that hit you on a regular basis. Your MTTD is three days, which would be a fantastic one.
But now I take half of those threats, and I block them immediately. My MTTD just went up because I no longer count them, and so that’s the problem with a metric like this. It actually does not incentivize the right behavior, which is preemptive security that stops the adversary from getting in the first place rather than reactive.
So, you’re measuring your reaction time.
[David Spark] So, you’re only measuring one half of your program.
[Andy Ellis] Right, and it’s the half that you want to have, honestly, going up, not down, which is sort of backwards, but it means you’re actually protecting your company more. One of the ones that I liked, and I didn’t like the way he phrased it, was percentage of security incidents tied to business critical objectives.
[David Spark] Yes.
[Andy Ellis] And I don’t like that as a percentage, but I like that as a concept to say, are these tied to objectives? But then further down was even better, and again, percentage I don’t like, of revenue-generating processes mapped to cyber risk controls.
Honestly, this is the number one metric you should start from, and it’s really qualitative. You should be able to walk into a board and say, “We are a business that sells X. Because we’re a business, we have a bunch of threats to our corporation, phishing, business email compromise, all of these things.
Here’s what we’re doing about those.” Because we produce X, we have additional risks. Like if I’m Vanta, one of my risks is, I have all the data about a whole bunch of customers. So, I am sure that Nathan talks about how are we protecting customer data?
You should be tying all of your risk controls back to the fundamental unacceptable losses of the business. Easiest way to do that is start from how you generate revenue. You exist, you sell a product, and you have customers.
[David Spark] Good point. All right.
[Nathan Hunstad] Yeah, absolutely agree with that. And the most important metrics that we have are all about tying back to business objectives. We tie things like security team activities to the revenue we have brought in by filling out questionnaires that our customers send us or meeting with them to talk about a security program.
We also tie those kinds of revenue-generating ideas around our marketing activities. So, even things like this, we tie very strongly to our security team objectives, which is not something that I think most teams do. So, I’m very aligned on the business objective side of things.
We also tie our metrics to just cost savings because unfortunately for most organizations, security teams are seen as a cost center, so the more efficiently we can do our activities, the better it is for the business. So, how many things are we automating?
How are we leveraging AI in our security team activities? So, we use many more of those kinds of typical business metrics versus things like MTTD/MTTR, which I wholeheartedly agree with your take on.
[Andy Ellis] This is why I love Nathan as a guest. He agrees with me very voluminously.
[Nathan Hunstad] I do.
[David Spark] Nathan, you got to get off of this.
[Nathan Hunstad] Yes. Well, I mean, unfortunately, I mean, we don’t even do phishing tests because we’ve just seen no value. Andy’s exactly right. You get to define your metric of success. And so, what’s the value in that?
[Andy Ellis] Right. And it’s adversarial. It makes your employees hate you.
[Nathan Hunstad] Yes. I have seen employees break down in tears because we’ve reached out to them after they’ve clicked a link.
[Andy Ellis] Mm-hmm.
[Nathan Hunstad] And it just breaks my heart and it’s not making us any more secure.
[David Spark] It’s making people miserable.
[Andy Ellis] It does. Well, and sometimes it makes the people who do them miserable. When I was at Akamai, we had a mailing list we called Social Engineering Alert, which anybody could opt into. But if you had a public phone number, like you were customer care, if you worked a reception desk, you were added to this list.
And the purpose of it was when you detected a social engineering attack on you, like somebody called and said, “Hi, this is so-and-so. I’ve got a meeting with the CEO, but I need his phone number.” All right? Your job was to say no and send mail to this mailing list.
Like don’t send it to the security team, send it to the people who are about to get called next. Whenever the IT security team would run a phishing test, the first person who would get it would send mail to this mailing list and say, “Hey, by the way, IT is running a phishing test.
Here’s what it looks like.” The IT folks would then be crying. They’re like, “You beat my phishing test.” I’m like, “Did we? I think we won. That’s the whole point of this.”
As a CISO, what do you think about this?
12:23.858
[David Spark] “In a pentest, you should be exploiting and demonstrating real world impact. Anything less is a vuln scan.” That’s how Nathaniel Shere of Skillable framed watered-down pentests he sees across the industry. So, what does an effective pentest actually look like in your organization?
By the way, this came up a lot at Black Hat. Are you getting vulnerability identification or true attack simulation? And once you have those findings, how do you operationalize remediation to get the most security bang for your buck? I’ll ask you, Nathan.
Because if you’re paying pentest prices but getting vuln scan results, you might be missing the whole darn point.
[Nathan Hunstad] Yeah, absolutely. And as a fellow Nathaniel, I wholeheartedly agree with what he said there.
[David Spark] So, do all the Nathaniels agree with each other?
[Nathan Hunstad] Yes, we have a shared Slack Nathaniel workspace where we talk about these kinds of things.
[David Spark] They can read each other’s minds too.
[Nathan Hunstad] [Laughter] Yes, absolutely. No, I do think that there are unfortunately a fair number of pentests that just turn out to be vulnerability scans, and so it’s very important that when you are trying to find a pentest vendor or engage in one of these activities, you define the scope, the objectives, and your expectations.
There’s likely going to be some kind of OAuth top 10 type results, especially if you have some gaps in your process or you’re a little bit immature, but the meat of the test results should be demonstrated findings that have an actual business impact.
So, an attack that exploits a vulnerability to gain some kind of valuable objective, and that objective can’t be something like denial of service because frankly, anyone can do that kind of attack if they pour the right resources into it or getting access to things that are of low value.
You need to demonstrate an attack against customer product data or financial data, something that attackers would actually take the time and money to try to get from you. And if you’re not getting those kinds of results, then you are definitely missing the mark.
[David Spark] That’s a good point. The pentest should actually go for the prize. What do you think about that, Andy?
[Andy Ellis] So, I don’t think that Nathan and Nathaniel agree as much as Nathan thinks they do.
[Nathan Hunstad] Hmm.
[David Spark] Really?
[Andy Ellis] I agree with Nathan, but not with Nathaniel.
[Nathan Hunstad] Hmm.
[Andy Ellis] And here’s the nuance, is that encoded in Nathaniel’s message is, “I want to break stuff,” right? The purpose of a pentest is to assess impact. So, I think if you just do a vuln scan and say there exists a vulnerability, that’s not a pentest.
If I exploit a vulnerability and get onto a system, I have now made it a pentest. I don’t have to damage the system to make this a valid pentest, and what I’m hearing in Nathaniel’s…
[David Spark] But can you do it to the point that you’re doing the equivalent of playing tag? You know what I mean? Like here, tag, I touched the prize data.
[Andy Ellis] Right, I mean, you play tag, this is not tackle football. And then you stop and you assess. And now there’s the point where you say, okay, here’s what we could do on the system. If we did this, what would happen? And when the security professionals all panic and say, “Oh, my God.
Don’t do that,” then you write that up as a, like, “This is a Tier 1 finding. You are all freaked out that we were going to turn off this machine.” Like, clearly that has serious business impact.
[David Spark] There’s difference between, by the way, a pentest and Chaos Monkey too.
[Andy Ellis] Except it’s not Chaos Monkey, it’s Chaos Evil Monkey.
[David Spark] [Laughter]
[Andy Ellis] Because I’ve met pentesters who really are not very reformed black hats, right?
[David Spark] Mm-hmm.
[Andy Ellis] I want to get paid to go break your stuff, which is different than I want to get paid to help make you better, right? By getting onto a system and everybody freaks out, like, okay, now we’ll go fix the system. And that is appropriate. Now, the one thing that people often get wrong when they’re doing a pentest is they don’t say, “Okay, what could you do laterally from here?” Right?
It’s reasonable to do a stop and say, okay, given you had access to the system, now what could you do non-destructive to the system further into my network? Because that’s really where you get to start to detect adversarial things. Okay, you’ve got login access on this machine.
Maybe [Laughter] I don’t want you on it. It’s my Active Directory controller. So, I’m going to give you a machine right next to it. Same network permissions, go do anything you want from it. That’s, I think, the place that good pentesting tends to get you is really in network and asset exploration.
[Nathan Hunstad] Yeah, I really like one thing that Andy brought up here, which is the interaction between pentesters and the targets because I think a lot of people also believe that a pentest has to be this isolated thing. You find a vendor, they attack you for two weeks, and you can’t talk in the middle.
But in that interaction is where you get a lot more value because you can say, “Oh, you came really close to this thing. Poke around here some more,” or “You’re off in the wrong direction.”
[David Spark] By the way, is it appropriate what you just described to say this is more of purple teaming? Because that’s what that’s been defined as, or no?
[Nathan Hunstad] I think some people have defined it as that, yes, and that’s where it comes back to you. You should really make sure that you define the scope and expectations ahead of time. And there’s a time and a place for each of these kinds of tests.
There is a time and a place for a pure black box, go at it, hit it hard, and see what we can detect, but I think there’s less of a time for that than most people realize. And you get a lot more value with collaborative purple team type of exercises where both sides can learn.
Your attackers can learn what the defenders care about, and the defenders can learn where their gaps are and where they need to pay a little bit more attention.
Sponsor – Vanta
18:12.287
[David Spark] Before I go on any further, I do want to tell you about our spectacular sponsor that would be Vanta. So, we all know in today’s fast changing digital world, proving your company is trustworthy isn’t just important for growth, it’s kind of essential, right?
That’s why Vanta, it rhymes with Santa, it’s easy to remember, is here. Vanta helps companies of all sizes get compliant fast and stay that way with industry-leading AI, automation, and continuous monitoring. So, whether you’re a startup tackling your first SOC 2 or ISO 27001, or an enterprise managing vendor risk, Vanta’s Trust Management Platform makes it quicker, easier, and more scalable.
Vanta also helps you complete security questionnaires up to five times faster so you can win bigger deals sooner.
The results? According to a recent IDC study, Vanta customers slash over $500,000 a year in costs and are three times more productive. Establishing trust isn’t optional. Vanta makes it automatic. Visit their website, vanta.com/CISO, and I want you to remember the /CISO part.
You remember vanta.com, you remember I told you it rhymes with Santa because they bring you gifts in a giant red bag, and they come down a chimney? That’s not what they do, but that’s how I picture it. It’s vanta.com/CISO, and if you add the /CISO, they know that we sent you there and that’s important.
So, go check out their site, vanta.com/CISO to get started today.
It’s time to play “What’s Worse?”
19:51.357
[David Spark] Nathan, I know you know how to play this game, but do you know how to play the game the way I like it? And the way I like it…
[Andy Ellis] That’s cheating.
[David Spark] …is you disagree with Andy. Whatever he says, disagree. Now…
[Nathan Hunstad] I’ll try.
[David Spark] …you’re free to say whatever you want, just as long as you disagree with Andy.
[Andy Ellis] Nathan is an awesome human being.
[David Spark] There’s no buttering him up. So, here we go. It comes from Jay Dance. He gives us a ton of phenomenal “What’s Worse?” scenarios. By the way, if you’re listening, send us in more. I’m always looking for more good ones. Here you go. These are two very crappy things with crappy results, Andy.
Number one, your business gets repeatedly attacked by malware from a malicious domain. When reported, the DNS domain register fails to take down the domain. Okay?
[Andy Ellis] Okay.
[David Spark] Number two, a malicious actor creates a website that mimics your business and scams your customers out of thousands. When reported, the web provider fails to take down the scam site. Which one is worse?
[Andy Ellis] So, first of all, there are vendors to solve both of these problems for you.
[David Spark] Yes, yes, yes.
[Andy Ellis] Since we do sponsorship, I’m not going to name the vendors, but you can guess who deals with DNS infrastructure, who will block your DNS for you. And then there are reputation-based vendors who will go after…
[David Spark] Let’s just say you don’t have either one and this is what’s happened.
[Andy Ellis] But let’s just say you don’t have either one of those solve that problem. Oh, wait, you’re not allowed to because we’re in a “What’s Worse?”
[David Spark] Yes.
[Andy Ellis] The second one is the worst one. Why? Neither one of these actually affect your business that much. Like you actually can protect yourself reasonably. The second one hurts customer trust.
[David Spark] Right. This is why I thought you were going to lean on this one.
[Andy Ellis] And if you have customers who are being scammed, the marketing impact from that one and the fact that you’re not dealing with that in a public way is actually more impactful. The security concerns here are honestly, like the first one is slightly worse from a security.
[David Spark] But it’s your business gets repeatedly attacked by malware. So, it does affect your business.
[Andy Ellis] Right. But like repeatedly attacked by malware does not sound like though we’re getting complete ransomware shut down. It’s an operational annoyance. I’m kind of defining it because Jay wasn’t specific about what the impact was.
[David Spark] He wasn’t specific, but this could be catastrophic. It could, we don’t know exactly.
[Andy Ellis] If it was catastrophic, he should have said so, and then I would change my mind. But I’m assuming these are both minor negligible things that if you were remotely competent, you would deal with either one of them by outsourcing it. You’re not, so you’re stuck with them.
[David Spark] [Laughter]
[Andy Ellis] The second one is the problem because you’re exposing your incompetence to your customer base.
[David Spark] But the thing is it’s a malicious actor and it scams your customers out of thousands, which is awful, but you’re not doing it. The other one is on your site.
[Andy Ellis] Yeah, but if your customer’s being scammed out of…
[David Spark] You didn’t deal with it on your own site. So, legitimate customers are coming to your site and they’re having problems.
[Andy Ellis] I know what you’re trying to do, David. You’re trying to convince Nathan. So, I get my counterbalance here, which is this. I’ve had my vendors and customers scammed out of money by somebody who pretended to be me.
[David Spark] Mm-hmm.
[Andy Ellis] And do you know at the end of the day who paid for that? I did.
[David Spark] Well, you’re being very kind.
[Andy Ellis] Because otherwise the reputational harm of this small vendor who lost $10,000 compared to a $2 billion company, massive. So, we ate it, right? Because either that or we’re losing more than $10,000 in reputation when they go tell people, “Hey, don’t do business with this company because they won’t make you whole.”
[David Spark] All right. Nathan, we’re throwing this to you. Do you agree or disagree with Andy?
[Nathan Hunstad] Yeah, yeah. David, I’d really like to help you out and not agree with Andy, but I have to agree with Andy. The second one is…
[David Spark] For the same reasons?
[Nathan Hunstad] For the same reasons, yes. And I mean, it really comes down to customer trust.
[David Spark] Mm-hmm.
[Nathan Hunstad] And if you lose that, that’s going to be far more detrimental to your business than if you’re getting repeatedly attacked by malware. I mean, I would quibble with the first scenario as well to say if you’re getting repeatedly attacked by malware, and you just can’t find any compensating controls to deal with that, I mean, do you even have a security team?
[David Spark] So, we’re assuming both situations are incompetent. [Laughter]
[Nathan Hunstad] Yes.
[Andy Ellis] The challenge of the second one is like the adversary could just move and it’s painful, but look, both of these – outsource the problem.
[David Spark] Yes.
[Andy Ellis] Like buy a proactive preemptive DNS service to deal with the first one, to just filter out domains for you. Second one is reputation monitoring services that will go chase these people down and chase down their vendors.
[Nathan Hunstad] I mean, honestly, for the first one, you can just, if you can’t do anything else and can’t find a vendor, just sinkhole it in your hosts file.
[Andy Ellis] I know.
[Laughter]
Please, enough! No, more!
24:46.489
[David Spark] Today’s topic in “Please, enough! No, more!” is AI in security operations. By the way, I came back from Black Hat, and I think I might’ve mentioned this in a previous show, but I heard it multiple times. There was no oxygen for anything that was not AI at Black Hat.
There was just nothing.
[Andy Ellis] Hey, in fairness, only a hundred vendors, no, it was 111 vendors, I think, mentioned AI on their booths, out of 359.
[David Spark] That’s on their booths, but…
[Andy Ellis] On their booth.
[David Spark] Trust me, they…
[Andy Ellis] [Laughter] It was mentioned elsewhere, but that’s the headliners of it.
[David Spark] Yeah, it’s like having a business not connected to the internet. That’s kind of the level of it at this point. All right, so Andy, I’m going to ask you, what have you heard enough about with AI in security operations and what do you want to hear a lot more?
[Andy Ellis] So, I am tired of hearing about LLMs are just going to replace the security operations staff. Like LLMs might replace their training, but if you want to replace tier one and tier two staff, you’re not replacing them with LLMs. You need to replace them with automation.
You might have LLMs build automation, but the expense that people are going through for unreliable actors, which is the definition of an LLM, right? We’re going to say, “Oh, we’re going to hand to an LLM, it sees the alert, and it’ll go deal with it.” No.
Have an LLM that might train an automation stack that will see an alert, knows what to do the right things, but you want repeatability in operations. LLMs are the antithesis of repeatability. So, stop telling me that your LLM is going to replace my Tier 1 person.
[David Spark] That’s a really good point because you can literally ask a question of an LLM and then five seconds later ask the same question, get a completely different answer.
[Andy Ellis] Exactly.
[David Spark] Yes, very good point. All right, I throw this one to you, Nathan. What have you heard enough about with regards to AI and security operations? What would you like to hear a lot more?
[Nathan Hunstad] Yeah, I would like to stop hearing about how vendors, to your point, every vendor is talking about AI and they’re trying to find some way to slap AI functionality onto their tool in strange ways. And whether that’s natural language search, whether that’s doing things around trying to triage alerts.
[David Spark] But just to push back a little bit here.
[Nathan Hunstad] Yeah.
[David Spark] Even in the smallest ways, AI can help a little bit. You know what I mean?
[Nathan Hunstad] It can.
[Andy Ellis] Yes.
[David Spark] And to not have it, to not do it at all, seems like you’re kind of behind the game. You’re not dealing with this right.
[Nathan Hunstad] Mm-hmm.
[Andy Ellis] Right, but one of the things Nathan just mentioned, natural language search. I think you should have an LLM as your interface to your knowledge base. If you’ve got a knowledge base, you should have an LLM that is the reference librarian for it.
So, somebody asks the question, the LLM interprets it, finds the answer, but then just hands them the answer out of the knowledge base, rather than trying to reinterpret it every time. You have static content for that reason, and that’s, I think, that may have been part of where Nathan was heading.
And that’s you’re just using AI as a tool. Don’t tell me you have an AI-enhanced knowledge base. No, you have a knowledge base that is using something akin to MCP to let me get access to it.
[Nathan Hunstad] Yeah, that’s exactly it. I think it’s much more about making your security team more efficient by having AI do things that are time consuming and not very valuable but have to get done. And sometimes searching for things from your SIEM is one of those activities, especially if you’re new to it, natural language search can help.
What I hate, taking this a bit further though, is having that as your only interface. And when you want to get very particular conditionals in your search query, doing that in a natural language search, and making it deterministic, which as you point out, GenAI is not made to do, is really frustrating.
So, I would like to see forcing everyone to use AI instead of just making it as an alternative to help you when you need additional help. I use AI all the time. And for certain things like generating a quick Python script to do a search of some IOCs using an API, fantastic.
I could do that myself, but it would take 10 times as much time to do so, and I’d rather be doing other things. So, that’s an excellent example of AI use, but positioning AI as doing the thinking for your team, that’s where you get into trouble because it’s just not consistent enough.
It’s not really designed for that when you are talking about most generative API tools.
[David Spark] Now, I do know how Vanta is taking advantage of using AI to facilitate compliance and regulations and essentially…
[Nathan Hunstad] Yes.
[David Spark] …also building up your trust center too, for that matter. What is it specifically doing with the operations center though?
[Nathan Hunstad] Yeah, with the operations center, I think that Vanta is still early in the process. If you want to consider things like your third-party risk management and those kinds of security operational tasks, Vanta does have AI capabilities to help flag vendors that may be higher risk or items from a SOC 2 report that you want to pay more attention to.
But Vanta isn’t like your typical SOC tool.
[David Spark] No, you’re not seen as that, no.
[Nathan Hunstad] But I mean, I do love the things that Vanta is integrating with AI because one of the things we do have is on our trust center, we have a chatbot. So, people can ask, “Hey, when was your last SOC 2?” or…
[David Spark] A natural language question.
[Nathan Hunstad] Yes, exactly. Were there any significant findings in your last pentest report? And those things, again, would you rather page through a 100-page SOC 2 report to find the one thing you’re looking for or just ask the chatbot the question and get the answer?
That’s a great example of how you can actually save time.
[David Spark] Now, Nathan, have you considered renaming this tool to Ask Jeeves?
[Nathan Hunstad] [Laughter] We would probably have to talk to our lawyers about that one. I have a sense that that may be taken.
[David Spark] That was Ask Jeeves’ selling point at the beginning and it couldn’t do it, Andy.
[Andy Ellis] That would have been great if Jeeves could have done it, but I think we leave Jeeves dead and buried.
[David Spark] Just stuck. If it just held out till AI really exploded, then it would have [Laughter] worked. They had a huge building, Ask Jeeves, over on the East Bay.
[Andy Ellis] They did. Oh, a lot of people had huge buildings back in those days.
[David Spark] Yeah, I know. [Laughter]
[Andy Ellis] VC money fell off the trees.
AI-infused security operations tip of the week – Anvilogic
31:46.656
[Voiceover] It’s time for this week’s security tip. This week’s AI-infused security operations tip is sponsored by Anvilogic.
[David Spark] Every SOC wants to boast smarter decisions and fewer false positives, but this is a process that starts long before AI enters the picture. In fact, it starts with data hygiene. That’s right. AI tools thrive on structure, consistency, and clarity, but some organizations muzzle their AI’s potential by feeding their models disorganized logs, untagged assets, and fragmented telemetry from mismatched tools.
It’s a new turn on garbage in, garbage out. When data is poor, AI learns bad habits.
Great AI security begins with clean inputs, accurate timestamps, consistent event formatting, and complete coverage of critical systems. This is followed by feature engineering, which is the art of turning raw activity into meaningful behavioral signals that a model can actually reason with.
As an example, instead of simply logging a failed login, a well-crafted feature might track frequency per user per hour, a pattern that could distinguish a human typo from a brute force attempt. SOC leaders who prioritize data quality find that their AI suddenly becomes more insightful, more reliable, and far less noisy, making their AI investment worthwhile.
[Voiceover] To learn more about saving costs and optimizing analyst capacity with a hybrid SIEM and data lake, go to anvilogic.com.
What works? What’s not working?
33:38.576
[David Spark] Threat intelligence seems like it should be a cornerstone of any mature security program, yet the numbers tell a very different story. A recent CSO Online piece highlights that while 93% of CISOs are concerned about dark web threats, 21% have no threat intelligence capability at all, and 46% don’t regularly consume threat intel reports.
The issue isn’t access to intelligence, it’s operationalizing it effectively. Yes, we have seen this. So, too many security teams are stuck using threat intel only at the tactical level, but few are leveraging it for incident response or strategic decision making.
Threat intelligence can be the difference between going reactive to proactive but can also turn into a massive time sink. That’s a major concern right there. So, Andy, I’ll start with you. What’s the biggest mistake you’ve made with threat intelligence, and what’s the one tip you’d give others to manage it more effectively?
[Andy Ellis] Ooh. That’s a really hard one. So, the biggest challenge of threat intelligence is thinking that the intelligence you have gathered, and the big mistake I’ve made, tried to sell it, that other people would also find it valuable.
[David Spark] Mm-hmm.
[Andy Ellis] But that’s the single biggest thing is threat intelligence has so much, to you, context that matters, that once you divorce it from that context, which you have to do. If I have, oh, I know this adversary attacked this system of mine in this way and compromised it, all I can do is pass on, “This is an adversary,” to Nathan.
Well, what’s Nathan supposed to do with that? Or maybe like, here’s what their TTPs are. But I can’t say, “By the way, they compromised this system of mine,” because my lawyers will never let me do that. So, I end up removing – and even if I did, it’s not clear Nathan could still use it.
So, there’s this very big cost to trying to share.
I think the biggest challenge is people are often using it tactically rather than strategically. You should use threat intelligence for context. Now, you should have somebody who’s out trolling the dark web, looking for people specifically trying to attack you.
Like that’s very tactical, actionable, but mostly you want to say, “Hey, what are people doing these days? Where are they pivoting to? What do I need to do? Are they talking about me? What are they saying?” Like that’s the sort of strategic inform, not your tactical defenses, but your whole strategy by paying attention to what adversaries are actually doing.
[David Spark] All right, I throw this to you, Nathan. Have you made a big mistake with threat intelligence yourself?
[Nathan Hunstad] Well, I don’t know if this would be considered a mistake or not, but long ago, I was actually on a threat intelligence team, and we did some amazing things. It was probably the most fun I’ve had in a role.
[David Spark] Does Vanta know this? [Laughter]
[Nathan Hunstad] This is not Vanta and did not break any laws.
[David Spark] But did Vanta know that you had more fun than at Vanta?
[Nathan Hunstad] [Laughter] Yeah.
[Andy Ellis] Oh, man. I’ll call Jadee up right now.
[Nathan Hunstad] Yes, yeah, please do. It was definitely a different kind of fun, let me put it that way.
[David Spark] Mm-hmm.
[Nathan Hunstad] But it was also frustrating and that’s the mistake part of it because to Andy’s point, we couldn’t really get any value or share what we were doing because the lawyers wouldn’t let us and because it was very limited in terms of what people would care about.
The other thing too, you would, well, maybe not be too surprised to hear this, but if you get some kind of intelligence that somebody else is being breached, they don’t want to hear about it necessarily unless you have all of the evidence because that puts them in a precarious position.
So, you have to be careful about what you share and a lot of times people aren’t necessarily thankful for what you’re telling them.
[David Spark] Very good point. But I’m thankful you told me that you had more fun at somewhere else that wasn’t Vanta.
[Nathan Hunstad] [Laughter]
[Andy Ellis] Oh, man. I’m going to go tell that to Jadee right now.
[David Spark] Remember, Vanta rhymes with Santa. So, how can it not be fun?
[Andy Ellis] I think it rhymes with Fanta. So, it’s very bubbly and fizzy and sweet.
[David Spark] It’s bubbly, Vanta, Santa. I like the idea of Santa coming down the chimney with a compliance offering or to help you with your compliance programming with a two-liter jug of orange Fanta soda.
[Andy Ellis] Yeah, I want to see Vanta Clause for your holiday.
[David Spark] Ooh. There you go. Dressed in purple, not red.
[Nathan Hunstad] Oh, yes, absolutely. I will talk to our marketing team.
[David Spark] Yes, get this done right away.
[Nathan Hunstad] Absolutely.
[David Spark] We demand it in fact.
[Andy Ellis] Yeah.
Closing
38:18.721
[David Spark] Go to Vanta’s website, vanta.com/CISO, not vanta.com/Santa, which I think will take you nowhere. Go to vanta.com/CISO to learn about all their awesome offerings specifically for your compliance program to save yourself a bundle in both money and time.
They’re connected, and Vanta will help you do just that. Nathan, thank you so much for coming back again and being on the show. Are you hiring over at Vanta and will you make it a more fun environment when someone comes and joins?
[Nathan Hunstad] [Laughter] Vanta is a fantastic environment. I do have tons of fun, and I want more people to join in because yes, we are hiring.
[David Spark] Awesome. So, I’m assuming there’s like a job board on vanta.com?
[Nathan Hunstad] Yes, there’s absolutely. If you go to the Careers link on vanta.com, you can see all the positions on my team and within the broader security organization.
[David Spark] And we will have a link to Nathan’s profile. I’m assuming people can reach out to you, yes?
[Nathan Hunstad] Yes, absolutely. Please do. And please mention that you heard me on the CISO Podcast and I will absolutely add you.
[David Spark] Yes, that is very key. Go to vanta.com/CISO, contact Nathan, tell him you heard him on the show. Is there anything else you would like to say about Vanta before we wrap this up?
[Nathan Hunstad] No, I am very happy to be at Vanta, and I think we are doing awesome things in AI. I’ve been at Vanta for a year and a half. It’s changed so much, and I can’t wait to see where we’re going.
[David Spark] Well, because AI has changed a lot too. [Laughter]
[Nathan Hunstad] Yes, it has changed so much, absolutely.
[Andy Ellis] Little do you know, David, but Nathan is actually a deep fake of the previous Nathan. He’s actually just an AI.
[David Spark] Really? We’ve had the deep fake Nathan this entire time? I’m 100% fooled.
[Andy Ellis] That’s why he agreed with me is I got to program that into the model – you must agree with Andy.
[Nathan Hunstad] Hey, you’re going to need to work on some prompt injection attacks to change that.
[David Spark] Oh, I definitely will. I will definitely have to work on that for the next episode. Thank you, everybody. We greatly appreciate your contributions and for listening to the CISO Series Podcast.
[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows – Super Cyber Friday, our virtual meetup, and Cyber Security Headlines Week in Review.
This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com.
Thank you for listening to the CISO Series Podcast.