We’re all drowning in AI hype. But now that the early days of being amazed with what consumer-grade LLMs can do are behind us, how do we find the actual value for organizations? It’s one thing to have a new way to scale an old process, but where is AI helping us solve things in new ways?
This week’s episode is hosted by me, David Spark, producer of CISO Series and Mike Johnson, CISO, Rivian. Joining us is Erwin Lopez, CISO, SLAC National Accelerator Laboratory.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, ThreatLocker
Full Transcript
[Voiceover] Biggest mistake I ever made in security. Go!
[Erwin Lopez] Can’t believe I’m admitting this, but [Laughter] right before vacation, I ended up blocking an entire top-level domain, co.uk, from my work specifically. So, anybody trying to go to any UK sites that were commercial, they were blocked, and that was on the day before I went on vacation.
[Laughter] They were blocked for 24 hours. [Laughter] That was in my early days.
[Voiceover] It’s time to begin the CISO Series Podcast.
[David Spark] Welcome to the CISO Series Podcast. My name is David Spark, I’m the producer of the CISO Series. And joining me as my co-host since episode number one, it’s Mike Johnson, the CISO of Rivian. Mike, say hello to the audience.
[Mike Johnson] Hello, audience. Great to be with you yet again.
[David Spark] By the way, for those of you who don’t think that he’s talking to you, he is. He’s talking specifically to you.
[Mike Johnson] I’m talking to you, you in particular. Thank you.
[David Spark] Yes. We’re available at CISOseries.com. If you don’t spend at least half your day there…
[Mike Johnson] What are you doing even?
[David Spark] What are you doing? I don’t know. Our sponsor for today’s episode, a phenomenal sponsor of the CISO Series, it is ThreatLocker Zero Trust Endpoint Protection Platform. That’s ThreatLocker. More about just that a little bit later in the show.
So, before we came on air, I was sort of telling a little bit of my history in tech media, and our guest, who I’ll introduce in a second, remembers me from that. So, I was on the early days of ZDTV, and I was also on TechTV on many of the programs there, but for those of you who ever listened to that network, which lasted a total of six years, was all about computers and the internet.
[Mike Johnson] It’s a good run.
[David Spark] Fascinating thing too. Who would have thought that people would be interested in watching videos about computers and the internet?
[Mike Johnson] [Laughter]
[David Spark] That was the theory back then. Anyways, went from 1998 is when it launched to 2004, I believe, is when it ended. I wrote the first 30 minutes that ever appeared on the network.
[Mike Johnson] Oh, wow.
[David Spark] And I was the second voice ever heard. What you heard is a person narrating what the network was going to be, and then you hear me saying, like talking the sound of a doctor saying, “It’s a brand new network!” like a baby boy.
[Mike Johnson] [Laughter]
[David Spark] And the idea was they took out a baby, but it had a giant red block on its head, which was the ZDTV logo at the time.
[Mike Johnson] Nice. I had no idea, I had no idea. That’s really cool.
[David Spark] So, there’s my sort of footprint in tech media history. I don’t think I have access to that 30 minutes because it was on, like, I had a copy of it on DVPro tape.
[Mike Johnson] Oh, my gosh.
[David Spark] Which I don’t know of anyone who had a DVPro deck or camera. Wasn’t digitized.
[Mike Johnson] You should get it digitized. Get it digitized, if you still have it.
[David Spark] You know what? I wouldn’t be surprised – I should search on YouTube – I wouldn’t be surprised if someone put the first half hour of ZDTV on YouTube somewhere.
[Mike Johnson] It’s probably there. Everything else is on YouTube.
[David Spark] Well, the first songs from MTV are up somewhere on YouTube.
[Mike Johnson] Mm-hmm.
[David Spark] Although ZDTV doesn’t have quite the legend that MTV has.
[Laughter]
[Mike Johnson] It’s not quite in the same category, but for certain folks…
[David Spark] Being that it’s not still around and MTV is still around.
[Mike Johnson] Well, is it though? Is MTV really still around?
[David Spark] Not what it was when we watched it back in the day.
[Mike Johnson] Yeah. There’s a thing called MTV, but it’s not MTV.
[David Spark] Let me tell you something. Can I tell you the moment that I knew I was turning into an old man?
[Mike Johnson] [Laughter] Please.
[David Spark] Here’s the moment. I was watching MTV, honest to God, these words came out of my mouth. I’m watching MTV, and I said to myself, “What are these stupid kids doing?”
[Mike Johnson] [Laughter]
[David Spark] I’m an old man now.
[Mike Johnson] I mean, that’s basically the kids these days.
[David Spark] That is the indicator.
[Mike Johnson] Yep, done.
[David Spark] All right. Let’s bring out our guest. Thrilled to have our guest on. Good friend of yours, I know. I know you know him very well.
[Mike Johnson] For quite some time.
[David Spark] He’s the CISO of the Slack National Accelerator Laboratory. None other than Erwin Lopez. Erwin, thank you so much for joining us.
[Erwin Lopez] Thank you for having me. It’s a pleasure to be here. Good to see you both. And again, I remember the good old days of TechTV, and I remember watching you, so this is a big deal for me. [Laughter]
Is AI going to help us, or hurt us?
5:07.301
[David Spark] “LLMs and machine learning are merely the conduit through which benefits to the customer are delivered,” Justin Warren from PivotNine tried to cut through the AI hype, highlighting the advances in cyber are incremental, such as using Copilot for documentation or Crogl for better incident response policies.
But GitHub’s MCP server was recently exploited through basic prompt poisoning. Essentially, AI makes it easy for anything, good or bad, to suddenly hit scale. And that’s kind of the history of the internet. So, the question isn’t whether AI helps or hurts.
It’s how do we separate those selling expensive ways to do the same old things from those actually solving valuable problems? Which, I don’t know if it’s expensive ways do the same old things. It might be same old things just at scale too, as well.
Mike, yes?
[Mike Johnson] Yeah, I think really, we’re in the early days right now. This is the experimentation phase, and that’s really kind of where we need to live for a while. It’s the same thing that we’ve done with cloud computing, the same thing with mobile devices.
We’re really experimenting right now. And some of the things that we find will be step-level improvements in our capabilities, the scale that you’re talking about. Some of these things are going to fail spectacularly.
[David Spark] Mm-hmm.
[Mike Johnson] Most are going to be somewhere in there and be in between huge improvements and spectacular failures. But I really think the thing that we need to get comfortable with is the idea of experimentation. It’s great to call out, “Hey, that failed.
Hey, there is a vulnerability in the GitHub MCP service.” We are going to see those as time go on, but we need to not let that stop the experimentation. We need to continue to try. Otherwise, we don’t know how these things will be used. Like it took a long time to figure out how to use a hammer.
[David Spark] For you, it took a long time, Mike? [Laughter]
[Mike Johnson] I hit myself, like, you know? It hurt a few times when I was learning how to use a hammer.
[David Spark] All right.
[Mike Johnson] We’re kind of in the hammer stage.
[David Spark] Let’s throw this to Erwin. Erwin, there are a lot of pros and cons, yes? Interested to know if you see yourselves creating more sandboxes to essentially uncover the pros and cons, yes?
[Erwin Lopez] I would agree with you, yes. I mean, from our perspective, really, AI is a force multiplier at the end of the day. It really, for us, it represents the evolution of what new security operations teams are going to be utilizing in terms of technologies, helping to not only synthesize data, looking at disparate data sets, and just trying to find that needle in the haystack, right?
But it also comes with a downside at the end of the day, right? I mean, bad guys are going to be utilizing the same tools that we’re trying to use against us. We open up our own attack service to new types of vulnerabilities that are out there just because we might not have the experience of being able to test these new AI capabilities, right?
Kind of like the GitHub MCP server and so forth. And also our user base utilizing things and entering more data than they should be entering into these public entities and so forth. So, that makes it kind of hard for us.
But let me share kind of a quick story about something that is near and dear to my heart. So, this is not in the cybersecurity realm specifically, but it has to do with disparate data sets and so forth. I’m going through some health issues and so forth, and I ended up getting recently a CT scan and a PET scan, and I’ve been entering all my data utilizing ChatGPT just to analyze the test results, imaging and so forth.
Again, at an anonymous level. I’m not entering my name or anything like that.
[David Spark] Mm-hmm.
[Erwin Lopez] Just sheer information and so forth. And in the last PET scan that was identified, they came out and said that there is what they call an attenuation error where they found something, but when they took out the filter, it wasn’t there anymore.
So, they labeled it as maybe a system error in terms of the imaging itself. But when I ended up putting into ChatGPT, it gave me a lot of different options based on the disparate data sets that were out there, and it said, “Hey, you know what? There could be a blood clot somewhere in your liver.” Okay?
That ended up giving me the option to basically talk to my doctor and say, “Hey, guess what? Could I get a liver MRI?” Which actually led to a real diagnosis that I did have a blood clot in my liver.
[Mike Johnson] Oh, wow.
[Erwin Lopez] Yeah. So, this was critical for us.
[David Spark] Wow. So, you used AIM LLMs.
[Erwin Lopez] Mm-hmm.
[David Spark] To self-diagnose and find something the doctor could not find?
[Erwin Lopez] Correct.
[David Spark] That’s unbelievable.
[Erwin Lopez] And that was just to ask, to just have the right conversation with the doctor to be able to get that test done. Otherwise, I would have had to’ve waited.
[David Spark] And also to get ahold of that data yourself, no one thinks to literally take the data from a report and stick it into a computer to give it any answer whatsoever.
[Erwin Lopez] Correct.
[David Spark] That’s amazing you even thought to do that.
[Erwin Lopez] So, I’ve been utilizing it to keep track of all of my tests from now on. And I’m not saying, you know, we’re not trying to go towards a Google doctor of the future or anything like that, right? But these are just, gives us the ability to have good conversations with a doctor and openingness.
[David Spark] Right. And that’s a really good point to make because one of the jokes I make about using the internet to try to self-diagnose is, what you use is you really just search for things, and they give you a laundry list of issues, and all of a sudden, your paranoia sets off.
But you are taking your own data and trying to understand it, which is very, very different than the way most people use the internet for health purposes.
[Erwin Lopez] Mm-hmm.
Why are CISOs leaving the profession?
10:47.313
[David Spark] Cybersecurity is a dynamic, challenging field. But what happens when you’ve done it all? And the biggest challenge becomes convincing your own organization to care. “I got tired of convincing people on the importance of security.” That’s from a CISO in the cybersecurity subreddit who’s tired of constantly having to sell security’s value to people who don’t want to hear it.
I’m sure neither of you have ever run into that problem.
[Laughter]
[David Spark] Now, some commenters didn’t want to hear it, calling it a First World problem and to retire, you know, telling the person to retire. But other commenters gave them more grace with one saying, “You will only know when you are in that situation.
It’s not arrogance or ignorance. It’s a disappointment, frustration, identity crisis, and burnout.” So, is CISO boredom – I don’t know if I’d call it boredom, although irritation, actually the work itself – or is it about endlessly repeating the same conversations with people who do not want to listen?
Which I know is kind of a core part of the job of being a CISO, Erwin, right?
[Erwin Lopez] I would agree with you. I mean, the constant need to sell security and translate risk for the business is exhausting sometimes because you’re having to repeat yourself multiple times and it’s just iterative scale of just the same thing over and over and over again.
But I see this as only one part of the job. There’s still a lot, other things that really kind of motivate me personally as a CISO, right? Being able not only to improve the program itself, build it, right? Also stand up maybe a new research capability within our own team to be able to kind of harness all these new technologies that are coming up.
At the same time, for me personally, staying connected to the technical expertise that I came from, right? Being able to maybe at times help out on incidents when I do have time, which is rarely available, but whenever I can, I do. And kind of quench that hunger that I have for that technical capability.
And that really what keeps me motivated to continue to move forward. But I would agree with you. I see why people are burning out because they’re having to have that same conversation over and over and over again. They feel that it’s not moving any forward as such.
[David Spark] All right, Mike. I know one of the things that you’ve said to get people to care is to make it as personal as possible. Does that always work, or I’m going to assume some people still don’t care?
[Mike Johnson] At the end of the day, it is our job to care about cybersecurity. That is the role that we take. And absolutely, it’s frustrating to talk to people and keep telling them the same thing and keep telling them that, “Hey, this behavior, it’s going to lead to a bad outcome.” But we’re not the only ones who face that.
Like this is not a unique problem to cybersecurity. Doctors, that’s very much what they’re doing.
[David Spark] Oh! My dad used to complain all the time, my dad who was a doctor, about his patients who wouldn’t take their medicine. Drove him crazy. [Laughter]
[Mike Johnson] Exactly. That’s the exact same thing that we’re dealing with here. And so, this isn’t unique to cybersecurity. This is something that, I understand burnout, and I understand how challenging that can be to go through, but we’re not going to change human nature, so we need to find our mechanisms to cope with it.
We need to take a step back, take a vacation, recognize that this is just how humans are. We need to figure out how to work with that, not continuing to beat our head against the wall and being mad that people aren’t doing what we tell them to. So, I appreciate burnout, I appreciate that it happens.
It sucks that it happens. But what I would encourage everyone to do is just some self-care, figure out how to deal with it because you’re not going to change it.
Sponsor – Threatlocker
14:43.491
[David Spark] Before we go on any further, I do want to tell you about our spectacular sponsor and that is ThreatLocker. Now, let me tell you, even the most reliable employees make mistakes. An unauthorized USB device or an accidental click can actually expose sensitive data and create serious risk.
It can be that simple. Now, traditional user-based access controls rely on trust, but trust isn’t security. So, ThreatLocker takes a different approach in securing your environment. They do it by enforcing program-based policies. It ensures only approved applications can access, read, or copy data.
Sensitive files stay locked down, while approved software continues to run without disruption. And when exceptions are necessary, administrators can approve them in seconds, keeping productivity high without sacrificing protection.
Also, with ThreatLocker, every action is logged in a detailed audit to capture the exact user, file, application, and device serial number. This is actually zero trust in action. This is how it works. It’s precise, it’s enforceable, and it’s simple to manage.
Discover how ThreatLocker can help you gain more control over your environment. Just go to their website and check it out. Go to this website, ThreatLocker.com/CISO. Do me a favor, add the /CISO. It’s the simplest way to let them know that you heard about ThreatLocker from us, the CISO Series, ThreatLocker.com/CISO.
It’s time to play “What’s Worse?”
16:19.581
[David Spark] All right, Erwin. You know how this game is played, right?
[Erwin Lopez] Yes.
[David Spark] Two bad scenarios. You have to pick which one’s worse, but I will make Mike answer first and you can agree or disagree with him. It comes from Jason Keirstead who’s currently working over at Simbian, and here are the two scenarios. Scenario number one, Mike.
[Mike Johnson] Okay.
[David Spark] You find out your product team has been using an AI provider that your team did not vet. What’s more, that provider just had a breach. You don’t know what, if any, confidential data was sent to the provider, nor how they protect it.
[Mike Johnson] Okay.
[David Spark] All right, so who knows what happens. That’s scenario A. Scenario B, you find out your product team has rolled out your own AI feature. The bad news? You found this out because it looks like the AI was breached through a prompt injection attack.
You don’t know yet what, if any, confidential data was exposed. All right. I want to point out that usually you go for the thing that you know more about, but in both cases, you don’t know what was exposed. So, you can’t lean on that, Mike, for this one.
[Mike Johnson] At the same time, it’s all relative, right?
[David Spark] Yes.
[Mike Johnson] So, one is, just again, to make sure that I understand the scenario and to also think it through.
[David Spark] Mm-hmm.
[Mike Johnson] One is your company is using a third-party LLM.
[David Spark] Correct. And you don’t know how data’s being handled.
[Mike Johnson] People are sending something to it.
[David Spark] Yes.
[Mike Johnson] And then a breach happened and so on and so forth.
[David Spark] Correct.
[Mike Johnson] The other is you’ve developed one yourself.
[David Spark] Right.
[Mike Johnson] And it has been breached, and you don’t know what’s been exposed.
[David Spark] It’s more or less breach within your four walls or outside of your four walls.
[Mike Johnson] Exactly. And so, the way that I think about this is what is the potential reputation impact of this? Again, bad scenarios. Great examples, Jason, like these both suck. But if I think about it’s somebody else’s breach or it’s my breach, while the data might be the same, the reputation impact is going to be greater if it’s my breach, and that’s what makes the headlines is my company is breached versus this other company was breached.
[David Spark] Now, let me also, I think there’s a good argument against that too. Erwin, do you agree or disagree with Mike?
[Erwin Lopez] I agree to a certain point. I think there’s a little bit more to it, specifically given the fact that if it’s my breach, it could lead to other nefarious activities like lateral movement and so forth, depending on what the capabilities were for the LLM and how it was cordoned off.
So, one could be worse than the other.
[David Spark] Well, hold it. Now I’m going to throw this out. If it’s your breach, yes, that’s a reputation impact. But at the same time, it’s your four walls. You conceivably could manage it better. If it’s outside of your four walls, it’s out of your hands, who the heck knows?
Or you could look at it it’s a breach, a breach, it’s out of our hands anyways here.
[Mike Johnson] Well, again, if we’re back to we don’t know what data is within either…
[David Spark] Right, in both situations.
[Mike Johnson] …in both situations, then the impact to data is the same at the end of the day because it’s basically everything, for all that you know. But I really think Erwin makes a really great point, and it’s not one that I thought of. If it’s within your own four walls, it could be even greater than just the prompt injection because it could spread within your environment and become an even bigger breach than just the information that the LLM had access to.
[David Spark] Good point. So, agreement, but disagreement on how bad it is, I think is what it is. But you come to agree with him. Now, I was just thinking, we’ve been doing “What’s Worse?” scenarios for more than seven years. Have you actually had an incident in the last seven years that mimicked one of the “What’s Worse?” scenarios we’ve had?
[Mike Johnson] I can’t think of any.
[David Spark] I know. It’d be hard to catalog because there’s been literally hundreds of them.
[Mike Johnson] Yeah.
[Laughter]
[Mike Johnson] There’s literally hundreds of “What’s Worse?” scenarios that we’ve been through, but part of that is because these tend to intentionally be extreme.
[David Spark] Although, every now and then we get one who goes, “Oh, yeah, this happens all the time,” kind of a thing.
[Mike Johnson] Yeah. And those are the ones that are more likely, but the majority of them are intentionally extreme so that we can really have a nuanced debate.
[David Spark] Mm-hmm.
[Mike Johnson] Because if one is extreme and the other one isn’t, then it becomes easy.
[David Spark] Yes. Well, that’s true. That’s the thing. If you send them in – by the way, please send in more “What’s Worse?” scenarios. We’re always looking for good ones.
[Mike Johnson] Please.
[David Spark] Yeah, you’re always looking back. And they could be balanced weak or balanced extreme. They don’t have to both be extreme. Two balanced “weak ones,” I’m using that relative, is also still a very good “What’s Worse?” scenario.
[Mike Johnson] Yeah. I think those are actually the ones that make us squirm more because it’s, “Oh, that hits a little bit too close to home. That actually could happen. Now we have to have a real debate about it,” versus the extremes.
[David Spark] Erwin, in times that you’ve listened, have you ever heard a “What’s Worse?” scenario and go, “Yep. That happened to me”?
[Erwin Lopez] [Laughter] In the past, yes. [Laughter]
[David Spark] Can you recall anything?
[Erwin Lopez] Not off the top of my head, but I remember I think a few times that it actually kind of came up that I was like, “Oh, great. Yeah, I think we’ve gone through that.” I mean, we’ve all gone through big incidents in the past where somebody’s clicked on something, and things moved laterally as such and took us a little bit of time to identify before we actually had the remediating, so.
[David Spark] I remember years ago, Mike, we had a scenario that a woman sent in that was an actual incident that she dealt with, and they were faced with two issues, and one of them was letting the attacker keep going on so the FBI could track them, and the other one was to shut them down to stop the bleeding.
And it was one of those “What’s Worse?” scenarios, and I would love it if our audience have some of these locked and ready to send to us, but it was one of those “What’s Worse?” scenarios where I could tell the answer to the story at the end. Like you got to pick which one to go with.
And I said, “And the end of the story is this.” And by the way, the end of the story in that case was they did let the attacker go on so the FBI could track them. And I remember you and Sounil Yu was on that, and you gave each other a high five because we recorded it in my house, I remember that.
[Laughter]
[Mike Johnson] You’ve got a better memory than I do, David. But that, yeah, absolutely. That is something that if ever you’re dealing with a real adversary, that is a decision you have to make.
[David Spark] Yeah, it’s a tough one.
[Mike Johnson] That’s a real one.
[David Spark] [Laughter] You don’t want them to keep with the F… I mean, I will just say I’m sure that hurts. By the way, Andy Ellis had a similar experience where he was working with the FBI. He had an adversary that was working for him. He had to keep putting them on the payroll [Laughter] so the FBI could track them.
It’s even worse.
[Mike Johnson] Yep, that’s painful.
[David Spark] That, you got to look at the person every day.
[Laughter]
[Mike Johnson] Yeah.
[David Spark] Oh. By the way, that’s a serious poker face you need to hold on to, too.
[Mike Johnson] I don’t want to play poker with Andy.
Once again, we’ve got identity issues.
23:40.919
[David Spark] “Threat actors aren’t hacking in anymore. They’re logging in.” That’s by Tom Etheridge from CrowdStrike in a recent CSO Online piece. By the way, he’s not the first one to say that. Many have said that. But we’ve heard it many times, yet attackers can establish a foothold and start moving laterally through networks before most secure teams even know they’re there.
That is a major problem. So, traditional static defenses like regular passwords and perimeter firewalls just aren’t effective as they used to be. If attackers are walking through the front door with stolen credentials, what’s the new playbook look like, Erwin?
[Erwin Lopez] So, we use the assume-breach high-fidelity surveillance to start out with. I mean, basically we understand that the traditional hardened perimeter, where we’re hard and crusty on the outside, soft and chew on the inside, no longer works, right?
I mean, basically bad guys are coming in fairly simple. So, utilizing a multi-factor authentication that’s phish resistant, utilizing our specific tool sets internally to be able to identify. So, like for example, our EDR capability plus identity protection, and our honeypots that we have set up to kind of detect that lateral movement as such are things that really kind of come to help us quite a bit.
In addition to that, having our specific business rule sets that we have set up to identify abnormal behavior from users is something that we have been concentrating on for the last couple of years. But again, it’s really the focus has now become the inside and how do we protect from a zero trust capability, right?
Assume basically breach, verify everything that’s going on from the inside itself.
[David Spark] All right. Mike, I throw it to you. It seems this kind of new playbook is not so new anymore. It’s pretty well known, yes?
[Mike Johnson] All right, there’s basically these actors out there that they’re called initial access brokers, and all they do is go around and collect credentials. They collect credentials, they collect session tokens, what have you, and then they sell them.
And that’s been going on for years now. And there’ve been so many breaches who’ve happened because of somebody bought a credential from one of these. So, this is not at all new.
[David Spark] By the way, are those initial access brokers, are they ever taken down? Because it seems they’re very much removed from the actual attack.
[Mike Johnson] They are usually somebody who has a strain of malware that they’ve managed to get spreading, and quite often those particular types of malware will be taken down.
[David Spark] Well, more I was thinking like cyber crime is kind of a low risk, high reward type venture. But I was thinking people who are in that aspect of cyber crime selling the credentials are often probably kind of removed from any sort of, I guess, criminal prosecution.
Yes or no? I don’t know.
[Mike Johnson] I don’t think they’re the ones that law enforcement goes after first.
[David Spark] No, but it would just be a good for, you know, this is like going to the source.
[Mike Johnson] Yes.
[David Spark] Let’s find out who’s making the drugs. Let’s start with them.
[Mike Johnson] Yes.
[David Spark] Then getting the dealer on the street.
[Mike Johnson] It’s how do you deal with your precursors rather than the actual drug manufacturers themselves? There have absolutely been some takedowns of them.
[David Spark] Okay, all right.
[Mike Johnson] But I think they are generally harder to get to.
[David Spark] Oh, I would assume so.
[Mike Johnson] For the exact reason that you’re saying, is they’re not the ones who are doing the actions on objectives, and therefore they’re not as loud and as visible to law enforcement and leaving as much of a trail.
Are we creating more problems?
27:30.600
[David Spark] “We build, we bond, and we can’t bear to let go. That’s a good summation of the IKEA effect from Steve Thomson of TJX Companies. We tend to overvalue things we’ve created ourselves even to our detriment. Ross Haleliuk of Venture in Security noted that security professionals often fall in love with their custom SIEM rules, SOAR playbooks, and detection logic.
They spend months crafting even when better off-the-shelf solutions emerge. Thomson lived this firsthand, spending three years trying to move his organization off a creaking on-prem Splunk deployment. The problem was the emotional investment in all those custom configurations made change feel impossible.
Letting go felt like admitting their work was worthless. So, Mike, as someone I know who likes to engineer first, you are very much in a build yourself when it comes to build versus buy. How do you determine when you need to reevaluate your decision to build, and have you ever ripped out something you and your team spent forever developing?
[Mike Johnson] Absolutely you have to make the reasoned decision in a build-versus-buy scenario, right? I’m not going to go and write my own EDR. That doesn’t make a whole lot of sense. But at the same time, there are other areas where I get value that it is first build the thing, and then that pays off for years down the road.
And those are the ones where we make the “Yeah, we should go build this scenario.”
[David Spark] No, but have you literally taken something down that you built yourself?
[Mike Johnson] Oh, yeah. You have to. You have to say, “Hey, this thing has outlived its usefulness.”
[David Spark] And does it hurt?
[Mike Johnson] No.
[David Spark] Because it’s like you watch your child grow up and it’s like, “Oh, okay, well.”
[Mike Johnson] So, you mentioned Sounil Yu earlier.
[David Spark] Yeah.
[Mike Johnson] Years ago, he introduced me to this term of pets versus cattle.
[David Spark] We did this on our show. I think he reintroduced it to both of us at the same time on the show.
[Mike Johnson] Yes, and that’s something that has always stuck with me. And these things that we go and build internally, we have to think of them as cattle. When they outlive their usefulness, it’s time to move on. We shouldn’t be attached to them.
[David Spark] I think the term you want to use is slaughter.
[Mike Johnson] I did not want to use that.
[David Spark] [Laughter]
[Mike Johnson] But going back to the scenario of the rules within Splunk, that’s not an example of build. That is example of lock-in to a vendor that you bought. That is actually one of the downsides of buy that you have to go into those decisions being very clear on.
If those rules had been written in a portable fashion that were easy to move on, they wouldn’t have been stuck on that on-prem Splunk environment. So, sometimes buy actually is the one that’s more difficult to move on from than a build scenario.
[David Spark] All right. Erwin, I’m sure you’ve faced this problem yourself. Did you ever feel pain getting rid of something you and your team built?
[Erwin Lopez] Sometimes it’s hard. I mean, I’ll be honest. You’ve put a lot of love and emotion and time into this, right? I mean, these are solutions that have gone through battle scars, right, from past incidents, business intelligence that you’ve added to it and so forth.
And then when it really starts to become a problem is when you start to have to pay the maintenance tax on it. How long is it taking to upgrade or keep up, right? What is the total cost of ownership of this? And then lastly is do we have a single point of failure?
Meaning how many people are actually managing the system? Do we only have one person or two people? And that’s really where for us, we make that decision to say, “Okay, well, maybe it might be time for us to look at an external solution that we might be able to basically move our rule sets and so forth.” But during that timeframe, it can be very painful, and I’ve had lots of experience going through that.
[David Spark] I want to talk about this just in general about rip and replace because I was having this conversation over at Black Hat. We’ve all faced a moment of rip and replace, and you see a better solution, but sometimes you think to yourself, “Is it really that much better?
Would it really make it…” And you sort of hold out, you hold out, and then you realize the sort of the getting better and how bad is sort of that gap is widening even more. And you’re like, “Okay, well, now we have to do it.” And then you also have the fear of making that leap because you know how much work it’s going to take to do it.
But once you get onto the other side, you’ve done the work, you get to the other side, you’ve replaced the thing that was kludgy that wasn’t allowing you to do it. You look back and you think, “Why didn’t I do this earlier?” Yes, Erwin, have you had that feeling?
[Erwin Lopez] Well, we have. We have. There’ve been times that we’ve gone through it. I mean, good example, we had a great SIEM capability that we had created that we had gone through, that it’s gone through multiple different iterations, and we’ve basically kept that up for years.
And just because it took a lot to keep this up and running, we ended up moving into, for example, a Splunk capability, right? That made it a lot easier, right, and gave us greater visibility to be able to not only search but also utilize it as a pre-SIEM tool.
[David Spark] Well, excellent.
Closing
33:01.338
[David Spark] Well, that brings us to the very end of this episode. I want to thank you, Erwin Lopez, who’s the CISO over at the SLAC National Accelerator Laboratory joined us. That was fantastic, Erwin. We’d love to have you back again. 1
[Erwin Lopez] I’d love to be back. And again, thank you for the invitation.
[David Spark] Of course. And Mike, thanks for making the introduction to Erwin as well. And I want to thank our sponsor. That would be ThreatLocker. They’ve been a phenomenal sponsor of the CISO Series, delivering zero trust in action. Remember, go to their website, threatlocker.com/CISO.
Remember to add that slash CISO, easiest way to let them know that you found out about ThreatLocker from the CISO Series. Thank you again, Mike. Thank you, Erwin. And thank you to our audience. We greatly appreciate your contributions – keep them coming, more “What’s Worse?” scenarios – and listening to the CISO Series.
[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows, Super Cyber Friday, our virtual meetup, and Cybersecurity Headlines Week in Review.
This show thrives on your input. Go to the participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David at CISOseries.com.
Thank you for listening to the CISO Series Podcast.