We know that phishing is a major threat vector. So why does it seem like phishing tests only make things worse?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark, the producer of CISO Series, and Edward Contreras, senior evp and CISO, Frost Bank. Joining us is Mark Eggleston, CISO, CSC.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Scanner

Full Transcript

Intro

0:00.000 

[David Spark] We know that phishing is a major threat factor. So why does it seem like phishing tests only make things worse? 

[Voiceover] You’re listening to Defense in Depth. 

[David Spark] Welcome to Defense in Depth. My name is David Spark, I’m the producer of the CISO Series. And joining me as my co-host for this very episode, it’s none other than Eddie Contreras, senior EVP and CISO over at Frost Bank. Eddie, say hello to the audience.

[Edward Contreras] Hello, audience. And David, thank you for having me back yet again. 

[David Spark] We love having you back. Our sponsor for today’s episode is Scanner, the security data layer for the era of agents. We will talk about just that a little bit later in the show. Let’s get to today’s topic at hand, Eddie. No one likes phishing tests, but are they immoral?

Dan Desko of Echelon shared a conversation with the CISO that made the case saying that a phishing test breaks trust with users. Now, is this a “What’s Worse?” game? You phish your employees or you don’t, which one is worse? Can phishes be done and still build trust with the security team?

It seems like they’re mutually exclusive. And can you not phish yet still build a resilient staff to phishing? Eddie, your thoughts?

[Edward Contreras] It’s funny that we’re still talking about phishing. I’m going to jump on my Wayback Machine just for a minute here, David. Mid-’90s, first phishing attack, right? AOL, someone wanted to not pay the $20, and so they use that AOL instant messenger to say, “I’m your support team.

Give me your credentials.” And 30 years later, here we are talking about phishing. The good thing is it’s evolved on what we have at our disposal from a tool perspective, but man, the tools aren’t always that great. And so, that’s the debate here. What tools do you use?

The testing, the mechanisms, and even how efficient is it? I looked at these quotes, and I even clicked on that link and I’m like, “Oh, I could stay on this link for about a good two hours,” because everybody had a great point, and it really just depends on the program.

So, I think it’s one of those, “It all depends.” And I know that’s a really bad answer. 

[David Spark] You know what? I know plenty about phishing and yet I technically have not been tested, and since we talk about it all the time, I’ll admit to something. I actually got phished last week, and it was a bad one. Like they got my email. But it was one of those things where I understood it because we talk about being heavily distracted and I was in monster heavy distraction mode.

Like, I dropped my car off, I was in a cafe. There was something that looked familiar, got it done, I was on the coffee house network. Like, it was a bunch of things I wasn’t paying… But the moment it happened, I immediately knew what I had done.

[Edward Contreras] It’s that Monday morning quarterback. 

[David Spark] But the good thing is I immediately went into fix mode and literally, I solved it in 60 seconds. Luckily, they didn’t have any automations in place, and I did change my password, put on MFA. I covered my butt. But man, it scared the crap out of me.

It really scared me. 

[Laughter] 

[Edward Contreras] Better than caffeine. 

[David Spark] Yes. 

[Crosstalk 00:03:08] 

[Laughter] 

[David Spark] It was like I could have just gotten upset and panicked but it was like, “No. Fix this now. Right now.” And the speed is always, I know, a key factor here. Anyways, we have a great CISO to bring in on this discussion for us to all discuss this.

I’m thrilled that he’s here. It is the CISO for the CSC, none other than Mark Eggleston. Mark, thank you for joining us again. 

[Mark Eggleston] Thank you, David. Thank you so much for having me, really appreciate it. 

Why are we blaming users? 

3:35.713 

[David Spark] Eirick Luraas of Raytheon said, “Phishing tests do not work. Educating employees works. It’s the education part of the phish test that works, not the phish part. The phish is a huge violation of trust and it creates an adversarial feeling between employees and security.” I will say, this is one of the major arguments against it, what Eric just said.

And Ryan Whalen of BSE Global said, “Catching people making a mistake and forcing them to sit through a mandatory 20-minute training because they clicked a link isn’t how you build a positive feedback loop. Most people today understand that punishing your dog for bad behavior doesn’t teach good behavior.

It just teaches them to hide their mistakes. People work the same way.” I think that’s kind of a good point. It’s like people don’t want to tell you they screwed up. They want to run away is what they want to do. How do you teach them not to run away and hide and be actually excited to tell you something happened, Eddie?

[Edward Contreras] Imagine sitting in front of your CEO trying to explain why productivity is down because employees are spending 20 minutes or longer on a phishing test, right? That’s brutal. I know times have changed. My hope is Ryan’s quoting something of an experience from the past and hopefully a long ago past.

Most phishing events now, you’re in and out of it in 30 seconds, maybe a minute, but 20 minutes is egregious. And I think you have to understand the goal of a phishing test, to your point, David, is are you learning something from it? And if it takes you 20 minutes to teach somebody, the conversation is what are you truly teaching them?

[Laughter] 

[David Spark] That’s a punishment at that point. [Laughter] 

[Edward Contreras] It is, it is. I mean, maybe the first two minutes, maybe the first three slides, but after that, you really are just bringing the hammer, and you can just picture like a principal with a ruler just smacking their hand as you read through your phishing test.

It truly is a punishment at that time. 

[Mark Eggleston] I would say also that’s like drawn and quartered, right? Not only did you fail the phishing test, but now we’re going to also subject you to 20 minutes. That’s just cruel and unusual punishment, I would think. David, you started off, too, saying that you got caught in a phishing test.

[David Spark] It wasn’t a test, it was a real phish! It was an actual, “Oh, I got nailed.” But I knew immediately what had happened, and I think that’s the thing is the awareness that I knew it, that it happened. Because first of all, it was one of the things, I was barely paying attention, but all the signs were there.

Like, I could read them all, but I just was like moment of weakness. 

[Mark Eggleston] Well, when you get that sensation, it’s like you have a instantaneous, you have hyper laser focus, like, “Oh, my gosh. Now I see it all,” right? And that’s a good thing. But I think what you said earlier, too, is like when you were asking our co-host here, what is it that triggered it for you?

And I think one of the bigger things about leadership that makes people see these things is when you have someone who has a self-deprecating sense of humor about themselves. And so, when I think about a phishing test, people should be clear that the CISO has been fooled on these things, too.

I often open up when I have sessions with people. I say, “Look, we’re not here about making people feel bad. I, too, have fallen for the phish, and here’s why.” It’s about getting better, getting more resilient. And it kind of puts them at ease, and then you can have more of the conversations, just hopefully not 20 minutes.

[David Spark] And by the way, that is kind of, I think, key, is when security professionals admit to being fallible themselves. And not just security professionals, just executives in general. Have yourself, Eddie, or others, spoken up about this? Like, “Hey, I fell for something once”?

[Edward Contreras] Oh, yeah, absolutely. And I think it is really about being human and think about reaching for a cup of coffee. And we’ve all done it. You watch it tip over in slow motion. You know where it happened. You know you can stop it. As an adult, you think, “Man, how foolish was that?

I just tipped over my cup of coffee.” But then we all have kids that drop cups of fluid all the time, and it’s like, okay, do you get mad at them? No, it’s life. And that’s what phishing is. 

[David Spark] That is a great example, the spilled cup of coffee. We all know how to hold and place a cup of coffee, but we spill it all the time. 

[Mark Eggleston] Yeah. I’d also add, David, too, that phishing tests or phishing clicks are not created equal. Most people have to open up an email to see a little bit more about it. Sometimes they might click the link. They might even open up the attachment, but it’s that third level when you start to enter your credentials.

That’s like the holy sin of phishing simulations, and that’s the ones that you really want to make sure you stop people from doing. 

There must be a better solution. 

8:15.195 

[David Spark] Tommy Ward of Waymark said, “I don’t do phishing tests on my users and won’t do so. My belief is that it is detrimental to building and maintaining trust. What I do instead is make sure they have defense in depth, utilizing technical controls to minimize the fakes that get through, privilege restrictions on workstations, endpoint protection, FIDO2 for MFA, backups, and incident response/recovery plans.” Andrew Kirch of Stoic Cybersecurity said, “I am aware of a study that shows that employees subjected to phishing tests are roughly half as likely to notify someone of an indicator of compromise, fearing they will get in trouble.

A better technical solution is single sign-on, two-factor authentication, and conditional access policies.” What Andrew says and Tommy says, it’s kind of what we were talking about, this fear of we’re creating our employees into adversaries. Isn’t it, Mark?

I mean, we definitely don’t want to do that. 

[Mark Eggleston] We don’t. I think it’s a balance, right? And all those controls that both Tommy and Andrew mentioned are really, really good ones, ones that we’ve used before. We had a code name for it. Internally, we called it Zscaler Jail. So, you can imagine what it was.

It wasn’t necessarily a sanction. It was a control, and that control limited folks to only business-applicable websites once they had a series of phishing failures. So, those technical controls work quite well, but they’re not meant to be punitive.

They’re meant to be protective, and that’s the big difference when you sell them. 

[David Spark] Eddie, your thoughts on this? Is [Inaudible 00:09:53] half as likely to do that, but is there a way to phish them, but let them know that they will not get penalized if they report? Like what we were just talking about in the last segment?

Is there a technique of just saying, “Hey, guys. Let me explain to you how it works when you tell us. We’re happy that you told us, not upset.” [Laughter] 

[Edward Contreras] It’s a reward system, right? 

[David Spark] Yeah.

[Edward Contreras] I mean, that’s really what you want to do. The quicker you can do it, the better the reward. But if I can talk to Tommy, to Andrew, you have to really think about it this way. As a CISO, you should assume at some point in time you’re going to sit in a courtroom, and you have to defend your program.

And if you have to defend your program, did you do all that you could do to protect your company? And if something as simple as a phishing test demonstrates due care, it’s a good test. Now, how you deliver that test, that’s up to the CISO to craft that.

But if you say, “I just didn’t do it because I don’t want to do trust,” or “I don’t want to do friction,” or “You know what? I got a report that says the numbers are not there.” You have to defend your position, and if you’re comfortable defending it, by all means, do it.

But you have to be comfortable to defend that position. 

[David Spark] That is an excellent point. That never comes up of you’ve got to prove you actually did something, not rather, “This is my philosophy in security.” 

Sponsor – Scanner 

11:11.776 

[David Spark] Before I go on any further, let me tell you about our great sponsor, and that is Scanner. Now, here’s a question for you. How much of your security data can you actually run detections on right now? I mean, all your security logs end up in cloud storage like AWS S3, but only maybe 10% also goes to your SIEM.

Splunk and Datadog are just too expensive for the rest. So, your detections only cover a slice of what’s actually happening. 

Scanner changes that. It’s a security data lake that indexes your logs directly in cloud storage and runs real-time detections on all of it. No pipelines, no re-ingestion, no scheme work, hundreds of detection rules on 100% of your data, not just the small subset that fits in your SIEM.

And when you need to investigate, searches come back in seconds, not hours. Teams at Ramp, BenchLink, and Lemonade use Scanner for detections, threat hunting, and incident response, and because queries come back fast enough to iterate on, most of the query volume in Scanner now comes from AI agents.

They do the log diving. Analysts, they make the decisions. It’s 100 times faster than traditional data lakes, 10 times cheaper than traditional SIEMs, and it’s loved by analysts, and it’s built for AI agents. You can check it all out at scanner.dev.

Go check them out and let them know you heard about them from the CISO Series. 

What should we be measuring? 

12:53.140 

[David Spark] Chris Spohr, CISO over at Republic Finance, said, “I have a contrary view. I phish test all employees monthly. You can train people all day long, but if you don’t measure the effectiveness by practical testing, then you are creating a false sense of security.

It took a period of time, but my phish scores are extremely good, which confirms my security awareness program is effective.” 

And Eric Benante of BAYADA Home Health Care said, “Social engineering and targeted phishing campaigns are considered to be the most effective attacks on businesses due to the success rate in compromising user credentials. If you’re a CISO whose company is compromised due to a phishing campaign, and there’s evidence that you haven’t taken the most basic steps to mitigate that threat,” per what you were saying, Eddie, “I could easily see you losing your job or worse.” The worse being being in a courtroom, there you go.

Eric goes on to say, “Internal phishing campaigns also provide you with metrics on how susceptible your user base is to phishing attacks.” 

And lastly, Joshua Cloud of NFI said, “If you can’t measure something, you can’t manage it. I don’t think testing the efficacy of a training program is an immoral practice at all. What you do with the results of that test is a different debate, but to not even know whether your training is working or needs adjustment seems silly to me.” All right.

So, all these quotes, the first two segments, people were against it, these people are very, very pro phishing, but they make it clear, like you got to do something, and you got to measure it, kind of like the rest of your security program. Yes, Eddie?

[Edward Contreras] Joshua may have an audit background, I like his quote. I was trying to unpack that a little bit. I was like, “Okay, there’s some audit language here,” and I’ll tell you, there’s a lot of similarities in these quotes if you try to unpack what they’re saying here.

By rewarding your employees for good behavior. And good behavior could be fast reporting, it could be transparency. It could be the fact that you saved the company by disclosing something. Those are simple things to do. And back to your example earlier where you got phished, David, by going through that and saying, “I fixed it within 60 seconds,” that’s a reward, even though it started on a bad footnote.

You can show and demonstrate we got through it. 

[David Spark] By the way, I was really shooken up after it. I should point out, I admitted to my team members that I did, and I was trying to focus on thinking, but I was shaken up. I’m like, “Holy crap. The house of cards could have come down!” [Laughter] Like, that’s really bad.

[Edward Contreras] And think about this, David, as well. Think about somebody who’s a junior employee in your program, who’s never been a part of security, that actually gets truly phished. Even if a test. They’ll have that same reaction. And so, you have to be able to understand what they’re going through and say, “Do I want to give them that 20 minute test?” No, I don’t.

I want to bring them back to, “Hey, let’s make this a positive experience.” 

[Mark Eggleston] Great ideas here. I mean, sentiment scoring’s another thing you can measure. Net promoter scoring, right? Create a speaker’s bureau and mark how many times that your group gets asked to present at other people’s town halls, other department sessions.

All those things are good things to measure too. You got to balance it. I mean, a lot of what we’re talking about here is behavioral. There’s a carrot and there’s a stick to use. So, make sure you’re using some good metrics to measure the good things you’re doing, and I think there’ll be a lot more forgiving when you go and then do a phishing test, and then you have to mandate training, or you mandate some internet restrictions, some of the other things.

It’s all about social exchange and making sure you have a credible reputation for your department, and it’ll be a lot more forgiving when you do the phishing test because as Joshua notes in here, you better measure these things. It’s really important.

[David Spark] And I want to go back to all of them were saying, like for all the people who are poo-pooing phishing tests, I’ll mention Mike Johnson, CISO of Rivian, my other co-host, he’s very much against it. If you’re not going to phish them, how else can you measure them?

Like, you got to measure how much stronger they are. I mean, heck, when you go to school, you take tests and you get grades on them. If you’re not going to test them, how else do you plan on grading them? I don’t know. I mean, is there another way, Eddie?

[Edward Contreras] Yeah. And think about like a car company. If you sit in a car, what happens when you turn it on and you put it in drive? The doors automatically lock. Why do they lock? No one’s going to steal nothing when you’re in there. It’s just habit.

It’s a very easy control to implement. It’s something that is second nature to the vehicle. Now, if a car didn’t auto lock when you put drive, and all of a sudden, a kid inside the car opens the door, well, why didn’t you do the auto lock? It’s such an easy feature to deploy.

[David Spark] This is to protect us. We’re talking about when the technical features fail. Like the auto lock is a technical feature that was added later. Something we used to do manually added later. So, that was to protect us like what you were saying.

But what happens when there are certain things we can’t fix with a technical control and we need humans? If we’re not going to phish test them, how do we know when they’re getting any better? Mark, I’ll let you have the last thought on this. 

[Mark Eggleston] Sure. There’s other solutions, right? UBEA has been out there for some time, User Behavior Entity Analysis. There’s also companies that have taken it a step further, said like, “Hmm, are you sure you want to add that new distribution list in this email you’re just about ready to send?” because it’s a colossal distribution list.

You might be over-permissive about what you’re going to share. Those are good solutions too. I mean, you can also score your LMS, your Learning Management System training, to get you some level of feedback on how your human firewall, forgive me for saying that, is operating.

But UBEA is a really good one too. And there’s other things you can also do instead of this that are beyond the human level. Like how about we make sure that we’re using phishing-resistant MFA, right? There’s a lot of other things you can do to harden to still reduce the susceptibility or increase your phishing resiliency.

How do we determine what’s most important? 

18:57.421 

[David Spark] Matt Seitz of the Santa Clara County Office of Education said, “I get the concern about tricking/trapping employees. Think of it as a fire drill. A drill isn’t about tricking employees into thinking there’s an actual fire. It’s about practicing so that people know what to do when a fire actually happens.

And if employees fall for the phish, view it as a training failure rather than an employee failure.” So, technically I’ve never been phish tested, but I definitely knew what to do after I got hit. Let me get one more quote in here. Emre Saglam of ANyscale said, “Instead of focusing only on preventing phishing, I believe it’s just as important, if not more, to teach employees what to do after they realize they’ve been phished.” Kind of like what I did.

“Assume breach. Someone at some point will click. What matters most is how quickly they respond and how comfortable they feel reporting it.” That is the quote that I think every CISO can get behind. Yes, Mark? 

[Mark Eggleston] Yes, these are strong quotes indeed. Practice makes perfect. I really like the idea of the fire drill. That really resonates with me, too, because that’s what you do. No one wants to go out in the winter and stand in the parking lot for several minutes.

[Laughter] 

[Mark Eggleston] But you got to do those things to make sure that you know how to do this and make your company more safe. The phishing simulations, to be proactively addressed, you’ve got to do some more of these testing, and you’ve got to make sure that you’re keeping ahead of that.

The other thing I’ll share here, there’s times where we’ll have some things coming into the SOC that look like it could be something related to phishing, but then I’ll get an email or two, or I’ll get a direct message, and I’ll use that bit of telemetry to then alert the SOC, and I say, “Wow, I think you’re on to something,” or “I think there is something else going on here.” So, you never know when that human report might be the extra little bit of evidence to help correlate something much deeper and much worse.

[David Spark] I want to just double down on the very last line, which I think I’ve heard many CISOs say – and I think it’s kind of a nice sort of conclusion and button to this – what matters most is how quickly they respond and how comfortable they feel reporting it.

I mean, forget all the phishing tests. If you could just get employees to that point, wouldn’t you feel really good about your security program and your team? 

[Edward Contreras] Oh, absolutely. We have a reward program, and we’ve implemented it at every company I’ve been a part of. You reward the employee. It’s worth the investment. We have a budget line specifically for that. It’s can you not only acknowledge them, reward them, but then praise them on a pedestal?

But you have to include not only their direct manager but their highest level executive and talk about, “Here’s what they did. Not only did they save our department, they saved the company.” And you have to be able to praise them in a way that the company recognizes – “We have a very efficient program, and this is a good example of how it’s working.” 

And so, I think that is something that companies need to really invest in is really understanding how to deliver that praise in a way that just reinforces that muscle memory. Because if you don’t do it and it goes unnoticed and it goes unrecognized and it’s a silent nod, it’s not going to be something that just is easy to report on.

People are going to look at it and it’s like, “Oh, nothing happened,” or “I feel bad. Maybe it wasn’t a great thing.” And so, you do have to shine a very spectacular light on it. 

[David Spark] I love the fact that you have a budget line item for just rewards. Mark? 

[Mark Eggleston] Yeah, I really like that too. When you’re incentivizing people to do that, that’s positive reinforcement. Believe it or not, that positive reinforcement is what helps shape human behavior much more effectively in the long term. The punishment only works for a short time, and you don’t hear that much with operant conditioning and other psychological principles being put into this, but that’s the truth out there.

So, yeah, I agree with Eddie wholeheartedly. 

Closing 

22:47.640

[David Spark] Well, this is excellent. I really enjoyed our conversation21, and I want to bring this to a conclusion right here where I liked pretty much all of these quotes. Everybody on both sides of the aisle makes a really good point. People who are pro-phishing and people who are against it.

But I do want to mention that I like what Emre said at the end, the very last line, what matters most is how quickly they respond and how comfortable they feel reporting it. But I’m going to start with you, Mark. Which quote was your favorite and why?

[Mark Eggleston] I’m going to give a shout out to Joshua Cloud. He’s a fellow peer in the Philadelphia network, but not just because of that, but his quote, I think, was very balanced and very practical, right? 

[David Spark] Remind us what he said. 

[Edward Contreras] Joshua Cloud was basically noting that it’s what you do with the results of that and you should be doing this because you have to have that measurement out there. So what do you do with the results? You shouldn’t be punitive, like we talked about earlier with the 20-minute training, but you should be measuring this, and you should be doing more positive things.

It’s kind of like social exchange theory. For every negative thing you might do, like a phishing sanction or a phishing restriction, make sure you’re doing at least three things that are more positive, too, and your workforce is going to continue to build trust, and they’re going to come to you and feel more comfortable learning that.

And that’s the other thing here too. I think we have to make sure that we’re focusing more on that reporting rate. Everybody focuses on a click rate. Not enough people are focusing on what’s the best percentage to have a report rate. Let’s get it way up there.

[Edward Contreras] That’s a good point. Eddie?

[Edward Contreras] it’s hard to go against Emre, and I agree, phenomenal, phenomenal quote. It’s how you come through the event. But I’m actually going to go with Matt. I love the fact that you build muscle memory, repetitive, repetitive, repetitive.

I’m a military person by trade, and you do things every single day until it just becomes muscle memory where you can do it in your sleep. And if you do that when it happens, it’s just second nature. So, yeah, I love Matt’s quote, that it is muscle memory.

[David Spark] I like your analogy, Eddie, about the spilled coffee cup. We’re all capable, but we also know how to clean it up. But we’ve all spilled a cup of coffee. It doesn’t mean that we’re incompetent. It just means we’re human, and we do make mistakes.

And we will make this mistake. It’s just knowing how to clean it up is really the trick. I think that’s it. It’s like maybe a little bit more testing of do you know what to do when this happens, and I think that’s what it comes down to. All right. Huge thanks to our sponsor, and that’d be Scanner.

Remember, it’s a security data lake that indexes your logs directly in cloud storage and runs real-time detections on all of it. Check them all out at scanner.dev. And let them know you heard about them from the CISO Series. I want to thank both Mark and Eddie.

Let me ask, both of you, are you hiring over at CSC and also at Frost Bank? Mark? 

[Mark Eggleston] Yes, indeed we are. We have a talented workforce and always looking to continue to add people to a great growing company. So, go on out to our website and check it out. 

[David Spark] All right. I’m assuming you have a Careers page of some sort, yes? 

[Mark Eggleston] We do. 

[David Spark] All right. 

[Mark Eggleston] We do. We have a Careers page. 

[David Spark] And we will have the link to Mark’s LinkedIn profile. So, if you want to reach out to him directly, say you heard him on the show, maybe it’ll help. Can’t guarantee it. I know Mark would like to hear from you. Eddie, are you hiring over there at Frost Bank?

[Edward Contreras] We are absolutely hiring, and we’re hiring in innovative sectors as well. If you want to be a part of an AI security team, please come over to our Careers page and absolutely reference me. 

[David Spark] Ah, definitely. And again, link to Eddie’s LinkedIn profile will also be available on the post for this episode. All right. Huge thanks to both of you. This was a really, really good discussion. And I want to thank our audience, as always, I say it all the time.

I do truly mean it. It’s not me just blowing hot air. I truly appreciate your contributions and for listening to Defense in Depth. 

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.

If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.