If all it took was effort to keep an organizations secure, we’d see a fewer breaches. Our industry doesn’t suffer from a lack of work ethic, but are we spending those energies without understanding what matters most?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark, the producer of CISO Series, and Steve Zalewski. Joining us is Terry O’Daniel, former CISO, Amplitude.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, SecurityPal
Full Transcript
Intro
0:00.000
[David Spark] If all it took was effort to keep an organization secure, we’d all see fewer breaches. Our industry doesn’t suffer from a lack of work ethic. But are we spending those energies without understanding what matters most?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark, producer of the CISO Series, and joining me on this very episode, he’s my co-host, it’s Steve Zalewski. Say hello to the audience.
[Steve Zalewski] Hello, audience.
[David Spark] And by the way, when you meet Steve, that is the way you have to greet him.
[Steve Zalewski] [Laughter]
[David Spark] Our sponsor for today’s episode is SecurityPal AI, 100x faster security reviews, powered by AI agents and expert humans. More about just that a little bit later in the show. But first, Steve, let’s talk about today’s topic at hand. “Attackers don’t care about your maturity model. They care about your misconfigured IAM role and the token your CICD forgot to rotate.” Now that quote came from Rinki Sethi, who’s a VP and CISO at Bill. Part of her larger point, that security teams don’t fail for lack of effort, rather, they’re chasing the wrong risks and alerts, which is a topic we hear all the time. By the way, there’s a slew of vendors trying to solve this problem. So, they don’t understand context is what the issue is. Do you agree with Rinki on this, that this lack of getting what matters most is the major problem that security teams face?
[Steve Zalewski] I think Rinki just really opened up a big can of worms because she just acknowledged a brutal truth, which is if what we’re doing isn’t working, then we better figure out a different way of tackling the problem. And that’s what she’s highlighting here is, folks, we have got to make some fundamental changes in how we think about success.
[David Spark] Very, very good point. Well, we’re going to talk about all of this with our guest today, a friend of yours, as I understand.
[Steve Zalewski] Mm-hmm.
[David Spark] He is the former CISO over at Amplitude, none other than Terry O’Daniel. Terry, thank you so much for joining us.
[Terry O’Daniel] Hey, thank you so much. It’s great to be here.
How do we approach governance?
2:13.373
[David Spark] Ayoub Fandi of GitLab said, “That’s where GRC can definitely help. A risk register that ties technical risks to engineering workflows. Even with good prioritization, if we don’t survey the stakeholder landscape, understand who’s moving the security needle and who we need to win over, we can have the right number one risk and still have no meaningful mitigation.” Very good point there. And Mike McGilvray of Prudent Technologies said, “It’s understanding what really constitutes material risk to the organization. If you’re not having conversations with your vendors and what’s important to you, and they can’t incorporate your context and priorities into their tooling, testing, alerting, and reporting, then you’re just doing the same old thing that isn’t working. There is a better way.”
All right. These two quotes match up, I think, quite nicely here because the first from Ayoub was saying, talk to your stakeholders. You don’t know the risks. They do. They know what is of great concern. And then Mike is saying, once you have that information, work with your tool providers to get it working better. What’s your take on this, Steve?
[Steve Zalewski] So, governance of risk and governance of compliance are two different things.
[David Spark] I must say, Ayoub mentioned GRC here. It was mostly leaning into risk here.
[Steve Zalewski] Yes. And so, I start with we’re focusing on risk now. That doesn’t mean that we protect everything equally, right? Risk is understanding what’s important, not everything that I can see. So, what I like about this governance conversation is what’s the value of a risk register? We’ve had them, but are we using them to explain to the business which lines of business are important to protect, and which are not? And therefore, the risk register is your demonstration of how you’re determining what’s important to protect, to be brilliant at the basics with regards to what’s most important to the company, not what’s most important to security.
[David Spark] That’s very important. What’s most important to the company? And the only way you can know that is you’re having frank conversation with the team, right, Terry?
[Terry O’Daniel] Yeah. I think there’s a challenge here in that most security professionals working in tech and software, which is my background, they don’t ask for a risk register. They asked, what are the top five things I need to worry about? What keeps you up at night? And I like Ayoub’s approach here because he’s not wrong. At the core, it is about understanding how to distribute this limited set of resources we have. How do we prioritize work? I think the term risk register has left a bad taste in a lot of CISOs’ mouths. And I think we have to find a way to get out of the weeds, get out of the details here.
And going into the second comment from Mike there about tooling, I think these do tie together because I think the challenge is we often start with the format we have, and most times for a risk register, it’s a spreadsheet. And we use that format and we let it guide us. Going back to Steve’s point, most security organizations don’t start from risk governance or the risk management lifecycle. They start from a business need, and that business need is almost always compliance. We need this certification to sell to these customers, to sell in this geo, etc. And that leads us to another point about how do you actually measure those risks so that you know, first of all, and you can tell your stakeholders where we’re succeeding and where we’re failing.
Where are we falling short?
6:11.825
[David Spark] Ashish Popli said, “There are clearly enough tools to do the identification part right of ‘what do we see?’ But turning that information into what will break us,” interesting take, “At the scale of an enterprise that constantly changes along with an evolving threat landscape isn’t always easy outside of obvious low-hanging fruit. So, the answer lies in modeling your attack surface in graphs and running algorithms that perform the analysis of what could break us, what has been attempted, what can I do to prevent it, and what can I do to detect?” All really good questions, starting with the what could break us. And Anthony Martin of Walmart said, “Security teams often have enough ‘tools’ but are limited in their effectiveness by ineffective communication of the really difficult subject matter in the appropriate business context.” Getting a lot of head nods here, by the way. “It definitely negatively impacts the trust that the core business has in the security teams.” So, I remember discussing this as well, this whole idea of just think about what are the things that could literally destroy us? That is really where you should start. Right, Terry?
[Terry O’Daniel] I think security is an existential risk to most businesses. It’s not fair. But from a security perspective, we have to get a lot of things right all the time, and when we get one thing wrong, it can have a catastrophic effect on our company, our customers. Even we can get in trouble with the law. Regulatory compliance can be an issue. And at the end of the day, we can do bad things and be poor stewards of data, whether that data is B2B or B2C. I don’t want to have our company show up on the front page of the New York Times for a bad reason. But also, I think security is about taking appropriate steps that match the risk appetite of the business. And I think that’s where sometimes we get into trouble as security professionals because we will tend to chase things that make sense to us.
Because I come from an infrastructure and production engineering background, I have a tendency to lean into DevSecOps, to lean into things like hardening base images, etc. But that might not be what business we’re in. So, I do think that starting by identifying the crown jewels, and that is not a security conversation. That is absolutely a business conversation. Let’s do a business impact analysis. That’s not a very [Laughter] sexy term and not a very fun, cutting-edge technical term, but it’s the truth. You have to have the business tell us what can we afford to do without, and what is absolutely essential to our operations and our reputation and our functioning. And if we don’t start from there, then sure, I’m going to show up and I’m going to say, “We need to make sure we take care of misconfigurations in our cloud surface and things like that.” And those are all probably true because I come to the table with that experience, but they may not be the real existential threats that that company should be worrying about in that business.
[David Spark] So, this topic has come up many times, Steve, before. And one of the things that I’ve found very interesting is the quote, “conversations happen,” but there isn’t a deep understanding. And this goes really to Anthony’s quote here of connecting the business concerns with the how that is getting protected through the cyber teams. We’ve gotten a lot of like, well, how long can you go without the internet, and shoot from the hip answers of like, “Oh, 24 hours, 48 hours,” like not even really thinking about it. And so, how do you go from that shoot from the hip response to the understanding the complexities of how the whole business, especially an enterprise, is interconnected and how you could sort of solve that or come to an answer of like how we can stay resilient up to, at worse, an hour of downtime. I mean, just making up a number, but whatever it is. Steve.
[Steve Zalewski] This gets back to what is the responsibility of the security organization? If the CISO believes it’s to build a maturity model into the organization and maximize maturity, and many of us do that, then what really Martin is calling out is your obligation is to understand the business better than the business knows itself because only in understanding the business can you figure out where security can have the maximum impact on what the crown jewels are. And nobody’s going to tell you what the crown jewels are because every line of business is going to argue they’re the crown jewels because they’re all arguing for their budgets.
[David Spark] Let me also just throw this in, and we’ve done a lot of these sort of first 90 days of CISO. You don’t learn that right away. They don’t go, “Oh, by the way, here’s all our crown jewels on a piece of paper.” That takes time to learn.
[Steve Zalewski] When I was at Levi’s, I’d tell people I came in thinking my job was to be able to provide security to the company. And my first 18 months was me understanding the creative process that goes into the jeans, the selling of jeans, and that a lot of that was just over time. Learning the business and touching the pieces and understanding how the creative process occurs is not just the quick 90 minute or 90 days, where’s your BIA? It’s you being able to reflect back, look, if I have to be brilliant at the basics and the human is my weakest link, then if I’m going to put a risk register together, it’s relative to what’s important to the business as I see the business. And if I’m not seeing it correctly, then I need to be corrected by the business so that my risk register adequately references the key posture I need to have. So, just that change of putting the business first, not putting security first, which is what I started with, is just something we’re starting to really come to grips with. And it’s what Rinki was saying, right? Which was, look, it’s not about maturity. It’s about being brilliant at the right basics to be able to protect those lines of business to the best of my ability.
[Terry O’Daniel] I love that. And I think it’s important to note that it’s easy for us to say, I need 18 months to really understand what the crown jewels are. I’ve worked at larger organizations like Salesforce, Netflix, but for the past five years, I’ve been working at smaller scale-ups and companies that have just gone public. Those companies have no patience to wait 18 months for me to figure out the business. So, what do you do in the meantime? You absolutely hit the right point, Steve. There are basics. We can measure our capability to detect certain attack paths. We can measure our capability to rapidly respond to an incident. Those are the basics. We can understand the architecture in general and where we have gaps.
Sponsor – SecurityPal AI
13:35.934
[David Spark] Before I go on any further, I do want to tell you about our spectacular sponsor, and that is SecurityPal AI. Now, I’m going to start with a little bit of a founder story here. So, the founder and CEO, Pukar Hamal, was about to close the biggest deal of his then-startup, different one, when all of a sudden, kaboom, a 200-page security questionnaire landed in his inbox. It was depressing at the moment, but it was an inspiration. It was the moment he knew that he had to create SecurityPal AI. Why? Because companies waste thousands of hours and millions of dollars on security assurance requests while juggling GDPR, FedRAMP, DORA, NIST2, the AIX, and a lot more. So, if you’re a CISO, CIO, or GRC exec, you know these headaches. Three-hundred-question reviews that stall deals, compliance requests that snowball, vendor approvals that grind to a halt. But don’t worry. SecurityPal AI and their adorable Himalayan mascot, the security yeti, they’re on the case.
SecurityPal AI solves that with a blend of fast, accurate AI, and a global team of certified security and GRC experts. You get 24-7 coverage, enterprise-grade precision, and turnaround times in – get this – hours, not weeks. So, whether you’re handling outbound questionnaires, inbound reviews, or third-party risk management, they – we’re talking about SecurityPal – eliminate the friction so your teams can focus on growth. Now Forbes has featured SecurityPal. Fast Company called them one of the most innovative AI companies of 2025. If you’re ready to eliminate friction and win trust faster, you got to go to their website. It’s securitypal.ai, and when you go there, let them know you learned about them from listening to the CISO Series.
What needs to be considered?
15:37.556
[David Spark] Jean-Philippe Martin of Start with WCPGW said, “Checkboxes are never a bad idea, especially before a release. Pilots have a checklist before flying. Firefighters go over their checklist before a mission. Because maturity model is orthogonal to best practice does not mean that it is a bad idea. Measuring efficiency/performance and measuring maturity are not mutually exclusive.” Ran Nahmias of Tamnoon said, “I believe maturity models still have a valuable place. Regulatory compliance is still a concern, and measuring success/proof of value for stakeholders outside of security becomes difficult without rubrics.” All right, I’ll go to you, Steve, on this. We’re shifting sort of our discussion here, just talking about like, well, the security team does sort of mature itself over time. And there is that classic question that comes from the non-security people of how secure are we, to which Mike Johnson, another co-host here on the CISO Series, would not really answer that, but would more talk about here’s where the maturity of the security program is. And I think you can have that conversation talking about what you’re doing and the business at the same time. Yes, Steve?
[Steve Zalewski] I’ve said this before. Is there a difference between measurement and metrics? I need to be able to measure things to be able to demonstrate that the security organization is being efficient. But the metrics, I’ve always said, is how am I demonstrating that the security organization is being effective improving the selling of jeans? So, a lot of people confuse the ability to generate numbers with the ability to generate a demonstrated value. I’ll start there. And I say, as long as you appreciate the difference, yes, we have to measure. We have tools to do that. But what I like about Jean-Philippe Martin, right, is checklists are what keep you alive. Okay? Follow the checklist, follow the process. So, then I would ask, I’d say, so, if you look at the security program, and you think about what Rinki said, is they’re looking for the easy ways in with the mistakes that are being made. How are you rethinking your security organization to understand what brilliant at the basics means for you? Right? Because it doesn’t mean one of everything. There’s plenty of tools. Are you thinking through IAM or SASE Edge or GRC relative to the business? And how are you now rethinking where maturity has to be, but where brilliant at the basics and automate, automate, automate are really what you’re driving at?
[Terry O’Daniel] I think that’s absolutely true. But I will layer in a little bit of my own experience in production engineering for Yahoo for five years. In production engineering, which is just ops, really, before we had DevOps, in production engineering, we live and die by telemetry and observability. So, we measure a lot at scale, but we always measure for a purpose. There’s no point in, as Steve called out, having these sets of numbers and measuring whether those numbers go up or down by a certain percentage. Who cares? That’s not our job, in security, at least. What that’s taught me is it’s important to understand what teams are measuring, but it’s also important to understand why they’re measuring those things.
One of the first questions I ask a security team when I join is, what are we measuring? And I remember at one place I worked, the answer was not much, but we have a few things we’re measuring pretty well. And one of them was the number of Dependabot alerts that people were finding in code. I said, “Oh, that’s great. That’s a good start.” Well, that can expose broken dependencies and things like that that could lead to vulnerabilities. So, I asked, “How are we using that number? Is it broken up by team?” “No.” “Is it broken up by which code base they’re in?” “No. It’s just one absolute number of Dependabot alerts.” And I was confused. I said, “Okay, but what does that number do?” And the answer was, “It goes up.”
[David Spark] [Laughter]
[Terry O’Daniel] Well, of course it goes up. I mean, yeah, we’re measuring something, but why? What are we doing with it? We’re doing nothing. We’re just watching as this number changes, and guess what? It’s going to go up. So, it was a wake-up call for me that the first question I ask shouldn’t be, what are we measuring? But what are we trying to accomplish with these measurements?
[Steve Zalewski] See, and I will come back to protect the brand, protect the workforce, protect the supply chain. Right? And I always tell people, start there. It’s the understanding why that’s important to the business and those metrics. Right? So, to your point here, this is why I go, that is the difference versus the how many people have taken my security awareness training in the last six months.
What must a security leader be able to do?
20:48.724
[David Spark] Nick Karaskiewicz of Baxter International said, “It’s deeper than just prioritization problems. It’s a culture problem. Security is a culture choice, which when evangelized appropriately, it becomes a proactive approach versus a reactive approach, punch list, or checkbox item.” I don’t think he’s going against the checklist sort of concept in the previous segment, but I understand where Nick is right here. And Thomas McCourt of Blue Team Tom Consulting said, “The number of companies that have cybersecurity folks that lack enough experience to be able to produce adequate effort is an extremely large problem. A lot of companies aren’t upskilling their workers. A lot of cybersecurity teams are interns that converted to full-time without experience outside of that one company, which may or not be drastically doing things different than industry standards. Most cybersecurity companies employ tools with a ‘set it and forget it’ mentality.”
By the way, side note, I put a post about this. There is nothing in cybersecurity that’s set it and forget. I was wondering if there was, and we all came to the assumption is there’s nothing. Going back to the quote, “This industry simply just needs to do better top to bottom, better processes, more accountability, and more transparency.” Well, that by the way, final quote there is, well, let’s just go dig into the archives of everything we’ve done on the CISO Series because we’ve brought all of that up before. But Terry, my question to you is, if you could start fixing one thing, where would you start?
[Terry O’Daniel] Wow. That’s a nice softball. Thank you. I think I’m going to go back to the original quote from Rinki that started us off and she called out security teams don’t fail through a lack of effort. I agree. I would add in another layer. Maybe it’s because I’m working in the rarefied air of the Silicon Valley in high-tech companies, but I don’t see security teams fail on technical merits either. Sure, I’m sure they exist. And that intern problem and the problem of security teams or security professionals who’ve only had experience at one company, those are very valid, but up leveling it to what we can actually control as CISOs, I see security teams most often fail for two reasons. We’re not very good at communicating. We’re not very good at managing the relationship with our stakeholders, which involves a whole bunch of communication and feedback loops. And we’re not very good at running programs and having the transparency and accountability to use those metrics to actually drive change. If you think that security can solve problems on its own, I think you’re mistaken.
I think security is a hub of the wheel function, and our job is to drive, exhort, evangelize with the rest of the company to drive change. If the finance team says we have this revenue recognition risk, sure. I can’t just take that to the engineering team. I have to understand and translate it into a language they’re going to understand, contextualize it so that it makes sense to them and they understand the value of solving the problem. But I can’t do that unless I’ve built a relationship with those teams. If I’ve built trust with those teams, if I’ve built feedback loops so that I’ve maybe started with low-hanging fruit and said, “Hey, here’s this kind of basic problem we need to solve together to understand how we work together.” So, I can show you the burndown chart I’m going to use to track progress so that we can build that sense of accountability so that if someone’s on the hook to do something, they actually understand that it’s a priority for the business, for their organization, not just my organization. So, I think security leaders have to be comfortable with transparency and accountability. And most importantly, we need to find a way to effectively communicate that to our stakeholders and help them understand their involvement in the process.
[David Spark] Steve, close us out here. Again, I’m going to ask the same question I asked Terry. There’s a lot here. If you could just start fixing one thing, what are you fixing?
[Steve Zalewski] I’m going to go from tactical to strategic for a minute, okay? It’s a culture problem. It absolutely is. But it isn’t that everybody needs to be part of the solution of cybersecurity. That is how we’ve made it a culture problem, but that’s not what it’s becoming. I would argue the practice of cybersecurity being 30, 35 years old, and the profession of a CISO being 20 years old, we need to, as a larger community, establish what good culture of the cybersecurity function looks like, and not just try to make everybody think cybersecurity is important. So, when I think about this problem, the one thing I need to fix is I think we need to acknowledge that as a profession, we’re teenagers. So, therefore, a lot of what we do isn’t the right thing, but we don’t know any better. And let’s look at ourselves and understand as we continue to grow up, what the profession of a CISO is. How do we know? What is CISO supposed to do? Where is the accountability and the culture to establish what a good one can do? And then if I look at the practice of cybersecurity, once I have professional CISOs, let’s do the same thing to figure out what the right culture is going forward.
Closing
26:46.327
[David Spark] Well, that brings us to the portion of the show where I ask both of you, and I’m going to start with you, Terry, which quote was your favorite and why?
[Terry O’Daniel] In Rinki’s original post, she has a phrase when she’s talking about the problems with our current approach. She says, “We measure posture, not pressure.” That really resonated with me because I believe that we are at our best as security leaders when we bring a combination of patience and persistence to the table. We have to show up with the right message, and usually the right message is we need to take care of the basics. We need to understand the fundamentals and we need to build our capability up from that. But we’re going to show up and we’re going to have this message that is not very fun for the rest of the organization to hear. Most organizations are trying to write code, ship code, and build product, and sell that product. So, if I’m asking them to do work that is not supporting that, it may feel to them like I’m taking them away from the core mission of the business.
That’s where the persistence comes in. I have to show up with those important messages that are fundamental, and I have to be willing to accept that I’m not going to win that battle every time. I might show up 99 times and the answer is, “Sorry, the product roadmap takes precedence. We have to ship these features.” But the hundredth time I show up with that same message, we have to focus on the fundamentals. We have to turn logging on our databases, things like that, it lands, and it sticks. So, I think we have to be comfortable with a little bit of failure and a little bit of resistance.
[David Spark] All right, Steve, your favorite quote and why.
[Steve Zalewski] Again, if I go back to Rinki, she says they don’t understand context. And I would argue for this whole show, right? You got to get the context right, and we’ve talked about different ways that context is incorporated into the conversation. So, with that, I’m actually going to look at Ashish Popli, which is his quote. And he says, “There are clearly enough tools to do the identification part right. What do we see? But what we have to understand is what will break us at the scale of the enterprise, and do we have first line of defense, last line of defense, and evidence of defense?” And I think at the end of the day, that is the materiality of what we’re trying to do.
[David Spark] Very good point. Well, that brings us to the very end of the show. And I want to thank our sponsor, SecurityPal AI. Remember, go to their website, securitypal.ai, for help in your GRC efforts, especially these last-minute requests that come in, the, “Geez, I can’t handle this. I don’t know about these laws and all this kind of stuff.” Guess what? They’re the experts and their AI tool’s going to make it all a lot better. Go check them out. And when you do, let them know you found out about them from the CISO Series. Thank you very much, Terry O’Daniel, who is the former CISO over at Amplitude, for joining us here. And also Steve Zalewski, always appreciate it as my guest or co-host, but you’re not the guest co-host.
[Laughter]
[David Spark] I just want to point that out. He is the regular co-host that is also my guest, but not the guest co-host. Thank you, everybody who is also listening to this show. We greatly, greatly appreciate your contributions. I know I say it at the very end of the show, and it sounds like it’s very pat, but honestly, the show does not exist without your contribution. So, that’s why I keep saying it again and again and again. So, thank you for your contributions, and also thank you for listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site, CISOseries.com, where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David at CISOseries.com. Thank you for listening to Defense in Depth.