Organizations don’t have the time to do in-depth vetting on every third-party. This leaves them turning to “better than nothing” security rating vendors. These might be fine for liability, but do these vendors actually help improve your understanding of third-party risk?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark, the producer of CISO Series, and Geoff Belknap. Joining us is Steve Knight, former CISO, Hyundai Capital America.
Got feedback? Join the conversation on LinkedIn.
A huge thanks to our sponsor, Formal
Full Transcript
Intro
0:00.000
[David Spark] Organizations don’t have the time to do in depth vetting on every third party. This leaves them turning to security ratings. These might be fine for liability, but do these third party rating vendors actually help improve your understanding of third party risk?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. And joining me for this very episode, you’ve loved him for many, many years, watching him grow up on television in front of your very eyes… It’s none other than Geoff Belknap.
[Geoff Belknap] Hey, everybody. I love you, too. I just want to be clear.
[David Spark] We all love you, Geoff. Our sponsor for today’s episode is Formal. Enforce least privilege on autopilot. Yeah, it’s a very interesting new take on PAM, privilege access management. We’ll talk about that a little bit later in the show. But first, Geoff, let’s talk about today’s topic. All organizations need to be able to assess third party risk at speed and scale. As Paul Valente of VISO Trust pointed out on LinkedIn, there is a gap in this process.
Sending out surveys and reading SOC2 reports is time intensive. Does anyone seek the insight and accuracy of security ratings? And whatever choice you use, it’s often just a point in time. Everything could change tomorrow. So, this is something kind of we’re all seeking out, Geoff. What do you say when someone asks, “What our best option for assessing third party risk?”
[Geoff Belknap] I think it is really a deeply philosophical question to figure out. Like, “What is your third party risk?” Everybody’s risk profile is different. Every vendor you bring into the organization brings something different to the organization. And what you’re doing with that vendor or that partner is different.
So, I think what most people need to do is figure out do you just need a very high level way to decide whether you should be doing business with this person, or do you really need to do the depth of understanding of how are you going to integrate this organization, or this process, or this product into your production environment. Those are two very different things, and they require two very different flows of work. And I think that’s one of the things we really need to get into here.
[David Spark] That’s a really good point you just made there, is it’s not what is the risk of working with them. How are you going to bring them in? Because they could be A+ stars all the way. But if they don’t integrate well with you, it’s not going to work for anybody.
[Geoff Belknap] That’s right.
[David Spark] This is great. So, we have a great guest here to help us with this very discussion. The former CISO over at Hyundai Capital America. None other than Steve Knight. Steve, thank you so much for joining us.
[Steve Knight] Hey. Thank you so much for having me. And, Geoff, I now have the Barney song in my head, so just so you know. I love you, and you love me.
[Geoff Belknap] I sure do, Steve.
[Steve Knight] Thank you. I appreciate that.
Would this work?
2:57.880
[David Spark] Christine Chalmers of Netflix said, “We’ve implemented a few filters that cut down on the amount of vendor evaluation we do. Vendors are tiered based on the data they process. We have a checklist of the top few reasons vendors fail our assessments, and we train everyone to ask those questions when they’re first evaluating options.” So, this seems kind of like a top of the funnel way of looking at vendor evaluation. And Mark Dunaisky of D3 Risk Management Group said, “How many people actually know how to read a SOC2 and get value out of it?
How useful are Security Risk Ratings platforms? Most companies cannot answer these questions. There are many companies who are only using Security Risk Rating platforms as their entire cyber third party program because ‘they are better than nothing.’” And I fear what Mark just said is the truth, and a lot of people look at it that way. Geoff?
[Geoff Belknap] Yeah, I think a lot of people are mistaken and have not taken Christine’s approach to this. And I think Mark is referring to the people who think every vendor that you buy from, you need to read their SOC2, get their audits, understand what their security operations response plan is. And you find out that you’re doing that for your janitorial company, and you probably don’t need that. What you really need is to be smart about it and follow Christine’s approach here, which is, “Hey, if you are hiring a company that’s going to clean dishes in your office, you don’t really need to know what their 24/7 SOC response program is because they’re not touching their data.”
So, there’s a filter you want to go through to figure out am I just buying paper from this company, in which case it’s great to know that they’re going to be an ongoing concern and that you can depend on whatever they’re bringing into your supply chain. But you’re not going to be sharing private information with them. And if you are going to be sharing private information with them, let’s say they’re processing data or running your payroll, or whatever it is, then you want to figure out how to read their SOC2 and really get into it. But you can’t apply the same level of evaluation to every level of risk that a vendor provides to your organization.
[David Spark] That’s a really good point, and that brings me to my question for you, Steve, and that is I’m just interested in the range, just so we can kind of put numbers around what Geoff just said. Can you think in your head the number of hours you spend integrating a vendor, the most difficult, the most complex one you ever had and the one that was the easiest. What would you say is the range of hours? Is it 1 to 100, or what is that do you think?
[Steve Knight] I think the most difficult… And mind you, in the programs that I’ve been in, we’ve been tiering vendors by criticality and impact on business operations. Because you want to be strategically aligned. More important, there’s the regulatory side of it, so you got to make sure you can check the box as well. So, those that come bearing gifts, those that come with a SOC2, those that send their CISO, not a subordinate within the organization, those that have a fundamental understanding of risk frameworks and they’ve implemented at least one in their environment, they come bearing gifts.
And so you can have a comprehensive conversation and actually get right to the point of what you’re looking for, which is kind of like helping your kids pick their future spouse. Can I trust you or not? And if I can trust you, tell me a little bit more about how you maintain your program on an ongoing basis? So, you get there by picking the right partners up front. The back end of that is you’re going to embed contractual controls, so you’re going to have a strong partnership with legal to make sure that you’ve got contract language that’s in there.
And then much to the chagrin of those that don’t think the tiering or the scoring systems are very good, you’ve got to have something that gives you some idea of posture. And so you’re going to be monitoring it on an ongoing basis. So the vendors that come to the table with the things that I’m looking for and can answer my questions rapidly, they get onboarded quickly. Those that don’t, got to go dig deeper, and ask more questions, and spend more time on my side.
How are the vendors handling this?
6:52.345
[David Spark] Simon Marvell of Acuity Risk Management said, “How about some old fashioned risk management where we identify vendors and risk scenarios that could cause truly material impacts on achievement of business objectives. Then work with the vendor to assess the risk, agree to specific controls to keep the risk within tolerance and the metrics to provide assurance that the controls are working.
Automate collection of these metrics, monitor them carefully, and address everything else through contract.” Sam Reddy of CISO InterSec said, “Metrics scorecard specific report is not silver bullet. It starts with the organization, integrated risk management, assessment for third parties to see how it can impact organization business, operations, and technology.” Steve, you pretty much said this in the last segment here. Very much a lot what Simon and Sam are talking about right now is…
And I like right at the beginning what Simon said. Who could cause truly material impact on the business? And one thing that we’ve heard from other CISOs asking vendors is, “Describe a worst day scenario to me and how you would handle it.” Have you had those conversations with your vendors?
[Steve Knight] Only the top tier ones. And what I mean by top tier… Not necessarily from say a ratings perspective, but let’s say the vendor in question is writing code for us, and we’re not getting an SBM, a software build materials, in that code. And worse, they’re coming in through let’s say a private VPN connection or over the internet into our environment.
And we’re worried that if something happens on their side, we need to be able to cut them off very quickly. So, sometimes it’s situational in the sense that I have to look at it from the perspective of what am I sharing with them? Let’s say PII data and massive amounts of it. Let’s say it’s intellectual property. Or the reverse. I’m getting from them in terms of a product or a service, and/or they’re also connecting into my environment. I think the summary here is it depends.
[David Spark] But the question of… And this is part of the trust. And I want to dig a little deeper on this is asking them, “Describe to me a horrible scenario that would happen.” You know. You just kind of panned it out. But from their viewpoint… Because they know horrible scenarios you may not be aware of and ask… Because everyone has bad ideas. And we talk about this – how you handle your bad day really kind of speaks a lot to your risk as well. Yes? And so that’s the question – are you having those conversations?
[Steve Knight] When we get into the discussion of if the vendor is having a bad day, how do they respond, first and foremost, what we’re going to find is most third party vendors that you do business with, unless they’re large organizations, don’t have a mature security program. The question you’re really looking for here is tell me about your incident response process. And more appropriately, let me then tie that into my incident response process as well as what the contract said in terms of my ability to audit and your ability or timeframe to respond.
Because really what we want to know here or what we want is awareness. Awareness is half the battle. So, if they get popped for any reason… They have a bad day. Regardless of how mature their program is, regardless of how good their incident response program is, I need to know, because I need to figure out how I’m going to respond to make sure I can shut the door if I need to.
[David Spark] All right, Geoff, just tagging off what Steve said, all this talk of ratings and what not, none of it means, I think, anything unless you know, A, how the data is shared and understand the incident response program. And I’m assuming you get into those conversations. Yes?
[Geoff Belknap] Yeah. You rarely get into it being direct like that. Because usually when we’re having this conversation, we’re having it at the very beginning of the relationship. And the whole reason we’re having a conversation is because somebody at the business said, “Wow, when you have a great day, you’re going to add a ton of value to whatever I’m doing, whatever I’m trying to accomplish.” They’re looking at the upside of everything.
And because sometimes in our job we have to be Debbie downer, we go, “Well, what if it doesn’t go great? What then? Am I just out of luck for whatever data or whatever services you’re providing, or worse, am I in peril now? Does my production or the services I’m providing to my customers now going to be impacted? Am I going to be popped because of you?” And really, Steve gets right to it. That’s what we’re trying to figure out is how screwed am I if you have a bad day.
Is it just a little bit of annoyance for me, or am I now going to have an even worse day? And the hard part is at the beginning of the relationship, it’s very difficult for a vendor to be able to be represent that to you in a way that you can really understand. And that’s why I say, at the beginning, you’re really just trying to figure out are they capable of having that conversation? Have they even thought about what it’s like when they’re having a bad day? If they are, you can build a relationship with them and figure that out over time depending on what you’re doing together.
If they have no idea what you’re talking about and SOC2 is not even something they’ve ever heard of before, you’re taking a bigger risk, and it’s really up to you to decide is it worth it. Are you going to be able to handle that risk? And that’s what this all comes down to.
[Steve Knight] What happens when you’re a newly minted CISO walking into a new organization, and you inherit the third party risk management program that’s already there?
[David Spark] Geoff knows that very well.
[Laughter]
[Steve Knight] You have not yet reviewed the contract language. The first thing you’re trying to do, of course, is figure out the maturity of your own program because you have to be able to respond if something happens to them. But now you start to look into this vendor that you do a lot of business with, and you find out those important questions haven’t been asked. Nor has the contract language set such that you have at least some way of responding when the bad event happens.
I think even in those situations where the risk is high, especially when the vendor is really impacting ongoing operations or strategic scale even, it still comes back to when are you or how are you going to know. And secondly, how good is your program that you can respond? Most mature organizations treat TPRM data like threat intelligence.
[Geoff Belknap] And that, my friends, is how you get ulcers.
Sponsor – Formal
13:01.860
[David Spark] Today’s sponsor is Formal. And if you’re a CISO trying to ahead of AI risks before they explode into incidents, well, this is for you. Now, let’s be honest. The security model for data access really hasn’t kept pace with how data is actually used today. AI agents, internal services, [Inaudible 00:13:23] are making calls to your most sensitive systems, and the legacy perimeter is nowhere near those requests. Formal gives you visibility and control where you need it most – at the point of data access. It sits between your AI agents, services, and data stores.
Snowflake, APIs, whatever. And inspects every request in real time. You get deep protocol level context – who made the call, what was requested, whether it involved PII or customer secrets, and what policies should apply. Now, here’s what’s critical for AI governance – Formal secures the model context protocol layer. The MCP server that feeds data to AI agents. You’re not trusting a black box. You’re governing exactly what goes in and out of it. You define and enforce policy, log access, stop leaks, and stay audit ready all without slowing down engineering. This is how security leads in the AI era.
Not with red tape but with infrastructure aware enforcement. If you want to get out of reactive mode and take control of AI and data flows before your board starts asking questions, go to Formal’s website. Visit joinformal.com. joinformal.com, where modern security starts.
What are the complaints?
14:53.925
[David Spark] Val Dobrushkin of AVA Compliance Solution said, “Security scoring services are a scam and do not provide any true measure of security as they have no inside auditing reporting or validation, and usually are fraught with false positives. SOC2 and other audit reports are our best option. Until we have continuous compliance monitoring via automation compliance tools that are visible to customers.” Which, by the way, we’re seeing these days.” John Overbaugh, who’s the CISO over at Alpine Investors, said, “I think these third party scans and reports are wholly misleading.
As a scan target, my scores and reported risks were always off. Just like the credit bureaus, corrections cores is impossible.” And Ira Winkler, the CISO over at CYE, said, “With a SOC2, at least you know their algorithms. With the scorecard vendors of the world, they have proprietary algorithms, and companies have to pay to have them address the shortfalls in their findings of the vendors.” All right, a lot of crapping on the score tools. Geoff, do you agree, or could you say something positive about them?
[Geoff Belknap] I’ll say this positive thing about the score card vendors. I think it’s a wonderful concept. But eventually what happens, like with all security tooling, is people put them on a pedestal and hope for way too much from them. I think the security scorecard vendors as a base level have a lot of value to add at…when we talked about earlier about tiering and sort of making quick decisions about where you want to deep dive into.
If you have a vendor who’s low risk and you want to just do a quick pass about whether this is something you worry about or not, a scorecard might be enough. If you are, again, sort of like we were talking about…if this is a vendor that now is going to operate infrastructure with you or be a shared part of your security threat model or your security risk model for customers, it’s irrelevant what the scorecard says because you’re going to have to spend time with them, talking about them, partnering with them.
And I think the other really important part of this is just if you are somebody who either makes or depends on these scorecards, I want you to just listen for a second to what this is like on the other side. You are trying to do business with customers. You are trying to do your best as the CISO for that. And you’re getting a report in your email that says you did all these things wrong.
And the reality is you would very much like to tell them to go stuff it because these things can be very inaccurate sometimes, but you really want to do business with your customers. So, you are forced into this really untenable position of having to go work with a third party that may not, in some of these cases, be interested in correcting what the scan says. Or in a very worst case, there may be some disreputable companies that want to charge you to fix the report even though it is inaccurate.
And that’s just a bad place for these people to be in. I think fortunately though, most of these vendors are more than willing to fix the report. It’s just the investment to go correct a thing even when you know you’re doing the right work.
[David Spark] Very good point. All right, Steve, you were nodding your head at some of these very negative things. I want your take on it. But like I asked Geoff, I wonder if you can say something positive about these scoring tools.
[Steve Knight] I was drawing the parallel between these scoring tools and credit reporting agencies.
[David Spark] Which was also said here as well.
[Steve Knight] And the pain is real. [Laughs] If you ever found something on your erroneous and you’ve ever had to go and actually, first of all, get a hold of someone, a human, and then begin the arduous process of fixing whatever the erroneous issue is, you know the pain. It’s real.
[David Spark] The thing is, this is a problem that comes up again, and again, and again. And you would think they would have some kind of more formal system to handle it, but it sounds like as if it’s a new one-off every time. Why isn’t there a more formal way to manage this?
[Steve Knight] I think because, from my perspective, I think we’re largely still looking at this as whether it’s a viable service or not. I mean let’s be clear here for a second – security ratings are not gospel, but they are useful trend indicators, like your credit score. They don’t tell the whole story, but they give you a signal that’s worth investigating, which leads to more direct questions. And those direct questions also call for more of a strong relationship with the vendor you’re doing business with.
So, if you have a strong relationship with your tier one vendors… Let’s say there’s five of them. And you’ve rated them and tiered them in such a way that you understand the impact of the business should something go wrong, you also should be able to pick up the phone and be able to have a conversation with that individual. If you see their score drop from 720 to 680, especially in the most recent update, you might call into question if you have a good relationship with the individual whether it’s real or not.
And so I think what is incumbent upon us as leaders is what kind of relationship do you really have with your tier one vendors so that you could have a very forthright conversation using these scoring services as a way to have a conversation.
[Geoff Belknap] Yeah. I think to that point, look, a number is a fine place to start, but it is not the definition. It’s not the end of the relationship. And I think as long as we’re using it as one component, there can be a lot of value from these scoring services if you contextualize the score.
What would a successful engagement look like?
20:10.860
[David Spark] Vladimir Yakovlev, CTO over at Higher Intelligence, said, “Security rating at the very least allows companies to decide if it is even feasible to consider a given vendor’s offering. At least there are active notifications in case of a rating drop, in which case alternatives may be considered as well as integrated process tracking for requests for remediations.” Just what we discussed.
Subarna Bhowmik of Deloitte said, “SOC2 reports are great avenues but not the complete package one should restrict themselves to. I feel questionnaires plus interview, validating policies and control effectiveness, plus independent reports reviews, SOC2, pen tests, DR tests, etc., can complete a risk assessment of vendors as add ons.”
All right, Geoff, I’ll start with you on this. Subarna, I think, kind of sums up kind of everything we talked about. Yes, you look at these rating reviews. Yes, you do interviews validating policies and control effectiveness. Yes, you look at their independent reports. If you look at all these elements, you can put together a story. My whole feeling is if you start working with a lot of vendors, that’s a ton of freaking work, isn’t it?
[Geoff Belknap] It’s a ton of work. And I think the really important thing here is if you’re going to use a short cut, you’re going to rely on a vendor, whether it’s a score or a vendor that does the evaluations for you, what you are really doing is not replacing the relationship building that’s required and the risk management that’s required. You’re accelerating some component of it. So, I think scorecards are fine for like an initial pass, especially if they’re low risk vendors or it’s a low risk engagement.
A scorecard is fine at long as you aren’t abusing them in a case where you say, “Hey, we run the score against everybody, and you have to fix it or do something unreasonable.” But you’re still going to come down to, as Subarna says, having discussions, reviewing things, seeing where things are at, how have things changed year over year, because that is what supply chain risk management is about. Like not a point in time but how are we doing together. And, again, it’s together.
Not, “You got a B-, and I expect you to get an A+.” It’s like, “Look, we’re working together. How’s that going? Do I feel good about this? What would I like you to do better?” Anything that you are doing to make that easier is great, but you have to look at it in the holistic package of what the outcome you’re trying to drive is.
[David Spark] Steve, I think the reason this topic just keeps coming up again, and again, and again is that we know it’s difficult. There’s a lot involved with it. We love the fact that there are these rating systems out because it gives a sort of initial understanding, and they do do a lot of leg work in a certain area for you. But it’s not enough to actually conduct business, is it? And I think that’s really where the concern is, yes, Steve?
[Steve Knight] I would tend to agree with you. And I want to change the narrative just for a minute. I think we need to shift from assessing vendors to assuring outcomes because I believe the leaders in the organization that you partner with are wanting that, very much so. They don’t care so much about the SOC2 report that you have to review. They don’t care so much about the conversation or the trust relationship that you have to build with the vendor and the security team to understand security program to know how it’s going to impact.
They just want to know that they’re… Like the quote earlier to decide even if it’s feasible to connect to a vendor. If the business comes to you and says, “I’m going to do business with this vendor,” they’re going to do business with the vendor. It’s up to you as the leader within the security program to figure out how you’re going to make sure that that relationship at your level is going to be able to be at least close to as trust as it possibly can be. Subarna makes a good point.
All these things that we have to review and go through and read, and the conversations we have to have are vitally important. But not to all of the TPRM, as Geoff was saying earlier. And, again, I’ll come back to it. We need to shift from assessing vendors. Or if you’re going to assess, it’s a finite amount. It’s a small amount that really are going to have an impact on operations. And we need to look more at assuring outcomes in these relationships.
Closing
24:20.900
[David Spark] That is a very, very good point. Well, that brings us to the end of this discussion. Now, I’m actually going to go to you, Steve, first. I always like to ask which quote was your favorite, and why. So, of all the quotes that I read, which one was your favorite, and why?
[Steve Knight] I think Subarna’s. It’s not just because it was the last discussed, but I think Subarna’s really is the one that I find most interesting. Because they seek to show what is required at the macro level, but the piece that was missing for me in that quote was but you’re going to apply it at the micro level to a finite group of vendors that could have significant impact in your environment.
I think the quote is spot on. It’s comprehensive. But it needs to be tailored based upon your risk threshold and the impact of those vendors on your ongoing operations.
[David Spark] All right, good choice. Geoff, your favorite quote and why?
[Geoff Belknap] I’m actually going to do a joint of two, but I’m going to go back to the very top of the show where we talked about what Christine Chalmers from Netflix and Mark from D3 Risk Management talked about, and I just want to reset the context. Whatever you’re doing for third party risk needs to be tiered and adjusted based on the risk. Either based on the risk of the vendor to your operations or based on what you perceive that vendor’s risk maturity to be about.
And that will tell you whether it’s okay to just use a score and then move on from that or where you really need to deep dive and spend your time. And a reminder that a risk rating, a number, a riskiness, a color, whatever it might be, is not the beginning and end of your risk management program. If that is what you are doing, you should stop, and you should just spin a wheel, throw a dice, guess, consult your spiritual advisor. That’s probably going to be a better impact than using only a rating.
What you are trying to do, to Steve’s point earlier, is build a long-term functioning relationship where you understand the security impact of that vendor on your operations, and you can’t do it with only a number.
[Steve Knight] I will add one thing to what Geoff said. TPRM is not just a checkbox. It’s an extension of your digital immune system. So, how healthy do you want to be?
[David Spark] Very, very good. How healthy do you want to be? That’s actually… We should have started with that. All right, let’s record this all over again.
[Laughter]
[Geoff Belknap] Take two, here we go.
[Steve Knight] Take two.
[David Spark] Hey, thank you very much. That was Steve Knight, former CISO over at Hyundai Capital America. And also Geoff Belknap, who you’ve loved, and you’ve watched grow up on television in front of all of our eyes. We loved it. Was it tough being a child actor, Geoff?
[Geoff Belknap] [Laughs] You know, I’ve blocked a lot of that out. With therapy, I think at some point we’ll be able to answer that question.
[David Spark] Compared to the other child actors, you faired quite well, I should mention.
[Geoff Belknap] I was the best one. That’s what my mom told me.
[Steve Knight] [Laughs]
[David Spark] Your mom was correct. Thank you very much, Steve. Thank you very much, Geoff. But also, a huge thanks to our sponsor, and that would be Formal. Remember, enforce least privilege on autopilot. Thank you so much for sponsoring this episode. Go to their website. Get a demo. Go to joinformal.com. To all of our listeners, we greatly appreciate your contributions and for listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please, write a review. Leave a comment on LinkedIn or on our site, CISOseries.com, where you’ll also see plenty of ways to participate including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at david@CISOseries.com. Thank you for listening to Defense in Depth.