Picking the wrong vendor can result in a costly transition. You have your roadmap and the vendor has its own. If you pick the wrong horse you could be ripping and replacing in 18 months. How do you avoid making such a costly mistake?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark, the producer of CISO Series, and Steve Zalewski. Joining us is Paul Guerra, former CISO, StockX.
Join the conversation on LinkedIn.
Huge thanks to our episode sponsor, Native Security
Full transcript
Intro
0:00.000
[David Spark] Picking the wrong vendor can result in a costly transition. You have your roadmap, and the vendor has its own. If you pick the wrong horse, you could be ripping and replacing in just 18 months. How do you avoid making such a costly mistake?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. And joining me as my cohost, it is Steve Zalewski. Say hello to the audience.
[Steve Zalewski] Hello, audience.
[David Spark] That is Steve Zalewski. You’re going to hear a lot more of that voice throughout the show. By the way, did you know that we are available at CISOseries.com, where you can find all of your favorite CISO Series programs. We have four other shows on our network, plus lots of videos, and articles, and tons of great stuff.
And you can sign up for our newsletter as well on our site or via LinkedIn. Whatever you like. Our sponsor for today’s episode is Native, the cloud security control plane for the enterprise. Unify, manage, and maximize built in cloud security controls and achieve secure by design consistency across cloud environments.
And we’re going to be talking about that a little bit later in the show. But first, Steve, let’s talk about the topic at hand.
No one signs a contract with a vendor thinking they’ll need to jump ship, but as Richard Stiennon of IT Harvest pointed out, it happens all the time. It wastes implementation and integration time and creates gaps in security while you switch, to say nothing of the wasted political capital.
A lot of this stems from incomplete market intelligence when they initially bought. As Stiennon said, “Due diligence isn’t just about evaluating vendors. It’s about knowing which vendors to evaluate in the first place.” Now, conventional thinking would lead you to believe the market leader would be the best, but your good pick could fail on you because their roadmap was looking out for their best interest.
“The vendor got acquired, and product development stopped. A better solution immerged, and they didn’t know about it. The ‘market leader’ turned out to be the marketing leader,” said Stiennon. Steve, this seems like a trap that far too many of us have fallen into.
Have you fallen into this trap yourself?
[Steve Zalewski] Oh, yes. And I think one of the things we want to talk about here is it used to be a trap. More and more I think we’re going to realize is it’s actually status quo, and it’s something we have to get better at. So, I’m really looking forward to this conversation.
[David Spark] That is a good point, and it’s a scary point as well.
[Laughter]
[Steve Zalewski] Yes.
[David Spark] Joining us for our conversation is someone we’ve yet to have on the show, so new sponsor, new guest. All wonderful. This person is the former CISO over at StockX, Paul Guerra. Paul, thank you so much for joining us.
[Paul Guerra] Hey, guys. Big fan of the show. Thank you so much for including me. And love the topic that we picked, so excited for the discussion.
Does anyone understand what’s going on?
2:57.385
[David Spark] Genevieve Bader of Starbucks said, “I’ve experienced this in several companies where they don’t vet the vendors thoroughly and end up regretting the lengthy contract they signed. Contracts are also not read carefully, which makes the vendor’s failures very hard to hold them accountable for.” And Anatoly Chikanov of Primary Venture Partners said, “Vendors have gotten way more flexibility with term duration.
I remember doing three-year terms all the time. But around the pandemic time, something changed, and I started to see many more 12-month commits and even 6-month commits.” That’s incredible. “Granted, yes, the price is a bit higher. I observed about a 10% higher on average.
But doing a one-year contract is a better option I think nowadays for most companies given how fast the pace of change is.” So, I think maybe what you said at the beginning of the show, Steve, has to do with Anatloy’s comment. Is like if everyone thinks this is what happens, we have to go to one-year commits here, don’t we?
[Steve Zalewski] So, I think what you’re saying is four or five different use cases. And we have now got more of these use cases coming up. And so for an example, a three-year deal, that’s what they want. If I’m limited from CapEx and OpEx, I almost have to do a three-year deal because it’s the only way I can get the revenue.
Because I have to be CapEx limited or OpEx limited. And yet I know I’m going to have to abandon it in two years. Another use case – so many in security start ups are new, and they want revenue. And they’re willing to be very flexible just to be able to say, “You’re a customer, and I’ll give you a six-month deal.” Okay?
So, again, a perspective there. And the third one I’ll offer is when we went with cloud and now when we’re doing AI, we’re pushing the security envelope of capability right to the edge with the business. And often times, the business is pivoting in 12 months, and we need some new technology because the use cases have changed, and therefore we have to pivot off of something even though it’s only 12 months old.
[David Spark] This endless pivoting, Paul, is this your life?
[Paul Guerra] A little bit. A little bit. So, I’d say a couple things. One is the market is moving lightning fast. Sometimes faster than enterprise buying cycles. So, by the time a tool is fully deployed, the problem you were trying to solve has already evolved from that point.
So, look, I think it’s a tough time to be a CISO. There’s too many options. It’s also a tough time to be a founder. New competitors sometimes every month depending on what domain you’re in. Or, look, it’s even harder to be a VC. Who’s going to win the category?
The valuations we’re seeing these days are crazy. Things are just evolving so quickly. I think when we talk to some of these vendors, even some of the startups, a lot of the roadmaps that they’re sharing with us, they’re aspirational documents, not so much commitments.
And then you start talking about acquisitions, platform pivots, pricing changes. Those are now normal and not necessarily just edge cases. So, an example. We had a leading AppSec vendor, and they increased their price by 35% year over year. No change.
No additional modules, no anything, no increased volume. Just 35% just to go and keep up. Sometimes the safe choice often creates the biggest lock in risk, and you just don’t know it at the time.
What problem is this solving?
6:35.021
[David Spark] Peter Margaris of Congruex said, “Isn’t there also a problem with enterprises evaluating vendors based on their magic quadrant rankings, miscellaneous reports, pay to play vendor rankings, etc.? Rather, what they should be doing is technical due diligence and evaluation based on what problem they are looking to solve.” Andrew Plato of Zanaciti said, “There’s also the big boy sales technique.
They wine and dine customers, telling them how massive and important they are and how awesome and respected the CISO/CIO will be if they implement the vendor’s product. ‘You’re a big boy if you buy our stuff.’” I like that.
[Laughter]
[David Spark] All right, good scenario. All right, a couple of things I want to bring up. So, Richard Stiennon who initiated this discussion… By the way, I consider him the only analyst that is truly counting every vendor in this space, and he knows he can’t keep up.
He’s at around 4,500, and he estimates he’s maybe 500 to 1,000 off because companies come and go, and it’s hard to know everybody. But given the number that Gartner measures and the number that he knows, he estimates Gartner only is covering about 3% in their magic quadrant in their analyst reports.
So, if you’re only relying on Gartner, you are looking at a very, very tiny percentage of the market. But at the same time, I remember talking to Wendy Nather, who at the time was the researcher for 451 Research. And it was her full-time job to be looking at companies and researching them.
Full time. And she was only looking at a tiny percentage of the market. So, this is a losing proposition any way you look at it. You’re never going to know everybody, are you, Paul?
[Paul Guerra] No, definitely not. So, kind of these magic quadrants, they do definitely optimize for procurement comfort, not necessarily operational fit. And some companies buy categories, not capabilities. And I think some of us do pigeonhole these vendors into these artificial domains.
So, not intentionally, but external inputs are by these categories, and sometimes our budget line items are also by those same categories. Vendors do optimize messaging for their largest addressable market, not necessarily our edge cases. So, my thought is if you can’t describe the problem you’re trying to solve in one sentence without buzzwords you’re probably not ready to buy something.
You have to know the exact problem and threats you’re solving for in depth. Needed detailed requirements, a testing plan, and a way to score the results at the end is imperative. And sometimes when you buy a platform, you need a specific tool. Some of the best vendors are often bad fits because they solve too many problems poorly, and then you’re left with some unfulfilled capabilities.
[David Spark] So, Steve, I’m sure you fell into the trap of, “I know these vendors. I’m sure there’s more out there. I don’t have time to look at them all or figure out who they all are.” And then also just defining your own problem, like Paul just said.
Just doing the matchmaking yourself is a ton of work. And I always point out this is only a part of a CISO’s job. It’s not your full time job, it’s only a part of it. What do you do in these sort of overwhelming situations, Steve?
[Steve Zalewski] It comes back to this – am I trying to pick the best solution? Am I trying to pick a good solution? Or am I trying to pick the right solution? Now, you can argue those three questions are all the same thing, but they’re not. Am I picking a solution because I have to make my boss happy?
Am I picking a solution because I have to make my board happy? Am I picking a solution because I have to make my line of business happy? Am I picking a solution because I have to make my auditor happy? Every one of those can result in a different outcome for the product that I pick, and the magic quadrant, like it or not, is a good indicator for the right answer for some of the ways that I pose the question.
And I think that’s what we’re getting at here is if the question changes because the requirements change, and it’s no longer about the best tool from a technical perspective, but you’re looking at the political landscape to pick. Or you’re looking at keeping your job, or you’re looking at downsizing requirements.
You have to appreciate the entire picture as to how you arrived at the conclusion rather than just simply saying, “Well, the magic quadrant is pay to play. And if you pick that, your job is safe,” that’s no longer true.
[David Spark] Well, they would want you to believe that. But one quote that Mike Johnson, who’s one of our cohosts for CISO Series Podcast and currently the CISO of Rivian, he said something that our community rallied behind. And that was, “I’m not going to ignore a vendor just because they do not show up in the magic quadrant.” And every one was applauding that tremendously because they don’t like the fact that it becomes sort of a gatekeeped experience.
Which I know Gartner would love people to believe that so more people see value in their product. I’ll ask kind of the same question I asked Steve to you, Paul. How do you sort of deal with that overwhelming feeling of, “I can’t know it all. I have to talk to so many people to try to figure out what matches my problem.
I don’t think I have the time because it’s only part of my job.”
[Paul Guerra] Yeah, great question. It’s tough. It’s tough. You have to know the vendors that are out there. You have to look at these external inputs. And yeah, as you mentioned before, it’s only 3% of really what’s out there. What I try to leverage quite a bit is some of my CISO peers and new, interesting tools that they’re implementing.
But we focus on the problem statements, not so much the solutions that are in place. “How did you solve X, Y, and Z? What kind of result did you get? What kind of outcomes did you get for your end customer at the end of the day? How easy was it for IT or platform engineering to go and implement?
What are the cost implications? Was it cheaper? Was it more expensive? So, it’s asking all those questions and understanding how all stakeholders are going to react to that decision. Because, Steve, you made a great point. You have a lot of people that want to weigh in on the decision, and it becomes not just technical anymore.
It’s financial. It’s political. It’s all these other pieces. So, yeah, it’s three dimensional these days.
[Steve Zalewski] And I want to chime in with one more thing. And I think Michael is correct. But here’s what we’re seeing in the industry, brutal truth – 4,500 products that we know about, another 500 to 1,000 in stealth mode. There’s no way when we need to now say, “Okay, we’ve got to put a guard rail around AI models,” that we can actually go out and do a traditional POC evaluation.
We’re reaching into our networks and saying, “Who’s solved this problem in the last six months? What did you do? Does it work?” And we’re basically prequalifying one or two products based on the network.
Sponsor – Native
13:41.474
[David Spark] Cloud providers ship powerful, built in controls. But most people struggle to turn secure intent into consistent enforcement across AWS, Azure, Google Cloud, and OCI. Different policy models force security teams into manual translation and one-off exceptions, which get brittle fast as accounts, services, APIs, and AI workloads change.
Native is the secure by design control plane for cloud security. It helps teams operationalize provider native enforcement, manage intents centrally, and roll out changes safely at scale. Native works through the cloud’s own mechanisms, so guard rails are enforced natively while teams can preview impact before deployment and reduce drift over time.
With Native, security isn’t bolted on after the fact. It becomes part of how you operate the cloud. Remember, go to native.security to learn more.
Would this work?
14:48.992
[David Spark] Dmitry Raidman of Cybeats said, “This mistake happens far too often. Sometimes it is due to budget or going after a lower cost solution. Other times, it is marketing fairy dust or high expectations set by sales teams. Regardless of the cause, the fall out is devastating for the organization and the internal champion who vouched for the tool.
My recommendation, look closely at the customer success team.” That’s a good point. “Their knowledge and their documentation before you sign. Keep in mind that if you dislike something during your POC or pilot, the friction will likely be five times worse after you buy the solution.” That’s a good point.
Sachin Bansal of SecurityScorecard said, “Here’s the board level gap – most organizations lack a documented exit trigger in their vendor governance. No defined KPI for time to rip out. I bake detection gap reviews into quarterly risk committee cadence so rip and replace becomes a governed decision, not a crises.
What metric would you hard code?” I kind of like that last one from Sachin. And it sounds easier said than done, doesn’t it, Steve?
[Steve Zalewski] [Laughs] Okay, maybe I’m going to cross the line. Okay? When you get married…
[David Spark] There is no rip and replace. [Laughs]
[Steve Zalewski] There’s a phrase that says it’s cheaper to keep her. Okay? Meaning the friction of doing a divorce with the business, to pull that technology out, to put something else in, it’s better to just get along and go along because the cost of rip and replace actually does more reputational damage to you and to your representation to the business than should you just stay the course.
Okay? GRC team likes it, you rationalize away the relationship, because the alternative is still a bar too high. And that is a terrible thing to say. But quite honestly, that’s what we’re up against. Because buying the technology is easy. It’s the nine months it takes to get it in and get it running right.
And then, okay, there’s potholes in the road, you tend to just find your way around the potholes rather than repaving the road.
[David Spark] That is a good metaphor – find your way around the potholes rather than repaving the road. So, again, like Sachin’s comment. But again, the reality of it, it’s like I’d love to do what you’re saying, but… Yes, Paul?
[Paul Guerra] Yeah, I think that’s why the POCs and pilot are so essential. And you have to set them up to be successful. So, if you don’t have your POC results like formally defined, they’ll lie to you unless you designed them to fail. Most POCs that you do, they’re guided tours, not a real stress test of the tool.
You talk to sales engineers, they’ll optimize for all the green checkmarks, not real world friction. And I love the comment around the customer success team. In my experience, that is an easy tell. Sales will show you the product at its best. Support will show you when the product breaks.
And a customer support team will show you what you’re living with after you’ve actually implemented, and you’re dealing with that pain. So, if something is annoying in these proof of concepts, think about it as operational debt after you purchase.
[David Spark] That’s a really good point. And honest to God, I wish I knew who said this, but I remember one of our guests said the question you should ask is, “Tell me about a time that things went really wrong and how you dealt with it. And also tell me how you deal with worst case scenario type things.” Like get the story of things going wrong out of them.
Steve, have you done that before?
[Steve Zalewski] Yeah. And here was a rule of thumb for me – I would want to meet whoever ran customer success. And if they weren’t an officer in the company, if they weren’t a VP, I’m not picking your product. Because customer success not patting me on the head and giving me a chocolate chip cookie.
Customer success is, “I’m not happy.” They’re calling the CEO and kicking the CTO out of bed to go fix the damn problem. Okay? Because they represent me, they don’t represent their company. And if I didn’t see that commitment in the company by making them an executive so that I can force that to happen, “Thank you very much.
Don’t let the door hit you in the butt.”
[David Spark] Similar story, Paul?
[Paul Guerra] Yeah, same thing. There was a couple companies that we looked at. We talked to their customer support team. We did not get the response that they wanted. They told us on the phone that they weren’t getting the right support. Things that were broken weren’t being prioritized by product or by engineering.
That told us everything we needed to know. Because I guarantee you, the first month in, we’d have issues. We’d have concerns. And we’d be one of those other people in the queue not getting prioritized.
Why is this an issue?
19:53.020
[David Spark] Sunil George of GlobalDots said, “The real damage isn’t the tool, it’s the time, effort, and credibility lost fixing the choice.” I would agree 100% on this. “Hope is simple. Look wider. Ask harder questions and plan for change, not perfection.” I think he sums it up really nicely.
Your fear isn’t the cost you laid out to buy this, it’s everything around it, isn’t it, Paul?
[Paul Guerra] 100%. So, ripping and replacing a new security tool sets the program back 12 to maybe even 18 months. Teams become now change averse to one bad bet, so they’re constantly now second guessing themselves. And then as CISOs, we lose leverage with the board and other C level leaders after these reversals are very visible to everyone else.
So, vendor turn, it just at the end of the day erodes trust. Even if the decision was reasonable.
[David Spark] Yeah, I got burned on a product a long time ago, and my whole thing now is know at the beginning how I get out of it. Steve, do you know at the beginning of purchase like, “How do I get out of this?”
[Steve Zalewski] So, there’s something I learned the hard way, which was security is the cost of doing business. It is a friction on an efficient business process. So, I stopped trying to make security important. And what I said to the business was, “For this amount of friction, I can give you this promise, insurance policy.” So that what I was already doing with the business is setting the stage for if the product became unusable, the friction was too high, the opportunity to turn it off and either accept the risk or go through an exercise to bring a new product in where the business was on board with me to go through the people and the process friction was the thing I learned.
Don’t make the decision yourself, take them on the journey with you so that if they go, “Yes, the friction on this tool now is so high that if you want to swap it out, we’ll agree with you, and we’ll go through the transition,” that was the secret sauce.
[David Spark] I think we’ve all learned something today. And that is that if you don’t want to get burned, ask the really tough questions up front. Don’t fall for the wonderful steak dinner, the great POC that never looks at the tough stuff. You do your own measurements, talk to the customer support team, and ask them what happens when things go really wrong and tell me a story of how you dealt with things when they go really wrong.
Paul, back me up on this. And what have I missed here?
[Paul Guerra] Some of the things I would agree upon… If we pick the wrong vendor, security leaders, we pay twice – once in budget, once in reputation. And then we also have to make sure… As we said earlier, vendors evolve faster than their contracts.
That’s why we’re going to one-year. And then attackers evolve faster than some of those vendors do. So, the buying strategy we have has to absolutely assume both.
[David Spark] I love that. Anything to add to what I said and what Paul said, Steve?
[Steve Zalewski] I would simply say you’re not wrong if you abandon a product before the license time is up. That more and more now, you have to be really sensitive to the environment that you’re operating under and be comfortable with realizing the world has changed, and you need to be able to change your ability to protect it.
Closing
23:33.650
[David Spark] All right, with that being said, I’m going to ask you first, Paul, of the quotes that I read here, all a bunch of great ones, which was your favorite, and why?
[Paul Guerra] So, I really like the last quote from Sunil, just talking about the damage it takes if the tool isn’t implemented correctly. And he’s talking about the time, the effort, the credibility lost. And the way I’d summarize it is we’re not going to always be right, but our goal isn’t to pick the perfect vendor, it’s to avoid the kind of mistakes that are expensive to unwind.
So, as long as we go through the proper process, we do our due diligence, we bring along all the stakeholders involved, and it’s all of our collective decision to move forward… If we’ve all made a collectively incorrect decision then we can go find the best solution in the future.
[David Spark] Good answer. All right, Steve, your favorite quote and why.
[Steve Zalewski] See, when you get a great answer like that, what do I do? But here’s what I’m going to do, I’m going with Sachin Bansal over at Security Scorecard.
[David Spark] The one just before. Yes, that was also good. The one I loved, difficult to implement.
[Steve Zalewski] Yeah, the one you love, difficult to implement. Right? Which is define the KPI for time to rip out. But the question was what metric would you hard coat, and my thought here is the metric to hard coat is what we talked about, is is the friction for the solution with the business too high that you want to make a joint decision to move on, not pick a traditional measurements of time to market or something else.
So, he’s absolutely right that we need to think it through but don’t think traditionally in understanding what the metric is.
[David Spark] Well, that brings us to the very tail end of the show. Thank you very much, Steve. Thank you very much, Paul. And huge thanks to our sponsor, and that is Native, the cloud security control plane for the enterprise. Remember, go to their site, native.security.
They’re the cloud security control plane for the enterprise. Unify, manage, and maximize built in cloud security controls and achieve secure by design consistency across cloud environments. native.security. Just go there. It’s spelled just the way I just said it.
Thank you, everybody. As I always say and always mean… No, I sometimes get into this very earnest voice. “I really mean this.” I’m not going to do that this time. I’m just going to say it straight out – we do appreciate your contributions and listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please, write a review. Leave a comment on LinkedIn or on our site, CISOseries.com, where you’ll also see plenty of ways to participate including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.