Penetration testing hasn’t fundamentally changed in decades, and it’s starting to show. You commission a test, wait for the report, and by the time it lands, the application has already changed. On the other side of the equation, pen test firms constantly rotate their bench, and the testers who did sharp work last year have left for somewhere else. Now add AI-generated code to the mix. Those models were trained on open source repositories full of known vulnerabilities, and that code is flowing directly into production. The attack surface is expanding faster than any group of human pen testers can cover.

In this episode, Nico Waisman, CISO at XBOW, explains how XBOW uses autonomous AI agents to run continuous, incremental penetration testing without triggering false-positive avalanches or taking down production systems. Joining him are Jacob Combs, CISO at Tandem Diabetes Care, and Davi Ottenheimer, president at Flying Penguin.

Want to know:

• Why can’t traditional pen tests keep up with modern attack surfaces?
• How XBOW’s attack credit model maps to the way security teams already size testing effort?
• What stops an autonomous pen testing agent from causing real damage in production?
• How incremental testing works when a new pull request changes the application?
• Where XBOW is headed on prompt injection and LLM-specific vulnerabilities?
• How you audit what the AI actually did during an assessment?
• What novel vulnerability chains are emerging as AI reasoning models get more capable?

Check out the episode for the answers you need.

Join the conversation on LinkedIn.

Huge thanks to our sponsor, XBOW

XBOW is an autonomous offensive security platform that operationalizes penetration testing as a continuous, AI-driven system. Leveraging multi-agent AI to emulate attacker workflows, XBOW autonomously discovers, exploits, and validates vulnerabilities across applications, providing reproducible proof of exploitability. XBOW enables enterprises to prioritize risk, compress time-to-remediation, and scale offensive security coverage.

Full transcript

[Music]

[Voiceover] Connecting security solutions with security leaders, Security You Should Know starts now. 

[Music]

[Rich Stroffolino] Welcome to Security You Should Know. I’m your host, Rich Stroffolino. Today, we’re going to be talking with XBOW and what they’re doing in autonomous offensive security. Now, the problem that they’re addressing, it seems more relevant than ever.

It’s that traditional pentesting, not really able to keep up with the pace of modern attack surfaces. Helping us get some answers to why this is a problem, what needs to change, and more about what XBOW is doing are Jacob Combs, the CISO over at Tandem Diabetes Care, and Davi Ottenheimer, president at Flyingpenguin.

Davi, I’m going to get started with you here. Where are traditional pentests maybe letting us down a bit? 

[Davi Ottenheimer] Ah, it’s a question of where to begin. There’s so many things that go wrong. I have the problem of snapshots, where you grab a thing that’s already decayed by the time you get your report back in. There’s a scope compression because of budgeting, very expensive, but you only get to look at a few things.

There’s a sort of monoculture of we’re actually getting a review of what they’re capable of, as opposed to what actually is wrong. And everybody seems to study at the same school, so you get the same capability or the same black hat report tested on you and not something novel or new.

The adversaries are always learning, but the pentest is over, [Laughter] and so by the time you actually get the attacks, they’re beyond what you just tested for. Yeah, where do I stop? 

[Rich Stroffolino] It’s rough when linear time is one of the failings that you have to kind of conquer. That one is always a bummer for me. Jacob, I’m going to go to you. Where are you seeing traditional pentestings maybe not keeping up with modernity here?

[Jacob Combs] Yeah, Davi said a lot of it really well. And one of them that I’m going to keep hitting on what you just said is this point-in-time picture that you get. You get a snapshot of something that’s happened at that moment. When things change, the dynamics change on both ends drastically over time.

The other one that I’m seeing that’s a little bit maddening actually over the past two years is the vendors that we work with continuously rotate their bench of testers. So they’ll come in on day one, they’ll be hotshots and be finding all kinds of things, but then a year later, you have to go find another vendor because they just lost their muster or their talent has left and started their own firm or gone somewhere else.

So that continuously rotating bench, what I’ve been trying to do is chase around the talent, but I can’t always find them, or they’re not as available or always within my price range. So that plus how expensive it is and how everyone relies on it as the last test before we go live in production is something that’s been, for me at least systemically, a problem over the past few years.

[Rich Stroffolino] Jacob, I love that human element of it. I don’t think we hear about that enough in terms of the very real human capital, right, that these firms have and then can go elsewhere, I think is something underappreciated perhaps in the industry.

Well, today we’re going to be talking with Nico Waisman, the CISO over at XBOW, to help us figure out what they’re doing, how they’re operating in this space, maybe helping to solve this problem. Nico, help us out here. We need to answer three essential questions before we dive deeper here.

So, can you explain the value of what XBOW is doing to my CEO? What does your solution do? What does it not do? And what is the pricing model? Can you help us out here? 

[Nico Waisman] Absolutely. So let me start with the first one. I think pretty much every CEO right now is focusing on how they can accelerate with AI, right? That’s in everyone’s mind. And the question on the security side is the same question as the CEO had, which is how we can accelerate security with AI.

Because the reality is that we are living in a world where a lot of code is being generated with AI. It wasn’t like that six months ago, but things completely flipped from one day to the other. And now we have way more code. Some of the models, all of the models actually, have been trained in a lot of open source code which has vulnerabilities by itself.

So we are seeing some of the code that is being generated by AI also having a lot of vulnerabilities. On top of it, it’s just like the surfaces continue growing. 

So, the question to how we can address that problem is now how we can better use AI ourself as security practitioners to figure out how to solve all these problems. And what we’re seeing is that we are using AI to find more vulnerabilities. Fantastic.

We all have different tools and find vulnerability. The question is how you prioritize what is the vulnerability you have to fix. And I think that penetration test has always been the answer to that, right? It has been around. Static analysis will find a lot of vulnerabilities, but what are the ones that actually can tell you if you’re at risk or not?

And this is where penetration tests came into place. 

Now, what XBOW is proposing here is let’s bring AI into penetration tests, which is the ultimate test in terms of being able to understand what is the real impact of a finding. Because exploit is kind of, in many ways, the way they show you risk, and how we can accelerate that.

As everyone said, not as a point in time, but more of something that continues. 

[Rich Stroffolino] And then in terms of pricing, what are we looking at? 

[Nico Waisman] It’s a consumption-based model. What we did in terms of pricing is quite interesting because we talk with security teams and you tell them like, “How big is your application?” And they will say, “I have no idea.” “Give me lines of code.” Sometimes it’s really hard to get the lines of code.

“Tell me how many dynamic pages are endpoints.” Another impossible question. So, we try to kind of like find a way where they were very comfortable. And for security teams and application security teams, they’re comfortable with the past, which is how many hours or weeks they have spent in the past assessing the security of that application.

And so what we came up with as a model, as a consumption model, is this idea of attack credit. So, one attack credit corresponds for the amount of work that a human will do in an hour. So it’s a unit of work mainly. And so like in your previous penetration tests, this application took two weeks to perform the penetration test.

That means for us, it’s like 80 attack credits. That doesn’t mean the pentest with XBOW is going to take two weeks. It’s actually going to take 12 hours or less. But the point is that we can size your application based on that effort. 

And this is also important because not every application has the same risk level. Like your crown jewels, you want to spend way more time. And this is why you’re actually doing a three, four weeks penetration test rather than one week. And then some of your internal applications, you might not need to spend that amount of time.

So, you can actually tell XBOW, like, “I only want to invest 40 attack credits here and just do a lightweight penetration test.” XBOW will focus on what is most important for the application or what are the areas that potentially can find more bugs, and if not, just leave them behind.

And if you obviously use more attack credits, you get like an overall penetration test. 

[Rich Stroffolino] All right. Well, we’ve gotten a taste. We’ve gotten a cursory overview what XBOW is doing here. I’m sure lots of questions here. Jacob, I’m going to start with you. What other questions do you have for Nico and for XBOW? 

[Jacob Combs] So, this is a pretty interesting configuration you have there in an interesting offering. The question I would have is, I’ve seen a lot of issues with companies where they’re using up their AI tokens in a very short time. Like an example I’ve seen is Coinbase went through their whole AI budget in a quarter when they meant to have it for the entire year.

How have you seen some of your customers or anyone leveraging the tokens or the attack credits you spoke of with your product? 

[Nico Waisman] Yeah, that’s a very good question. Generally, we see most of our customers using between 4 to 120 attack credits per application. That is for like the baseline penetration test. This is something that I didn’t mention before, but one of the kind of like big advantage of XBOW is that once you run baseline penetration tests, then later rather than rerunning a full penetration test, you can run an incremental assessment.

So, let’s say you started with 80 attack credits, and then as we discussed before, penetration tests are point in time, and you don’t want it to be in a point in time. Like tomorrow, there’s a new PR that gets deployed. That means that the application completely changed in terms of security.

What we are also doing at XBOW is doing this idea of incremental penetration test. So rather than using another 80 attack credits to go and do a penetration test, we only test the deltas between the new version and the previous version. So you’re using maybe, for a new assessment, 2 or 3 attack credits rather than spending the 80 to 40.

[Rich Stroffolino] Davi, I’m going to get you in there. What questions do you have for Nico and about XBOW? 

[Davi Ottenheimer] I kind of want to key in on the token thing again. For me, if you have a vendor who’s defining the unit of measure, if you said it was a week, your week and my week are both measured in time, but what commitments have you made to really be sure that you aren’t taking a credit as an opaque unit that rewards inefficiency instead of results?

Or what we’ve seen in the AI industry is tokens are abused like crazy to benefit the vendor and to harm the customer. But let me throw in another one there too. In AI space – this is actually related – how do you test for the integrity of systems or integrity breaches?

Because we always test around confidentiality breaches, availability breaches, and things would be bad if the system leaked or went offline, but where are you testing for the system producing bad answers, like a token being rapidly inflated artificially?

[Nico Waisman] So, the answer to that, and I mean, you can ask the same question to a penetration testing team, right? At the end of the day, what exactly were we able to accomplish during the period of the assessment, right? That’s basically the question you’re asking.

And we pray around being able to answer that question for our customers, which honestly is a big question in the penetration testing world. And in order to do that, what we do is we collect everything that the AI did. We collect every network package.

We collect every log from the agent. And at the end of the penetration test, you can have a pretty good idea and full auditability of everything that happened through the penetration test. Every attack that was performed to each of the individual endpoints, you can search, you can have a very beautiful tree map that you can see in a demo of all the different endpoints and what was XBOW able to test on each of those individual endpoints.

So you’re given a good understanding of how much was invested during the assessment in comparison with the consumption that happened during the assessment. 

[Davi Ottenheimer] Okay. I want to go a little deeper into the pentesting itself of environments or code that needs to surface that things can be inaccurate. In other words, there’s a trend now in the industry, I find, that people are getting things that are not…

It’s a supply chain issue that we’ve known for a long time, but they’re not getting what they ordered. So if you are getting water out of a pipe and then you get contaminants in the pipe, there’s a problem in our physical world. But in our code, we’re depending on a lot of information flows now that are getting contaminated by the systems we depend on.

And how are you addressing testing for that in a way that people can say, “Okay, you’ve tested it to make sure that we have protections, controls against integrity breaches”? 

[Nico Waisman] That is an amazing question. And to be honest, before getting to XBOW, I was the CISO at Lyft, and it was one of our biggest burden, the supply chain. But the big problem with supply chains, especially the tools around supply chain, is that there’s a huge amount of false positives.

And that has a burden to engineer because as you mentioned, maybe the supply chain is tainted, but really not touching your code at all. Like you have a library that has a vulnerability or something else, and maybe that vulnerability is not really flowing through your application.

Maybe you’re not even touching that part of the application that has a vulnerability. 

So at the end of the day, the question is around what is the real risk that you have by using this old library? We struggled a lot with that when we were at Lyft because we will present the vulnerability to the engineers, and the engineer will be like, “You know what?

Yeah, I’m not using that actual library. It’s in my code, but I’m not using it.” But once you report it, you start the SLA and you start all that motion that requires someone to do it. So, we believe that this is why penetration tests is kind of like the tool to prioritization because you can say – if XBOW actually finds a way to exploit that vulnerability – then now you can say that the risk is there versus having something that is potentially vulnerable or not.

Having said that, we are not specifically focused in the supply chain. The supply chains came into place based where XBOW is able to attack any of those libraries that might have vulnerabilities or not. But we are not specifically looking at your code and reviewing the supply chain.

[Davi Ottenheimer] What about prompt injection then? I mean, the supply chain itself is a poisoning issue, but what about poisoning the, or, you know, like SQL injection used to be the thing we talked about. Well, now we’re injecting the prompts and now we have MCP transport integrity and tools that can be the open cloud.

Classic example is that the tools are so poisoned, they’re stealing all of your tokens and keys, and they tell your car to crash into a pole because there’s no way to stop them from changing all the things they were programmed to do. Someone’s changed all the programs, I should say, or changed the tools, changed the skills under the covers, and you have no idea.

How are you testing for that? 

[Nico Waisman] Yeah, that’s a huge concern for us. We right now are slightly touching that surface, and we want to invest most, we have in our roadmap both for Q2 and Q3, to start focusing more on LLM type of vulnerability, prompt injections. Everyone is using agents nowadays, so that’s clearly the next step.

Everyone is interacting anyway with agents, so that’s absolutely what we’re going to aim for. It’s a complex vulnerability to find because there’s a lot of contextual business that the LLM or the product needs to understand to be able to separate the needle from the haystack.

Like finding the true positives in all those false positives. If you run it without properly doing validation and fine tune it for these kind of like vulnerabilities, you will end up with a lot of false positives. And this is what we are super worried about it as part of XBOW.

But we believe that with the new modern models, GPT 5.5, Opus 4.6, and 4.7, we’re seeing the value of the high-reasoning models, and we know that we can attack that problem. It’s just a matter of having the time to properly implement it and test it, which we have currently going on.

[Jacob Combs] I like that because one of the things I worry about with a tool like this is another pile of vulnerabilities that my software development and IT teams have to deal with. Just adds another bit of noise to them, so that’s great. You kind of filter and use the intelligence to make it more practical.

I have a question on the other end, actually, is about one of the things that you’ve seen in the news, right, is that someone’s using an agent and it takes down their whole system or deletes the production database and all the backups or whatever. And a tool like this, as a CISO, I’m concerned about it being overreaching and doing too much and maybe damaging something or something along those lines or causing a denial of service.

Do you have protections or even like the Tesla FSD, do you have a ludicrous mode you can set up or is there a more safe conservative mode as well? How does that work? 

[Nico Waisman] A hundred percent. So, when we started XBOW around two years and a half ago, we were playing around with GPT 3.5. We didn’t even know GPT was good at it. But when we started seeing some good results out of it, right away we saw that when, for example, like GPT will try to find an SQL injection, it will try things such as like drop table.

So, we were [Laughter] like, “We’re not able to put anything in front of our customer.” If it’s like some of the test cases, it’s like dropping table. So, we build safeguard from the early go at XBOW. And we do it in, I would say, in three different ways.

First of all, we do it at the network level. Like we have a proxy where everything that XBOW do goes through that proxy and that basically stop XBOW from going out of bounds, right? That’s number one. 

Second one is the way we find vulnerabilities is more focused on trying to be as harmless as possible. Just to give you an example, for SQL injection, what we do is rather than trying to say dump me a table or a database or a username, we try to do injection bugs.

So, we try to insert like slips within the string of a query. So, we kind of like measure the response rather than seeing the results from what you’re getting out of an SQL injection. That’s our measure. So every vulnerability class that we look for, we try to do it as harmless as possible.

Number three is, on top of every action that an agent do, we had another small SLLM that is focused 100% on safety. So we review every action that’s performed and confirm where that is a safe comment or not. And now, there’s obviously a balance there, right?

Like sometimes it’s like you have to figure it out whether the attack you are doing might actually not be harmful or is part of the penetration test. So, we are constantly working around safety into a balance between what a pentester will do and what are the things that you should not be doing.

[Jacob Combs] Yeah, I love that. It’s a great look into your agent architecture you’re building there. So, on the back ends, a lot of my trouble is, as a CISO, is that I’m having to deliver all these vulnerabilities and communicate, especially a pentest, communicate to the developer what the problem is.

How do you do that? Do you have like a last mile handoff piece that makes this easier for us or how does XBOW stand out? 

[Nico Waisman] So, we have an API that you can use and insert in your workflow. I know that everyone is selling their own vulnerability management tool. We also have integration with like GitHub and some other ticketing tools. But I think most important, what people find more useful has been on the API side.

One of the things that we say about engineers, which I like a lot, is that one of the big burdens that I used to have as a CISO was that we are not providing enough information for engineers to act on. Or like sometimes you have to convince the engineer that is a true positive versus a false positive.

One of the things that we do, and this is part of just running a penetration test, is that every vulnerability that XBOW finds has an exploit or a POC associated with that. So if the engineer is kind of like trying to challenge the vulnerability, you can be like, “Just follow the three or four steps that XBOW is providing, and you will see yourself what is the impact.” 

[Jacob Combs] Is there like a rerun capability? So, where like the developers say, “Okay, I made the patch.” We can rerun it and validate it? 

[Nico Waisman] Yes, yes, absolutely. And the way we do it is kind of interesting because obviously, we provide all the details on how we reproduce the vulnerability to an agent, but what we tell the engineer is like, “Go and find that vulnerability.” And as a result, what’s happening is it’s not going to just reproduce the vulnerability.

It will try to find a way around the patch to see if it can actually trigger again. And we have a bunch of interesting examples where when we are with a customer and the customer told us they fixed the vulnerability, we run it and what ended up happening is that they put a rule in front of the WAF to specifically stop that vulnerability.

What ended up happening is XBOW ended up mutating the string to bypass the WAF and triggering the root cause. 

[Davi Ottenheimer] Oh, I love that. That’s great. That escalated quickly from harmless to exploit.

[Jacob Combs] Yep.

[Davi Ottenheimer] I’m not sure I believe your definition of harmless is the same as mine, but… [Laughter] 

[Nico Waisman] I mean, let me tell you one thing. I always say that to customers, “This is a penetration test.” So you should be aware a penetration test is basically affecting the functionality of the environment, and we try to make it as harmless as possible, but there’s always a risk of something going the wrong way.

[Davi Ottenheimer] I guess my question would be do you have a safety LLM? So, I would question is the safety LLM working if you’re able to get your job done? Because have you captured the safety LLM in the way that it’s trained on the same basis as the other LLM and there’s no real safety at that point?

[Nico Waisman] Well, that’s why we use two different models for that. Because if you use the same one, they’re probably thinking across the same lines. So, we have one whose kind of like objective is to find bug and another one whose objective is specifically focused on that.

They’re completely separate from the product itself. 

[Davi Ottenheimer] Sounds like the safety one is losing. That’s all I’m saying. 

[Nico Waisman] It should be, it should be. [Laughter] 

[Rich Stroffolino] All right, Nico, before we get out of here, what’s one thing we didn’t ask about that we need to know? 

[Nico Waisman] I guess the big question that some of our customers are asking is how are you finding novel vulnerabilities, and what we are seeing is that as the progression of the models are coming, we’re seeing some kind of like spark of creativity on the models that are suddenly changing vulnerabilities or finding things that people have never thought of.

Or even exploiting vulnerabilities in a way that maybe some pentesters have never thought of. So we are slowly seeing, and not slowly, quite fast actually, seeing some spark of creativity in LLMs that are taking it outside of your low-hanging fruit and taking us more into advanced research vulnerabilities.

[Music]

[Rich Stroffolino] Well, that’s just about it for this episode of Security You Should Know. To learn more, head on over to xbow.com. And if you have any questions or feedback for this show, send them to us, feedback@CISOseries.com. A huge thanks to Jacob and Davi for helping us learn more about XBOW.

And thank you so much, Nico Waisman, for your time and being game to answer all of these questions. And thank you for listening to Security You Should Know. 

[Voiceover] That wraps up another episode of Security You Should Know. If you like this program, please subscribe, tell your friends, and leave us a review. All companies showcased on this program are sponsors of CISO Series. If your company would like to be spotlighted and interviewed by our security leaders, go to our contact page on CISOseries.com or just email us at info@CISOseries.com.

Thank you for listening to Security You Should Know, connecting security solutions with security leaders.

[Music]