Just when we haven’t solved the proliferation of Shadow IT, we’re now dealing with Shadow AI. While much is the same, the “newness” is the scale and speed of AI advancement. We’ve been talking about digital transformation in the enterprise for several decades now. Is AI throwing a wrench into our very linear management process?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark, the producer of CISO Series, and Geoff Belknap. Joining us is our sponsored guest, Kara Sprague, CEO, HackerOne.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, HackerOne
Full Transcript
Intro
0:40.643
[David Spark] We’re all rushing to use the latest AI tools, so how do we start to manage the risk these underlying AI models are bringing into our organizations?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark, I’m the producer of the CISO Series. And joining me, it’s a CISO himself, it’s none other than Geoff Belknap. Geoff, say hello to the audience.
[Geoff Belknap] Hey, everybody. Good to see you again.
[David Spark] That’s Geoff. You’re going to hear that voice many times through this show. Our sponsor for today’s episode is HackerOne – the offensive security platform for the AI era. We’re going to learn more about that all throughout the show. In fact, they’re responsible for bringing our guests, but before I introduce our guest, let’s talk about today’s topic.
Just when we haven’t solved the proliferation of shadow IT, we’re now dealing with shadow AI. Now, while much is the same, the “newness,” I would say, is the scale and speed of AI advancement. We’ve been talking about digital transformation in the enterprise for several decades now, actually.
I’ll ask you, Geoff, you’ve gone through much of this. Definitely shadow IT. I don’t think we’ve solved it yet either, but there are solutions. Do you think AI is throwing a wrench into our very linear management process?
[Geoff Belknap] It’s interesting. I think shadow IT has really gone through the three-act play, and now we’re in a position where we largely understand how to manage this. This is a solvable problem, if not a solved problem in many organizations. AI is a little bit different.
We’re earlier in the stages. We don’t fully understand what the full shape and size of those problems are yet. In shadow IT, we really, we sort of understood we needed authentication, we needed some policy applied, we need to be able to manage the lifecycle of those things.
We’re not there yet with AI, and I think that’s the thing that’s throwing the wrench in. We recognize a lot of value, but we haven’t really identified everything that’s going to be a problem or could not be a problem down the road.
[David Spark] I think the reason for that, what you just said, is the people using AI don’t even know what they’re doing with it yet, and you can’t solve that problem until they figure out that.
[Geoff Belknap] I think it’s one of those rare instances where everybody recognizes there is value here, and we have rapidly put it in people’s hands before we really understand the full size and shape of that.
[David Spark] To help us unpack this quite complex problem, we hope to have an answer in the next 25 to 30 minutes, we’re going to find out. [Laughter]
[Geoff Belknap] I can’t wait to hear the solution.
[David Spark] All right. Our sponsor guest is actually the CEO of HackerOne. They are dealing with this very issue, and we have the head honcho with us. I’m very thrilled that she’s here with us. It’s Kara Sprague. Kara, thank you so much for joining us.
[Kara Sprague] Thanks so much for having me.
Is this problem solvable?
3:39.754
[David Spark] Shyama Rose, CISO over at Affirm, said, “This is not the first time we’ve had to adapt rapidly. Previous shifts like SaaS sprawl and decentralized API use taught us how innovation often outpaces governance. Each wave forces us to rethink oversight without abandoning our core principles.” That’s a good point there.
She goes on to say, “AI adoption is happening quickly, often outside of security’s line of sight, but fundamentally, embedded AI is still just a third-party dependency. It must be evaluated, challenged, and governed like any other technology. Novelty should never bypass scrutiny.
We also need scalable playbooks that address both tooling and human behavior because shadow AI is as much about people as it is about products.” Also a good point.
And lastly, Patrick McFadden of Thinking OS said, “Third-party AI isn’t a vendor risk problem. It’s a cognition control problem. Shadow IT became shadow AI. Features became agents. Plugins became inference surfaces. If your organization doesn’t control how cognition enters the system, it doesn’t control the system.” So, Geoff, both cases here with Shyama and Patrick saying, “We’ve seen this before.” But I like the way Patrick said it.
It’s just about how we’re bringing knowledge into the system and how we have to control it. But I must say, the way AI does it is kind of different, yes?
[Geoff Belknap] Yeah, I mean, Shyama has exactly the same thinking that I have on this. We constantly have to adapt rapidly. That really is the job. It’s a matter of figuring out how do we adopt some transformative technology in a way where we can manage the risk as best possible so that our organization can take advantage of it.
The hard part about it is, like I was referencing before, we don’t really fully understand AI yet or what the full benefit or full risk is. And so, I think Patrick might be thinking certainly ahead of me. I don’t disagree it’s a cognition that AI brings to you, but I don’t have a way to tag, sort, identify, or alert negative cognition or malicious cognition.
So, that is the size and shape of the problem. I don’t have the tools yet for that. And I think that’s the inherent issue and why I’m really excited to hear what our guest has to say because when I don’t have specific or purpose-built tools for a problem, what I need then is smart people that are thinking outside the box to bring me problems that I haven’t identified yet.
[David Spark] All right, Kara. I’m throwing this to you. How much of this can we look to our past experience and say, “Hey, just do what you did before,” and how much of it is, “Oh, no. We have to shift gears unlike how we’ve dealt with this before”?
[Kara Sprague] Well, I think it’s true what Shyama says. History has definitely taught us that every wave of innovation stretches the boundaries of governance. And we saw with SaaS and API sprawl that we had to rethink control, but we had to really figure out how to do it without stifling innovation.
But with AI, it’s not that we’re just managing new tools. We’re also managing entirely new forms of cognition that’s embedded into workflows, interfaces, and infrastructure. So, while AI components are technically third-party dependencies, they behave nothing like traditional software.
They adapt, they infer, they act autonomously, and they’re often doing this without clear boundaries or visibility.
And I’ve even seen some estimates that by 2028, over 80% of enterprise applications are going to include some sort of embedded AI, but unlike those APIs or SaaS features, these elements are going to retrain and evolve continuously. They behave non-deterministically.
They introduce emergent behavior that a governance checklist simply can’t predict. So, I fully agree with Shyama that we can’t have novelty bypassing scrutiny, but I think the scrutiny in this case really has to evolve. And what does that look like?
At least from what we see at HackerOne, that involves building playbooks that have to go beyond the policy, and actively simulating the misuse and manipulation. That means embedding red teaming, and as Geoff suggested, red teaming of a diverse group of people into your procurement and product workflows.
It also means treating AI not just as a dependency, but as more of a dynamic and evolving attack surface that touches both technology and behavior.
How do we handle this?
8:06.915
[David Spark] Matt Doughty of Prefactor said, “Human-centric identity models simply aren’t designed for the unique nature of AI agents. Managing agents like employees is a shortcut that won’t scale. When agents act independently, the traditional questions of ‘who defines access, manages the blast radius, and audits what happened’ becomes exponentially more complex.
Every agent needs its own distinct identity. Think of it as agent’s passport explicitly scoped to its function, inherently portable across environments, and fully auditable for every action it takes.”
Now, Mike Toole of Blumira said, “Give users access to better, easier options internally we control and heavily test. Then moving connections and OAuth grants to an allowless model to limit ‘random shiny AI product testing.’ If I can keep half of my horses in the barn, I still have half of my horses versus opening the door and hoping for the best.” So, I’m getting the sense from both Matt and Mike, they think this is more of an identity issue and managing access, and Mike has the hope of, well, the company and the security team, they’ll define what AI tools you use first.
And Mike says, “Even if I can just get half of the people that agree, that’s better than nothing.” What do you think, Kara?
[Kara Sprague] I think that’s part of the problem, what both Mike and Matt are talking about, and certainly our OAuth systems are going to have to evolve to take an account of agents. But this unauthorized use of third-party AI systems is not the biggest issue I see.
What I’m seeing, especially with first-party AI systems – like imagine every enterprise today is adopting AI into their applications, and so everybody effectively is becoming a first-party owner of an AI system – they’re often just as risky, if not more so.
And what we’re seeing from HackerOne’s AI red-teaming engagements across Fortune 500 and other critical infrastructure environments, our independent researchers are routinely uncovering these critical and high-severity vulnerabilities in production AI deployments.
These aren’t theoretical flaws. They’re real-world attack vectors, and it means that attackers can extract sensitive training data, such as PII and credentials. It means that they’re able to bypass access controls through prompt manipulation and context injection.
They’re subverting downstream workflows. They’re even creating persistence by embedding malicious logic into the model weights and plugins. And what’s even more alarming than that is you see how often these systems reach production without going through some sort of mature offensive security review.
So, what I see as a lesson here is that trusting an AI system just because it’s first-party, we’ve built it homegrown, it’s like trusting custom code just because it was written in-house. And so, what we would advise from a HackerOne perspective is that every AI system, even the ones that you’re procuring and you’re using independently kind of through an API, or the ones that you’re embedding into your applications and using as a first-party system, those need to be extensively tested.
[David Spark] Kara brings up a really good point, Geoff, because I have heard so many “experts,” not even experts, just people fearful of what could happen. And yet, it’s a talk all theoretical, not what’s actually happening. And really, yes, you’ve got to protect for the future, but really what’s happening now, the current attacks is really what you’ve got to focus on.
So, isn’t a more realistic testing philosophy with AI better than a theoretical, Geoff?
[Geoff Belknap] Absolutely. I think if you’re not using AI today, I’m sorry for you, you’re missing out, but also then don’t worry about it. Then you can write blog posts about it being Skynet or the end of the world. And certainly, I think we need to have philosophical discussions and technical discussions.
But if your organization is using AI today, you need to really find a way to cut through the emotion and the speculation and cut to what you’re actually dealing with today. We had exactly the same class of issues where emotion and fear were getting in the way when SaaS and IaaS were really coming around and being a big part of how enterprises do business.
And today, I can’t think of an enterprise where maybe 5, 10 years ago, you were like, “Oh, we’ll never go to cloud. Cloud’s super insecure. We can’t operate with cloud.” Today, I don’t know a business that can operate without cloud in some way in their organization.
[David Spark] That’s a good point.
[Geoff Belknap] It’s the same with AI. And the fear before was, “Oh, well, all our data’ll get stolen,” or “We’ll have all these problems.” And certainly, we have had challenges in the cloud space and some of them still persist, but in AI, the problems we have right now are not dissimilar than other problems we’ve had before.
We’ve got identity issues. We constantly have identity issues in the tech space where we have to evolve how we think about that and how we manage it. I think these are not intractable problems in the AI space to think about it. And we have your typical software dependency issues.
When people start adopting new technology, unfortunately, they start with YOLO downloading stuff from models or MCPs or whatever they might be from all over the internet, and that’s scary.
But that is a problem that your security department knows how to deal with today. It is only a massive problem if you are ignoring the fact that people are downloading random models, tools, MCPs, whatever from the internet. The unique and novel things are hard, and they are challenging, and I think they’re exactly what Kara talked about, where you can have prompt injections, you can have unexpected responses, you can have privacy or trust issues or things like that.
But I think if you focus on those issues and treat the other problems like problems you know how to solve today, that is a reasonable place to be.
Sponsor – HackerOne
14:01.322
[David Spark] Before I go on any further, I do have to tell you about our spectacular sponsor, HackerOne. We all know, as we’ve been discussing in this entire episode, AI is changing everything. Your product, your infrastructure, your threat models. And the only way to stay ahead is with security that’s just as smart.
HackerOne combines cutting-edge AI with human ingenuity to protect your systems from evolving threats in the Age of AI. It’s today’s conversation. So, AI is at an unprecedented pace. Cyber criminals are exploiting AI to launch faster, larger-scale attacks.
Meanwhile, organizations are rapidly integrating AI into every layer of operations, escalating risk, and calling for increased regulations. Bottom line – offensive AI is outpacing defensive strategies. Enterprises need a proactive approach to AI security and safety.
So, this is where HackerOne comes in, is the global leader in offensive security solutions. Trusted by AI innovators like Anthropic, Adobe, and Snap, HackerOne helps customers find and fix vulnerabilities across the software development lifecycle. So, the HackerOne platform offers bug bounty, vulnerability disclosure, pentesting, AI red teaming, and code security, powered by AI and the world’s largest community of security researchers that give organizations an unmatched edge.
Whether you’re testing large language models or identifying bugs before production, that’s what HackerOne does for you. So, AI needs both protection and participation. HackerOne uniquely offers both. Testing your models in ways internal teams simply can’t.
You can find out more by just going to the website, hackerone.com or hacker1.com. You’ll find it. You’ll get there. And when you get there, let them know you heard about them from the CISO Series.
Nothing will happen until we take action.
16:10.556
[David Spark] Jared Mendenhall of CyberCISO said, “Ideally, AI tools should follow the same evaluation, onboarding, and hardening processes we use for any new service. But in reality, that’s becoming harder to sustain. Existing tool services and workflows are actively being reshaped by AI.” Talk about that all the time.
“Executives are pushing for rapid adoption. Existing vendors are quietly embedding AI into platforms, often without advance notice. As a result, security is frequently left in a reactionary mode, left to assess risk and find security solutions post-deployment.
This isn’t a breakdown of process. It’s a case of security process simply being overrun by momentum, and we need to adapt accordingly.”
Brock Roderick of TSB New Zealand said, “The best you can do is to find your organization’s appetite for AI risk, create a safe environment for the business to meet their needs by assessing specific solutions, approving those designs with known security controls in place, and blocking/monitoring the rest as best as you can.” All right, Geoff, this has got to be happening to everybody in that the playbook or the plans they had before have just changed because, oh, what do you know?
All the tools we have are now embedding AI, and there are AI things going in that we didn’t know about, we didn’t plan on, we did not sign up for, and it’s just it’s now happening whether you like it or not. How has that thrown a wrench into anything?
Or maybe not at all.
[Geoff Belknap] My experience with this is really interesting. I work for an organization that’s both adopting rapidly AI technologies, but also building them and trying to deploy them to customers, so I live in a world where having very similar conversations to what Brock is talking about – how do I build an environment that is safe for developers and regular business operations folks to experiment with AI?
Whether it be our own or third-party AI and systems that are based on that. And as everyone has experienced, everybody and their brother, including us, sorry, including my employer, is adding AI to everything. So, sometimes, whether you like it or not, you’re experimenting with AI.
Sometimes it may be a surprise.
At the end of the day, though, all of these things are largely either applications that you run, that you are downloading, and you know how to deal with those, or they are web-based apps, and you know how to deal with those. There is very little here that is absolutely brand-new.
And I think you really do have to just create what’s our risk appetite, how do we build some guardrails around it, and you have to understand that if you don’t build guardrails to be able to go fast, people will just bypass security. I do think this will slow down a little bit as people start to understand where the sharp edges are, but I just have to remind all of my colleagues and peers, slowing the process down will make it worse, not better.
[David Spark] Good point. Kara, I’m throwing this to you. I can imagine that you have seen this shift in behavior that as these apps become AI-capable, the whole security landscape starts shifting, or what gets protected starts shifting. So, maybe what happened last year, now that with the AI enabled, it’s a whole different story this year.
Have you seen that?
[Kara Sprague] Definitely, definitely. And HackerOne is very much like Geoff’s organization, which is we’re both embedding AI into the technology that we are using, but we’re also adopting it aggressively across our organization, so we see this every day.
I really don’t think that this is a process failure. I would describe it more as a velocity mismatch between innovation and control.
[David Spark] That’s a polite way of putting it, by the way, Kara.
[Laughter]
[David Spark] A velocity mismatch. That, by the way, is my favorite so far. Go ahead.
[Kara Sprague] Security teams were built to manage change, and what we’re seeing is different now is the rate and opacity of that change. AI is getting embedded into these tools invisibly, oftentimes without an explicit knowledge. It’s getting activated by default, and it’s evolving via this continuous retraining.
And so, what used to be a quarterly product update now turns into a rolling wave of, I’ll use Patrick’s term, cognitive upgrades. And that has risk implications that emerge downstream, not at deployment. The result here is that the security team is forced into post-deployment triage.
And what we see, and our companies need to do to adapt, is we need to shift from this static gatekeeping role that many security teams have been playing to much more real-time validation and adversarial simulation.
So, I’ll hit it again – red teaming AI behaviors is critically important. You have to do the behavior, not just the interfaces or APIs. This is about continuous trust evaluation, not just a one-time onboarding exercise. And it’s about monitoring the autonomously for emergent behavior and not just doing static policy enforcement.
So, the answer here, I don’t think is slowing down adoption. Much like Geoff said, I don’t think you can slow down the adoption. It’s about how do you embed safe and secure deployment practices through offensive security programs that enable you to test the effectiveness of your controls, inform your defense, and rapid response approaches to this innovation.
[David Spark] Kara, let me ask you a follow-up question. You gave kind of three points of how you have to look at this differently. From your would-be customers, whomever you’re talking to, what either do they struggle to do or becomes the biggest shock to them?
[Kara Sprague] I think what they see already and seems to fit naturally into existing security testing mechanisms is looking for security flaws and, for example, the APIs and the systems around the AI model. Where we’re seeing the truly unique items is when we’re testing the emergent behavior of models.
I’ll give a couple examples. We are working together with one customer that has a big consumer application, and one of the values that they’ve seen out of their AI red team engagements, they had one person who is a photographer as one of the researchers, and they ended up using a bunch of very specific photography language to end up getting the model to produce images that this client would not want shown outside, so it violated their safety practices.
Another example that came out of another customer is they had another person who was a Shakespeare aficionado and started quoting a bunch of Shakespearean terms into the model and jailbroke the model.
[David Spark] [Laughter]
[Kara Sprague] And so, what’s really surprising and interesting here is you’re seeing these researchers, and this is why we advocate for diverse researchers, you have researchers with this really idiosyncratic domain expertise that are able to make these models produce behaviors that the customers or the people that are using the models in their applications would not want that to happen.
Does it play nicely with others?
22:52.678
[David Spark] Christian Rose of TSB New Zealand said, “I spend a lot of my day disabling AI tool sets within apps, but educating users about it, the risks, not just of data, but of the queries they are typing into AI tool sets is still key. Anyone who has kids know telling people they can’t use X means they will simply find a why they can use.
So, there is the illusion of control. Nobody can access AI from their corporate devices, so they just do it from their personal phones instead, and you lose visibility of what they are doing. I argue that any forward-thinking company today needs to provide an approved tool internally, correctly implemented and monitored to make sure data is correctly labeled, but channel everyone through something you know about so you know about it and can deal with the inevitable issues when they will come up.
Blocking is no longer enough. We have to adapt.”
So, Geoff, most people are not blocking things like in the past. I think there was a lot of like, “We’re not doing the cloud,” and then finally everybody came on board. Very small percentage, but again, all businesses are different. banks, healthcare organizations different than a retail store.
This all sounds good, but easier said than done, yes?
[Geoff Belknap] Yeah, absolutely. And I think my big message here is please don’t make your teams block all the AI things. That is not a great use of time. What is a great use of time is exactly what Christian’s talking about here, which is give people a paved path.
Give them a path to experiment and a path to use these things in a way that you feel is safe for you.
[David Spark] By the way, can that path be two-way? If you’re giving a paved path, that means someone has to be the AI czar, if you will, and find all the tools that everybody wants to use and know what everybody wants and needs it. But rather, couldn’t it be kind of a virtuous cycle where people can recommend things, you’re learning about it, then you’re like, “Okay.” Like, yes, it could go in that way?
[Geoff Belknap] Absolutely. I would be shocked if you’re a significant organization right now if you don’t have somebody who’s responsible for figuring out how AI’s going to work into your environment. And if you don’t, you either have to decide you’re going to block out everything that’s new and adopt it 10 years later, or you have to just accept that this change is coming, and you’re going to have to adapt.
I just don’t see how you can put your head in the sand right now. If for nothing else, other than, like I said, everybody and their brother is putting AI in their apps, you’re not going to be able to catch up with turning it off everywhere. And I do think, frankly, while there are significant risks in AI, depending on how you deploy it, most of those risks are not the end-of-the-world scenario that everyone’s worried about.
So, I think safe experimentation and being reasonable about how you respond to security threats or how you test things is really the way to go here. Just like you couldn’t succeed blocking all the cloud or SaaS apps from your organization, this is going to be just as futile.
[David Spark] Kara, from what I’ve heard you say so far, this sounds great, wonderful, sure. But if you don’t test real-world attacks, this doesn’t matter. You could do it this way, you could do it the other way, whatever. But if you don’t test real-world attacks, that’s really what it boils down to.
Yes?
[Kara Sprague] I would even go a step further than that, which is to say in this kind of environment where really, really the horses are out of the barn, blocking isn’t going to be effective, and you want your employees, in order to really thrive in this era, you want your employees experimenting with these new technologies.
The only approach really is to be doing a lot more testing and a lot of adversarial testing to make sure that what’s actually happening in your environment on that day is something that is not going to be disclosing or creating a massive security breach for you.
There’s not really a structured process you can put around it. You can pave a path and put in place a well-architected AI environment, say, “These are the safe ones to use,” but you’re still going to have a bunch of things that get past that.
[David Spark] Very, very good.
Closing
27:01.118
[David Spark] Well, that will bring us to this portion of the show where I’m going to start with you, Kara, and I’ll ask you which of these quotes was your favorite and why?
[Kara Sprague] Yeah, the quote I think that really stuck with me was the quote from Patrick, and the reason why that stuck out for me was because he was the one that really articulated what is different about these AI systems from prior generations of technology.
So, yes, it’s true. AI is developing and evolving much more rapidly, but I think what’s even more important and different about what AI is doing is that this is a decision-making surface area. And so, this isn’t just like a governance gap or a new technology or needing your process to move more quickly.
We’ve now got things in our environments that can be used to shape, mislead, or overwhelm our human and machine cognition. And so, this is a really different kind of security flaw and security issue that organizations are going to have to deal with.
[David Spark] Very good. Geoff, your favorite quote and why.
[Geoff Belknap] As always, we’ve got some great input here from Shyama and others. I’m going to go with Mike Toole from BlueMira, and I’m going to have to find a way to work in keeping half of my horses in the barn while still having the barn door open.
I think this is, as much as I hate to admit it, kind of the policy I’m taking, which is I think we need to try to build paved paths so people can experiment safely, and we can understand both the value and the risk that we’re taking with AI, and I don’t think we can do that without taking some risk, right?
Our job here is to mitigate risk and to manage risk, not to avoid it altogether, and so I think this is the way we’re really going to have to learn.
[David Spark] Well, that brings us to the tail end of the show. Our sponsor has been Kara’s company, HackerOne. By the way, you just go to hacker o-n-e. I believe I went to Hacker and put the number 1 .com would I also get there? Yes, you can do the number 1 or type o-n-e.
So, HackerOne, any way you want to spell it or write it out with a number, you’ll get there. Hackerone.com, the offensive security platform for the AI era. Thank you so much, Kara. Now, anything special you want to tell about HackerOne for our audience?
By the way, if you want to reach Kara, we’ll have her LinkedIn profile linked to on the blog post for this episode. And, oh, are you hiring, Kara?
[Kara Sprague] Yes, we are hiring. And a very cool thing about HackerOne, we both in-hire and we have many roles open for our company, but equally so, we want to recruit as many security researchers to come to the HackerOne platform as possible. So, if you are interested in breaking things, if you are a creative person, and you want to be part of this generation of researchers that red team all of these AI systems, please come to HackerOne.
[David Spark] They would love to have you there. That’s awesome. Well, thank you very much, Kara. Thank you very much, Geoff. And thank you to our audience. As I always say, and I always truly mean it, see, you notice I purposely slow down to know that I’m not rushing through this and actually care about what I’m saying here, Geoff.
We greatly appreciate your contributions and for listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.