Every company wants AI agents in production… yesterday. The trouble is that these agents are a new class of digital worker, and most organizations have no equivalent of an HR file for them. We know who our employees are. We have controls for applications, infrastructure, and service accounts. But ask a security team who created a particular agent, what data it can reach, what actions it can take, or what evidence exists after it acts, and the answers get thin fast. Agents are non-deterministic by nature. They stitch together a handful of perfectly normal permissions and produce a deeply abnormal outcome. They move at machine speed and machine scale, which means when something goes wrong, it goes wrong a thousand times before anyone notices. Identity, authorization, observability, and policy enforcement were built for humans and service accounts. None of that translates cleanly to a workforce that writes its own code, calls its own tools, and makes its own decisions.

In this episode, Venkat Siva, co-founder and CEO at CompFly AI, explains how his platform gives security, engineering, and business teams a control plane for autonomous AI agents across their full lifecycle. CompFly discovers agents, assigns each one a verifiable distributed identity, runs adversarial and safety simulations before launch, enforces deterministic policies at runtime through a gateway, and produces immutable audit logs for compliance teams after the fact. Joining him are Mike Lockhart, CISO at EagleView, and Gary Chan, System VP and CISO at SSM Health.

Want to know:

• Why is safely deploying AI agents still an open question for so many organizations?
• How do you govern an agent that can stitch normal permissions into abnormal outcomes?
• How does CompFly plug into a stack that already includes Bedrock, Copilot Studio, LangChain, and custom harnesses?
• What happens when an agent action gets blocked, and what does the audit trail look like?
• How do you validate that the policies CompFly recommends are the right ones for your agents?
• Where do the boundaries sit when agents have code execution capabilities of their own?
• Who watches the watcher, and how does CompFly itself stay trustworthy?

Check out the episode for the answers you need.

Join the conversation on LinkedIn.

Huge thank you to our sponsor, CompFly AI

Full Transcript

[Music]

[Voiceover] Connecting security solutions with security leaders, Security You Should Know starts now. 

[Rich Stroffolino] Welcome to Security You Should Know. I’m your host, Rich Stroffolino. Today, we’re talking with CompFly AI and what they are doing in identity and access management. Now, we’ve talked about identity and access management before, but the problem that they’re specifically addressing is a big one, never felt more relevant, and it’s all about the problem of how do you safely deploy AI agents?

Everybody’s wanted them since yesterday or even before then. Doing it safely still feels like an open question for many organizations. Helping us figure out why this is so hard are Mike Lockhart, CISO over at Eagleview, and Gary Chan, system VP and CISO over at SSM Health.

Mike, I’m going to start with you. Why are we having trouble safely deploying AI agents at this moment? 

[Mike Lockhart] It spans a spectrum of issues that all of us are trying to tackle today, but it really breaks down to authentication and identity. How do we prove who you are, what you are? A lot of agentic capabilities have been bolted on to pre-existing design patterns that weren’t really intended for agentic use.

The agents are nondeterministic. There are things that they can do that go outside of the scope of the guardrails that we deploy, so drift and decomposition are huge issues. And those are just two of the things that are part of a larger spectrum of problems that we encounter of observability, auditing, monitoring, judging, but it really starts with identity and authorization as the starting point and then making sure your agent actually behaves in the way that you expect it to.

[Rich Stroffolino] Gary, I’m going to come to you. From your perspective, why are we struggling in this moment with deploying these AI agents? 

[Gary Chan] So, first off, I agree with everything Mike said, but because I want to have some variety, I’m going to focus on something else. I’ll mention speed, and this is speed just for deployment as well as the speed at which the LLM or whatever AI agent, how it operates.

And because of that speed, it can do so much in such a short period of time that you’ve got to be sure that it’s right because otherwise, it’s going to do a lot of really bad really fast. And then, of course, there’s the scale as well. So, if you have multiple instances of it, you can now do way, way, way more.

So, I would say speed is just like a huge aspect of why we got to get it right because otherwise, it’s just going to blow up. 

[Rich Stroffolino] Well, today we’re going to be talking with Venkat Siva, the co-founder and CEO over at CompFly AI. Super excited for this conversation. But to start out, we need the answers to some preliminaries here. So, here’s our classic trio of questions for Security You Should Know.

How do I explain the value of your solution to a CEO? What does your solution do, and what does it not do? And what is the pricing model? Venkat, can you help us out here? 

[Venkatesh Siva] For sure. Thank you for having me, Rich. Companies are about to introduce a new class of digital workers. These AI agents do not just answer questions. They can access data, they can call tools, write code, trigger workflows, and make decisions at machine speed, like Gary and Mike was talking about here.

So risk is not just about that the AI can answer something bad. The risk is largely about these autonomous agents taking an action that the businesses cannot control, see, or explain afterwards. Today we do have controls for employees, for example, applications, infrastructures, and service accounts, but most companies do not have a control plane for these agents to know who created these agents, what can it access, what actions can it take, and what evidence do we have.

CompFly helps businesses move faster with AI without asking security to blindly trust it. We give security, engineering, and business teams a way to say, “Yes, you can go and deploy these agents, but every agent has to be known, tested, governed, and auditable.”

So what do we specifically do? We actually help govern autonomous AI agents across their lifecycle before they launch, while they are acting, and after they act. Before they go live, we help discover these agents, understand its owner, purpose, permission, tools, and then we simulate realistic and adversarial scenarios so teams can catch these risky behaviors before production.

That matters because agents can stitch a bunch of normal permissions together to actually create a very abnormal outcome. While agent is live, CompFly helps enforce policies. Should this agent be allowed to access this record, call this API, or approve this particular action?

And afterward, CompFly gives security compliance and audit team a way to actually have these evidences to know what happened and what policies were applied. 

What do we don’t do? We are not replacing LLMs. We are not replacing agent frameworks. We are not trying to replace DLPs, SIEMs, GRCs. CompFly sits across these environments to actually make these agents visible, testable, policy-bound, and auditable.

We are also not treating AI governance as a one-time questioner. Agents are dynamic. They evolve, the prompts change, they have emergent behaviors. So, we treat governance as something that is continuous from development to simulation to runtime decisions to audit evidence.

Our pricing model is a base platform fee plus a usage for simulated and governed actions. Base platform fee covers everything from a control plane perspective. What’s the discovery, inventory, integrations, and visibility? The usage is largely tied to where CompFly is actively testing or controlling agent autonomy.

[Rich Stroffolino] All right, so I feel like we have a pretty good overview there. Thank you so much. I know a lot to go through, but I feel like that really helps kind of know where you’re operating. I love the specificity of what you are not doing.

I feel like that is so incredibly valuable in this space when it seems like everybody wants to boil the ocean all at once here. So, Gary, I’m going to start with you, though. I’m sure you have a lot of questions here, getting into the specifics here.

What other questions do you have about what CompFly AI is doing? 

[Gary Chan] Yeah, well, I’m going to ask the first question that every engineer is going to ask. Can you tell me a little bit more about the technical architecture? Like does this sit in the middle, work in parallel? Just help me understand a little bit high-level, technically, how this thing is going to work. 

[Venkatesh Siva] What we have, Gary, is there is an SDK. We call it FlyEdge. It’s super trivial to sort of integrate with agent frameworks. We support all of the different agent frameworks…not all of them. Most popular ones, right, like LangChain, LangGraph, AutoGen, CrewAI, and a few others.

The engineers added asset dependency and dropped a single line so that we understand, we start to understand this agent deeply. And this information go to the cloud so that we can start to assess these agents, understand these agents, what are the tools’ calls it has?

What data does it access? What skills does it have? And what we do is we start intercepting these calls at the time the agent is making an action. Either it’s a tool call or LLM, and then actually see, should we allow this or not? Then how we do that is basically a gateway that is constantly verifying policies on these systems, and we make sure these policies are deterministic in a way.

The signals are not deterministic; the policies are deterministic, so you actually are relying on things that you know can be sort of monitored and actioned upon. 

[Gary Chan] Just one quick follow-up. So, let’s say that it does a block. So, can I go in and find out why it blocked it? Like is there some sort of explainable…? Does it have an audit trail, for example? 

[Venkatesh Siva] 100%, Gary, right? So, what we do is we actually sort of look at what is this agent doing right now? What is the context that it actually have? What is the trajectory of all the tool calls it has done? And we sort of keep them as audit logs, and the policies that we are basically looking at and the policy that is allowing it to allow or deny.

Especially if it’s denied, we kind of exactly capture why it’s denied and what time and create this immutable log that can be used by audit and compliance and other teams as well. 

[Rich Stroffolino] All right, Mike, get in there. What other questions do you have for CompFly AI? 

[Mike Lockhart] So, my question’s around portability. And to put that portability question into context, our organization, we develop on a constituency of core platforms. We have AWS Agent Core, we have Copilot Studio, we have additional layers on top of that, such as N8n, and then we also have custom frameworks and harnesses as well, too, where we have done native Bedrock integrations as well.

So, when you’re encountering a situation with an engineering organization where you have to find a way to effectively plug in and horizontally scale across a multitude of different foundational frameworks, as well as a larger constituency of sort of the code-first frameworks, what’s your enablement process?

How do you tie in? And how do you define success with an organization that’s still exploring the art of the possible, but also developing on the fly across all of these different platforms? 

[Venkatesh Siva] That’s a great question, Mike, and I can assure you you’re not alone. As you talk to the community, so many people are going through sort of the same thing. From an AWS Bedrock, so the way we essentially do this is the discovery part of the system initially understands all the different providers.

We have these platform providers and there are model providers. The platform providers, based on the platform providers, we can start to understand what agents are getting deployed in each of these platforms. And that effectively provides us a discovery of these are the agents that are actually there.

And the moment that we discover, then there is an onboarding thing. It’s a few steps that basically allows people to quickly onboard this agent into the platform. And as a part of the onboarding, we create a DID. DID is a unique identity that we have for this agent.

I mean, historically, people were just using model’s API key to identify an agent. Of course, that has a lot of sort of downside. A DID is a distributed ID and that is sort of it is signed and there’s a key for it and we can basically verify these signatures along the way.

So what we do is we, the moment the DID gets added to the system, it can send the calls, can send these informations to the cloud, and then we can start to actually infer what’s happening and then apply policies on that. 

[Mike Lockhart] Now, I mean, I think that gives me a good starting point there too. And so adding onto that, right? What’s the journey look like though once you’re initially plugged in? What is the expectation for an engineering organization as far as dipping their toe in the water to starting to tread water and all the way to swim, sort of that crawl, walk, run mentality, getting all the way up to we have now fully integrated this, and what’s the value proposition at the end of that enablement journey?

[Venkatesh Siva] So, the way we do this is there is what’s called an agent guard. The agent guard allows us to essentially define what are the base controls that we want to put in place. And the base controls are largely divided by three. There is what models and tools are sanctioned for this organization, what data safety and security policies that I want to apply on these agents, and then what data sensitive policies that I want to apply for these agents.

These are largely provided by the platform for organizations, so they can effectively… And these are based on, the security and the safety is largely based on OAuth Top 10. The data sensitivity is based on data exfiltration, keywords exfiltration, secret exfiltration, and so forth.

So these things are offered for the company. So, the company can actually, as they onboard into the platform, they would look at these base policies to make sure that they are aligned with the company’s policy, and once it’s basically done, you’re effectively doing this for all of the agents to come.

So, as agents get onboarded, we start to understand what these agents’ behaviors and the tools and the systems are. And the moment that happens, these agents go through a simulation phase. We have got two modes of simulation. There is a security simulation and a safety simulation.

Security largely is using OWASP but uses the MITRE techniques to sort of attack these agents for adversarial sort of scenarios, and it actually provides you specifics about what are the weaknesses of these. It’s not more a binary about sort of this is good versus bad.

It’s more about this is where we see these agents’ weaknesses are, and this is where we see the agents as good. 

And wherever these weaknesses, we also say, is this refusal categorical? Is it hard refusal, soft refusal? Or is it just evading some of these requests? And provide them recommendations on what to essentially fix. And some of these fixes could translate to potential policies that are specific or controls specific for this agent, versus it could potentially be an application that you want to apply across all of the agents as well.

So, the moment you do the simulation, you understand the developer could fix a few things, the security team could fix a few things. And then the actual guard essentially picks up, and the guard essentially has this baseline and agent specific controls.

And that’s how you continue to sort of really make sure this agent is behaving the way it should be. 

[Gary Chan] I’d like to dig into that a little bit more. So, if I were to give a policy, right, and it runs it against it, CompFly will do all that and it says it’s good or bad, or maybe not good and bad, but it gives that analysis, right? How do you know that it’s correct?

Is there some sort of validation where a human kind of looks at it? And if so, does that mean that your team always has to be part of it? Or is this completely autonomous? How do you know that what it’s doing is right? 

[Venkatesh Siva] The base policies are specific policies. Like I said, this is about sort of PII. Are there specific PIIs going out or not? Are there keywords? Is there jailbreaking? Is there prompt injection and those kinds of things? Those are pretty sort of standard ones.

And then for agent-specific ones, Gary, these ones are largely tool specific and, in some cases, sort of LLM specific. So, we sort of intercept these things before and after tool calls and before and after model calls. So, in tool calls, we actually show recommendations of based on the behaviors that we have seen or based on the simulations that we have run, we think these policies actually help you guys, right?

And those policies effectively are like there and then it’s effectively a prompt where it tells you why it basically thinks those policies are good ones for these agents, and people can work with the system to sort of really make sure that these policies are accurate.

And then the moment the accuracy happens, we actually generate code behind the scenes to make sure that these policies can be applied before tool calls.

I think the interesting thing with tool calls is tool calls have structured schema for what goes in and what goes out. And of course, with the context as signal that we essentially get, we can apply these things at runtime. And of course, people can actually test them, right?

Like there is a playground that we essentially have for each of these agents where you can actually test these policies before you actually say, “Yep, I want to publish these policies out to this agent right now.” 

[Mike Lockhart] So, here’s a follow on question to that then from the policy aspect, how much governance can CompFly provide for agents that also have code execution capabilities? To put that into context, if you have an agent that is helping to develop, say, like geospatial referential code for a customer that has complex questions, right?

Very relevant to some of the things that we do inside of our platform here. How would you assure that there is good governance controls and good policy controls there to make sure that code development, execution, and destruction is safely governed?

[Venkatesh Siva] Just to understand that question, Mike, better. So this is effectively an application that is providing code to build these applications that are geospatial specific. Am I right? 

[Mike Lockhart] Yes, that’s correct, right. So, we have a new product that we’re launching and we’re in the geospatial space and our agents do a lot in that particular domain. So part of that is helping answer complex questions, right? And it goes beyond that too.

It could be internal tooling as well where you want agents to be able to help address code development, right? It’s maybe low-hanging fruit where you have agents that maybe are writing some code changes for PRs, need to do some safe space explosion.

We obviously want that to stay constrained, right? And this is where the autonomy component really comes into play with these LLMs. You are effectively delegating power to them to do things on your behalf and that power spans a broad spectrum, one of which is writing code.

Now certain platforms like AgentCore provide some safe spaces but other places don’t. We also just want to think about where we draw boundaries and limitations. So it’s understanding if or how CompFly comes into that picture or if it doesn’t, also just understanding sort of where the threshold lies.

[Venkatesh Siva] No, for sure, of course. I think there is. I think code execution has a bunch of different sort of variables in it including sandboxing sort of where this is run because this is sort of generated code in that sense. We are not focusing on sort of code sandboxing for code execution.

What we can do is sort of make sure that when the agent is generating a code, there are plugs where you could plug in to basically say, okay, this is a code I… I mean, the good part of the code is when the code is generated, it is deterministic at the end of the day.

The code essentially works or doesn’t work. But the functionality of the code itself, I think this is something that we will potentially need to work with you a little bit closer to sort of understand this use case a little bit deeper, but in general, there is an escape hatch for these policies.

The escape hatch allows you to sort of plug in what you want to do beyond what the capabilities of the platform are. And that’s effectively how you essentially look at this as, okay, this is the codes that have generated but like what are the deterministic aspect of this that we can actually truly validate at the end of the code generation itself in this case here?

[Gary Chan] I’ve got a question. How do you discover autonomous agents or shadow agents? So, do I need to know about it and then tell CompFly or can CompFly go out and find all the ones in my environment? 

[Venkatesh Siva] Yeah, that’s a great question, Gary. So, there is, I think that’s… There’s no one vendor that can actually discover these shadow agents 100% today in the market, and we are not an exception in that case. I think what we can is there is sort of known unknowns and there are sort of unknown unknowns.

I think we go after the known unknowns. That means these agents, let’s say, if they are largely deployed in a cloud environment or in a platform provider in this environment, like let’s say just Bedrock in this case or OpenAI directly or Vertex or one of these, then we can sort of identify those because we have hooks to those platforms to actually go and identify those.

But if somebody is running an agent, let’s say an OpenClaw on their local laptop or a MacBook, then we wouldn’t be able to sort of do that today, right? So, that’s something that we’re not going after today. 

[Mike Lockhart] Who’s watching the watcher is my question, right? And so there’s this delegation of authority with these platforms now, CompFly and others, that we’re asking them to help make sure that we’re behaving in a safe way. But I just want to get an understanding at a high level and not so much platform-level trust, but just your design modalities and like how are you assuring that the things you’re building are going to be trustworthy and provide the governance that we’re expecting or are not going to introduce new potential undue risk into our organization? 

[Venkatesh Siva] Yeah, that’s a great question. So, from a deployment perspective, I mean, we are, of course, doing our SOC 2 from a certification perspective. But beyond that, I think the folks that are developing this are sort of really industry veterans that have essentially worked on regulated industries in the past.

So we are making sure our environment is sort of really constrained from a security and a safety perspective. But in addition to that, from a deployment perspective, the key components of the data plane can actually be deployed in a bring your own cloud.

Effectively, you could effectively give your cloud, and we can start to deploy our data plane there. That’s how we are sort of trying to protect this. And most of the way that you provide access is through workload identity. So that means you still hold a way instead of sort of really giving us keys.

So we make sure that we’ll provide these, potentially that’s the way we are effectively going forward for getting access to your environment and to actually discover things and apply policies. 

[Rich Stroffolino] All right, Venkat. What’s one thing we didn’t ask about that we need to know about CompFly AI? 

[Venkatesh Siva] Yeah, I think the big part of this is I think if you look at traditional AI governance, traditionally AI governance typically starts with sort of policies and then goes to questionnaire and then an approval flow. I mean, they’re all important, but they’re not just enough for agents and agents take actions.

We talked about how sort of emergent they are with sort of tools and the way they touch systems. So, governance has to be closer to run time, and it has to be continuous. It has to be sort of technical and evidence based, and that is where CompFly truly fits in this ecosystem.

[Music]

[Rich Stroffolino] Well, that’s just about it for this episode of Security You Should Know. To learn more, head on over to compfly.ai. If you have any feedback for this show, send it to us, feedback@ciso-dev.davidspark.dcgws.com. Thanks to Mike and Gary for helping us learn more about what CompFly AI is doing in a fascinating and fast-moving space.

And thank you so much, Venkat, for your time and being game to answer all of these questions. And thank you for listening to Security You Should Know. 

[Voiceover] That wraps up another episode of Security You Should Know. If you like this program, please subscribe, tell your friends, and leave us a review. All companies showcased on this program are sponsors of CISO Series. If your company would like to be spotlighted and interviewed by our security leaders, go to our contact page on CISOseries.com or just email us at info@CISOseries.com.

Thank you for listening to Security You Should Know, connecting security solutions with security leaders.