SOC analysts don’t need more alerts. They need fewer. But the AI tools vendors have shipped to address that have often delivered the opposite: new categories of noise, and a quietly growing accountability trap. When AI runs an investigation and gets it wrong, the analyst still owns the outcome. The analyst didn’t dig the hole, but they’re the one filling it.
In this episode, Grant Oviatt, vp of product and co-founder at Prophet Security, explains how his platform deploys AI agents to investigate and respond to alerts the way a skilled analyst would, using REST API integrations across existing security tools rather than absorbing all your data into another SIEM. Joining him are Will Gregorian, CISO at Galileo Medical, and Howard Holton, CEO at GigaOm.
Want to know:
- Why are AI-powered SOC tools adding to analyst frustration rather than reducing it?
- When an AI agent makes a bad call on an investigation, who actually owns that failure?
- How does Prophet Security’s audit trail let you trace every query, piece of evidence, and reasoning step an agent used?
- Why is Prophet Security using frontier models rather than training its own, and how does security-specific context change the outcome?
- What does giving an AI agent remediation authority look like in practice, and where does Prophet Security draw the line?
- How long does it realistically take to go from contract to running Prophet Security against live alerts?
Check out the episode for the answers you need.
Join the conversation on LinkedIn.
Huge thanks to our episode sponser, Prophet Security
Full Transcript
[Rich Stroffolino] Welcome to Security You Should Know. I’m your host, Rich Stroffolino. Today we’re gonna be talking with Prophet Security and what they’re doing in the security operations and automations space. Very hot area, lots of excitement there, and we need it because they’re addressing a very big problem in the industry, and that is SOC analyst overload.
Heard a lot about it. I see it every day it seems like on LinkedIn, all my social medias. Helping us find out why this is such a problem are Will Gregorian, the CISO over at Galileo Medical, and Howard Holton, the CEO at GigaOm. Will, let me start with you. From what you’re seeing, why historically have we seen SOC analyst overload, and why does it feel so acute now?
[Will Gregorian] Well, it’s always been acute. If you go to the r/cybersecurity on Reddit, almost every day there is somebody who says, “I am tired of the SOC, I am tired of the NOC,” because they have way too much signal, not enough time. And I use signal because it’s an industry buzzword. Everybody gets it. Somebody implements a system, somebody has to go in there and monitor it, and that’s what happens.
So we have a whole bunch of human beings that are just fatigued from all of the noise that is being generated right now, and forever, as long as I’ve been in the industry.
[Rich Stroffolino] Howard, what about for you? Is this a forever problem, and has anything… I mean, I’ll just drop the AI buzzword in here. Is that just massively increasing the signal and causing even more overload?
[Howard Holton] So first, we’ve always had this problem, it will always be a problem, it will never be solved. While I appreciate the potential that AI brings, AI brings the infinite problem to both sides of the equation, right? As long as whatever tool you’re using to speed up your own efficiency is able to be used by the attackers, it ends up being an infinite game.
That’s not a good thing or a bad thing, that’s just the reality of the situation. To piggyback on what Will said, I’m always a fan when someone brings up a subreddit. The other thing, and I just wrote a blog on this although we haven’t published it yet, that I’m seeing within the cyberspace is analysts complaining about AI, and they’re complaining about AI from two different sides, right?
Side one is, “Why in the hell did my organization decide we could run AI without ever talking to a cybersecurity person?” And then the other side is, “Why did my vendor turn on AI that just creates more noise,” right? So I think those two problems we really need to address and figure out how are we gonna bring the people along so this actually works, so we’re not leaving people behind, and so we’re not just shoving another tool down their throat.
[Rich Stroffolino] All right, I feel like we have an understanding of where the state of this problem is at, so that’s why I’m excited to be talking with Grant Oviatt, the VP of Product and Co-founder at Prophet Security. Grant, you gotta help us out here. Help us out with our three essential questions, the classics of Security You Should Know.
How do I explain the value of your solution to my CEO? What does your solution do? What does it not do? And what is the pricing model? Can you help us out here?
[Grant Oviatt] Yeah, absolutely. So to the CEO out there, what we do is we reduce the risk of threat activity in your environment and the operational cost of doing investigations for your security team by using AI agents to do the work instead of humans to do the work, and use humans for what humans are best at. What we do is we take alerts from your security tools, we investigate them just like an expert would, and then we respond along with you.
So we can go and take response actions on your behalf to contain threat activity in your environment and help take out the trash of all the noisy stuff that’s going on. What we don’t do: we’re not a SIEM, so I’m not interested in taking all your data, all your telemetry, and absorbing that. Wherever your data lives is where we want to investigate from.
We use queries just like an analyst would instead of just consuming a stream of your data. No one really wants that in this day and age. And pricing model: usage-based on an annual subscription term. So you pay for the work unit. Number of investigations is how we look at it. So how much work are you going to ask Prophet to do over the course of the year? That’s how we price it.
[Rich Stroffolino] I love anytime we are talking to a company and they say, “We are not a SIEM.” That is the start of a very good answer for so many security vendors. Thank you for that, because I feel like that truly helps to specify where you’re operating, and that is truly, truly meaningful.
Thank you so much for answering those preliminary questions. So we got a taste of what Prophet is all about here. I’m sure our panelists have a lot of questions. Howard, let me start with you. What other questions do you have for Grant and about Prophet Security?
[Howard Holton] So I’ll start with one. One of the complaints that I’m seeing from people using AI SOC tools is, great, another place that I get blamed when things go wrong. The challenge with AI, I contend, isn’t that it’s worse than humans. The challenge is you’ve removed the agency from humans, and in some cases, maybe many cases, not made it better.
And what I mean by that is if the AI makes a mistake, there’s no one to blame but the analyst. The analyst still gets the blame. If I make a mistake as the analyst, as the human at the keyboard, and you blame me, it’s fine. I can accept the mistake because I had agency in making that mistake. If AI makes a mistake and I still get blamed for it, I get all of the risk, none of the reward, and it’s frustrating.
And then the second thing is when AI makes a mistake, it’s like having someone come out to repair a busted sewer main, and the way that AI decided to repair it was to dig a septic system. No, no, no, no, no, no, no. I just wanted you to connect me back to the sewer main. Why are you digging a hole I now have to fill, and then I have to do the real work anyhow?
So it’s those two, call them the two sides of the same coin.
[Grant Oviatt] Makes sense. So it’s accountability at one level, right? The buck stops with the analyst when you’re actually doing the work. But what happens when AI agents are doing the work? Who has the responsibility there? Something that we really lean into is transparency and auditability in what AI agents are doing in order to make things explainable so that you can blame AI agents appropriately and coach them.
There’s one thing if it’s just a black box and shows a red banner, this is bad, and you go do all this remediation and it’s wrong. But if you can’t look at the queries that it issued or the reasoning as to why it made these specific decisions, just like you would a human, or in fact it may have a better audit trail than a human would, you at least have some level of culpability of, “Hey, the AI agent made a mistake in reasoning right here.”
And then I think the next piece, and what you’re asking for, is that instead of building a septic system when you need to fix the sewer main, I need to be able to deliver specific context or guidance and coach this AI agent to say, “Don’t make this reasoning mistake. Here’s exactly how I expect you to handle this situation.”
And as a consumer, I expect to see that adaptation in place or the back testing of that to take place. And that’s how we think about the problem.
[Will Gregorian] Yeah, I think one of the basic questions that I would always ask any service provider who’s obviously playing around with AI is, what does your model training look like? What is it based on? But also transparency within the model as well as the bias and fairness within the model that makes these decisions around what is legitimate, real or not.
[Grant Oviatt] It’s a good question. So we’re not training our own models, in full transparency. I think with the age of frontier models and how fast they’re moving, that’s a bit of a fool’s errand. And so we are bottling lightning in a way and looking at across all frontier model providers, have a team that’s constantly doing evaluations on what problems these model providers are best at in framing of security.
It’s really interesting when you wake up one morning and LLMs are better by 2% at solving one problem, but worse at another. And so there’s a lot of specificity or work that we have to provide, of these specific tasks are meant for these types of models. With that comes context. So you give a problem to Claude on security analysis, it’s a coin flip whether you’re gonna get the right answer or not.
And so that’s where all that security-specific training comes in, actually in the form of context, of how at the point of decision, when asking an investigative question, can you have that security knowledge, both philosophy and what are the questions that a reasonable practitioner would answer? And then the understanding of what tools to query across the environment. So it’s effectively the secret sauce of how do we put that context in time with the frontier models in order to get the right outcomes for customers.
[Howard Holton] I’m really glad you guys aren’t training your own model. I agree it’s a complete fool’s errand. And I always have to question the intelligence of a provider that says, “No, we’ve trained our own model.” First, do you even know what it means? Because most of the time, no. And second, why are you wasting your time?
Almost nearly trillion-dollar companies are investing all of their time and energy in that exact thing. Why are you doing it?
[Grant Oviatt] Totally agree.
[Will Gregorian] Tell me about the audit control capabilities. What does that look like?
[Grant Oviatt] So all the actions that AI agents perform in your environment, there’s an audit log for all the atomic actions. Let’s say remediation investigation started, the queries that were issued and were executed, even the evidence that was used to go and make a decision. So you can see what was in context effectively for this agent to go and make a call, and then the questions and rationale for why it came to a specific conclusion.
So I like to say if you wanna, quote-unquote, “repeat the experiment” and go line by line and just pull all the data yourself and go through the analysis, you totally can. I think that level of transparency is actually demanded of the community to make sure that if I’m trusting this agent to be on guard when I’m asleep, I better be able to follow its logic even better than the person sitting at the desk next to me.
So we spend a lot of energy on that audit trail.
[Will Gregorian] And you mentioned the community, so the first question I should have probably asked you was, who’s your core customer base? Who are you after?
[Grant Oviatt] Yeah. Core customers tend to be, in this day and age, we’re talking about analyst overload, so you have an alert volume problem. That’s typically the customers that we see. Visibility is typically not an issue. You’ve invested in a security operations team, but just can’t keep up with the deluge of alerts.
So that’s typically organizations of a couple thousand people or more. I would say we skew more towards enterprise these days, but our pricing model is usage-based, so we try to make ourselves accessible to anyone that would need us.
[Howard Holton] So it’s really the public-private data contained within the system, right? There’s a ton of value in being able to share semi-anonymous or completely anonymous data about how to solve things while keeping the customer secure. If you share nothing, if customers share nothing about how to resolve an issue or the different variety of problems seen, this whole system stays dumber longer.
How do you manage that, and what are you doing about it?
[Grant Oviatt] Yeah, it’s a really interesting problem. On the one hand, there’s the concern of data contamination, right? With anyone and third-party risk, if my data is being consumed by other vendors, how are you using my raw data to train LLMs is a common question that we see. We’re a single-tenant architecture, very strictly enforced.
We don’t use raw data to train anyone’s model. We view Prophet AI as a single-tenant instance. It’s an AI employee of that organization that lives and dies with them, all the context or training in the form of additional guidance or instructions that have been provided end there. Where we haven’t crossed yet is what are learnings that are safe to extract that our customers are comfortable with that we can share across that community.
And truthfully, our stance today has been nothing. Let’s err on the side of data privacy today, and then eventually allow customers to opt in to specific content that they’re comfortable with sharing across their community or specific peers. But right now we’ve operated in a more restricted methodology and allowed individual organizations to iterate very quickly on their instance of Prophet.
[Howard Holton] That tends to be the response I get, and I’ll be honest, I do not like it. I understand why it happens, but I do not like the response. And the reason I don’t like the response is you can’t see everything, and I can’t see everything. And if the only sources of data I have are your master list and my minor list, right?
Then there’s gonna be these huge open white spaces of time between when Will’s organization sees something and my organization sees something, and those bubble up and equalize. We’ve gotta figure out how to get over this, security through obscurity is almost what it is, which every security practitioner will tell you is not security.
We’ve gotta figure out a way to do that, and I’d really like to see it, and it has to come from a vendor, really. I’m happy to jump on the bandwagon. I’m happy to broadcast it, but we need vendors that take a real leading view of this is how we’re doing it, this is how we’re managing that anonymization and pseudonymization so your data isn’t shared to give you some confidence.
But really, guys, we as an entire community need to stand up and say, “Enough is enough. We’re not going to get smarter by not sharing.” All the attackers share everything, or near enough.
[Will Gregorian] Yeah, I’ve had a situation where I’ve actually gotten rid of a vendor because of that same exact reason. But I wholeheartedly agree with that stance. In fact, that question has been brought up in several other conversations previously in my past lives, and the answer and the default is, “Well, we can’t do that.”
And I do think that cybersecurity ultimately is a data science issue. If we don’t have it, how would we know what we’re measuring, right? It’s a collective. When… Let’s face it, from an industry perspective, we all talk to each other behind closed doors. It’s the same approach. Maybe anonymize the data, but let’s just get over with it.
Let’s start sharing the actionable information that makes us better at what we do from a defense perspective.
[Grant Oviatt] Yeah.
[Howard Holton] I would also say that if the data that’s going to the LLM can’t be anonymized, don’t we have a bigger problem anyhow if you’re using frontier models? I mean, because now I have to trust OpenAI, I don’t know why I would do that, and I have to trust Claude, and I don’t know why I would do that.
They’ve both shown that they’re not perfectly trustworthy, let’s say. They’re not terrible, but they’ve shown they’re perfectly not trustworthy. And if Prophet was able to say, “This is why and how we share data and how we anonymize it,” it would also give me the confidence to know, well, if you’re doing it there, then we can ask the question of are you anonymizing it before it goes to the model?
And you’d have very good answers to those questions as well, right?
[Will Gregorian] Wouldn’t that also be something that we would actually influence and control from an integration perspective? So one question: what does the integration ecosystem look like, first and foremost?
[Grant Oviatt] Yeah. Before I get to integration ecosystem, I definitely want to talk about the stuff you all dislike. I think it’s a fair take. I also think sharing information, anonymized or not, without organization consent is also a bad slope to go down, right? Of just
[Howard Holton] Agreed. Yeah.
[Will Gregorian] Yeah.
[Grant Oviatt] pilfering data and saying, “Oh, we used it in a beneficial way,” is not a fair
[Howard Holton] I don’t think… Yeah, I don’t think Will or I are suggesting that in any way, shape, or form. And I would agree with you, right? You can’t do it without consent. But we also have to start… we really have to, just like we’re hammering you, right, we have to hammer our peers and go, “Guys, you’ve gotta get over this share nothing thing or you make all of us worse.”
That’s all I think Will and I are saying.
[Grant Oviatt] Yeah, I think it’s also interesting how does an analyst find novel activity on their first day, right? We’re also moving from an indicator-based organization where that threat intel sharing becomes really meaningful because there’s an atomic indicator that has a short TTL that the community needs to know about quickly, to we’re teaching AI agents how to be analysts and think through investigative methodology.
The important thing is the coaching. Does the AI agent know how to handle novel situations and follow a robust investigative process to where knowing the latest domain becomes much, much less relevant to getting to the right outcome than following standard investigative methodology that experts would adhere to?
I don’t discount the community aspect of how do you anonymize data and get better threat intel, that’s more organization-oriented to what’s happening in industry. Totally heard. We have some thoughts around there that we could talk about too. But just from how does this impact outcomes of other organizations, from your worldview and macro worldview to what may be observed that other people haven’t, our observations are that the focus is on building a really great analyst.
[Howard Holton] My only caution there is the reason atomic works is it’s faster than any investigator, right? Knowing this is bad, turn it off in 10 seconds, you’re not gonna have an investigator that’s faster than that, right? So if the goal is to make a very good investigator, the investigative process used by AI today and how that is evolving is still a process of investigation rather than answer, right?
I don’t have to answer if I ask AI what color is the sky, it’s going to say, “It is normally blue. Is your sky blue? If not, danger may be ahead.” That’s different than, I don’t know, take a picture, upload the picture, I’ll compare that to the palette of colors that I’m aware of… You know what I mean? The speed of resolution on things where we have an atomic solution is faster than the things where we do not.
So I don’t want to position ourselves to say everything should be an investigation,
[Grant Oviatt] Fair.
[Howard Holton] right? When we can get to an answer faster, we need to get to the answer fast.
[Grant Oviatt] Yeah, and then it’s how long did it take you to get that threat intel is the other piece too. And so are you only gathering threat intel during waking hours? Is it specific to your organization? That’s another agentic problem that we look to solve, right? Of how do you customize this more to your organization, be looking for those indicators when folks are asleep, right?
And pull that into the investigative process to go and perform that containment. So I think sharing and specificity are interesting when you look at an organization specifically. Most organizations don’t have threat intel companies surrounding them around how do we gather all of this, but in the world of AI, you could.
We do.
[Howard Holton] Yeah. I think it’s a both. I don’t think it’s a one or the other, right?
[Grant Oviatt] Yeah, reasonable.
[Howard Holton] And that’s the advantage to AI, right? The advantage to AI is not the volume and velocity. Sure, that’s a benefit, but the actual advantage is the personalization, right? AI can consider so many more variables than any human ever could, and AI really starts to multiply its value exponentially when it gets really personal, and so you need to enable both sides of that coin.
[Grant Oviatt] Yeah, I totally agree. I think the customization and the personalization is the huge value to AI.
[Rich Stroffolino] Well, do you want to get in with your integration question?
[Will Gregorian] Yeah, and this conversation dovetails into that. So I am interested to understand what the integrations look like.
[Grant Oviatt] Yeah. So for us, it’s effectively REST API integrations with your security tools. If you have data stored in an S3 bucket or a SIEM, doesn’t really bother us, but you connect via REST APIs, then we go pull the information at investigation time.
[Will Gregorian] Got it. So it’s freeform.
[Grant Oviatt] Yeah, if you wanted to send a webhook, you totally could and just send some payload over. We can perform an investigation from that.
[Will Gregorian] What’s the mean time from procurement to implementation? What would that look like?
[Grant Oviatt] Depends on the organization. I would say less than a day typically. It’s just API keys that you’re going to provision. That said, for large, complex enterprises, could be a week. You’re sharing the responsibility across a few different teams, but it’s effectively how fast can you plug in API keys across your tool set is how long it takes to integrate.
[Will Gregorian] And you’re not writing back with the API keys, right?
[Grant Oviatt] We can. In some cases, organizations may want us to close out alerts in their upstream tools, so for a CrowdStrike or a Wiz, for instance. We go do the investigation, we close it out in their product, and so there’s a write-back there.
[Howard Holton] And is that write-back another REST response? Is it an interpretable response?
[Grant Oviatt] Yes.
[Howard Holton] So if I have a governance tool that I’m using to onboard and off-board data, I can insert that governance, that same level of governance that I’m using enterprise-wide, and you have no problem with that? I really like that as a component.
[Grant Oviatt] Yeah, no issue.
[Howard Holton] Right. It also might help to give customers the ability to address the… First, it’s a great observability point, but also the anonymization. If they’re comfortable using tools, they can inject those at that point and take advantage of it there.
[Rich Stroffolino] All right, Grant, what’s one thing we didn’t ask about that we need to know?
[Grant Oviatt] I think one piece, going off the REST API components, is how do we handle remediation in a world of agentic AI, right? And I think giving agents just domain admin access and saying, “Figure it out,” is just not the right call. So for us it’s all atomic REST APIs that you’re giving an agent permission to, and then describing the situations that it can perform these actions in, and then being able to back test over things that you’ve seen historically to see, yep, this checked the box or this didn’t, whether it’s notifications or otherwise.
And so actually going more from an orchestration perspective there than a full agentic lean so that we can enforce specific API calls that are being issued. The effect is the same. You want resilient, robust response delivered at machine speed, but the control is at a higher degree today. So just an area, talking about remediation.
[Rich Stroffolino] Well, that’s just about it for this episode of Security You Should Know. To learn more, head on over to prophetsecurity.ai. If you have any feedback for this show, please send it to us at feedback@ciso-dev.davidspark.dcgws.com. A big thanks to Will and Howard for helping us learn more about what Prophet Security is all about, and thank you, Grant, for your time and being game to answer all of these questions.
And thank you for listening to Security You Should Know.