You can put practically anything into an LLM and the output will look acceptable if not fantastic. Unlike most things in technology, beautiful results with AI can lead you astray. How do we know if we’re being led down the wrong path, and how can we prevent it from happening?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark, the producer of CISO Series, and Geoff Belknap. Joining us is our sponsored guest, Matt Goodrich, director of information security, Alteryx.

Join the conversation on LinkedIn

Huge thanks to our sponsor, Alteryx

Full transcript

Intro

0:00.000

[David Spark] You can put practically anything into an LLM, and the output will look acceptable, if not fantastic. Unlike most things in technology, beautiful results with AI can lead you astray. So, how do we know if we’re being led down the wrong path and how can we prevent it from actually happening?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark, I’m the producer of the CISO Series, and joining me as my co-host, it’s none other than Geoff Belknap. Geoff, say hello to the audience.

[Geoff Belknap] Hey, audience. It’s me, Geoff Belknap.

[David Spark] You will recognize that voice because you will hear it all through this recording. By the way, if you’re not familiar with our other four shows on the CISO Series network, why not go to CISOseries.com and check them out? We’ve got lots of wonderful shows. While this show is very, very good and everyone loves Geoff, we have other co-hosts and other guests that people like as well. Do you believe that, Geoff?

[Geoff Belknap] I, in fact, like many of the other co-hosts, better than I like myself sometimes.

[David Spark] Oh.

[Geoff Belknap] As a person, I think, just listening to the shows, I’m like, “Ah, this person has a very different opinion than I do,” and I find that fascinating.

[David Spark] The differing opinions is one of the reasons our fans like us. They get to hear a milieu of people working in different industries and different opinions and that’s why it’s popular. If everyone agreed, it’d be very boring. Our sponsor for today’s episode is Alteryx, where analytics, automation, and AI come together. We’re going to be talking a little bit more about that actually throughout the show, but also in the middle of the show when I’ll tell you specifically about what Alteryx does.

But first, let’s get to our topic at hand, and I’m calling up one of our absolutely favorite guests, Davi Ottenheimer, who has provided such fantastic insight. He’s the VP of Digital Ethics and Trust over at Inrupt. He is very wary of our reliance on what AI tells us. Since LLMs operate so efficiently and can generate impressive results, the algorithm does not care about your input, and that can be dangerous. The old saying of garbage in, garbage out doesn’t really hold up. As Davi says, it’s now garbage in, beautiful out. You’ll really never know what’s garbage unless you deal with it at the input stage. But even that has its issues in that we all have our own inherent biases that will skew the results. So, Geoff, how do you keep track of your data inputs, so you’re not surprised or really don’t even realize that your output is deceiving you?

[Geoff Belknap] I worry less about being misled by outputs[Laughter] as much as I worry about being sort of seduced by the polish. I think just being a generally cynical person, I’m like, “Man, this seems too good. This is a little too confirming of my biases.” And what I really try to do personally, because I use AI every single day, is to make sure that what I’m reading is sourced, and I will go taste test the sources of what I’m reading. Because sometimes I find out, depending on the model I’m using, it’s just looking at the last email I sent about something or the conversation I had right before that that I wanted to go ask it about, and it can trick you into believing your own nonsense a little too often.

[David Spark] Yes, and that’s what we all have to be very wary of. And luckily, we have someone from Alteryx to help us in this discussion. Our sponsored guest, the director of information security at Alteryx, none other than Matt Goodrich. Matt, thank you so much for joining us.

[Matt Goodrich] Thanks for having me, David, and great to be here with you, Geoff.

This is not just a security issue.

3:30.912

[David Spark] Nrupak S. of Cole said, “What defines trust? What can be trusted and what cannot? Is an internet article a good source? Is it a good source if it’s a research paper? Similarly, within the security data, is data relevant? Trust challenges have always been there. Different security tools bring context with data to help the decision maker make the right decision. The LLM will have to present context with the data and reasoning why the data was thought to be relative to the question.” I like that last line.

And Ravi Char of NXP Semiconductors said, “Bruce Schneier has pointed out that while AI poses confidentiality and classic security challenges, it is much more an integrity problem. Even with strong input, output, and model hygiene, it still takes human judgment to validate results, catch inconsistencies, and resolve the trust and integrity issues that AI cannot detect on its own. AIs don’t face consequences, but we humans do.” I like that line too. So, two really great quotes, and in fact, the ends of both of these quotes are my favorites here, Geoff, in that the need to present context with the data and the need for the human to get involved with it. And you kind of made a reference to that at the very beginning of, I want to know where the source is so I can go back and look at it.

[Geoff Belknap] Yeah, I think the difference between letting a computer that is trained on a bunch of internet data make decisions for you and you making better informed decisions is questioning information that’s presented to you. I think whether you are in a meeting and somebody is telling you their take on something, or whether you are in a chat window with a large language model, your follow-up to anything that seems very interesting or new to you should be, “Oh, how did you come to that?” or “What made you say that? What information are you basing that on?” I think precision Q&A or precision question and answer is just as useful for human beings as it is for LLMs, and it helps you avoid this trap.

[David Spark] Matt, I actually just had a conversation about the need for the human to be in the loop, but that is really vague. Where does the human come into the loop here?

[Matt Goodrich] I think the human can come into the loop at a number of different points, depending on your use of AI. There can be a human in the loop in preparing your data for consumption by AI. There can be a human in the loop in looking at the outputs of data after AI has generated it. Or, in some cases, in a properly constructed system with boundaries and guidelines for the AI, you may not actually need a human in the loop at all, especially if no decisions are being made.

[David Spark] So, I mean, maybe this is a situation, and correct me if I’m wrong here, maybe this is something you’ve discovered at your own business is, you put the human in the loop in as many places as possible and start pulling them back when no longer needed, and I think that’s the hope of AI here, isn’t it?

[Matt Goodrich] Theoretically. I would say, depending on the gravitas of the output of the AI and what it’s going to be used for, I would say, has a pretty good impact on whether or not a human actually needs to be there. And the more trust you can build in that AI, the more it can cite its sources or go through a double verification method, maybe using another piece of AI, something like that. The more trust you build in that system, the more you’re likely to be able to take the human out at some point.

[David Spark] So, the thing is, while we’re talking about we’re fearing about AI running wild, is we can create systems with AI to let AI check itself. There isn’t a fear of that sort of running rampant. I mean, I can think of a lot of sci-fi movies coming in here.

[Matt Goodrich] I’ve probably seen the same movies. I do think properly scoped and bounded AI that maybe is just for a single purpose would be good at detecting things like bias or anomalies, checking sources, these sorts of things. I think we’re in the early stages of really being able to rely on something like double verification using multiple different AI systems, but I think that’s only going to get better over time.

[David Spark] Geoff, were you about to say something?

[Geoff Belknap] I was going to sayI strongly agree. What I have found is, if you’re breaking up the problem you’re trying to solve, and especially if you’re using multiple agents, you can use agents that didn’t generate the information that you’re looking for to check it and to be a really skeptical checker of facts. And I think what I’ve learned is, certainly AIs can hallucinate. Most of the models that professionals are using don’t have this problem so much, but what they do have is the ability to mislead you into different directions, and I think if you’re using it smartly, you can certainly use AI to help avoid the pitfalls as well as cause them.

Why is this so darn hard?

8:19.307

[David Spark] Louis Zhang of AIA Australia said, “AI is non-deterministic, so hallucination is inevitable, even with flawless input data. Consistently challenge both the output and the reasoning process and constantly verify the underlying sources of truth. AI is like a fresh graduate intern – quick, smart, andbroadly knowledgeable, but with limited context and memory and prone to mistakes. Treat AI as a valuable thought partner and an extra pair of hands that boost productivity, but never the ‘decision maker’ in the loop.” Well, that’s sort of a strike against agentic AI there, but Louis brings up a really good thing that we talk about a lot of treating AI like a new employee.

And Adrienne Young of Team Carney said, “It takes quite a bit of time to audit the LLM to ensure that it is in fact ‘right,’ and it can get very murky. Sometimes it’s just easier doing it myself and not having to second-guess. I don’t want to be, ‘that guy’ who can’t answer where the conclusion shown on a PowerPoint slide comes from.” So, that’s really interesting. So, Louis brings up the need to check all your work, which is going to take a lot of work. And then Adrienne’s like, “Well, if you check all the work, it’s going to be just as long as if I just did it myself.” Matt, what’s your take?

[Matt Goodrich] Yeah, both of these quotes actually really resonated with me. AI is non-deterministic by nature. So, you can’t treat its output like a finished product. You have to treat it like you would a bright and inexperienced intern. You wouldn’t let an intern pull together a report and send it out to the C-level. You’d probably want to have your eyes on it first to validate what you’re seeing and talk about the outputs and why the outputs are the way they are. Interns can be helpful, fast, and capable, but they still need oversight. Ultimately, I think the trick isn’t to try to eliminate some of these hallucinations or these jumps. It’s try to design workflows where AI has a very clear role. The data’s vetted. The human remains the decision maker. And then when we do that, we actually get the acceleration benefits of using AI. And when you don’t, then you just end up second-guessing yourself over and over and over.

[David Spark] You say you use AI every single day now, Geoff, but have you fallen into this trap that Adrienne says, which is, “Ah, it’s easier if I just do it myself”?

[Geoff Belknap] I think for some things, absolutely. At this point, it really is, you know, if I go back to the intern analogy, I think that might be very generous. I think in many cases, some of the very smart people I work with day-to-day on AI security remind me that it really is like a four- or five-year-old cognitively with just an incredible vocabulary and just an amazing ability to produce polished garbage results. So, if an intern you’re working with produces a slide deck that looks amazing, I think you’re inherently sort of driven to go like, “Let’s just double-check.” Polished results does not mean there are valuable information in there.

And so, in this case, if you really think about how you’re using it, like if I’m using it to make a decision about when I should calendar a meeting, I can trust AI to do that. If I’m using it to make a decision about a strategy I should implement, or whether something is or is not a vulnerability or something like that, like, that really has to be you. I think there are higher-order things AI can make decisions about, but then what you really need to be sure of is, are you getting a log or a record of how it made that decision? Can you go back and analyze it later to make sure it’s not wrong? And so, it really is just a matter of keeping the right context and not just giving yourself over to like cruise control. I think in your car, cruise control can keep you going straight and it can definitely feel like it’s self-driving, butas soon as you hit a curve, you might get a surprise.

Sponsor – Alteryx

12:17.246

[David Spark] Who’s our sponsor this week? Well, our sponsor this week is Alteryx and let me tell you about them because they’re doing something very cool on the topic we are talking about, which is managing your data before your tools. So, organizations are moving fast to adopt AI, but most still struggle with a fundamental problem. They cannot explain how these systems make decisions, what we’re talking about. As AI agents begin acting inside business processes, CISOs must ensure not just security, but governance. The How to Trust AI eBook from Alteryx actually makes the issue clear. Without explainability, lineage, and visibility into the data powering AI models, teams cannot verify or audit outputs, which stalls adoption and introduces risk. So, that risk grows with agentic AI. Now these systems perceive, decide, act, and learn inside enterprise environments. If you cannot trace which data an agent accessed, how it was transformed, or why it took a specific action, you cannot put it anywhere near compliance workflows, customer-facing decisions, or operational controls. Very good point.

So, Alteryx solves this problem with its AI Data Clearinghouse. It turns raw siloed inputs into governed AI-ready data and provides complete transparency across the entire pipeline. You can see where data came from, how it was prepared, and what logic was applied. You can even inspect an agent’s reasoning after the fact and enforce guardrails to keep actions within approved boundaries. Now for CISOs, this transforms AI from an opaque risk into a controllable, traceable, auditable capability. Now, to learn how an AI Data Clearinghouse helps you govern the machines, just go to their website, alteryx.com.

How do we approach governance?

14:18.986

[David Spark] Khash Kiani said, “We’ll never have perfectly sanitized security data and that’s not the point. The real challenge is designing AI that can reason over imperfect data with the governance and guardrails enterprises actually need. That’s how you earn trust of garbage in is not the main risk, unguarded out is.” Interesting take. And Abilash Prabhakaranof Grain Management said, “I’m all about guarded outputs. With the goal of disproving exploitability, my AI runs freely within bounds to verify breaches in cloud environments, way more accurate than chasing exploit proofs.” So, I’ll start with you, Geoff, here. This whole idea about guarded outputs, I like that, but jeez, if your input is a mess, I mean, this goes back to the old shift-left philosophy of that seems like a nightmare [Laughter] if you’re only dealing with that.

[Geoff Belknap] Well, it does andit doesn’t. I’ll sayI think Khash does a really important service for us here.

[David Spark] Khash said the first quote here about garbage in is not the main risk, unguarded out is too.

[Geoff Belknap] Let me go back to the very beginning because I think I just found my favorite quote, “We’ll never have perfectly sanitized security data,” and I just want to stress that regardless of whether we’re using AI or not, or we’re back to just like reading log lines with grep and sed and awk, the input that you have as a security professional is never perfectly sanitized or even accurate or well-structured. I mean, I have lots of complaints, but we’ll do a therapy show another time. So, one of the huge benefits in a use case like that of AI is AI can help you structure some of that data or even identify where the data’s inconsistent or where you’re missing some stuff, and that’s super beneficial.

And I will say, this is the first time I’ve heard the concept of unguarded out, and I think that is a really cool idea because the way I think about it is like, you cannot blindly trust the output of an AI in generic use cases, right? Depending on what you’re trying to do. So, if you are in a security professional role and you’re using AI as I do, it may be your research partner. It may be a thought partner, somebody to bounce ideas off. It may help you accelerate what you’re doing, but it is not replacing you. And I think you have to remember that as a user of AI, that you still are responsible for making decisions. The AI is not accountable for a negative outcome. You are.And then you have to act accordingly.

[David Spark] I want to go to you, Matt, on this, about this need for to essentially have guarded outputs and that we don’t expect to have perfect security data. Where are we dealing with this problem? I mean, I’m questioning whether we’re dealing at the front end or the back end or what’s going on.

[Matt Goodrich] I think having perfect data is never going to happen, right? Any AI that expects perfect data coming in the front door or really anybody, regardless of using AI or not, that is expecting perfect data, they’re going to be upset because it’s never going to happen. An AI that expects perfect data is going to be unusable. On the other side, though, I think the governance of what comes out is actually what builds trust in the AI. AI is great at doing things like confidence scoring. You can have it grade the confidence of its output itself based upon its sources and the confidence of the data. You can have AI tell you that it had to make a cognitive leap to be able to connect two pieces of data. Those are all beginning steps of governance. And then you start to add on audit trails and triggers for human escalation and redaction policies based upon stuff that you may not actually want the thing to spit out but might actually exist in the input data. So, looking at what’s coming out this side is still an important step, and I think the governance and that control and structures you can put around the output are ultimately going to help gain the trust of the user of that AI system.

[Geoff Belknap] I think this is really important, Matt. And I just want to underscore, the goal shouldn’t be to make sure that only perfect information makes it into the AI. The goal should be to make sure that we understand what came out of it and that we understand how it came to that decision or how it’s informing your choices.

What do most people think it is, and what’s the reality?

18:37.976

[David Spark] Robert Wood of Sidekick Security said, “Something to consider here is whether people are primed to just treat what comes out of LLMs as great because it looks polished. These kinds of tools have not done the collective us any favors in terms of encouraging and refining critical thinking and deep investigative skills. So, while I think there’s huge potential for LLMs turning messy data into useful insights, the more important question is, how do we know if it’s right?” And pretty much the theme of today’s episode.

Yael Citro of Ox Security said, “If the 1983 masterpiece War Games taught us anything, it’s that blind trust and automated output is exactly what almost triggers [Laughter] nuclear war. A confident computer with bad data is far more dangerous than a hesitant human with good judgment.” This is, by the way, just a great closing segment, both of these quotes here, sort of summing up our conversation here. And I think we kind of agree with all of this stuff in that we want to know why it’s right. And going back to your last comment in the last segment, Geoff, you said, “We just want to know how it came to this decision,” but I’m going to toss it to you, Matt. Is that really just the best way to look at this – how did it come to this decision?

[Matt Goodrich] I think that’s one aspect. We gain a lot of trust by understanding how the AI came to a certain conclusion. For me, what I think would gain more trust is a bit more of a humble AI, one that admits uncertainty, that may show its evidence of how it came to its conclusions but maybe could surface alternatives or ask for clarification – and granted, there are LLMs out there that do this fairly well today – or that flag questionable inputs and work with you more as a thought partner than sending out this beautiful polished nugget of, “Yes, this is exactly what happened,” with such certainty because that just builds a whole lot of cognitive bias that this magic AI system must be way smarter than I am because I don’t know that I would have come to that same conclusion.

[David Spark] Geoff, what do you think? Is there sort of like a definition we can put here of saying, “As long as you have this and you’re looking at this, this is the way to approach it,” in terms of the way the output looks, the input looks, the connections of the dots. Like,what would be the thing that would make us all happy? My guess, by the way, your answer today may be very different than your answer next year.

[Geoff Belknap] Yes, like emphatically yes. And I think my answer six months or even a year ago would have been very different than what I’m about to say, which is, like, first of all, I just want to go back to what Matt said, thinking about that while he was talking. We have definitely improved LLMs that will now not just flatter you and make you feel great about yourself, like, “Oh, what a great question, you’re so smart, what a good boy.” And now they will give you direct, concise answers. I think it would be amazing if it would come back with a response that said, “I’m unsure. Here’s some information, but this is probably a low-confidence answer,” which I think I could do with some clever prompting, but that would be really cool by default.

I think the real answer here is, if you are using a general AI, like I do sometimes, just use Copilot or whatever model I might be using to ask questions and do research, you should treat that like you’re doing research, especially on the internet or in a library, where you have to make decisions about the veracity of this information, about the timeliness. Are you reading about a conflict and this information that you’re consuming was published pre-conflict or by one of the actors that has a bias in that conflict? You end up in this place where what we have learned is not all humans are good at critical thinking and making informed judgments. AI does not always make that better. It is not any different than a search engine when it comes to that, you just have to be thoughtful about the purpose you are using AI for at the time. If the AI is there to help you rebook a flight or something like that, you can be pretty confident about its outputs. If you are saying, “Dear AI, do I have cancer or not?”

[David Spark] [Laughter]

[Geoff Belknap] You should get a professional opinion. And also just don’t ask AI or the internet about medical questions. That’s not a good idea.

[David Spark] What are you talking about? People do that all the time. They play internet doctor.

[Geoff Belknap] Yes. My family calls me Dr. Google; however, [Laughter] don’t do what I do. If there is something that is really important when it comes to life safety or like your career safety, be skeptical about the outputs you’ve got. Be thoughtful about whether you’re asking an expert system in this very narrow field or whether you’re asking a very general large language model and understand that ultimately, you’re accountable for decisions that you make, not the place or the computer that helped you make the decision.

[David Spark] Excellent, excellent point.

Closing

23:29.799

[David Spark] This brings us to the point of the show, Matt, where I’m going to ask you, which quote was your favorite and why. I loved a lot of these. Just pick one. Tell me why you like it.

[Matt Goodrich] I liked a lot of the quotes. I think they all resonated with me quite a bit. The quote from Louis Zhang about the interns. I know I keep coming back to the interns, but I actually liked Geoff’s adaptation of it a little bit more. It’s like a five-year-old that can use really fancy words. Both of those resonate a lot with me because they are powerful. They can be smart. They can have really good insights that maybe I wouldn’t have come up with by myself, but they’re lacking experience and deeper context.

[David Spark] Exactly, good point.All right, now Geoff, your favorite quote and why.

[Geoff Belknap] This show, and I know I always say this, and I just want to be clear, like I always mean it because all of your quotes, everybody who contributes are really important to us and great. I think we have two really good competitors, so I’m going to pick both of them, but I would say my number one favorite quote contextually is Khash.

[David Spark] In the third segment, yes.

[Geoff Belknap] Who said, “We’ll never have perfectly sanitized security data and that’s not the point. The real challenge is designing AI that can reason over imperfect data with the governance and guardrails enterprises actually need.” And I think this – Khash, I love you, I’m going to abuse this and use this in some of my regular arguments day to day – this is the real point, right? We want imperfect data and we want AI that we can trust to help us reason over imperfect data. What we don’t want is imperfect, well-polished garbage answers on top of it. What we need to know is, “Here’s an imperfect answer based on imperfect data.” I think that really helps. And then I think strong runner-up is Yael calling back to War Games who – hey, folks, if you haven’t seen it, don’t watch the current one. Go find a movie with a guy named Matthew Broderick in it. Go watch that one and learn what happens.

[David Spark] They made a remake of War Games?

[Geoff Belknap] I thought they did. Maybe they were planning on it, whatever. But if you find War Games…

[David Spark] The original one with Matthew Broderick, it’s a great one. A very young Matthew Broderick.

[Geoff Belknap] That’s a great one. If they’re using a phone that looks very foreign to you that has a cable that’s not a power cable or USB cable, you’re watching the right movie. And the important moral of that story, my friends, is that you shouldn’t blindly trust computers to make important decisions for you. Inputs, not decision makers.

[David Spark] Very good point. All right, that brings us to the very end of the show. I want to thank your company, Matt – Alteryx – for sponsoring this episode. Remember where analytics, automation, and AI come together. That’s alteryx.com. That is Alteryx. And Matt, thank you very much for coming here. For those of you who don’t know, this is what Alteryx does. They can deal with your data. So, if you’re interested in doing just that, please contact Alteryx. Can they reach out to you, Matt?

[Matt Goodrich] Yeah, please do.

[David Spark] All right, we will have the link to Matt’s profile on LinkedIn on the blog post for this very episode. You will find it there. Thank you very much, Matt. Thank you very much, Geoff. And thank you, audience. As always, as I always say, we greatly appreciate…

[Geoff Belknap] And we always mean it.

[David Spark] And we mean it. We greatly appreciate your contributions and for listening to Defense In Depth.

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.