Is It Possible to Inject Integrity Into AI?

When it comes to generative AI systems, often we’re concerned about the quality and reliability of the output. But do we risk losing sight of the integrity of these systems when we only focus on outputs?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap). Joining us is Davi Ottenheimer, vp, trust and digital ethics, Inrupt. Sir Tim Berners-Lee co-founded Inrupt to provide enterprise-grade software and services for the Solid Protocol. You can find their open positions here.

Be sure to check out Davi’s last appearance on Defense in Depth, “Machine Learning Failures.”

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Concentric AI

Full Transcript

Intro

0:00.000

[David Spark] When it comes to generative AI systems, often we’re concerned about the quality and reliability of the output, but do we risk losing sight of the integrity of these systems when we only focus on the outputs?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. And joining me for this very episode, you’ve heard him before. Get ready, you’re going to hear him again. His name is Geoff Belknap. Geoff, say hello to the audience.

[Geoff Belknap] Hey, everybody. This is definitely me and not an AI repli, repli, repli, repli, replication.

[David Spark] Very good. I want you to know that I have heard AI replications of other podcasters before. It is impossible to tell the difference from just listening, so I know that’s going to be done sooner or later. Our sponsor for today’s episode is Concentric AI. Find and protect your data with Concentric AI, a brand new sponsor of the CISO Series.

Welcome. And we are going to talk more about how they find and protect your data later in the show. But first, Geoff, let’s talk about the topic at hand. What are the integrity controls around AI systems? When we talk about development of these models, we often get back to very reductive garbage in, garbage out mentality.

But our guest, Davi Ottenheimer of Inrupt, argues that’s hardly an objective measure with many systems able to be tuned to produce seemingly impressive outputs yet have plenty of bias or really crappy inputs. So, I’ll just start with you, Geoff, and ask, what does an AI pipeline look like with integrity checks?

[Geoff Belknap] That is a great question that I’m really hoping our guest can answer. But I think along this line, the stage where we’re at with AI is definitely at the early enough stages where nobody really knows. But I think the important part of this is there are plenty of people that have an idea of what this can and probably should look like, but we’re probably in the phase where we’re overfocused on just getting… Sorry, let me say that again.

We’re probably overfocused on the phase where we just want to get cool output. We just want to make it do tricks. And it’s easy to sort of look at neat tricks that AI does, and it’s hard to sort of look under the hood and go, “Okay, well, what’s driving those tricks?” So, I’m hopeful today we can really get into some conversation about how we think about what’s behind the magic.

[David Spark] Very good point there. And I am thrilled to have this guest on. And, by the way, there is no other guest better to speak on this topic than who we have on right now, who has been studying it for so long. One of the only people I know really leading the charge on this discussion. And also truly one of our favorite guests on the CISO Series.

It’s the VP of trust and digital ethics over at Inrupt, Davi Ottenheimer. Davi, thank you so much for joining us.

[Davi Ottenheimer] Thank you for having me. It’s great to be here. Although I question your integrity with those statements about how great I am. I just don’t know if I can trust you.

[David Spark] You are. Don’t argue with me on that. That’s the number one thing you can’t argue with.

Where do we begin?

3:10.316

[David Spark] Lars Paul Hansen of Danske Bank said, “Implementing integrity controls in AI is akin got equipping it with the dependable compass and map. These controls guide AI through the intricate landscape of data, ensuring ethical considerations are met and trustworthy outcomes are achieved. With these controls, AI operates blindly, increasing the risk of biases, mistrust, and inevitable disorder.” Kind of a nice way of putting it.

And Ed Contreras, who’s the CISO over at Frost Bank, said, “Similar to development, peer reviews are critical to validation. I can see a similar control for AI where automated peer reviews bumped up against a security LLM can provide ‘higher credibility’ in the output. The goal of AI should be greater efficiency and confidence in the output.” So, I’m going to start with you, Geoff, on this.

I kind of like these sort of setup answers in here in that they’re sort of describing like, “Hey, AI is doing some magical things but don’t be dazzled by it.” And you addressed it in the beginning as well.

[Geoff Belknap] Yeah. Look, I think we’ve all run into one of those scenarios where we’re in a room, and there is somebody speaking, and everyone is just wrapped with what they are saying. And then there might be one or two of us, maybe like Davi and myself, that are sitting in the back, looking at each other, going, “Everything this person is saying is wrong.” However, they’re saying it with such confidence and with such impact that everyone around them is impressed by what they’re saying and walking away, going, “Wow, the sky is green.

That’s amazing. I didn’t realize that. I thought it was blue, but really it’s green.”

[David Spark] We have seen this with amazing orders in the past.

[Geoff Belknap] Exactly. So, AI and certainly LLMs and chatbots really kind of suffer from the same thing. And I think we are in a unique position where we can with AI have some idea of why it might be saying these things and make sure that those things that are being said are as accurate, or as unbiased, or at least sort of the flavor that we want them to be said.

Unlike humans, where we really have no idea why anybody says anything sometimes. We have the ability to add some controls here, and I think honestly think we should.

[David Spark] So, Davi, one of the things that you had said in an earlier episode we did… We did an episode with Davi on essentially machine learning failures, which we’ll link to it on this episode. It’s a great episode as highly. I highly recommend you listen to that as well. But one of the things you said, Davi, which I alluded to here, is the old line of garbage in, garbage out.

You say actually it’s quite the opposite with AI. It’s garbage in, and it’s beautiful out. Explain.

[Davi Ottenheimer] Well, it’s not always the opposite. But let’s start with the premise that when you take compost, which is literally garbage, and you throw it out and then you grow an apple seed in it into a tree and then you get apples out of it, we have principles in which we use garbage, and we get great results.

Another example is you come into a hospital… Maybe even more applicable to the AI revolution. You come into a hospital, and you say, “I’ve got pain.” They say, “What kind of pain?” You say, “Ow pain.” They say, “Well, tell us where it is and what it is.” And you can’t. You’re like, “It’s like eight out of a ten, and it’s in this part of my body.” And you’re kind of giving them almost nothing to work on.

And yet they are experts, and they go in, and they say, “That’s really almost unhelpful information. But you know what? I’m going to go right into surgery and do a tumor removal.” So, you can actually come into this with… And intelligence agencies know this as well. You can come into this with almost useless information, but there’s a seed of truth that they home in on, and they get a beautiful result out of it.

And that’s really a better model for how artificial intelligence should work than the garbage in, garbage out you hear over, and over, and over again where everybody says, “Go in with the answer you want.” and then you get the answer you want coming out. And that, to me, is almost… That’s a tautology.

That’s meaningless.

[David Spark] But one of the other things that I found interesting… And there’s lots of examples of it, we talked in the past, is all three of us I hope are well meaning people. And our listeners, too, well meaning as well. But if each one of us was to be in control of an LLM the net results would be different because my bias, your bias, Geoff’s bias, even though not negative, not showing any malintent, is going to result in different outputs, isn’t it?

And so this is where I think integrity jumps in. Yes, Davi?

[Davi Ottenheimer] Yes. And no. So, there are some people that you do want to have influence the outcomes and some people you don’t. You would restrict them because they’re intentionally trying to poison or affect the system. The same way you can have lots of people who are banking, and some people are doing exchanges with the teller, and some people have a gun that are robbing the teller.

And you have to sort of… And we’re very familiar with this in security, but you have to think of it in terms of absolute and relative. For me, it was very useful to study philosophy and use that as a guide because from Descartes on they talked about what is intelligence, versus going to the Pope and saying, “What is the meaning of life?” The Pope would give you this dictum or say, “This is how God thinks,” and suddenly people could think for themselves.

But why could they think for themselves? Because they had either inherited concepts or rights or controlled concepts of rights.

In other words, there’s a balance here. And people either, for me, get on an extreme or the other. The same way we get on an extreme of service side computing versus client side computing. If you’re on the client side entirely, it’s like I control everything. Whatever I decide is right. But that doesn’t work very well in security because sometimes the server has got to just say, “Nope, you can’t do that.” And that’s sever side computing where you say, “You have to follow the rules exactly as they’re laid out for everyone who connects to me.” But you can’t have that 100% because sometimes you don’t want to have the server know everything.

There’s secrets that you give to the client that only they can know and some things that are relative that only they control. And anthropologists have said about this in the past, we have time. Everybody knows in investigations you have to have an absolute sense of time in order to piece together what happened in any place in the world.

But we also have time zones. So, to say that I’m in Central European Time, and you’re in Pacific Time is to say we have different experiences that affect our reality when we’re having some intelligent conversations. But we’re also all participating in time, and you can’t just… Apologies to Einstein, but you can’t just make up time and say it’s all relative.

What aspects haven’t been considered?

9:22.434

[David Spark] Steve Zalewski, the other cohost of this show, said, “I would argue the problem is not one of integrity but rather it is curated validity of the data you are letting it ingest. You have to have some governance in place to prevent egregious abuse of the learning models. So, while integrity is important, curated validity means you’re applying the appropriate business filters to obtain output that is more likely to meet the expectations of the business.” And Aaron Stanley of dbt Labs said, “Integrity is only the central problem in an AI system where the output is expected to conform to some idea of true or right, but not all AI systems need or want to be right.

If everything people did was totally grounded in integrity, how would that impact innovation, art, fun? Some applications will need these controls, but a key risk of believing that they are somehow integral to the development of AI systems is that we stifle the development of the systems themselves.” I like this last quote from Aaron arguing that integrity could actually be damaging to creativity.

What do you think of that, Geoff?

[Geoff Belknap] I disagree. I think in this case where we’re talking about integrity, unlike cyber security where we sort of try to box everything into a zero or a one… It’s either a good packet or a bad packet. When we’re talking about integrity, especially with an intelligence system, what we’re talking about what’s the intent.

Are we trying to do the right thing? Are we trying to be…I’m going to use the word good for some values of good. Or are we trying to deceive, or do we even…? Does the AI model not even have any idea whether it’s hallucinating or not? I think what we’re looking for here is to make sure that at least the model is trying to do the best with what it can with the information that it has.

That it’s trying to give you accurate information even if it knows it might be wrong.

It’s trying to be clear about whether it has fulsome information or not. I think, again, this is sort of the difference between the guy speaking to the room very confidently about a subject they know nothing about. There’s a difference… Integrity is saying, “Look, I could be wrong, but I’m going to say these things.

I’m going to have strong convictions that might be weakly held.” And the lack of integrity is, “No, I’m 100% right because I thought of these things in my brain, and I am infallible.” And so when we talk about integrity in the system, it’s just like let’s just try to be clear about what’s happening here.

Let’s do our best.

[David Spark] That’s a good point that Geoff just made, Davi, is that people could feed information and saying, “I’m maintaining my integrity.” Saying, “I’m not 100% sure of my information,” versus the person who is just spouting whatever that could be completely bogus and demanding that everything they say is correct.

What do you think of that model?”

[Davi Ottenheimer] This is fascinating when you get into what happens in history. What we’re dealing with is a domain shift. And when there’s a domain shift in technology, there is a tendency to seek for new science. We want to have new ways of understanding. So, when airplanes start to fly, we have to rethink everything because there’s this new technology, and now people can be in the air.

And before, they couldn’t be in the air. And classic phase over, and over, and over again, what you find is that there’s a mysticism that rises out of this, and snake oil rises out of this. And people start saying things that are inherently untrue because the science isn’t there yet. There hasn’t been a build of authority yet for people to trust what they should and shouldn’t think.

And you actually see this in the early air ships where there were people who would sell things…

There was a guy who literally in San Francisco said that he was going to build an air ship that would be across the country in an afternoon, and everyone should jump on his ship. And as soon as it launched, it crashed, and it was lucky it didn’t kill everybody on board because the guy was just totally making everything up.

It didn’t do anything he said it did. And that’s what I think of when you say, “Okay, can’t we be creative without integrity?” Well, you can. You know? But asking me like, “Why can’t I create art without any integrity?” And the answer is because art is really an exercise in integrity. You establish a domain, and then you sort of play out who is the best at it.

That’s what you see over and over again.

And the fact that you would go in and say, “This is art, and no one can judge me,” is controlling the narrative to the point where you can define bad as good, and that’s not actually how anything useful works. But it’s especially dangerous when we talk about healthcare, security, law, transit where you have strict integrity controls and adherence to a factual correct state.

Because if you don’t do that, people die. The consequences are horrible. Whereas you want to be creative and play music in a new field and say, “This is jazz. This isn’t… Or this is a particular form of jazz. This is bebop.” Well, sure. But let’s define that. So, a group of authorities come together, and they define, “This is good bebop.

This is bad bebop.” You still have integrity within these fields. It just takes time for that domain shift to play out. And that’s where the snake oil salesman come in. And we see a lot of this. People selling stuff that’s just garbage.

Sponsor – Concentric AI

14:18.476

[David Spark] Before I go on any further, I do want to tell you about our absolutely spectacular sponsor and new sponsor. That would be Concentric AI. So, as the leader in AI driven data security posture management or DSPM, Concentric AI understands that Gen AI tools like Microsoft Copilot are creating new data protection challenges that were unheard of not that long ago.

That’s why they’re developing a cutting edge solution designed to protect your most sensitive data no matter where it resides. On premises, in the cloud, or within SaaS applications. So, today’s data protection solutions need to go beyond just identifying risks. Concentric AI was designed to do more.

It proactively and automatically discovers, classifies, and remediates at risk data across your entire organization.

Plus it makes sure that any content generated or accessed by Copilot remains protected from unauthorized access or accidental exposure. Now, whether it’s structured or unstructured data, Concentric AI adapts to your unique environment, providing the intelligence and oversight you need to stay compliant and secure.

So, by integrating seamlessly with Microsoft Copilot, Concentric AI empowers your organization to harness the full power of AI technologies without compromising on security. Trust Concentric AI to keep your data safe so you can focus on innovation, growth, and success. I mean, that’s why you’re trying to use AI, right?

So, go over to their website. Visit concentric.ai today. Do there to discover how you can protect your PII, PHI, and intellectual property with ease.

What’s the best tool for the job?

16:20.759

[David Spark] Jared Mendenhall, who’s the CISO over at Impossible Foods, said, “I question whether an AI system is the best guardrail for itself. While improving data quality, establishing ethical guidelines, and citing sources are all crucial for enhancing AI output, AI systems can still hallucinate.

We often lack control over much of the data in use and what queries were used to generate the output. In short, programming the AI to manage its own integrity control feels like a fox guarding the hen house situation. Therefore, third party validation is essential for a true integrity check.” We will come back to that.

James Bowie, the CISO over at Tampa General Hospital said, “Misinformation is already bad enough without AI. Ai will help exponentially distribute misinformation without proper controls for garbage answers and validation.

We are in the very early stages of this journey. It is going to be a wild ride and an interesting one. AI is one of those tools that has the capacity for achieving great and terrible things at the same time.” I think we can all agree to that very last statement there, Davi. So, I want to reference both James and Jared’s comments here.

Third party validation is essential for true integrity check, and the fact that we are going to constantly have problems with just the validity of information… Forget about integrity but validity of information without some kind of check. So, who is that third party? What is the regulation if there needs to be a regulation?

Where are we operating here, Davi?

[Davi Ottenheimer] This is familiar to me in the context of internal audit versus external audit. It’s not that you don’t want it to manage the integrity of itself. It’s that you obviously have a conflict of interest there that you have to manage as well. So, you have an external pen test on top of your internal security measures, and AI is no different than this.

It really needs a lot of internal integrity controls, and it is guarding itself. But then you validate those by systematic spot checks. You go back, and you look at where they might have had problems. ISO is a good example of this. When you work through an ISO program, it’s like prove to me that all of the checks you’re doing are actually working, and somebody can do that.

So, the third party validation can be that when you’re looking at the tools.

But that being said, this is a bit of a weird perspective for me because I’m starting to think that people are thinking of AI so much as like this external thing that’s way outside of their realm as opposed to augmentation of their self, and that’s a mistake. I don’t like digital twins. I don’t like the idea of our bodies, ourselves being separated from us.

And I don’t like the tools being things that we don’t hold in our own hands. The hammer is in your head, you’re using it. So, who validates that the hammer has integrity? Well, you do. And you do it with the vendor who sold it to you, and you hold yourself and the vendor accountable. You’re both responsible there, and that’s how it should work.

[David Spark] So, Geoff, let me… Is this…? I’m going to boil it down to a very, very simple example. You write something. You edit it. You have your Grammarly check it. You have your spell checker check it. But even if…because you did everything, you hand it off to someone else, and you go, “Hey, would you look at this for me as well?” I mean is it as simple as that?

[Geoff Belknap] I think with some reflection, what people forget and what people are very want to forget sometimes is you have personal agency. And maybe a better way to say that is personal responsibility. When you’re using an AI tool, or a Copilot, or a chatbot, or whatever it might be, I think Davi’s example is apt.

It is affectively acting like a hammer. Now, it might be a very complicated hammer, and you might have no idea what’s inside of it exactly. But you are responsible for that output. We are not yet and may never be at the point where AI is just so much smarter than you that, “Don’t worry. You give me some grunts and clicks, and I will translate that into a language that everybody else can understand for you.

And you don’t ever have to think about what you’re saying, what you’re doing, what the impact of the words that you’re having are.”

[David Spark] I feel a lot of people are falling into that trap. That’s a good point, Geoff.

[Geoff Belknap] Yeah. And I think what happens is when you have sophisticated enough technology that might just be one percent more than what you’re aware of… It doesn’t have to be like alien technology. People immediately sort of all into that trap. It’s like, “Oh, great. I don’t have to think anymore.” And the reality is like, no, you do have to think.

You either have to think about how you, as an individual, are using this tool to produce a document, write content, edit your own emails, whatever it might be. But you also have to think about if you’re creating these tools, how do you build trust and integrity into them?

I think it’s okay that we don’t all know how to do that yet, but I think we also have to look for solutions, and frameworks, and tooling, and consulting, or whatever those options are that are out there today. And to be clear, they’re thin. There are not a ton of options. There’s not a single, industry decided upon standard to use yet.

But that doesn’t absolve you of the need to look into these things. It’s a little like going like, “Well, there’s no standard authentication model yet, so let’s just not use passwords.” Like obviously you need to secure whatever you’re building, but you have to…as somebody who is building something that other people are going to use or somebody who is using something that somebody built for you to use, you have to be thoughtful about how that gets used and what the second order impact of that is.

[Davi Ottenheimer] I just want to add to that. I mean this is fundamentally the core of the issue for me is that we have this model for so many other forms of technology. And when we get into AI… I’ll have to bring up Tesla as the example. It is such bad technology, and it’s very easy for them to say in the car space that the people who are driving are at fault.

They’re always trying to prove that somebody was drinking right after they said, “Use this when you’re drinking.” And you think there’s a sort of merger here where they’re saying, “Make sure you’re drunk when you use the car because we can always blame you when our stuff fails.” And that’s an escape.

It’s a loophole in our way of thinking about these AI systems where we try to make the people as weak and useless as possible who are using them so that we can blame them when these things fail, and the people aren’t standing up and saying, “Hold on. That technology isn’t working right. It’s really not doing what you promised.”

And the way to get around that, to close the loophole, for me, is to put the power back into the hands of the people to actually present themselves to the AI with far more control of their own destiny. If they are in charge of their data, and they control their data, and they agree to or don’t agree to use their data.

In other words, if I can get in the Tesla or not get in the Tesla and I have a choice, I’m going to walk away from the bad AI and walk towards the good AI. But I can only do that if we do have a standard, to your point, Geoff, that’s like a data standard. That’s why I joined Tim Berners-Lee in his quest to fix the web by having a standard.

He wrote a standard solid that really fixes this problem. So, the way that you get to the transparency we were talking about, the accountability after hallucination is to really move towards a better web that gives you that control back where you don’t get locked into garbage tools that hurt you.

I didn’t think of these options.

23:23.712

[David Spark] Phillip Miller, CISO over at Qurple, said, “I am not convinced ‘integrity’ is the best measure because AI is unable to match a human-form of integrity, nor can it be validated in the sense that a building has ‘structural integrity’. That said, there is merit to the concept of a set of quality measures that an individual or enterprise can use to factor into access or usage controls.

Similar to peer reviews of research papers. Multiple factors could be produced with various (open) formulas and then leveraged by other systems in selection processes.” And Nir Rothenberg, who’s the CISO over at Rapyd, said, “We’ve been dealing with ‘garbage in, garbage out’ for decades in computing.

AI is just a more complex version of the same problem. The market will self-correct. Companies that build trustworthy AI will thrive. Those that don’t will fail. No need for endless academic debates. Let the market sort it out.”

Okay, I want to start with Nir’s comment here. Let the market sort it out. But I’m going to come to you last here. I’m starting with you, Geoff. We have seen the market sorted out. We have seen politics where things go in a very wrong direction. And I know that’s always subjective to people, what is wrong and what is right.

But that could be very, very dangerous, yes or no, if we just let the market sort it out, Geoff?

[Geoff Belknap] Yeah. Look, free market capitalism is not a system by which we decide what is the most just, integral, or verdant thing in the world. And to be clear, I’m not speaking against it, just let’s be clear about what it’s good at. It’s good at figuring out what do people want to buy the most, not what the best thing for society or the advancement of our race is.

[David Spark] By the way, it’s why we still have regulations.

[Geoff Belknap] Exactly. This is why we figure out how much lead is acceptable in our food or something like that. Because otherwise we would just buy whatever the cheapest or tastiest thing is, right? And I think certainly there is a lot of value here in understanding what people want to buy. But I think ultimately when we build things, it is incumbent on us as the people that build them, that build tools and build things that improve people’s life, or enhance the economy, or make businesses easier, and better, and more efficient at what they do…it’s incumbent upon us to make sure that we’re building things that are sustainable.

And to be clear, what I mean by sustainable are like it’s not a one-time thing. It doesn’t just burn through humans to do the work. It’s incumbent upon us to make sure that they help protect the economy that we’re trying to enhance and doesn’t abuse the people that use it.

So, I think in this case, even though we might not know the best way or the most perfect way to ensure the integrity of a model or ensure the integrity of output, we got to start someplace. We shouldn’t just throw our hands up and go, “Eh. If people really want seatbelts they’ll make us invent seatbelts, or airbags, or crumple zones, or whatever.” I just can’t stress enough especially as security professionals, it’s sort of on us to build those things in before people know to ask for it.

Otherwise… If you’re the inventor of the airbag, you’re not going to wait for somebody to ask you to invent an airbag. You’re going to look for a need in the market and then fill it.

[David Spark] It is the classic case of the old line of if Henry Ford asked people what they wanted they would say faster horses. But, Davi, I’m going to throw this to you. And I know you kind of reacted very much to Nir’s final quote about letting the market sort it out. I’m getting some sense that there is some space in between government regulations and market sorting it out.

And we’ve seen this with just sort of organizations like trade groups that sort of self-regulate. I don’t know. Where is this space that we can operate in that we can sort of build this integrity level, I guess?

[Davi Ottenheimer] There’s just so many levels of wrong in this comment, I don’t know where to begin.

[David Spark] My comment or Nir’s comment? Hold on, whose comment?

[Davi Ottenheimer] No, Nir’s comment. Let me just clarify. I mean the elephant in the room to me is you shouldn’t be allowed to be a CISO if you think the market should just sort itself out because the CISO role is anti-free market. It’s like you’re saying, “I’m going to put rules and regulations in place.” Even if you’re the most extreme form of CISO, your job itself depends on being a regulator of the information space that you’re working in.

You literally are putting controls in place. So, that’s not free market, open, anybody should do anything, the market will sort it out. That’s like, “Okay, we got to get some rules in here. We’ve got to put some policies in place, and we got to sort of lock it down,” because you can’t just let anything go.

And to put a more fine point on it, law and order is what allows markets to flourish. You can’t have markets if you don’t have law and order because criminals step in. And next thing you know, it’s just mob rule. And we know historically this has been tried time, and time, and time again.

When you allow things to just devolve, fall apart, if you allow toddlers to rule because parents aren’t allowed to, you don’t get a great kid. You actually get kind of the opposite. And people say, “Whoa, that’s not a good parent. They let their kid just do absolutely everything they wanted because they’ll figure it out.” And if you put it back in the AI context, it’s no parent who knows what they’re doing is going to say to their kid, “Go out there and absorb everything.

As much data as possible of all content because you’ll figure out everything just the way you need to.” That’s just not how to parent, and that’s basically what AI is. It’s like a toddler entering the space. And what you do instead, a good parent would say, “I’m going to put you in front of the best teachers.

I’m going to put you around the best friends. I’m going to get the best quality in your life by regulating and then giving you the safe space from which you can work in.” Because if I give you sort of a sandbox or a playground with a place you can fall on then you’re going to be able to grow and learn in a safe way.

But if I put you into a place where anything can happen, you’re dead by the time you’re five. It just doesn’t play out that well. So, that’s why… I could go on forever about this, but the fact that we have people in modern society saying the market self corrects shows us that we’ve had a failure of education.

In the security industry itself but also broadly that this just doesn’t work. This just isn’t true. It’s like people saying the earth is flat. And what do we do with those people when they go around saying the earth is flat?

[David Spark] We don’t believe them, or we make a mockery, or some… But the thing is some people believe them. That’s the thing that I’m always amazed by. If I watch a video on YouTube of someone claiming the earth is flat and I watch a second video of someone claiming the earth is flat, YouTube thinks I’m really into the earth is flat.

And all of a sudden, my world, my consumption environment is videos and people making some type of logical argument of the earth is flat. And because my view is that, I’m going to start to assume it because there’s lots of people telling me that and lots of very convincing people, too. So, we can fall into this trap pretty darn easily, can’t we, Davi?

[Davi Ottenheimer] We can fall into it easily if we don’t have the right controls in place. And so one of the things that people talked about many years ago… George Lakoff, I think, in “Metaphors We Live By…” I was forced to read this back in the early days of my disinformation learning. I mean, to be frank, I learned military history and disinformation tactics that were used in World War I and World War II because the methods that were used then are so seminal in today’s debates.

Like Woodrow Wilson has a propaganda office that basically changed the outcome of World War I by running it as a government office, and he nationalized the phones so that the tapped everything and listened to everything. And if people understand our history, and how we’re manipulated, and why we’re manipulated, and the methods that are used, they would see that… Again, back to George Lakoff’s point.

The way you handle disinformation is you sort of put it into a truth sandwich. The way you avoid that is by saying first, “I’m going to tell you something that’s totally bonkers, wrong.”

And then you tell it. And then at the end you go, “Now, why that’s wrong is because of this, this, and this reason.” Now you’ve put it between these two truths before and after, so now you can share that information. But if you just tee up the information with no controls around it, no safety, you absolutely are going to spend a lot of money and time cleaning up the giant mess you’ve just created for yourself.

It’s not unlike putting vulnerabilities into code. If I release code with a bunch of criticals, I’m going to spend all of my time and money trying to clean up all that mess. We should not be doing this with the way we handle information, and yet we still do it. Like, “Let’s talk about the earth is flat.

Go.”

And even playing field, let’s just say… It’s like, “Let me just put the secrets, and the passwords, and the tokens into the URL and just have it stuffed into URI, and let’s see how that goes. I’m sure that’ll work out great. The market will figure out how to make that safe.” It doesn’t work out. You’ve got to put in place hard rules based on science that this is expensive, so fix it now.

And when they say we shouldn’t have lots of academic debate, in fact an academic debate is the place to figure this stuff out, but it has an end. It’s the experts debate it, and they get to an end point. And that’s why we don’t think the earth is round. We do think… I mean we…

[Laughter]

[Davi Ottenheimer] We don’t think the earth is flat. We think it is around. And people are always going to disagree, but we have an academic environment for those people to go to make their case, and generally they just don’t get anywhere. They don’t get degrees in flat earth.

[David Spark] No, you can’t get a degree in flat earth.

Closing

32:24.168

[David Spark] Davi, I always enjoy having you on the show. I thoroughly enjoy sort of opening my thoughts and ideas on topics like this. It’s fascinating. Thank you. Okay, we have come to the portion of our show where I ask you which was your favorite quote and why. Davi, I will ask you first. Which quote was your favorite and why?

[Davi Ottenheimer] I got to say Jared Mendenhall, Impossible Foods, because the first thing that came to my mind when he said, “I question whether AI systems are the best guardrail for itself…” The first thing that came to mind was if you said to your let’s say AI chef, “I’m really hungry tonight, and I think… I don’t care what you make but as long as it has meat in it.

I want some meat. I’m feeling like I need some protein.” And it goes away and comes back and says, “Here’s your dish. What do you think?” And you go, “Oh, this is delicious. A little chewy. But where’s my cat?”

[David Spark] [Laughs]

[Davi Ottenheimer] An example of how you just can’t let the thing figure out for itself what to do. You really got to get the controls and guardrails in place of what’s allowed from your perspective because maybe you want it to chop up your dog, but maybe you don’t. I mean it’s just kind of within the guardrails of the society you live in, the groups you live, but also your own personal preferences.

You’ve got to manage that thing very closely.

[Geoff Belknap] Although to be clear, I’m pretty sure that cats and dogs are not part of Impossible Foods.

[David Spark] Impossible Foods. Yes, they are a non… They are all plant based products. All right, Geoff, your favorite quote and why.

[Geoff Belknap] I am going to go with this young upstart, Steve Zalewski. And I think his quote here… “I’d argue the problem is not one of integrity. Rather it’s curated validity of the data you’re letting it ingest.” I think Steve has a point here, and I think this is one of the areas I tell people is a very easy place to start, is understand what you’re training your model against.

And I think just blindly training your model or your AI system against everything you can possibly find and assuming that it’s going to come out and give unbiased, very clear factual answers is assuming that everything on the internet is unbiased, very clear, and factual. Which if you believe that, I have a bridge to sell you, or maybe some CISO Series coin, or something like that.

[David Spark] Or some magic beans.

[Geoff Belknap] Or whatever. Some digital magic bean coin. One of the easiest things we can do is sort of understand what we are using to train our model with and understand how that might have an impact on the output. If you have nowhere else to start, that is a good place to begin.

[David Spark] Awesome. Thank you very much. Davi, I always enjoy having you on the show, and I thoroughly enjoyed sort of your opening my thoughts and ideas on topics like this. It’s fascinating. Thank you, as always. I’m going to let you have the very last word here. I first want to thank our sponsor, and that is Concentric AI.

Find and protect your data with Concentric AI. If you’ve got Microsoft Copilot in your environment, you owe it to yourself to look at what they’re doing over at Concentric AI. Go to concentric.ai. It’s as simple as that. Check it out. Geoff, thank you as always for being a part of the episode. Davi, are you hiring over at Inrupt?

[Davi Ottenheimer] We are. We’re always looking for great people, but in particular we’re looking for some great engineers. So, hit me up if you want something.

[David Spark] Are you just davi@inrupt.com? Is that your address?

[Davi Ottenheimer] Correct. Yeah, davi@inrupt.com.

[David Spark] All right. So, davi@inrupt.com. Go send him a message. But I’m assuming you have a job board on Inrupt, the website, yes?

[Davi Ottenheimer] We do. Inrupt.com, yeah. So, first check out solidproject.org first to see what Tim Berners-Lee has invented and we’ve been working on for so many years, to fix the Web3, make it better. And then after you check out solidproject.org, check out inrupt.com to see how we’re doing an implementation of that solid protocol.

[David Spark] I will add links to all of that on our site, plus the previous episode Davi was on of Defense in Depth. He goes back many years about machine learning failures that I absolutely adore. Thank you very much, Davi. Thank you very much, Geoff. And thank you to our audience as well. We greatly appreciate your contributions and listening to Defense in Depth.

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site, ciso-dev.davidspark.dcgws.com, where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.

If you’re interested in sponsoring the podcast, contact David Spark directly at david@ciso-dev.davidspark.dcgws.com. Thank you for listening to Defense in Depth.