Managing the Risk of GenAI Tools

We know new generative AI tools come with risk. What are you doing NOW to manage those?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Joining us is our sponsored guest Karthik Krishnan, founder and CEO, Concentric AI.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Concentric AI

Full Transcript

Intro 

0:00.000

[David Spark] We know new generative AI tools come with risk. What are you doing now to manage those risks? 

[Voiceover] You’re listening to Defense in Depth. 

[David Spark] Welcome to Defense in Depth. My name is David Spark, producer of the CISO Series. And you’re excited. Are you excited? Because I’ve got one of your favorites here. It’s none other than Steve Zalewski as my co-host. Steve, say hello to the audience. 

[Steve Zalewski] Hello, audience. 

[David Spark] Ah. Our sponsor for today’s episode, Steve, is none other than Concentric AI, and in fact, they’re responsible for bringing our guest today, who we’ll introduce in just a moment. I do want to mention to our audience, if you haven’t been to our website, CISOseries.com, you should check it out.

We’ve got a lot more programs there and lots more great content on CISOseries.com. All right. We’re going to be talking about AI here, Steve. That has come up in a few conversations you’ve had in the past, correct? 

[Steve Zalewski] Once or twice. 

[David Spark] It’ll come up today.

[Laughter] 

[David Spark] Sometimes lost in the AI gold rush, is that while a lot of organizations want to embrace tools like Copilot, we’re still wrapping our heads around the managing of the risk of things like Copilot. So, rather than fret about what can go wrong, what are we doing as an industry to actually reduce the risks with these tools and actually get the productivity gains they promise?

So, we know there are productivity gains. We know there are risks. Instead of complaining, what can we do to deal with this? Steve, this was a question you posed to the LinkedIn community. 

[Steve Zalewski] Yes. And the context here is for the last year, every panel, every conversation, right, every ask is, what are you doing around generative AI? And the frustration I got was, “We’re talking about it, but we’re not doing much about it,” and I just felt like it was time to kind of let everybody see that we’re having the same conversations and look at where the technology kind of bar is right now against just guardrails.

[David Spark] That is a very, very good point. And guardrails, by the way, is part of the solution, and we are going to discuss that. And in fact, we have the perfect guest to discuss this because he is literally in the category of doing something about AI. Very excited to have him. Our sponsor guest, he’s from Concentric AI, couldn’t ask for a better person.

The founder and CEO, Karthik Krishnan. Karthik, thank you so much for joining us. 

[Karthik Krishnan] Great to be here. Thank you for having me.

We’ve seen this one before. 

2:38.357

[David Spark] Ed Covert over at Bowhead Specialty said, “This is a risk that should be managed like any other. The first step would be understanding why the organization is concerned and then articulating the risk so that users understand why the organization thinks the risk is important. If you can’t do that, everything is just good money after bad.

And do they care about loss of revenue, loss of IP data, etc.?” This, I think, probably frames this issue as sort of simply “and as well as we could want it right now.” But let me also add what Mauricio Ortiz of Merck said, “Work on having an AI inventory with sanctioned AI tools/solutions to help you track areas and their risks.

It would be highly recommended that you hire a vendor to assist you with an AI risk assessment to work on areas for improvement.” Again, we’ve talked to a lot as well. This is not something you can do alone, is it, Steve? 

[Steve Zalewski] Well, and the two of these also are talking about the what should we do and how should we do it versus what we can do, given we don’t have any money or resources to do it with. So, to your point here, Ed simply saying is, “Is this any different than any other DLP problem?” Great way to position, right, coming out of the gate as to what guardrails might look like and whether you need new technology or is this just more of the same?

And then when you look at Mauricio, he’s like, “Okay, this is net new. You need to go hire an expert to help you have this conversation.” And I go, “Both are true.” And so therefore, it got back to the we can talk about it, or we can do something about it, and which of these even, right, is the one that you want to pursue if you have very limited money and very little people?

[David Spark] Karthik, I got to imagine that you’ve had kind of the same conversation over and over with potential customers. Like, they are in this realm that we talked about before, which they’ve seen every damn panel out there, possibly, talking about this and talking about this and talking about this.

And they realize they need some help, and they want to get to the [Inaudible 00:05:03] face because maybe the business is running amok without them, and you’re like, “Oh, geez, we have to do something quickly.” So, what is this? Let me ask more of the situation. What is the situation potential customers come to you with this problem?

Where are they? 

[Karthik Krishnan] I want to just zoom out a little bit and place this in the context of everything that we have seen over decades and decades. Right? You can go back to mobile laptops when they came out, where people moved away from static desktops to mobility for productivity reasons. Then the cloud happened.

And GenAI is, while it’s a fascinating new topic, it’s along the long lines of human desire for continued ways to automate things, to become more productive, and this is just one more technology evolution, if you will, that’s happening. And productivity will trump everything else. Right? So productivity will find a way to get to the hands of users and employees who need them, and then the enterprises have to respond to the risks and everything that happens as a consequence.

And so the reason I’m pointing that out is, yeah, there’s a lot of mental gyrations that our customers are having around, “Oh, my God. GenAI tools, our customers, our employees are going to start using them, and what’s going to happen to security and the risk? But if you zoom out, you’ll realize that this is just one more of those, and it’s fairly inevitable, an inevitable thing that’s going to happen.

[David Spark] What is that conversation they’re having of, all right, I know I need to do this. I know productivity is king. It is not cybersecurity. Where do I go from now? What is that conversation? 

[Karthik Krishnan] Yeah, the conversation is very simple, it is how do I allow my employees to be more productive while still making sure that I am not going to place sensitive data of mine at risk from data leakage? For example, if my employees want information that is going to make their jobs more efficient, I don’t want them ending up getting access to my sensitive financial data or critical intellectual property or customer information.

And so how do you make sure that you’re able to create an appropriate fencing between the kinds of information that will make them more productive and the kinds that will potentially place enterprise data at risk from leakage and serious damage to the company itself? 

How do we approach governance? 

7:38.864

[David Spark] Matt Konwiser of IBM said, “Protecting the data in the model is well covered by traditional controls and the new NIST framework recommendations. The clients I’ve spoken to are more concerned about the models being socially engineered since models cannot use their “spidey sense” to distinguish an imposter or corrupted employee from a standard daily job function prompt.

AI governance has to be a pipeline-level protection, not model protection.” That’s an interesting point right there. 

Let me throw out David F. of Mastermind, who throws out something else here. “The situation highlights why it’s critical to have governance programs that adapt as the risk landscape changes. While some SIEM solutions are designed specifically for monitoring AI application usage, they must provide continuous and relevant visibility into the tech stock to be effective.

Similarly, I’ve seen organizations adopt generic acceptable use policies and distribute them to [Inaudible 00:08:44] without proper training on data protection risks and the potential breaches associated with unauthorized LLMs.” 

So, what they’re talking about is what is the program you have to put in place, and the SIEM is just sort of an example here. But the idea is that you can’t have a generic model and hope that works. It has to look at that very specific pipeline, which is an AI pipeline is going to be very different than a human pipeline, correct, Karthik?

[Karthik Krishnan] Absolutely. So, when, for example, your employees ask one of your internal LLMs a question like, “Hey, how are we doing this quarter revenue-wise?” The model’s not putting out a result in a vacuum. It’s leveraging the available knowledge base within your enterprise to be able to provide that information back to that employee.

And so insofar as you have got sensitive information within your environment around sales data, revenue data, financial data, that’s the information that’s going to be leveraged. And so the pipeline point is spot on because if you were to ask an LLM a question, it’s basically leveraging that information.

And so understanding across the entire pipeline how that LLM…what information the LLM is accessing to be able to provide a contextual answer back to your employees is super critical, and so looking at the entire pipeline is super important. 

[David Spark] So, Steve, I’m going back to an earlier question I came with is you can’t do this alone because your average person doesn’t know what the LLM is asking or what the employees are asking of the LLM. Like, they don’t know, so they don’t have the history of dealing with AI tools and LLMs to be able to do this on their own.

Or is there some way you can do it on their own? Like, I think that’s my question is you would need a tool to manage this for you, or am I wrong? 

[Steve Zalewski] Here’s my frustration building, [Inaudible 00:10:41] here’s the analogy I use. We’re parents and the business is the child, and we’re trying to teach the child that a stove is hot, and you shouldn’t touch it if it’s hot. Okay? Well, unfortunately, some kids are going to touch that stove and learn the hard way that it’s hot.

Others will listen to our warning. Okay? Well, the guardrails that we’re putting in is we’re telling all of the children, “Hey, be careful, the stove could be hot.” Okay? But the business, to what Karthik is talking about, needs to get productivity in, so there’s a whole bunch of them touching that stove hoping it’s not going to be hot.

And that’s where I was pushing on. What are the technical controls? What are the real things that we’re doing not to prevent the children from getting burned but to manage the burn? Okay? Get them away from the stove when they are initially getting their first-degree burn. Okay? And my point here being most of those kids, meaning most of the business, as they’re touching GenAI are realizing it isn’t all it’s made up to be, and they are becoming dissatisfied.

And so let’s go ahead and let them touch the stove if we know we can get it away from them before they do any real damage. That’s the transition that I was trying to get to. 

[David Spark] Karthik, actually, you probably have some insight on how people are using Copilot, other GenAI tools, that many of us don’t know, haven’t seen before. So, my question is, what have you learned about how people are using this? And are they going into situations where they’re touching the stove like Steve just played out?

[Karthik Krishnan] Absolutely. So, we have financial services customers looking at Copilot, and they have researchers like their equity traders and so on needing to leverage Copilot to do a bunch of research on companies and so on. But while they’re doing that, they have to make sure that if they’ve got any sort of sensitive information within the company, that that’s not something that these language models end up leveraging as part of the research that they’re actually presenting back to their own employees.

And so, perfect example of how you need to find a way to get them to be able to use those tools and yet not necessarily get burnt along the way. 

[Steve Zalewski] So, Karthik, let me ask you this, right? Which was we talk about garbage in, garbage out. What we’re saying here is we need truth in, truth out. Okay? But what we’re seeing with GenAI is you can put truth in, but there’s no guarantee that what comes out is true. It’s a story, right?

And that the business now has to determine whether that’s truthful. And that’s where I was getting at, which was they’re touching the stove, and they’re realizing that what’s coming out, even though it’s based on truth, okay, is not something they can count on. And so therefore they become disillusioned with the storytelling versus the truth telling.

True? 

[Karthik Krishnan] That’s totally fair. I think in the end, these models have to… We’re in the early days of this journey. And so as much as, like everything else, there’s a huge promise and then you’ll go through your trough of disillusionment. And then eventually these tools will end up, almost inevitably, radical technology such as this will under deliver in the short term and over deliver in the long term.

And in my view, this is no different. 

Sponsor – Concentric AI 

14:23.839

[David Spark] Before we go any further, I do want to talk about Karthik’s awesome company, and that is Concentric, and we love having Concentric on board. So, thank you for being an awesome sponsor of the CISO Series. So, as the leader in AI-driven Data Security Posture Management or DSPM, Concentric AI understands that GenAI tools like Microsoft Copilot are creating new data protection challenges that were unheard of not long ago.

That’s why they’ve developed a cutting-edge solution designed to protect your most sensitive data, no matter where it resides – on-premises, in the cloud, or within SaaS applications. So, today’s data protection solutions need to go beyond just identifying risks. 

Concentric AI was designed to do more. It proactively and automatically discovers, classifies, and remediates at-risk data across your entire organization. Plus it makes sure that any content generated or accessed by Copilot remains protected from unauthorized access or accidental exposure. Now, whether it’s structured or unstructured data, Concentric AI adapts to your unique environment, providing the intelligence and oversight you need to stay compliant and secure.

Now by integrating seamlessly with Microsoft Copilot, Concentric AI empowers your organization to harness the full potential of AI technologies without compromising on security. Trust Concentric AI to keep your data safe so you can focus on innovation, growth, and success. Visit Concentric AI today to discover how you can protect your PII, PHI, and intellectual property with ease.

Just go to their website, that’s concentric.ai. Go there now. 

What aspects haven’t been considered? 

16:14.648

[David Spark] Chad Beckman of TrustMAPP said, “Some best practices from a data officer.” This is what he learned. “Step one, ensure the data you will be using AI with is the proper layout and is scrubbed from PII and other confidential information.” Karthik is nodding his head here, he agrees. “Protect this data store accordingly and keep it outside of production systems for now.

Now step two, leverage AI to internal use only, refine the process, and test the outputs, including data security. Step three, once a level of confidence is achieved, extend the use of AI to trusted external parties. And step four, after further confidence and validation is achieved to the data returned and the controls around the data, now a business can expose the AI to any applicable external parties.” 

And then Brian Clark of Info-Tech Research Group said, “It is my opinion that this begins with solid enterprise and application architecture. You can write the policy you desire, but if you lack the ability to monitor and enforce written policy, it will be circumvented.” All right, there was a lot of nodding of your head through all of this, Karthik.

It sounds like you very much agree with… Now, Chad learned this from another data privacy officer or a data officer. These seem like pretty darn good steps, like get your systems in line, test it out internally, and then slowly release it externally. You agree, yes? 

[Karthik Krishnan] Absolutely. Any security program, it’s usually the following sequence, and there’s no getting around that. First, discover, know what’s happening within your environment. Second, put an effective risk monitoring program around the things you care about. And then if you see things at risk, you protect and remediate.

And then God forbid, you have to go back. You can also do some sort of post-forensics investigation. And if you lay this out in the context of this, the first thing you have to do is understand what’s actually happening within your environment. Like what sorts of requests are getting presented to these language models?

And the consequence of those requests, what sorts of information are these models accessing? Are they accessing generic information to be able to summarize what happened in a meeting the previous day? Or are they accessing your sensitive financial data or business confidential data and intellectual property or customer data and presenting it back to your customers?

And once you have a good sense of what’s actually happening within your environment, which is the discovery phase, now you can actually effectively monitor that and make sure that you’re able to put the guardrails around the data that you want to fence off. If it is sensitive information, you want to perhaps classify it.

You may want to put some appropriate permissions control around it. And you can help identify the information that’s of consequence to you that you want to keep outside of the purview of these models. And on the other side, as it spits out data, you want to have a sense of, hey, what is it throwing out to my customers?

Is it giving out sensitive information that potentially they shouldn’t be able to access? And so make sure that you’re able to classify that. And so putting exactly like the gentleman said, you can have all the use policies that you want around what needs to happen within your environment, but if you don’t have a program that allows you to effectively discover, monitor, and enforce, you’re going to be out of luck.

[David Spark] All right. Well, basic policies. And also I’m going to toss to Brian’s comment here, of like, yeah, have all the policies you want, but if you’re not actually doing it, per what Karthik just said, I mean, it’s pointless. I mean, this is not any news to our audience at all, Steve. 

[Steve Zalewski] So, this is what I was getting at. If I can wave my magic wand, okay, both Chad and Brian are right. This is how to do it. So, if you only listen to me, I will guarantee that you will be the best adult you can be. But the business is a wayward child. And what we’re calling out over and over is not that we don’t know how to solve the problem if everybody would only do as I say, it’s how do we solve the problem when the wayward children aren’t interested in having to listen to what we say, but we’re still obligated to protect them, right?

And this is where it gets back to, hey, how much of that data, now that I’m exposing it as a data service, not as a network edge, is unclassified because it never had to be, and it wasn’t an issue because the business wouldn’t classify it for us, but we hid it behind the network edge. Now it’s all exposed.

Even worse, that data like an email address can be public knowledge, it can be private knowledge, it can be PII knowledge, depending upon the context that it’s being exposed. We don’t know how to do that. The business isn’t interested, and yet security is trying to enforce it. So, this is where I got back to why we’re having this episode is so it’s a DLP problem, it’s a DSPM problem, meaning it’s not something we haven’t solved before.

We’re just solving it at a different edge, and therefore we just have to do the best we can, but we can’t try to prevent the child from getting burnt. 

What are the elements that make a great solution? 

21:44.402

[David Spark] Robert D. Brown III of Resilience said, “One very important activity that all decision makers need to fulfill prior to engaging in any initiative is a very clear framing of the objectives and goals of the initiatives. That’s the reason humans make decisions – to achieve human-centered objectives and goals according to their values frame.

Those are not clarified beforehand. You will not take a coherent set of appropriate actions on purpose to achieve them. Not engaging in this framing exercise is generally the reason almost all projects fail. So, the first risk companies need to address is the possibility of solving for an irrelevant problem with an ineffective or worse solution.

Solve for business relevance first, and the ability to address almost all other relevant risks generally comes into sharp focus. The framing will also yield the important KPIs and guardrails needed to make sure that the initiative is leading to the desired outcomes.” Wow, that is on the money because he sort of does the ultimate shift left of, “Well, why the heck do you want to use AI?

For what darn reason?” Like not just because it’s cool and everyone’s talking about it, right, Steve? 

[Steve Zalewski] Yes, and if I stand with my analogy, so it’s, let’s not worry about getting burned on the stove. Do you need a stove? If you have no need for the stove, you don’t need to be in the kitchen. And the way I really interpret this is what we talked about. Let the business decide how important this technology is to their ability to be both efficient – productivity – and effective – revenue generating – and let’s let them make up their own minds, right?

And a lot of the conversations have been, which is why we put guardrails up primarily, which is just policy and very little technology, because we really don’t know how many of those business use cases are going to get through the gate that actually then want to go into production. Then we truly can use our DLP and DSPM, which will probably cover 80% of the problem.

[David Spark] Karthik, correct me if I’m wrong. Have you had customers come to you and says, “The business is using AI. We don’t know why, but we need to protect it.” Do you have you had that conversation? 

[Karthik Krishnan] 100%. 

[Laughter] 

[David Spark] Oh, really? So, they are not following Robert Brown’s advice, which is what they should be doing. 

[Karthik Krishnan] We had a hedge fund where the CEO decided he had to roll out Copilot to everybody, and then the CISO came to us completely freaked out saying, “Look, the CEO wants to do this. I have got to get a data governance program and technologies in place, or I am just going to be dealing with data leakage issues all day long.” And this happens all the time.

In fact, another CISO told me his CEO had rolled out Copilot to 30,000 users within his organization, and they were having a reduction in workforce meeting. 

[David Spark] Hold it. They went from zero to 30,000 just like that? 

[Karthik Krishnan] Zero to 30,000 in one of the fastest [Inaudible 00:25:02] ever. 

[David Spark] Holy moly!

[Karthik Krishnan] And they were having a reduction in workforce conversation, and somebody said, “Should I record this meeting?” And he was shrieking saying, “Absolutely not! Because I have no way of knowing if Copilot’s going to pick this up and then end up broadcasting it within my entire company.” And so it happens a lot.

[David Spark] Yeah, okay. So, this is the common behavior of this. 

Closing 

25:25.457

[David Spark] All right, let’s bring this show to a close. This was great, guys. This was fantastic. What a great conversation. All right, this is where I’m going to ask you, Karthik, which quote was your favorite and why? 

[Karthik Krishnan] Ed Covert’s point completely resonated with me when he was talking about understand what risks these things really present. Understanding is this loss of revenue? Is this loss of data? What sorts of data matters to you? At the end of the day, one of the founders of my prior company told me, “Security is like brakes in a car.

It’s not meant to slow you down. It’s meant to get you to go faster.” 

[David Spark] We’ve used this metaphor many times, yes. 

[Karthik Krishnan] [Laughter] And so the idea really here is knowing what risks these things present yourselves to will prevent you from looking like the party pooper and looking more like the business enabler, except you’re going to be able to do this in a secure fashion. 

[David Spark] Very, very good point. All right, Steve, I throw it to you. Which quote was your favorite and why? 

[Steve Zalewski] All right, so I’m going to dovetail on Ed’s risk, and I’m going to work with Robert Brown as my choice.

[David Spark] So, opening and closing quotes here. 

[Steve Zalewski] And in particular, what he says, right? Which was, so the first risk companies need to address is the possibility of solving for an irrelevant problem with an ineffective solution. Okay? Solving for business relevance first, and the ability to address almost all the other relevant risks generally comes into focus.

And I would say great way to close.

[David Spark] I love it. Yeah, well, there’s a reason we put them as a closing quote. We have rationales among our madness. By the way, I got to give it to the great production staff that puts these shows together. So, not all my credit. Awesome talent here at CISO Series. All right, with that being said, let’s wrap this show up.

I got to thank you, Karthik. That was fantastic. I’m going to let you have the final word. I want to thank your company, Concentric AI. Remember, concentric.ai, go to that site. Go check out what they’re doing. If you’re using AI in your environment, you owe yourself to see what they’re doing, especially Copilot for that matter.

All right, do you have any special offers? Anything you want to offer to our audience? Oh, also we always like to ask, are you hiring? Let’s hear the final word, Karthik. 

[Karthik Krishnan] Yes, we are hiring. We’re in a period of dramatic growth, and my appeal to your audience is if you’re interested in technologies like GenAI technologies that can actually protect GenAI in discovering, monitoring, and protect your data, come visit us at concentric.ai. Give us a holler, and we’d love to learn about your requirements and see if we can be of help.


[David Spark] Awesome. Well, thank you very much. Thank you to Concentric AI. Thank you, Steve. And thank you, audience. We greatly appreciate your contributions and for listening to Defense in Depth. 

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.

If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.