Large companies aren’t traditionally known for being the fastest to adopt new technologies. So why is cybersecurity the exception? It seems like the larger or more highly regulated an enterprise, the more likely they are to be a cybersecurity startup early adopter. What is it about this technology that flips the typical adoption curve?

This week’s episode is hosted by me, David Spark (@dspark), producer of CISO Series and Andy Ellis (@csoandy), operating partner, YL Ventures. Joining us is Justin Somaini, partner, YL Ventures.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Vanta

Full Transcript

Intro

0:00.000

[Voiceover] What’s a great approach from a security vendor? Go!

[Justin Somaini] My tactic and advice that I would give to sales teams is to make their customer champions of success within their organizations. If they’re able to do that, you win a customer and a relationship for life.

[Voiceover] It’s time to begin the CISO Seies Podcast.

[David Spark] Welcome to the CISO Series Podcast. My name is David Spark. I am the producer of the CISO Series. And joining me as my cohost, it’s none other than the operating partner of YL Ventures, Andy Ellis. Andy, say hello to the audience.

[Andy Ellis] Good morning, folks. Or depending on when you’re listening, good afternoon, good evening, or goodnight.

[David Spark] Our sponsor for today’s episode is Vanta, your security and compliance verified. And more about that later in a show. And a huge thanks to Vanta because they have been a phenomenal sponsor of the CISO Series. Andy, we put this call out to the community to ask them how successful you are at getting people to agree with you on the, “What’s Worse,” game.

[Andy Ellis] Yep.

[David Spark] And you believe that it’s a very high percentage of people agree with you. Well, we have a fan who did just that. They actually went back through four years’ worth of our episodes, which is, I think, how long we’ve been doing transcripts, so they wouldn’t actually have to listen to all of these, to determine your success rate.

Huge thanks to Derek Clauver [Phonetic 00:01:35] of Oksoro [Phonetic 00:01:36] I believe is how you pronounce his company’s name, who has done this work. And, Andy, we have the numbers now.

[Andy Ellis] Excellent. I got a sneak peek at the spreadsheet, so I’m very excited.

[David Spark] Yes. So, the numbers are that essentially about three-quarters of the time people agree with you.

[Andy Ellis] Yep, but I’ve been getting worse, I noticed.

[David Spark] Noticed, yes. So, Andy, you did do the stats here. I noticed that in 2021, you were at three-quarters. 2022 jumped dramatically to about an 87% agreement. Then sliding in 2023 to about 69%. And it’s looking really bad in 2024 where you’re just essentially the flip of a coin at 53%.

[Andy Ellis] Basically. I need to go back and look at this year’s because I went and looked at 2022 to figure out what happened and all the ones that were disagreeing with me. I did notice that I don’t play the audience sometimes very well. There was a couple places where we had a sponsored guest, and the, “What’s Worse,” scenario… Like one of the situations played to what they sold, but I went for the other one.

And they were like, “Well, of course I can’t take that. I have to go and take mine.” And so I need to think a little bit more about who I’m competing against.

[David Spark] That rarely… I think I can only think of one time that that scenario came about.

[Andy Ellis] Well, it may be the same one I found, but I’m going to go look and see if I’ve got any more like that. But I did find we had two times where the guest refused to play the game.

[David Spark] Yeah, that’s irritating, I must say.

[Andy Ellis] Well, they were both fascinating. It was Renee Guttmann who basically said, “These are both awful scenarios, and the fact that they’re in either one of them is my fault. So, I’m just going to take the blame for having a bad security program that got us here.” I think that was a really beautiful way to answer it.

[David Spark] [Laughs]

[Andy Ellis] And Jason Witty, because we gave unrealistic scenarios. And he said, “I don’t live in a fantasy world, so I’m not going to play.”

[David Spark] Bruce Schneier, his response to that has been, “Don’t fight the hypothetical.”

[Andy Ellis] Yep, but we actually almost talked him around to agreeing with me, but we ran out of time, and we just moved on. So…

[David Spark] The best example of this… And I’m trying to remember. It was actually the CISO of Atlassian, and I’m going to blank out on his name right here. But we were recording Atlassian with the other cohost, Mike Johnson, and Mike answered one way, the Atlassian CISO said… Oh, Adrian Ludwig. He’s no longer the Atlassian CISO.

[Andy Ellis] Yep.

[David Spark] But he said something else. And Mike said, “I want to change my answer given what Adrian…” [Laughs]

[Andy Ellis] Wow.

[David Spark] And I said, “No dice. You have to stick with [Inaudible 00:04:22]

[Andy Ellis] Yeah, but I did go… Just so you know, the very first one he detected was when I was a guest, and Mike was the host. So, I did not use that in my summary.

[David Spark] Oh.

[Andy Ellis] Because the fact that there was disagreement when I went after Mike isn’t really the same…

[Crosstalk 00:04:34]

[David Spark] See, the thing is also, it’s like you’re the house in that…

[Andy Ellis] Right.

[David Spark] …you get to answer first. And if it’s an easy scenario then they’ll always do it. And often we have easy scenarios, which I try not to create.

[Andy Ellis] Yeah. So, I think we’ve been getting better scenarios, so there’s more disagreement.

[David Spark] Let’s hope. By the way, call out to the audience. Send us more, “What’s Worse,” scenarios. Enough of this. We will have the, “What’s Worse,” game later in the show. Let’s bring our guest on, who you work with, Andy.

[Andy Ellis] I do.

[David Spark] This person has actually been a multi-time CISO. And like you, he’s a partner over at YL Ventures. It is none other than Justin Somaini. Justin, thank you so much for joining us.

[Justin Somaini] Thank you very much and great to be here.

How can we secure new technology without creating new risks?

5:19.572

[David Spark] With a lot of technology, large and regulated enterprises can be slow to adapt. Think about how long the corporate mandated Blackberry stuck in the enterprise while SMBs were quicker with BYOD. But Ross Haleliuk points out that this is the opposite with cyber security, where large enterprises are early adopters for new startups.

But solving a niche product for an enterprise may not be what the mass market needs. This leads to an adoption chasm for startups to the rest of the market. I’m going to start with you, Andy, on this. How does a startup avoid a success milestone of landing a huge enterprise customer, hampering their further growth?

And by the way, I want to know if you believe that this actually happens.

[Andy Ellis] Yeah, I want to challenge the premise here, which is there’s a lot of technologies that first came into large enterprises and then worked down market. If you think about cell phones as an example. There used to be a briefcase you carried around with a phone hanging out the side of it. This was not being adopted by small consumers first.

And so I think what you really have to look at is what is the ICP, the ideal customer profile, for a given solution coming to market? And what’s really important is that a startup understands who they’re trying to sell to because what gets in their way is not, “Did I get a big enterprise first?” It’s, “Did my first customer not actually match my ICP, and am I now designing for the wrong market?” Like if your ICP is the Fortune 500, that’s great.

Then you need to make sure that you don’t get a design partner who’s a little mom and pop shop because they’re not going to help you iterate and build a great product for your Fortune 500. Conversely, you might get one niche in the Fortune 500 that thinks you’re great. They want to buy in early. And they’re going to keep you busy with customized feature requests that only they need, but it turns out you’re a mass market play.

That you need to go to the 9,000 companies in the US that are only a thousand employees or greater, not just that Fortune 500, but you’re busy building the wrong features. So, that’s really what you have to look out for is that mismatch. I don’t think that it is one before the other necessarily.

[David Spark] All right, Justin, I’m going to throw this to you. We’ve brought this subject up before. And the common response we hear from CISOs about working with startups is they will be very responsive to our needs, and they solve some niche problem, and that’s why we want to work with them. Because if they’re working with a much larger level vendor, they’re just not going to get that kind of service, sort of the tripping over themselves to make them happy, because they know landing this customer would be a big coup.

What say you to that? And I know you have actually passionate feelings about this topic?

[Justin Somaini] Yeah, I would say yes, but. And so when you look at a vendor startup if you will and its ability to land a large customer, there could very well be and the majority are depreciating returns on that. One, the sales cycle is so long that that startup of four or five people, it takes forever in order to land them.

And so what that means is they’re missing other opportunities that are probably more relevant in that ICP that Andy spoke about. Two, the requirements of the demands that they are… Let’s take a large bank and a financial service.

The requirements of injecting into that customer is going to be so significant that it’s probably going to take engineering time away from those feature and functionality that they could serve in a larger base. And as a result of that, they are not going to hit their revenue targets and goals to get to that next funding round.

And so, yes, a large company can be massively helpful. But in a lot of cases, it can be detrimental to their performance. The trick is do you have a customer, large or small, that is very aware of the development process that it’s really more of a partnership at these early, early stages than it is a vendor.

Yes, they will be receptive to your ideas, but they’ve got to build a company and a wide base. And so it’s a give and take.

[David Spark] Let me ask you about… You mentioned detrimental because this was the premise of Ross’ article about…saying that there’s a chasm. Do you agree that there’s a chasm in that…? And this goes back to what you said, Andy, of they’re so tailoring for this enterpriser that they’re missing the mark that they were originally planning.

Do you agree that chasm does come along?

[Justin Somaini] Absolutely. I see it a lot of times when you have a huge company, huge logo, and of course it’s positive in so many ways. However, you’re not going to win your entire business on one customer. You need to build a base. And so therefore you need to make sure that that focus for your engineering team is building that project base for the large masses.

At the same time, hopefully servicing the large companies on what they need. But sometimes, in a lot of cases, your startup is going to have to wait maybe a couple of years down the line to be able to hit those larger companies with regulatory and larger needs.

[David Spark] Okay, quickly, Andy or Justin, have either of you…because you’ve worked with a lot of startups…ever seen a startup say no to an enterprise customer? Like, “We simply can’t do it. We’re not going to be able to deliver. It pains us to say it, but we’re saying no to you.” You’ve seen it? Yes or no?

[Justin Somaini] Absolutely, yes.

[Andy Ellis] Yeah. I’ve seen it in multiple places in the sales pipeline. We’ve seen people do the qualification step and just say, “We’re not even going to pursue this. You showed up. You wanted to talk to us, and we already know we can’t deal with your sales process.” All the way to firing a customer because you’re upside down on the customer.

What many startups don’t get is they don’t realize the volume that a large enterprise will bring to you. And they sign a deal, and they’re so excited. They’re like, “This is a million dollar a year deal. This is amazing.” And it costs you three million dollars a year to service them, and so at some point you have to fire them.

[Justin Somaini] I mean there is no better large company than DISA. However, the hurdles to get DISA as a customer are pretty significant, and so that’s probably not your first customer but maybe down the line. But not right now.

Let’s talk community.

11:52.526

[David Spark] We hear a lot about people wanting to “give back” to the cyber security community, but what does that really mean? So, that question came up on the cyber security subreddit recently, and the most common answer was mentorship – helping people learn and getting a foot in the door. Others pointed out things that help share knowledge but also may raise your profile like blogging or being active on LinkedIn.

So, I’m curious, Justin, where do you see the most value for yourself in giving back to the community? And here’s the key thing is what do you think we can do to incentivize more of it? More giving back in whatever way.

[Justin Somaini] Yeah, I’m a big believer, especially at this later stage of life, in the giving back kind of model where I think that comes into a couple of different components. One, how do we help the next generation of cyber security practitioners that are just now graduating college, supporting them, helping them with, “This is what a career track could look like in a whole bunch of different variants in cyber security.” I remember myself back in those days, there was a lot of things I didn’t know.

Maybe even still to this day. So, the ability to kind of help educate early on so that they could be better than the generation that is now. The second one is I’m a big believer in you don’t have to look very far in order to mentor people. You can look in your own team in your own organization. And it doesn’t have to be in cyber.

It can be in a lot of different things.

I mean I’ve been in cyber, and now Andy and I are in VC. And so how do we bring our totality of experience and probably a lot of scar tissue along the way to help those that are just learning, and growing, and coming into their own with this themselves. And so that is probably the two biggest things.

The last one is I’m a big believer in unrepresented individuals in cyber. In STEM as a whole but cyber, of course, as a subcomponent of that. And how do we participate more thoughtfully in those organizations to create a healthier playing field for all of us?

[David Spark] All right, Andy, I throw this to you. Giving back. You create an endless amount of content, so you are just in terms of blogging, writing, being on the show, your book, what not. Is there something additional you do? And, again, I want to go to the incentivizing thing. Where is the area to incentivize people to do more of it?

[Andy Ellis] I worry about the incentivization, honestly. I see a lot of people who blog to raise their profile, and they’re not actually thinking about, “Is this useful content? Am I actually helping people?” Because they get stuck on this wheel of, “Oh, I have to keep churning out content.” I write a weekly leadership newsletter, except some weeks I don’t write one because I don’t actually have a thing to say, and I don’t want to waste people’s time and energy.

And so we need to think about how do we be thoughtful about what we do. But I want to dig in on one of the things that Justin just said because this is the biggest way you can benefit the rest of the community is in your own team. I’ve built a security team, and I’ve followed those people as they move out into different places in their career, and I’m watching the impact that they’re having.

And if you want to have impact, start with the people you know. How can you help improve them? How can they see that you actually walk the walk. That you’re not just, “Oh, I’m writing some content to go out and get engagement,” but, I’m actually developing the people around me so that you have built a culture of development so that people want to pay it forward.

Let’s not worry about the incentivization of paying it back. Let’s worry about how do we build the culture where you say, “I benefited from somebody else helping me, and so I’m going to go help other people. Not because I get kudos, not because I’m going to get engagement but because it’s the right thing to do.” I sit on a number of nonprofit boards that almost nobody outside of those boards even know that I do because I’m giving back through those communities because I want to help those missions, those causes, and that’s what I think we need more incentive for people to do.

Sponsor – Vanta

16:08.416

[David Spark] Before I go on any further, I do want to mention Vanta, who has been absolutely a spectacular sponsor of the CISO Series. Great supporter of everything that we do here. So, whether you’re starting or scaling your security program, Vanta helps you automate compliance across frameworks like SOC 2, ISO 27001, and more.

With Vanta, you can streamline security reviews by automating questionnaires and demonstrating your security posture with a customer facing trust center. I know you’ve been reading about these. They’re hot. They’re popular. Vanta has got one. So, over 7,000 global companies like Atlassian, Flo Health, and Quora use Vanta to manage risk and prove security.

So, our listeners of the CISO Series get $1,000 off of Vanta at vanta.com/ciso. So, go to their website, vanta.com/ciso. Check them out and get your discount.

It’s time to play, “What’s Worse?”

17:11.112

[David Spark] All right, it’s time to play, “What’s Worse?”

[Andy Ellis] Ooh.

[David Spark] All right, here we go. We discussed this at the beginning. No pressure to you, Justin, to disagree, but I would like it if you would disagree with whatever Andy says.

[Andy Ellis] But Justin talks to me more frequently than he talks to you, David.

[Laughter]

[Andy Ellis] Although Justin and I actually healthily…we often disagree reflexively just to try to find the synergistic answer.

[David Spark] Really? So, it’s conceivable that Justin will agree with you here now. And then a week later, he’ll go, “You know, Andy, when we were doing the podcast recording, I have a second thought about it. I now disagree with you.” Could that actually happen?

[Andy Ellis] No, no, it’s more likely to be the other way around. He’ll disagree with me here and later he’ll be like, “No. No, I think I agree with you.” But often what happens… The challenge we have here is that there is no healthy compromise. We get two scenarios, and we’re not allowed to modify them to come up with what’s reasonable in the middle.

We have to pick from these two.

[David Spark] Yeah, and a lot of times people say, “Oh, well, I’ll pick that because then I will fix it by doing this.” I’m like, “No, there’s no fixing it. It is what it is.” [Laughs]

[Justin Somaini] That’s fine. That’s ad hoc debate class 101. It’s going to be great. 

[Andy Ellis] Yeah.

[David Spark] All right, so this comes from… And every now and then we pick one from BadThingsDaily on… It hurts me to say this. X. Not Twitter but X.

[Andy Ellis] We can call it Twitter still if we want.

[David Spark] [Laughs]

[Andy Ellis] X is pronounced Twitter.

[David Spark] There you go. All right, so here are the two scenarios. A researcher poking for Log4j exposures in your product has found callbacks coming from systems you don’t recognize and can’t identify. Not good. Or the administrator account to your organization’s cloud email is no longer accessible.

The recovery email belongs to a former employee on a custom domain running on a personal email server. Which of these two is worse?

[Justin Somaini] The admin one. The admin one…

[Crosstalk 00:19:16]

[David Spark] Hold on. I make Andy answer first.

[Andy Ellis] I was supposed to answer first, but Justin already agrees with me, so I’m golden.

[Laughter]

[Justin Somaini] No, no, no, you disagree with me. You’re just coming into realization about it.

[David Spark] [Laughs]

[Andy Ellis] With Log4j, we deal with this all the time. The admin one, oh, this is brutal.

[Justin Somaini] No, I think the admin one is easier.

[Andy Ellis] Oh, you think it’s easier?

[David Spark] Hold on. Wait. Hold it. Hold it. Wait. So, again, the game is, “What’s Worse?” Which scenario is worse? So, you think the worse scenario is the Log4j scenario, Justin?

[Justin Somaini] Correct.

[David Spark] All right. All right, so we have disagreement. Excellent. All right. Hold on. You’re going to flesh these out. I’m going to start with you, Andy. Andy?

[Andy Ellis] So, these are differently bad, which is one of the things that makes, “What’s Worse,” scenarios fantastic. Like what is worse for you to have to deal with as the CISO is going to be the admin situation because you have to do really strange and arcane things to recover it. Because I’m going to assume your cloud provider is not going to be helpful, so you have to go hunt down this former employee, do something, deal with it.

At the end of the day, it’s quiet. Nobody will ever know this happened. And so you’ll have stories over drinks forever. The flip scenario, like, “Oh, wait, you actually have a compromised Log4j system,” well, this is Solar Winds all over again, functionally, which means I’ve actually just talked myself into agreeing with Justin.

But Justin hasn’t technically answered yet, so he can change his mind.

[David Spark] Wait a second. No. So, you’re switching your answer mid…

[Andy Ellis] Because I hadn’t… Normally I talk myself through it. And so here’s the deal. You’re in the Solar Winds situation of your product, which assume you’ve hit product market fit, so is widely deployed in a lot of enterprises, turns out to now be the avenue that everybody is getting compromised.

If a researcher found it, that meant they have access to your product which means it’s in enterprises, it’s not just in your own service. So, you’re going to have a lot of bad days. And who knows? You might be prosecuted by the SEC on various spurious grounds going forward. So, I think I kind of have to go with that one as being worse.

[David Spark] Okay, so you’re saying the SEC. And I’m going to summarize, it’s going to be longer and more painful than the first scenario. I’m getting that from…

[Andy Ellis] Yes.

[David Spark] Okay. All right. Justin, now that he agrees with you, because you are obviously the wiser person on the microphone right now… [Laughs]

[Andy Ellis] He’s got more gray hair than I do. Of course he’s wiser.

[Justin Somaini] Dare I say, he originally saw the error of his ways and then converted into greatness. And so I appreciate that about Andy. For a different tact, I believe in the goodness of human nature. And so whoever this employee is, he or she will generally do the right thing when reached out to.

Our legal team will have a conversation with them about the…and impress upon them about the needs of making sure that this account is handed back over to the rightful and just owners.

[David Spark] You’re going to have the legal team make the first contact? That’s my question.

[Andy Ellis] God, not the legal team making the first call.

[David Spark] No. No, you don’t make the first call the legal team.

[Justin Somaini] It’s a legal issue. It’s not a security issue.

[David Spark] Yeah, but as someone who worked with this person, you should make the first call.

[Andy Ellis] No, it’s an HR issue. Like I’m going to have somebody give a call and say, “Hey, by the way, we forgot to do this in offboarding. Can we just fix this now?” And if they say no then legal is like five minutes later giving him a phone call.

[David Spark] Yeah, yeah. Legal is not the first call.

[Justin Somaini] I don’t think HR should be the first call.

[Andy Ellis] No, no, it’s an HR issue, not that HR would make the call. I would have an executive.

[David Spark] I think someone who worked with this person should be the first call.

[Andy Ellis] Yeah. I mean just to be clear, I’ve been that employee that I have left holding social media credentials for a company I worked with, and sometimes you just wait. You’re like, “Are you ever going to realize that you don’t have the credentials anymore?”

[Justin Somaini] Andy, are you inferring that our legal industry is full of individuals that don’t have empathy, or kindness, or a healthy relationship with others?

[Andy Ellis] I am saying that our legal industry is full of people whose fiduciary obligation is to their employer and not to humanity.

[David Spark] Let me ask you, have either of you either done this for a person or been the person? And my question is was there kind of a smoothing process? Like, “Hey, we haven’t talked in a while. Let me take you out to lunch.” And then over lunch…

[Andy Ellis] No. Very straight and direct. “We screwed up. When you left, we forgot to reclaim this. Somebody didn’t notice. Can you help us recover it?” I’ve been on both sides of this one. Never a problem. I literally had this… In fact, honestly with the social media account where literally I reached out to the person.

They texted me the password. I dealt with it right there, and we were fine.

[Justin Somaini] Honesty and transparency is generally the best practice in these cases, yeah.

[Andy Ellis] Yeah. He agreed with me. I agreed with Justin. Justin had to jump in fast to make sure I agreed.

[David Spark] All right.

[Andy Ellis] But the dynamic you just hard is actually a very common one between Justin and I, which is we dive in, and we might start, but we’re exploring. And we’re going to end up mostly in the same place. It helps really nicely with ideation.

[Justin Somaini] So, basically we’re right for different reasons.

[Andy Ellis] Yes.

[David Spark] Well, I’m going to chalk this up as someone disagreeing with you.

[Andy Ellis] No, no, no. This is somebody agreed with me.

[David Spark] No. No. No. It’s you coming around because someone else was wiser than you.

[Andy Ellis] Someone else cheated and gave out his answer before I could talk myself through it.

[David Spark] [Laughs]

[Justin Somaini] I love the spin. I love it.

Why has this topic suddenly become the center of attention?

24:50.194

[David Spark] When it comes to offensive security, are we putting the cart before the horse? We talk about wanting to build resiliency, but it often feels like we bring in things like red teaming too late in the process to have them be affective. Granted that in a greenfield deployment you’d still need a baseline of tooling, as outlined by Danny Magallane on LinkedIn.

He names EDR, IDS, SIEM, and IAM as sort of table stakes. But if we aim to do offensive security as early as possible… Again, this is sort of… I know that a lot of people have issue with that term but essentially it’s a form of pen testing, if you will.

[Andy Ellis] Then call it pen testing. It’s not offensive security. Sorry. I used to be a part of an organization that did offensive security professionally. What people call offensive security today is not offensive security. I know we recently talked about stupid hills to die on. This feels like one.

[David Spark] I think offensive security is just pen testers without deodorant. What do you think?

[Andy Ellis] Okay, that’s pretty offensive, but we can take it.

[David Spark] There you go. Yeah.

[Andy Ellis] If you want to call yourself offensive security, that’s what we’re going to assume you are.

[David Spark] All right. All right. But here’s my question. I’m going to start with you actually, Justin, on this one. If we were to do let’s call it pen testing, offensive security, whatever the heck term you want to use. Red teaming, whatever. If you were to do it extremely early in the build out of your security program, what could that tell you?

What would be the value of doing it early? And would you recommend that?

[Justin Somaini] Well, only in one situation is where you need political support. Because pen testing/red teaming, which I started in security doing exactly that many, many, many years ago, is a mile deep and not a mile wide. And so you’re not going to get a holistic view of your issues, etc., but you will find a bunch of good things that could be the wow factor to really be able to present in front of a CEO, a head of engineering, a head of infrastructure, whatever it is to make them realize the possibility of bad.

And so if you need that, absolutely start with it. Get that wow factor to kind of loosen the chain, if you will, of support. But if not, I think there’s probably bigger bang for the buck if you’re starting to build out a security program.

[David Spark] Very, very good argument. All right, now I throw this one to you, Andy. What do you think about doing pen testing, because you prefer that term, extremely early on in the security build out process?

[Andy Ellis] Yeah, I think it’s mostly a waste of time, except in the circumstance Justin just talked about. But if you’re going to do that, you probably already know the pen testers are going to find. 80% of it you should already know, and you should already have said, “Hey, these are things you should go fix.” Just even from lightweight analysis.

Like, “Oh, I looked at your architectural diagram. There’s a brute force attack available against passwords right here. I don’t need to test it.” That’s the first thing my pen testers are going to go find. And so when they ignore you, you now bring in the pen testers, and they’re going to basically create this list that is 80% overlap with what you already told them to go do.

And you argument above that to executives is, “Why are we not listening to our in-house folks? Why do we have to pay money to bring in somebody from outside to embarrass us when we already knew better?”

[David Spark] But that happens a lot with all consulting. [Laughs]

[Andy Ellis] Oh… As somebody who does consulting on the side, I will tell you, this is… I get paid to be the outside voice telling you what your insider already knew. But I want it to not be about trivial things, and often we use pen testers to tell us about trivial things. The most important thing you should do if you engage in a pen tester is when you constrain them, which you will do… Everybody does.

They say…

[David Spark] Yeah, of course.

[Andy Ellis] “…you can’t attack this. You can’t attack that.” Whatever it’s going to be. “Oh, you just got credentials. Well, you’re not allowed to do this other thing next.” That list of constraints is something you should publish when you bring your pen test report to whomever it’s going for. You just say, “And by the way, here is how badly we had to hobble the pen testers to make this report only this small.”

[David Spark] Let me throw out this argument… And I don’t know if this is valid or not, but I’m throwing this out because I’m just arguing for the side of, “Yeah, this will be a good idea.” Is there anything that a pen tester could tell you early that would open your eyes to any architectural design that I’m like, “Oh, if we do this early on then…” If you didn’t deal with it early on, you see it later then becomes sort of painful and costly.

Or that doesn’t exist?

[Andy Ellis] Almost never because early on there’s going to be so many problems that an architectural flaw isn’t going to show up necessarily as this fundamental issue. Like that’s when you got a pent tester who’s like, “Oh, I’ve got to go work for it. I may be able to look at the system diagrams.” If your pen tester can work without system diagrams they’re unlikely to find serious architectural defects.

[Justin Somaini] I think you’ve got a different problem if you’re including it as part of the SDLC, or the release train, or whatever kind of process of deployment that you’re talking about…development and release. You don’t have enough people to keep up with the releases that a company has, and so how are you going to have a heavily labor oriented process maintain the release train cycle, speed, and velocity that a company has now a days?

I mean we’re talking about release trains that are happening 15 times a day for SaaS services, and so it just doesn’t scale. And so you need to be thoughtful about where you put red teaming. I think it can add value, and I agree with Andy’s point. I think it can add value, but it’s a scalpel of a tool, not a general purpose one that we generally need for most things.

Is AI going to help us or hurt us?

30:51.272

[David Spark] I don’t know what phase of the gen AI hype cycle we’re in currently. I’ve actually lost track. But one thing this new tech wave is making very obvious is that companies can struggle to balance embracing innovation with privacy considerations. Christopher Burgess at CSO Online outlined this as a compromise from embracing the gen AI tech, like having AI chat bots in Zoom meetings versus privacy concerns that could either skirt regulations or just rub employees the wrong way.

So, I’ll start with you, Andy, on this. How do we create a framework for this compromise? Does it exist? Where we’re not missing out on technical innovation without giving up on privacy? And that seems like kind of the big one right here, but you can point out any others if you’d like.

[Andy Ellis] What’s fascinating here is the tech that’s problematic was not gen AI. It was recording the Zoom meeting. What gen AI did was made that useful. We’ve always had the ability to record Zoom meetings, at leas for a very long time, but what did people do? You had this recoding that was mostly useless, and occasionally you’d like, “Oh, we recorded our training so that Justin who didn’t show up can listen to the training later.” Which Justin was like, “Yeah, you think I’m going to listen to the training?

No, I’m not really going to do so.” But now we have a chat bot that will listen to the training for Justin and give him the summary that says, “Yeah, Andy pontificated for 60 minutes. He said one useful thing. Here’s what it is. Now go on with the rest of your day.” The problem here is that the AI made it useful to record, not that AI recorded.

I think we should separate these concepts to say if we’re going to record a thing, that’s where the privacy consideration came in. Consideration did not come in because we let an AI chat bot interact with it.

[David Spark] I’m going to argue with you a little on this. You are technically right. And if we were in front of a judge, yes. But the usefulness now makes it a privacy concern in that now my privacy is far more easily exposed than it was before.

[Andy Ellis] Right, so we’ve changed the incentives, and that’s going to be what’s important. But I think the challenge is nobody sits and talks about privacy from a first principle and says, “How should we think about privacy in our business? Should we record our meetings? Should we take notes in our meetings?

What should we do?” Because if you had ever had that conversation then people wouldn’t just turn on a chatbot willy nilly, because they’d say, “Oh, I know that in a one on one, we do no record one on ones, period, ever because that’s a corporate rule.” And so the idea that a chatbot is going to record what I say to Justin, and then that’s available to somebody else would never happen.

Because that’s my issue is not the chatbot making it available to the people in the meeting, it’s now it’s a persistent recording available to everybody else. Like the transcripts for this show. We know it’s going to be transcribed. We’re intentionally recording this. But I don’t want the transcripts of every call I’ve ever had published.

[David Spark] Good point. All right, Andy makes a good point. Do you have another point to make, or do you want to double down on what Andy said?

[Justin Somaini] Well, I think it’s a little bit more nuanced, because the question is is AI going to help or hurt us. There is a lot of different modules or algorithms in AI. I think we’re hyper inflating now a days that gen AI is AI, and that’s just not really true. I think… And I also agree with Andy that it is a privacy issue.

However, with AI and that analysis, it’s not the transcribing I see as the issue. It’s how can I attach the information of that transcription, for example, into a larger data set to make inferences or insightfulness that I otherwise would not be able to do. So, for example, when I was at Yahoo, we had AI ethics on search around the information you could identify about a persona based upon their browsing habits and everything else.

And I think there is a notorious article about somebody saying, “Hey, I was informed that I was pregnant by a chat or something.”

[Andy Ellis] It’s an apocryphal story. I know where it comes from.

[Justin Somaini] Yeah. And so whether that’s real or not, the situation, I think, is really the point, and that is really the concern that I think we should be thinking about, which is how can we make inferences or actual factual inferences that might be a little bit creepy and that we might not want to do.

So, it’s the ethics around it. The second point that I would make is I think that the helping or hurting us around gen AI… Right now we’re in kind of a hype cycle of the use cases of gen AI, and so I don’t think it’s going to have as much of a panacea of intelligence to everything that people might otherwise think, but there are some really good use cases for it.

But I think that it’s going to be tempered with the reality of the next couple of years. But having said that, it is exciting for an algorithm, but I think we should take a step back and look at a lot of the other algorithms that are massively valuable as well.

[David Spark] Well, excellent, both of you, gentleman. That topic alone could be an entire episode, or it could be a whole podcast series on its own, but we’re going to leave that discussion right there.

Closing

36:23.095

[David Spark] I want to thank you, Justin, for coming on, and I’m going to let you have the very last word here. But first, I do want to thank our sponsor, Vanta. Vanta.com. Get $1,000 off on your Vanta service for all your compliance needs. Vanta.com/ciso. Go check them out. Andy, as always, thank you so much.

Justin, any last words? Any interesting things you’re working on that you want to tell us? I’m assuming from both of you, YL Ventures…your YL Ventures companies are hiring, yes? The portfolio companies. They should just go to your site if someone is looking for a job?

[Justin Somaini] I’m sure they are hiring, and there’s key roles, but I think they’re more focused on how they do outreach to potential individuals that might actually see their solutions adding value into their companies. And to that point, maybe a little bit of a segue, to any security leader out there that is really interested in our portfolio companies or just the innovation that we see in the market as a whole, my ask is reach out to me.

Send me a LinkedIn request. Send me an email. By all means, we’d love to have that conversation with you.

[David Spark] Excellent. Well, thank you very much, Justin. Thank you very much, Andy. And thank you to our audience. We greatly appreciate your contributions. More, “What’s Worse,” scenarios please. Send them in. Really tough ones for Andy. By the way, when you send them in, they don’t have to be the flip side of the same problem.

You can have one scenario and then one completely different scenario like what we sort of threw out today. Send those in. We greatly appreciate it, and we’ll give you full credit. You don’t have to. Sometimes people like to be anonymous. That happens every now and then because they just don’t want to be associated with our program I think is…

[Crosstalk 00:38:10]

[Andy Ellis] I don’t know why somebody wouldn’t want to be associated with our program.

[David Spark] Honestly, I think some people feel that they’re going to get repercussions at their job.

[Andy Ellis] Right. Because some companies do actually very much care about how their name ever gets used in public.

[David Spark] I know. We have seen this problem before. Thank you, everybody, for your contributions and for listening to the CISO Series Podcast.

[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, ciso-dev.davidspark.dcgws.com. Please join us on Fridays for our live shows, Super Cyber Friday, our Virtual Meetup, and Cyber Security Headlines Week In Review. This show thrives on your input.

Go to the participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at david@ciso-dev.davidspark.dcgws.com. Thank you for listening to the CISO Series Podcast.