As security professionals, we know a lot of the things we lack visibility into that can cause security issues. That alone is enough to keep your team occupied. But what about the things you don’t even know about in the first place?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap), CISO, LinkedIn. Joining us is our sponsored guest, Himaja Motheram, Censys.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, Censys
Full Transcript
[David Spark] Security professionals fear not what they don’t know but rather the unknown unknowns, those items they can’t see because they’re not even looking. And will that thing they don’t even know to look at ever cause a security issue? How can one create a security program around unknown problems?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. And joining me for this very episode, I know you’re a big fan, his name is Geoff Belknap. He’s the CISO of LinkedIn. Everyone who listens is a big fan of yours, Geoff.
[Geoff Belknap] I don’t believe you. But, hey, I appreciate any of you and all of you, even if you don’t appreciate me.
[David Spark] I have spoken to every single listener, and they all tell me they’re big fans of Geoff Belknap. I will say this – if you are not a big fan of Geoff Belknap, just let me know. Or better, keep it to yourself.
[Geoff Belknap] I was going to say, better yet, don’t tell me. It’s fine. It’s fine. I don’t need to know.
[David Spark] I don’t need anything negative.
[Geoff Belknap] It’s good.
[David Spark] Hey, our sponsor for today’s episode who has been a phenomenal sponsor of the CISO Series, I’m so thrilled to have them onboard, it is Censys. They are the leading internet intelligent platform for threat hunting and exposure management. And they’re responsible for bringing our guest today, for which speaking of being a big fan… It appears our guest has lots of fans back at the office, as I have learned because I met a bunch of them when we did a live show in Nashville.
All right. But I’ll get to that in a second. I first want to talk about the topic at hand.
[Geoff Belknap] We’re going to keep it a mystery, okay.
[David Spark] Yes. Not that long. It’s in seconds we’re going to meet this guest. I know you know who it is. It’s our audience who doesn’t know yet. Although if they read the blog post or the description on the podcast episode, they would actually find it out. So, it’s not that much of a mystery.
[Geoff Belknap] This is such a buildup. I cannot wait for this guest. I don’t know who it’s going to be, but it’s going to be amazing.
[David Spark] Geoff, don’t lie to the audience.
[Geoff Belknap] [Laughs]
[David Spark] You do know who it is.
[Geoff Belknap] It’s true. I still think it’s going to be amazing.
[David Spark] All right, Geoff, let’s get to the topic at hand. It will be amazing, by the way. On LinkedIn, you asked the community, “Do you have a strategy for uncovering unknown unknowns? And if so, which do you prioritize? Then how do you communicate those priorities upwards in your organization?” And this is the part I was also interested in knowing, and you asked as well, “And can security leaders build this into their program when they’re still struggling to manage their known unknowns?” We got a lot of feedback on this, which I thought was kind of fascinating.
[Geoff Belknap] We did.
[David Spark] What was your take?
[Geoff Belknap] [Laughs] My take is, boy, I have enough problems I know about. Do I really want to know about the problems I don’t know about?
[David Spark] Could you even build a program for that?
[Geoff Belknap] Absolutely. And I think the answer is personally I don’t want to know. And professionally I kind of have to know. So, I think it was a really cool question to ask everybody because I learned some things here, and I learned a lot of people have different approaches to this. And certainly it just recognizes that this is an important part of this process.
There’s a lot of stuff for us to stress about. But unfortunately we always need to be looking for what’s that thing we don’t know we’re supposed to be worrying about currently, and do we have to worry about it? What do we do next? And I think our guest will be a great person to help us think about that.
[David Spark] By the way, I think it’s probably the most classic answer to the class question of what keeps you up at night. It’s what is it I’m not worrying about that I should be worrying about. The unknown unknowns. And yes, we have the perfect guest for this, who is going to help us make sense of this discussion.
It’s not that we’re going to all of a sudden clear up all the unknown unknowns, but how we deal with something that’s always going to be an existing problem. Our sponsored guest, who comes from Censys, who is the security researcher over there, it is none other than Himaja Motheram. Himaja, thank you so much for joining us.
[Himaja Motheram] Thank you for having me. I’m so excited to be here.
Why is this relevant?
4:15.197
[David Spark] Matt Holland over at seedata.io said, “The minute you anticipate a problem or detect its existence you’ve moved it out of unknown/unknown quadrant, therefore doing anything in our domain could be considered a strategy. It’s more of a conceptual space that represents, ‘We know we don’t know everything, so expect a little chaos from time to time.’ It’s why we talk about probability, not certainty.” And Edwin Covert of Bowhead Specialty said, “Dealing with the unknown unknowns comes down to resilience.
All you can do is prepare as best as you can to survive in the face of a determined adversary.” So, I fear that that may be the whole discussion, the last thing that Edwin said, was it’s really just a discussion about resiliency. Is that what this is, Geoff?
[Geoff Belknap] That’s definitely how I think about it. I have been quoted before saying I don’t think perfect prevention is doable. But I think what’s even better than perfect prevention of everything that could go wrong is being able to respond to that quickly and efficiently. I think resiliency is kind of the word you would use for that.
If I don’t know there’s a certain vulnerability out there, and certainly there are lots of vulns out there that everyone doesn’t know about yet. But I know there are exploits. There are problems you can have. And I have a robust way to respond to those. I have a plan. I have a strategy. That is what’s going to give me the edge over the long-term where you’re not playing a game where you know all the adversaries.
So, I think here, preparation is key, and I think everybody had lots of good ideas. Like Matt Holland here, in terms of how you think about the unknown unknowns and how you prepare for that. And this is what it’s all about. It’s what are you going to do about it once you know.
[David Spark] It is kind of two sides of the coin you have to deal with, Himaja. And both Matt and Edwin bring up a good point, is that the moment you sense it, it’s now not unknown. And Edwin says no matter what the situation, you have to have a resiliency program. So, we’re attacking from two angles, yes, Himaja?
[Himaja Motheram] Absolutely. But I kind of have a counter to that, which is that I think one of the reasons why the fact of the presence of unknown unknowns causes so many people to lose sleep at night I think is this idea that there is no strategy, there is no perfect strategy for them. But what resonates with me about these two quotes is that an organization’s security strategy for responding to known unknowns is also your strategy for unknown unknowns.
Because typically after you discover something like this, the first question or one of the first questions you ask is where are my affected assets.
That key preparation and knowing how your own network is mapped out, knowing that data is something you can prepare in advance… But the reality is that a lot of orgs don’t have the tools to understand that question. But it is definitely something that you can proactively respond to.
[Geoff Belknap] I think that’s a great point, that there are so many things that are unknown that there are important things that just don’t have to be unknown. What assets you have, where they are, what the current state of them are, that’s a really important step in this process.
[Himaja Motheram] Absolutely.
[David Spark] You said just moments ago, Himaja, that the program you’ve got for unknown unknowns is really on the back of known unknowns. Does it, one, sort of start revealing the other? Yes?
[Himaja Motheram] Yes. And something that I thought touched on this in other responses is that a lot of the things that we think are unknown unknowns are actually just unknown to us, to individuals. Usually there is somebody in the org who has been noticing something weird, an unrecognized IP, some weird activity.
And so usually unknown unknowns are known by somebody or known unknowns can pivot to unknown unknowns. But having that basic security hygiene in place for responding to the known unknowns… How many times can I say that phrase? And the low hanging fruit kind of to improve your security posture really goes a long way to minimizing the impact of unknown unknowns.
What’s going on?
8:43.806
[David Spark] Jonathan Waldrop of Expel said, “This is where building a culture of security and not making security a roadblock is really important. If the security team is the firefighter then the rest of your company needs to be Smokey the Bear and work to prevent fires.” That is a really good take on the sort of uncovering the unknown unknowns.
And Jovica Ilic of WIM Security said, “I frame it as a subset of security culture. I usually call it the neighborhood watch. The basic strategy is to get everyone in the organization to report things they don’t understand, don’t work as expected. Make sure the security team has the capacity and competence to follow up on all of the reports.” So, this is a really interesting angle about going after the unknown unknowns, and that is use all the eyes and ears you have in the organization to help you.
Yes, Himaja?
[Himaja Motheram] 100%. And this touches on what I mentioned earlier about having kind of a listening ear towards all parts of your organization, because you’d be surprised at how many things that are categorized as unknown are actually things that someone has noticed. A story that we see play out quite a bit when looking at internet infrastructure is these old unpatched things on the internet – things that a patch was released for years ago that are still sitting out unpatched and outdated.
And to me, that is a clear as day sign of an incomplete security culture. That low hanging fruit would be prioritized in a healthy security environment. And I think that really highlights the importance of things like getting executive buy in to have resources and also making sure that every organization realizes that the IT organization is not a subset.
Everybody is a part of that. Because layer eight is the user. And I think everybody, regardless of industry, needs to have a little bit of fluency in security because it always ties in to these business critical operations.
[David Spark] Geoff, have you ever had success with sort of the general populous behaving in the if you see something, say something? Have they seen something that your team has never been able to see?
[Geoff Belknap] All the time. I think the title for this section really should just be relationships matter. There’s a bunch of reasons, like Jonathan says here, why security shouldn’t be the blocker. Security should not be viewed in any organization as the team that stops all forward progress. Because what you need is a positive working relationship with as many stakeholders in your organization as possible because you want them to reach out.
You want you to be the first place they think of when they see something weird.
No amount of putting up posters that say, “See something, say something,” or whatever slogan you can steal from TSA is going to replace the fact that when your partners see you as a productive, generative partner to complete their business, to make them successful, they’re going to reach out to you.
And I think Himaja put it perfectly. You have to build that layer eight security. I think for me, layer eight is politics, religion, and sports. It’s all the human aspects of building relationships. You have to invest in those because you want people to come flag stuff to you when stuff is weird because that’s how you’re going to learn about brand new things.
Or I think to Himaja’s point, brand new old things that somebody forgot about, and they built some marketing website, and they just forgot to take it down and never patched it. Those are the things that you need help in your organization identifying and mitigating.
Sponsor – Censys
12:31.178
[David Spark] Before I go on any further, I do want to tell you about the incredibly cool things that Censys is doing. Now, protecting your company from a cyber attack is a pretty monumental task. If there was ever a line that was preached to our audience, that is it right there. So, not only do you have to stay a step ahead of the threat actors who, let’s face it, are getting increasingly good at what they do, you have to secure a technology landscape that’s becoming more vast, complex, and fragmented.
So, again, it’s tough all the way around. So, think about all of your company’s internet connected tech.
We’re talking about assets living in the cloud, your software and web properties, remote devices, not to mention all of the shadow IT you don’t even know about. As your digital footprint grows, it becomes more challenging to identify, monitor, and defend all that you own. Just one unknown or undermanaged asset can be an attacker’s point of entry to your network.
That’s why continuous visibility, that’s key, into your entire attack surface and larger threat landscape is critical.
To prevent an attack, you need visibility that’s informed by a comprehensive, highly contextualized set of internet intelligence for both proactive and reactive security analysis at scale. We’ve been talking about this. So, you need visibility into all of the exposures an attacker could exploit. And this, everything I just described, is exactly the kind of visibility Censys provides.
With the Censys Internet Intelligence Platform, your security team can access the most comprehensive, accurate, and up to date internet data available so that you can take down threats in as close to real time as possible with no deployment or configuration required. Governments, enterprises, and researchers around the world use Censys to defend their attack surfaces and hunt for threats, including the US government and over half of the fortune 500.
Wow. So, you can learn more about Censys on their website. And it’s not spelled like the US census, so listen. It’s censys.com. Go there and check it out.
What aspects haven’t been considered?
14:57.038
[David Spark] Rocky DeStefano of RiskOne said, “This is where you want human creativity to be highly encouraged and have some dedicated time to think strategically. Most successful current unknown unknowns strategies are about adding thoughtful perspective.” Aldo Febro, PhD over at Continuant, said, “Former risk council representing various stakeholders.
Ask them whether they have risks that they’d like to discuss. They might surface topics you never thought of.” And Andrew Hendela of Karambit.AI said, “Once cyber criminals are caught doing a new tactic, technique, procedure, TTP, then it becomes a known unknown that needs to be dealt with. The challenge is to discover the unknown unknowns first and try to improve before they can be used against you.” So, this is a really interesting take here, Geoff, in that the success here comes from just creative thinking about what are we not thinking about, which I’m guessing is how you uncover unknown unknowns.
[Geoff Belknap] It’s important to be a part of your strategy, an intentional part of your strategy. We often talk about in security leadership that you need to pick investments and places where you’re going to invest your time, or your resources, or your people. And uncovering sort of unknown unknowns is a place that you need to invest time, and that’s not just sitting around, googling around.
But it’s like… Well, like several people suggested here – put a panel together, query your stakeholders, have regular time to sort of pipeline and ingest new information to figure out if you need to reprioritize something, if you need to re-rank your risks, if you need to create a new risk, and then build a new strategy around it.
Just make sure it’s an intentional approach to the problem.
[David Spark] What we realize in going back to the original question is does anyone actually have an unknown unknown strategy because maybe they’re still struggling too much for known unknowns. And what Rocky [Phonetic 00:17:13] is saying here is even though you’ve got that known unknown strategy and you’re still struggling with it, you got to set aside time to look at what you’re not thinking about.
Himaja?
[Himaja Motheram] I completely echo that thinking. I think what’s unique in the distinction between known unknowns and unknown unknowns here is that known unknowns are an increasingly solvable problem using technology. External attack surface inventory and exposure management tools are evolving. Intrusion detection systems are evolving.
However, I think unknown unknowns are a space where humans are still outperforming machines and anomaly detection fancy algorithms by far because of this creativity and this thinking like an attacker that is required, honestly, to really think about what sorts of situations to prepare for.
[David Spark] Geoff, let me ask you, do you have a formalized, “Hey, it’s time to start thinking creatively about attacks?” or does it just sort of happen sort of naturally over time?
[Geoff Belknap] At Microsoft, enterprise wide we spend time every year sort of looking at what do we think the top risks and threats are from a security perspective, and we have a fairly formal process where we go through and asses those, and talk about if there are new risks that we need to put on there, if we need to sort of rebalance things.
And the idea is that that sort of helps everybody who’s in a security leadership position across the entire enterprise make decisions about what they’re going to prioritize, what they’re going to invest in, what do they need to hurry up and do more of if they don’t have coverage of a certain risk. And I think that’s really valuable.
But this is a very, very mature, large organization. You don’t have to have that formal of process, but I do think it makes sense to at least a couple times a year sit down and ask yourselves, “What are we not thinking of? What’s new that we haven’t spent any time on?”
[David Spark] Have you actually reflected and gone back like, “Last year, this is what we said, and we were right/wrong.” When you do this year’s meeting, looking back at least year, do you do that?
[Geoff Belknap] Absolutely. And I think that’s probably a less formal process, although it’s part of the risk review. It’s interesting. The same thing happens. Everybody puts out these top ten predictions for the new year post. I love to go back and go, “Which ones were right?” The bottom line, I think, and weird how it is for me to talk about this… An AI and an LLM is not going to solve this problem for you.
This is just a placeholder for you to remember that to Himaja’s point, you’re already struggling with the things you know about patching or Shadow IT. But you also have to keep thinking about the things you aren’t focused on yet. And even though the game is very hard to play, and it’s very hard to make progress, the adversary is advancing with or without you.
They’re not working on your timeline. So, you have to constantly be bringing in new data to your thinking.
[David Spark] You know what we’ve discovered on these top ten lists of like what to expect next year? We realized, you know what the top ten list is? Take last year and just add more. It’s just more of that.
[Geoff Belknap] Just this year, just add AI to it, and you’ll probably be dead on. But also 90% inaccurate.
What else are we missing?
20:32.397
[David Spark] Benjamin Purgason of LinkedIn said, “Having a firm understanding of what we’re trying to prevent or protect allows us to conduct a pre-mortem…” I thought that’s interesting. “During the pre-mortem we challenge ourselves to identify all the ways our proposed security control could fail.
This gives us a list of concerns to address as part of our design.” That’s a very interesting take and a more sort of structured way of thinking, I guess. And Jordan Wigley, Field CISO over at SimSpace, said, “The ability to simulate your ‘worst day in the office’ before it actually happens, helps to uncover some gaps in playbooks/procedures and even tech/security stack in advance.
There’s no good way to account for every unknown, obviously, but being prepared mentally and procedurally to remain calm and follow standard operating procedures is when it counts most.” So, this is interesting. This is two different things. One is building out your security program, and then at the same time, making yourself resilient, kind of the where we started here, Himaja.
I kind of really love this sort of pre-mortem concept that both Jordan and Benjamin are talking about here because it gives greater structure to our specific problems. There could be unknown unknowns that could affect others, but what are our specific issues. Is that a good plan, Himaja?
[Himaja Motheram] Absolutely. Any sort of proactive thinking on that part is definitely applicable in the majority of situations. Where I think… And this has been on my mind recently. Especially with supply chain attacks in recent years, I’m thinking most recently of the MOVEit file transfer attack that affected lots of organizations, through their adjacent organizations or partnerships that weren’t really in their internal tech stack but that they were impacted through the supply chain, I think that’s where this sort of pre-mortem strategy does have a gap.
And this is one of those unknown unknowns that I think I’m still kind of ruminating on how to really adequately respond to. I think to Jordan’s quote, the real baseline…the only way to respond to these is to be prepared because there are always things that you cannot pre-mortem.
[David Spark] Good point. Geoff, I love this kind of plan of let’s just look at what could be our worst day. I’m assuming the classic line of, “Think like the bad guy…” You must do that, yes?
[Geoff Belknap] Absolutely. And I think a lot of people do this, and some people call it tabletopping. Sometimes you’re threat modeling. I think to Ben’s point here, a lot of times a big part of our pre-mortem is just laying down to paper what you were considering when you built this so that as you reapproach it in the future or as you’re sort of thinking about what you want to invest in, you understand that you didn’t consider this new risk that you didn’t know about at the time, and you might need to go back to that thing and make some changes if it’s an important control in your environment or it’s an important part of observability or your detection strategy you have to think about that.
I think it’s important to create a historical record of what you considered and didn’t consider so that you’re thoughtful about not resting on your laurels but making sure that the infrastructure you’ve put in place to solve problems is not going to suddenly let you down because your adversary, or your risk, or your that model has shifted.
[David Spark] One great quote we got from Hadas Cassorla, who has been the show many times, is that one of the things that is missing from a lot of tabletop exercises is random activities. Because nothing goes as linear as you think ever.
[Geoff Belknap] Just nothing ever happens the way you expect it to. I think the important thing that a lot of executives that I do tabletops with miss is the adversary is not…does not feel bound by your policies. The adversary does not feel bound by your timeline. The adversary is not going to constrain themselves to what you’re good or bad at.
They’re just going to… They’re there for their reasons, and they don’t have to tell you or participate in your response. And you have to inject that a little bit when you practice or rehearse these things.
[David Spark] Himaja, I’ll let you have the last thought to conclude this.
[Himaja Motheram] I think this is one of those things that is so much easier to say than do. And unfortunately will probably only get harder, because I’m noticing that increasingly from the perspective of these software vendors, the onus is more and more on the user or on the end product to really have those extra thoughts about security.
Like things are less and less secure out of the box just from my perspective. And so I think, again, simulating your worst possible day, zero trust architectures will really go a long way in this space.
Closing
25:43.194
[David Spark] Excellent. Well, that concludes the regular portion of our programming. Now we’d like to come to the part where I ask both of you which quote was your favorite, and why. And, Himaja, I will start with you. Whose quote did you like the best, and why?
[Himaja Motheram] I think that actually the first quote that we discussed in this episode by Matt Holland about why we talk about probability not certainty in this space and expecting a little chaos from time to time, that really resonated with me. Because, again, I think there’s this misconception that a secure organization means being prepared for absolutely everything, having a strategy for absolutely everything.
And while we can always strive towards that ideal, that’s just… It’s probably not possible. But that kind of preparation mindset and expecting that chaos can go a long way towards being as prepared as you can.
[David Spark] Geoff, your favorite quote, and why.
[Geoff Belknap] Boy, there’s a lot of really good ones in here, but I’m going to go with Dr. Aldo Febro from Continuant who talks about forming a risk council and getting various stakeholders together to talk about these things. And we at LinkedIn…we have a formal part of our culture that recognizes relationships matter.
I think this extends to a lot of things. But this topic specifically, it’s really important to build those relationships, even if you don’t value them or you’re not good at that. you have to sort of build a process so that you can bring people together and bring new information, and share some thoughts about what’s going on, what’s different, what’s new in your industry, are there new attacks, this is a new regulation, is something going on these the security team has not been aware of.
That is super valuable as you build out your strategy and execute your program.
[David Spark] Excellent. Well, that comes to the very end of our show. And I want to thank Himaja for joining us today. It was Himaja Motheram. She is the security researcher over at Censys. And Censys is our sponsor, the leading internet intelligence platform for threat hunting and exposure. Go to their website.
It’s censys.com. Now, Himaja, I’m going to let you have the very last word here. If you’re hiring or you want to make an offer for our audience, or how they should check out more of Censys. And if people want to reach out to you… By the way, Himaja has the coolest Twitter handle or X handle as well call it, @himajedi.
Pretty cool. I was quite impressed by that. Smar thinking there. Himaja, any last words?
[Himaja Motheram] Yes. The short of it is that you can’t defend what you don’t know about. And if you’d like to get a little bit more sleep at night, Censys does have the best data out there to go to understand what is out on the internet and get information about your external facing blind spots. So, you can go visit our website at censys.com to schedule a demo and see what we can do for your organization.
[David Spark] Sounds excellent. Thank you, again. Thank you to Censys as well, and thank you to our audience. We greatly appreciate your contributions. And if you see a really great conversation online, predominantly LinkedIn is what we like the best, but we’ll accept it from anywhere, if you think it’s a really good conversation, you can say, “Hey, this would make a great episode of Defense in Depth.” Let us know, because we can make that happen.
Thank you for contributing and listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site, ciso-dev.davidspark.dcgws.com, where you’ll also see plenty of ways to participate including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at david@ciso-dev.davidspark.dcgws.com. Thank you for listening to Defense in Depth.