It’s one thing to protect your data within your four walls. But when data leaks increasingly come from third-parties, what can you do to protect your organization?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap), CISO, LinkedIn. Joining me is our sponsored guest, Mackenzie Jackson, developer advocate, GitGuardian.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, GitGuardian
Full Transcript
[David Spark] It’s one thing to protect your data within your four walls, but when data leaks increasingly come from third parties, what can you do to protect your organization?
[Voiceover] You’re listening to Defense In Depth.
[David Spark] Welcome to Defense In Depth. My name is David Spark. I’m the producer of the CISO Series. It’s your favorite of the series that are called CISO. I believe that’s true with you, Geoff. Yes?
[Geoff Belknap] This particular episode is my favorite of all the ones we’ve ever done today.
[David Spark] That is Geoff Belknap. He’s a CISO over at LinkedIn and also the co-host here of Defense In Depth, the show you are listening to right now. We have a website called CISOseries.com that has many of our other programs as well. If you’re interested, check them out.
Our sponsor for today’s episode is GitGuardian. Keep secrets out of your source code. We’ll be talking about that very subject with somebody from GitGuardian. But first, Geoff, let’s discuss our topic at hand. Data protection within your organization is not easy. We talk about this on many of our programs, and the principles are understood.
However, the same can’t be said for protecting the data you give to third parties or you get from them. When this data leaks, especially organizational secrets, it generally isn’t a one-off event. Rather, it gets repackaged and sold, meaning you’re now dealing with a data leak lifecycle.
According to a recent report by our sponsor, one in ten developers are leaking secrets. So it seems it’s inevitable, whether we’re doing it or not. We all have leaked secrets. So my question is, how are we addressing this problem? How do we begin to think about addressing this problem, Geoff?
[Geoff Belknap] I think the first step is to admit that you have this problem, whether you know it or not. If you have developers, you have this problem. The next step is to really think about how you’re going to solve this. A lot of people solve this simply by building it into their development pipeline.
[David Spark] The person who’s going to help us with that very topic is our sponsor guest today from GitGuardian. He’s been on the show before. He’s phenomenal. He is their developer advocate, so this guy essentially speaks the truth to developers and deals with developers. He’s your developer’s best friend.
It’s none other than Mackenzie Jackson. Mackenzie, thank you so much for joining us.
[Mackenzie Jackson] Thank you. That was one of the best intros I’ve ever had. I’m going to bring you around next time I meet my wife’s parents or something, so you can introduce me exactly like that.
This isn’t just a security issue
3:03.562
[David Spark] Dutch Schwartz from AWS said, “This is a systemic issue with how software is developed, provisioned, audited, and managed. We need to make it really easy for software development teams to do the right thing and very hard to do the wrong thing. That’s it, in a nutshell, because the opposite is true. That’s on the security community, not on developers.
The historical evidence suggests that tools alone will not solve this. This is a human-centric issue and therefore a cultural issue.“
Ian Poynter of Kalahari Security said, “When the software development culture in your organization becomes security-aware and has been provided with the tools to achieve it, security will become second nature for all software teams.”
This is an interesting theory. Could this truly just be solved by culture, Geoff?
[Geoff Belknap] Theoretically, a lot of things in security could be solved by everyone just always doing the right thing and never making a mistake. So I think that is very…
[David Spark] Simple as that. Wrap it up.
[Geoff Belknap] Just never do anything wrong. And frankly, I can go home. It’ll be great. A culture has a lot to do with this. I agree largely with what Dutch said here, although I disagree strongly that it’s on the security community, not on developers.
The thing that is on developers, and I think developers have skin in the game here, is we have to build a culture where if you’re a developer of any size or scale, whether it be LinkedIn-sized, or you’re just two people in a startup trying to build a product, if you develop this habit where you feel like, “If I’m handling a credential, a secret, a key, an API key, whatever it might be, you have a strong security culture,” if your immediate thought is, “Ick, I shouldn’t be touching this.
I shouldn’t be copying this. And I certainly shouldn’t be putting it into my code, hard-coded.” If that is the feeling and the emotion that you feel, great. You have built a strong security culture in your developer community. If your thought is, “I just got to get this done. I’m just going to paste this in.
It’s just for a test, not for prod,” you are the reason I’m spending tons of money on FTE and software and tooling to fight against people like you. Now, I don’t think individuals should feel shame, but I do think we have to look, on the security side, we’ve got to get a lot better about knowing this problem exists and addressing it and on the culture side for developer teams and especially, developer productivity teams.
We really got to be thinking in an aligned way that handling credentials in code is just never acceptable.
[David Spark] Mackenzie, this ball is in your court. It would sound nice if we could fix this through culture, wouldn’t it?
[Mackenzie Jackson] Of course. And I don’t want to underplay the importance of secure culture that it has. However, history tells us that culture alone doesn’t solve this. What’s interesting about this problem set and lots of problem sets in security is that from a technological point of view, this is a solved problem.
There’s no reason why we have to have hard coded secrets. There’s no reason why they need to leak. There is a chain of tools that we can access from secrets managers to store them with fine-grain access, to distribute them securely, to even not have them at all and have dynamic and just in time and all of this.
I may like, not even just from a culture point of view, from a technological point of view, all of this is solved. However, there’s still a huge and massive problem. So therefore, of course, this isn’t going to be enough. So it’s interesting to think about that problem, but it also creates that disconnect, I think, with a lot of people where people that understand the technology, people that understand that there’s no need for this are just scratching their head going, “Why on earth is this a problem that shouldn’t exist?” but not actually facing why this actually does happen, that it is going to continue to happen.
And we actually need to build up layers to understand the problem, look into it, be able to fix it, and solve it when it inevitably does happen. Have a leak because it’s going to happen. If everyone’s going to happen. So it is interesting. I love what Dutch said about it needs to be hard to do the insecure thing and easy to do a secure thing because this is one of my biggest gripes.
That I have out there is that a lot of tools want a developer to be able to use the system as fast as possible. And they have techniques to do this. For example, there might be downloadable configuration files. Let’s say you’re a cloud infrastructure or something. And they give you a downloadable config file that you can plug right into your project, any language.
And it just works because these keys are hardcoded. And then the reason they do this is so that you can very quickly use their tool so that you’re not even going to get a chance to look at a competitor because theirs is just working, but that is incredibly insecure.
What are the elements that make a great solution?
8:10.373
[David Spark] Erik Bloch of Atlassian said, “I found that far more compromises via credentials come from third parties than in-house, if you have a reasonable SSO MFA platform set up, but yet third parties access your stats or other applications without requiring them to use MFA. It is a prescription to get owned.
In this case, it’s a gap in corporate enterprise security controls, which normally comes as a function of maturity. So for some instances, this is simply a gap that hasn’t been addressed yet. In others, it’s a wholesale failure of controls, static creds in code, for example. So I doubt there is a one size fits all.
Mark Fireman of Entro Security said, “Organizations usually use multiple vaults, sometimes more than five different vaults. If it’s a large organization, this results in secrets sprawl. This is something Mackenzie knows plenty of. Where organizations have more secrets than they can handle. They are completely unaware of how many secrets are active, how many are exposed, and what the risk level of each secret is.
One of the main reasons secrets are exposed is the lack of proper management and protection. Why is that? Because in many cases, R&D teams are the ones responsible for creating and storing secrets, but they are not the ones who are responsible for securing them. There is zero oversight.” I think Mark here is speaking your language, Mackenzie.
[Mackenzie Jackson] There are a lot of things in there that I can resonate with. We look at what makes a good solution for all this, there’s multiple solutions. It’s not one solution here and you have to have secrets. A lot of people are talking about secrets managers in this course, and it’s really important.
And it’s important to use the correct manager that your team’s going to use, but you also need secrets detection. And then more than that, like when you’re looking for this, you’re going to have secret leaks. You’re going to have secret sprawl. How do you deal with that? So there needs to be a remediation component where you can follow the path down on some of the cool stuff that we’ve just integrated with CyberArk, which is a secrets manager to be able to do secrets detection, to know when a secret from your vault has leaked out in public to be able to do that securely and having that follow through all the way down to remediation.
These are the things that good tools need to have because secret sprawl is such a big problem. And if you’re in a five-person startup, maybe you haven’t experienced what it’s like to find a secret and have no idea what it gives access to.
But it’s a scary problem to have when it does happen, and it will happen. Are you going to break production by rotating a secret that’s leaked out? These are problems that we all face regularly.
[David Spark] That actually, I didn’t even think about that, but that’s a really good point, Geoff. That must give you a heart attack when something is out and you’re like, what is this giving access to? And now it’s public. Have you run into this?
[Geoff Belknap] I haven’t run into that specific problem. In our case, we have very regimented secrets and credentials, so we generally know what they’re all for. But I think there is an example of if you find an SSH key, you don’t know what that has access to. It’s just a private key. And that’s fairly common, I think, not for me, in general, just to be clear.
Everything, listen, everything I do for you is wonderful and perfect, and it has never been mistaken.
[David Spark] No, by the way, I am not referring to anything you’re doing at your current employer. I’m just talking about in general, in your history of working in cyber.
[Geoff Belknap] Of course, all good. I think the thing I really think about here, specific to the kind of things Mackenzie was saying, like the elements that are really important as part of just solving this problem in general is understanding the entire lifecycle of secrets. So somebody needs to generate those.
Somebody needs to protect those. Somebody needs to use those. And at each stage of that lifecycle, you really need some automation and you need some really strong developer-centric UI, whether that be like an API they can call, or whether that be a UI they can click around, it needs to be dead simple to be able to generate a secret to be able to generate a little bit of code snippet or whatever they need so that they can incorporate that secret into their code without hardcoding it.
And then there needs to be automation so that it can be automatically rotated if it gets exposed and you can understand who owns that secret, who requested it, what was it for. And then when it gets found, you need to have some way, like I said, to automatically rotate it. So when the UI is strong for that, your developers will use it when the UI or the automation you build or buy makes it easy to request secrets and integrate them into your code without having to know how to protect them or how to implement them in a secure way.
You are building that paved path that Dutch talked about in the previous section. And you are making your program and your organization trillion times more secure than it was before you had a solution like that.
Sponsor – GitGuardian
13:21.501
Let me ask you a question. How confident are you that your company credentials haven’t leaked? I think we stated at the beginning of the show, chances are they have, whether you did it or not.
Malicious actors are using leaked secrets for lateral movement, and GitGuardian’s mission is to help security teams stop them. GitGuardian’s platform facilitates a secure software development lifecycle for dev, sec, and ops teams. Some people like to call it DevSecOps.
In the event of a hardcoded credential, the GitGuardian platform promptly alerts security teams and helps fix the issue by directly involving the developer. Improving your mean time to remediate and keep your secrets secret. For companies with multi-cloud and multi VCS environments, GitGuardian is key to building software securely and adds a layer of protection by securing your infrastructure as code files.
Powerful features like honey tokens prioritize incident remediation, focusing your efforts where they’re most needed. So go to GitGuardian.com andstart scanning all your repositories, Jira, Slack, all during your 14-day trial.
You’ll discover where the hardcoded credentials are in your organization, or you can use the Community Edition for your open source and personal projects completely free of charge. Remember, that’s GitGuardian.com.
Who can solve this?
15:11.023
[David Spark] Mauricio Ortiz of Merck said, “We definitely need an industry shift to prevent secret leaks. Current software development practices are increasing the secret sprawl instead of minimizing it due to sloppiness or complexities.” The tools to scan for secrets stored or left in code can help to minimize it. The key is to improve better coding practices and processes to force it. Regular secret rotation or expiration would leave secrets useless even if found in code repositories.” Interesting points there.
Randall Hettinger of Permiso Security said, “Keep your key safe. Compromised credentials will continue to be a driving factor behind attacks in the near future. In addition to using a secrets vault, companies should prioritize developing more visibility into how their secrets are being used, who has provisioned them, who has access to them, and where they are being shared.”
It just seems like everyone’s saying, “Let’s just practice better behavior and put our secrets in a nice safe.”
That would be a good idea, Geoff.
[Geoff Belknap] I think that’s a great idea and very useful, but in this particular case, if you keep something like cash in the safe and it walks away, it hurts and that’s bad, but it’s not immediately damaging to your entire business and your entire family. In this case, you keep a credential in a safe and it walks away.
You need to know what’s going on with that credential. That thing can do a lot more damage than just the fact that you lost it. And the chances are you’re not going to know you lost it until it’s too late unless you make sure that you’re doing kind of the other thing that we talked about in a previous segment, which is doing some scanning.
If you are not currently scanning your code commits or your code repository, and by the way, not just where your code goes, but everywhere else your developers talk to other developers like Slack, like email, like wiki pages, like JIRA tickets, for example, you are almost certainly leaking secrets and not knowing about it, and you won’t find out about it until an adversary abuses that secret, or potentially an auditor flags the issue.
And that’s not what we’re going for here.
[David Spark] Mackenzie, why is locking things in a vault such a troublesome thing? It just seems it’s a simple security concept that everyone should get. Yes?
[Mackenzie Jackson] Yeah, but you need to get things out of the vault. And this is where the problem comes is that a lot of secrets you have a distribution problem. And where secrets come out, if you make something hard to get, developers are very thrifty. They can have the secrets.txt file. So they don’t need to talk to anyone, they can hardcode something and never let anyone else see it, and even delete it, but it remains out there in the ether.
These secrets are super slippery and they end up everywhere. Geoff says something amazing that I want to touch on a little bit back. And he said that developers have to have a great experience for them to be able to use tools and use them properly. And it’s so true. And I find that with a lot of security products, the people buying them are security engineers.
And sometimes the people using them are developers, but vendors will make a product for the people buying it because they understand it. It doesn’t mean that the people that are using it are going to have a good experience from it. So having a good developer experience and not only being able to store these secrets and retrieve them securely, so they don’t have to hardcode them, but also making sure that the tools that are designed to scan for these secrets, so that when they get alerts, are designed with developers in mind as well as security teams in mind with experiences so that developers can actually be part of the problem.
We all talk about shift left, right? But a lot of people think that’s shove left. It’s not shove left; it doesn’t work. Shift left is about empowering developers.
To take charge of security and you do that by giving them tools that fit into their workflow that speak to them and that enable them to move left rather than just saying, “Use this tool,
To take charge of security and you do that by giving them tools that fit into their workflow that speak to them and that enable them to move left rather than just saying, “Use this tool, use this thing now.” And that’s why I loved what Geoff said. And this is who can solve this as a big problem.
It’s everyone can. Developers, security teams, everyone.
What are we going to do now?
19:36.818
[David Spark] Amit Arora of AWS said, “Instead of thinking about how can we protect secrets, the time has come to just assume that a secret is stolen.” That’s a bold statement. Any activities happening in your environment, a backdoor opened, a new role created, things normally an attacker can do, can be monitored.
To guess backwards, if a secret is stolen, extrapolating it globally signals around stolen secrets, then can go to a centralized bucket from where enterprises can subscribe and then continuously monitor every activity instead of protecting the key, protect the kingdom.”
So for Amit, I think both are actually happening.
You’re still trying to protect the kingdom. Yeah, I mean, nobody’s security program is only to protect secrets and then just leave everything else open. Yes, Geoff?
[Geoff Belknap] Yeah, that kind of misses the point here. we talk about protecting the secret and detecting the secret and giving people a great experience because this is part of defense in depth. We want to protect the secret. We want to know if it gets leaked, but this is not the only control that we have.
We do have anti-abuse, typically you want to have anti-abuse technology. You want to figure out if somebody’s cramming the secret in. you want to have some detection that understands if the secret’s being used from a place that isn’t usually used. You also want to look for data exfiltration.
Like all of the standard stuff applies, but I think the really important message here is people need to hear you are not doing everything you can. If you are not building some automation and a good experience around how to get and rotate secrets, how to store and protect those secrets and how to detect if they eventually, and they will, leave the confines of those protections.
It’s the same as DLP used to be a conversation about this. You want to look and see if any other information is leaving the organization, but this is just adding one more layer to the depth that your security program already has.
[David Spark] I throw this to you, Mackenzie. To throw out the secret protection and only protect the kingdom, let’s just for a philosophical exercise, you let’s say we did this. You’re just creating, this is not a shift left situation. This is a shift right situation. Let’s make the problem way more difficult to the right.
Isn’t it?
[Mackenzie Jackson] Yeah. And there is actually an example of where this happens in the real world. And one of that is Firebase keys. So Firebase keys have had all kinds of trouble protecting the keys. And they said, okay, now we’re not going to consider these keys sensitive. And instead, we’re going to use a whole bunch of security rules to make sure that only specific services can access and Firebase is messaging.
So that specific services, trusted services can message the system. And that works well if everything, and everything is configured perfectly, right? And only then, and you don’t have any misconfigurations and there’s no ambiguity in anything. So that in itself, in my opinion, it’s not really insecure from an architectural point of view, from Firebase’s point of view, because if you do it correctly, it is secure.
However, you’re removing layers of defenses. The idea here is to protect the kingdom, but don’t protect the key. I say protect the kingdom and protect the key. And this is all about what zero trust is about. Okay. Just because you have an API key doesn’t mean I’m going to give you all the data, right?
And just because you’re coming from the right service, but you don’t have an API key, it doesn’t mean I’m going to give you the data. You have to have all of those things. So that, and that’s really good. And there are a couple of things, just, if you want to know, okay, what can you be doing in the situation, rotating secrets regularly, if possible, dynamic secrets.
That’s an amazing one. Make sure you whitelist services, make sure that everyone’s trained on the vaults that you actually using. Sounds crazy, but trust me, a lot of people don’t even know how to manage secrets properly. So all of these little things, there’s lots of practical things, but it’s defense in depth.
It’s a layered approach. It’s zero trust. It’s, I don’t disagree with protecting the kingdom. I just disagree with don’t protect the key.
[David Spark] You know what it’s, essentially saying let’s eliminate a layer. I’m like, yeah, you could do that. You just put a lot more work on the other layers is what happens. And true. Everything you said is right. It’s just, let’s try to keep all the layers intact. So not to overtax any one layer over the other.
Close
24:24.765
[David Spark] All right. That brings us to the end of our discussion here. And Mackenzie, this is where I’m going to throw it to you. We always like to ask which quote was your favorite and why? So I’ll let you choose first. Which quote was your favorite and why?
[Mackenzie Jackson] I really liked the quote from Maurizio Ortiz. We definitely need an industry shift to prevent secret leakage. Current software development practices are increasing the secret sprawl instead of minimizing it due to sloppiness or complexities. And whilst, and the kind of the general feel of the thing is that the systems are getting more complex and I like what he’s saying that, and we do need to shift to more prevention, more detection, and understand it and use tools to be able to find secrets, prevent secrets, and store our secrets.
[David Spark] Geoff, your favorite quote and why?
[Geoff Belknap] Everything I said was my favorite.
[David Spark] As is mine, Geoff.
[Geoff Belknap] But if I had to pick one person who wasn’t me, you were our guest. I think I would pick Randall Hedinger from Permiso security. Keep your keys safe. Compromised credentials will continue to be a driving factor behind attacks in the near future. In addition to using a secrets vault, companies should prioritize developing more visibility into how their secrets are being used, who has provisioned them, who has access to them, and where they are being shared.
I think this is a great point, and I would go one step further here, and I think we talked about this already, do some scanning, find out where they’re being stored because just because you generate it automatically, and just because you have a key vault, does not mean that for convenience sake, again, just for convenience sake, somebody didn’t paste that into a wiki or a SharePoint or an email or Slack channel to be helpful.
It is again, as Mackenzie talked about, developers are trying to do stuff quickly, efficiently, and effectively, and sometimes mistakes can be made. Building a fantastic program that assumes occasionally people will make mistakes will make you a happier security leader.
[David Spark] That brings us to the very end of the show. I want to thank our guest, Mackenzie Jackson, who is a developer advocate over at GitGuardian and why not? I think GitGuardian, our awesome sponsor, keeps secrets out of your source code. If you just want to scan your open-source and personal projects, you can use their free community edition, but check out their website, GitGuardian.com.
Super easy. We also have a link on our website. You could click on that as well. Thank you, Geoff, as always. And let me throw this to you, Mackenzie. Is there anything you would like our audience to know? Any special offer you would like? I know you, by the way, love communicating with developers. If you are a developer listening or you run a security team that works with developers and wants a great ally like Mackenzie, please reach out.
Mackenzie, anything to add?
[Mackenzie Jackson] Now, I just want to add in there that if you are a developer, there are some amazing tools that we have. An open-source tool called GG shield, which will help you keep your secrets safe. You can install things like git hooks. You can create things like honey tokens, fake credentials to trap attackers.
And also, we have a new service out there called has my secret leaked.
You can find out if your secrets have ever leaked in a very secure way. I don’t have time to go into it, but I’ll leave it there.
[David Spark] Is that like the, have I been pwned?
[Mackenzie Jackson] Yeah. And is it, have I been pawned inside and we use the same technology to make sure that, we never release, that we don’t ever see the secrets that you’re leaving. But so yeah, very similar to very similar. And we, we, worked with Troy to, build a similar kind of function.
[David Spark] Ah, there you go. By the way, I don’t want any crap from the community for me pronouncing it pawned or pwned. I don’t…
[Mackenzie Jackson] I, yeah.
[Geoff Belknap] We can mispronounce it correctly each time.
[David Spark] Are you, I’ll go back and forth each time. Then they’ll never be able to catch me,
[Mackenzie Jackson] Geoff, seeing as you, you are one. How, do you pronounce CISO though? We’ll see. So it’ll all right. How do you pronounce the title? Cause this is another…
[Geoff Belknap] I prefer people just call me sir,
[Mackenzie Jackson] Easy. Yeah. I can pronounce that all day,
[Geoff Belknap] or just never call me at all. That would be my second preference. That’s great.
[Mackenzie Jackson] sir, or nothing. Okay.
[Geoff Belknap] But I say CISO.
[Mackenzie Jackson] See you soon.
[David Spark] There you go. I just said it for you. Hey, everybody. Thank you so much. I greatly appreciate it. Your contribution to and listening to Defense In Depth.
[Voiceover] We’ve reached the end of Defense In Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn, or on our site, CISOseries.com, where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at david@CISOseries.com. Thank you for listening to Defense In Depth.