00:00:00
00:00:01
Container security with Wiz
937 Plays9 days ago
In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin talk to Ofir Cohen, CTO of Container Security at Wiz. The discussion focuses on the challenges with the cloud native security ecosystem, how organizations can improve their security posture, how developers can do less with more, and how Wiz helps organizations avoid security incidents.
Check out our website at https://kubernetesbytes.com/
Cloud Native News:
- https://blog.kubecost.com/blog/2.4-release-highlights/
- https://www.theregister.com/2024/09/26/critical_nvidia_bug_container_escape
- https://techcrunch.com/2024/09/18/edera-is-building-a-better-kubernetes-and-ai-security-solution-from-the-ground-up/
- https://www.cncf.io/blog/2024/10/01/karmada-v1-11-version-released-new-cross-cluster-rolling-upgrade-capability-for-workload/
- https://github.com/atlassian/escalator/?tab=readme-ov-file
- https://cybenari.com/2024/08/whats-the-worst-place-to-leave-your-secrets/
- https://thehackernews.com/2024/10/new-cryptojacking-attack-targets-docker.html
Show links:
- https://www.wiz.io/customers
- https://www.wiz.io/blog
- https://peach.wiz.io
- https://www.linkedin.com/in/cohen-ofir/
Timestamps:
- 00:01:43 Cloud Native News
- 00:12:56 Interview with Ofir
- 00:56:58 Key takeaways
Transcript
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts. We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:30
Speaker
Good morning, good afternoon, and good evening wherever you are. We're coming to you from Boston, Massachusetts. Today is October 4th, 2024. Hope everyone is doing well and staying safe. Let's dive into it, Bob. And how's it going?
00:00:44
Speaker
I'm doing good. Yeah. It's Friday, I guess. Just has been a busy week for me. So yeah, I'm glad it's Friday. Let's just leave it out there. I'm always glad it's Friday.
00:00:57
Speaker
came what you It's already October, which is kind of wild to me, but um and I'm all for the season. I don't know about anyone else, but fall is awesome. If I get out of this weather all the time, like slightly chilly in the morning and then it kind of gets nice in the afternoon. Hell yeah. Yeah.
00:01:14
Speaker
I know. Agreed. I'm just saying. 50 to fifty to ah high 60s. I think that's the perfect range. It goes below 50 like, no, come on. Yeah. Fun stuff, fun stuff. Well, um we do have a really fun episode um coming up. We're going to dive into the news, but we do have Ofer Cohen from Wizz.
00:01:35
Speaker
introduce him in just a few minutes. But why don't we dive into our news section, Bob, and you want to kick us off? Yeah, sure. So I had a couple of news things to to talk about. ah Cube cost, right? I know last time we covered that they're getting acquired by IBM. ah They do have a new release out. yeah So if you are a kubecost customer or if you are looking at them ah version 2.4 is out it has a few things one is oracle cloud support so in addition to aws azure and google cloud they can now support oracle cloud as well i'm sure they'll add ibm next but then ah they tell that before the 2.4 release all they did was ah they had some integrations for discovering gpu so
00:02:19
Speaker
If you request it for X number of GPU resources, they'll show you that these have been requested, but it didn't do any real-time monitoring and it didn't have those capabilities. With this release, they fixed that so they can actually show you real-time GPU utilization, um if if there is any waste, if you can maybe scale down, if you are using fractional GPUs, and it shows you the efficiency level. so If ah you you are running at 10 percent efficiency levels, maybe you can consolidate that workload on a different cluster. so CubeCourse 2.4 will start allow you to do all of those things. In addition to them, ah they also added the ability to create budgets. ah So you can create a budget per team, per user. It's pretty flexible. ah but And then when you are getting close or you are over budget, you can get alerts by emails, webhooks, Slack or Teams. I have a feeling there's going to be a lot of email and Slack messages based on those budgets.
00:03:15
Speaker
you know ah But I think the the whole point of ah helping organizations identify the costs or the money that they're spending, I think having these budgets and having those annoying notifications will definitely like, I don't know, put it front and center rather than at the end of something that people look at at the end of the month. or end Sure, yeah. Awareness is always good. Yeah. No, but I agree with you. like A lot of Slack messages are going to get triggered by this.
00:03:40
Speaker
So that's update from Cube cost. I think another update was ah talking about GPUs, right? Nvidia GPU operator and the container toolkit that they have is widely used. Like if you're using Nvidia GPUs anywhere with Kubernetes, that's the toolkit and operator that you're using.
00:03:55
Speaker
They had a critical vulnerability of released last week are on, I think, the 26th of September. I know we are a couple of weeks late or a week late in talking about it, but ah that's how that's when our episode is coming out. ah There is a CVE called CVE-2024-0132. It can lead to remote code execution, denial of service, escalation of privileges, information disclosure, and data tampering. I think that's the bingo. like All five things. like That's pretty but bad. one Yeah. So NVIDIA already has a fix for it. So if you, this vulnerability exists in versions up to 1.6.1 and the GPU operator 24.6.1. So if you just update it to container toolkit 1.16.2 and operator 24.6.2, you should be good that there is a fix and patch available. And we can thank, I think this was the first one that brought out this CVE or disclosed to CVE. So I'm glad we are having offer on. Very fitting. Yeah.
00:04:51
Speaker
And then finally, the last piece I had was um a new startup in our community's ecosystem, Edira. ah e d era I'm not sure if I'm saying it right, but They came out of stealth, at least for me, and they they raised they disclosed that they raised a $5 million seed round to build a new hypervisor for Kubernetes containers and AI workloads. So covering all the different keywords. But that their whole approach is trying to ah improve performance for your containers by not either having ah and sorry not sacrificing performance or not having to use ah infrastructure that has hardware virtualization enabled. So what they are trying to do is
00:05:31
Speaker
ah bring out a new hypervisor. So I think oh one of the co-founders started rewriting Zen in Rust to help solve these things for the company that she was working on. And again, all of this was open source. And then I think her and other co-founders, again, all female co-founder teams. So that's awesome. um They decided that, okay, there is more value than that one use case. So now they have a new company focused on building that new hypervisor. So I'm looking forward to it.
00:05:58
Speaker
ah keeping an eye out on on the company. ah In the list of angel investors, a familiar name, Joe Beta, popped out. so I'm sure there there is some ah value ah in it if Joe Beta is giving you some angel funding. so um yeah We'll have a link in the show notes if you want to check out the company. Yeah, I'm curious. you know They mentioned other tools like Kata Containers and stuff like that, you know arguing that bolt yeah they're kind of like a bolt-on way.
00:06:22
Speaker
Which you know i think introducing a whole new hypervisor is always a feeding of itself but yeah um yeah how do you work with like this will be helpful for on prem customers maybe but again.
00:06:36
Speaker
Organizations have a lot of ah inertia in adopting something new. as Anyways, they're not adopting Kubernetes. Changing the hypervisor itself ah might be a ah bigger uplift, but and we'll see how the seed a a series A, series B, and the product market fit looks like. Again, this is just seed downside. I just wanted to share. It because ah it is a Kubernetes project in and a lot of ways too, right? I have to read more about it. but um ah like yeah So Zen in Rust, I think, i I forgot the name of the co-founder that started the whole project, but she she started it as an open source thing and then they're like, okay, let's just build up a company around it. So one of the things.
00:07:13
Speaker
Cool stuff. Cool stuff. Well, uh, you know, I liked that you tapped into the Nvidia container toolkit bug and whiz being sort of the one there. I did have of an article on here that was about, um, crypto jacking, uh, Docker targets. So like the Nvidia bug is, uh, basically, uh, a user is escaping the container yeah and and doing some real damage, but this article,
00:07:39
Speaker
Uh, on the hacker news, I just wanted to make note of it is really about, um, how there's sort of a ah new campaign targeting exposed Docker engine APIs, which basically will like run a bunch of scripts and spread throughout your containers and find other API endpoints and those kinds of things. It just goes to show that, you know,
00:08:01
Speaker
misconfigurations, something we'll probably bring up with oppher is ah is still something that, you know, um is constantly being um kind of like tapped into by bad actors. And yeah this article, um as well as another one that I read was titled the worst place to leave your secrets. And this was an interesting sort of research project, which was pretty much um taking a whole bunch of different ah methods of leaving sort of Canary tokens. So like purposely leaving things out there and seeing how quickly they get pulled in. So things in GitHub, things in um GitLab. ah
00:08:42
Speaker
um other ones in terms of like in Docker Hub or package managers and just like wait and see when things ah get tapped into, right? Docker Hub actually took like seven days for it to to get accessed in this article. um But something like NPM was like minutes.
00:09:01
Speaker
or something like that obviously so you know it's it's not like if if bad actors will come after your stuff it's like they already are yeah so just so sorry if it's been yeah anyway we'll include those two articles because I thought they were interesting um the The other one I had is Carmata 1.11 is released. So, Carmata is an incubating project um really around multi-cluster, multi-container, and multi-endpoint management, orchestration, and those kind of things. So, 1.11 does release some cool new features. ah One is the rolling upgrade feature, which is um
00:09:41
Speaker
So if you have like three different clouds, you can say, have a new version of this container or this application, roll it out to cluster A in this cloud first, and then I will delegate when the other ones um can ah do that and and all from a single sort of a configuration file across your different environments, which is pretty cool. So haven't used this yeah I haven't used the project myself, but these types of things I imagine will become more and more popular.
00:10:10
Speaker
No, agreed. right I think in addition to the multicloud use case, you might have different geos in the same cloud. So serving application an instance of your application serving the US market, another one serving the European market, you can just roll out changes at the same time.
00:10:24
Speaker
man that That makes it simpler. I'm hoping it it proceeds quickly from the incubating phase to like a more matured and graduated project. But no, this is a cool find. Thanks. Yeah, absolutely. And then one other project that I came across was Altejian created a new horizontal um auto scaler for Kubernetes called Escalator. um We've talked about horizontal ah scaling before on the show, but this is a new project that I just came across, um ah which does some interesting things. So if you're into, um you know, auto scaling and AWS, I think is sort of the cloud provider it supports at the moment. ah Cool, cool, interesting project, open source. You can go check it out. We'll put it in the links as well.
00:11:11
Speaker
and Escalator, yeah, which makes me think of like scaling up, not necessari necessarily out, but yeah it does support up as well. I know. they they like They need more marketing help. It's so hard to come up with names, right? yeah True.
00:11:26
Speaker
An escalator is awesome. but it's more An escalator is terrifying if you really think about it. I mean, it's one of the most terrifying things we deal with in everyday life. I feel like. Why? just like I don't know. Maybe an unrational fear of being sucked into it or like, you ever see those videos that say like, here's what happened. Here's what can happen to you on an escalator. If you get like piece of clothes or like a heel, uh, it's terrifying. Don't watch those videos if you don't want that. I've seen those videos. Yes.
00:11:51
Speaker
But hasn't ever actually happened? like Oh, for sure. For sure. You just don't want to know when it happens. That's true. Yeah. I'm i'm always careful. like like Obviously, I don't have clothes that go a lot beyond my feet. Not wearing anything that's bad, but yeah.
00:12:08
Speaker
Interesting. Maybe I'm living in ah and a bubble, but speaking of escalators, I went to a mall in this area recently that had a separate escalator for a cart. Have you ever seen that? Yeah, I like that. Yeah. know Maybe I just, I'm not used to like huge malls because I don't like going to them. You can put your whole cart and it brings it up separately if there's two floors to it.
00:12:29
Speaker
I mean, it makes total sense. I saw that in like the Target or in New Hampshire. Those are fun. That's the one I'm talking about. That's literally the one. Anyway, totally off topic on escalators.
00:12:42
Speaker
um Let's get back on topic. ah We are going to have a discussion with Ofer Cohen, the CTO of Container Security at Wizz. We're really excited about this conversation because security is always an awesome topic. so Without further ado, let's get over on the show. All right, Ofer, welcome to Kubernetes Bites. It's so good to have you on the show. Let's start with giving our audience a little bit about who you are and what you do. so Hi, Ryan. Hi, Bavin. I'm excited to be here. I'm a huge fan of Kubernetes Bites podcast. You guys are doing a great job. Thank you. Welcome. I'm Ofer Cohen and the CTO of Kubernetes and Container Security in WIS, filling a customer-facing role.
00:13:23
Speaker
My role involves um product management and engineering and solution architecture and essentially ensuring that Wizz customers can deploy Wizz at scale and how can we improve the maturity of the product by designing new features, affecting the old map and prototyping new technologies and tools.
00:13:42
Speaker
That's awesome. like I'm so glad. like I know we have spoken and connected in the past over LinkedIn because you you reached out but ah ah to talk about a specific episode. But man, your role sounds interesting like being being that customer facing. So I think that brings me to my next challenge ah question. like ah What are some of the challenges that you see in the cloud native ecosystem today? like When you're talking to customers, I know right now you're in the Bay Area as well. So what gets brought up like the top three things like, oh, these are the things we need to solve for. Right. so If you look at the enterprise landscape and which has over 40% of the Fortune 100 customers, most most of them will have um their own base image. This is just one example. i mean Container image security and vulnerabilities management is just one pillar out of so many. We have identity and access management. How do how do my employees access my cloud assets? And then we also have workload identities.
00:14:39
Speaker
How do my pods access databases and my S3 buckets? right So there are so many pillars and each organization and enterprise and startup, whatever, each one of them have more or less the same concerns, but in in a different um and a different risk appetite.
00:14:58
Speaker
it Gotcha. So like ah you you mentioned a couple of things, right like signing images and things like that. So are they already like do they already have policies that maybe they have based it on what security was in a non-cloud native traditional world and they're just looking for newer versions? Or they're looking to vendors and industry leaders ways to tell them, OK, what should I be even doing? like ah Where are they in this cycle?
00:15:23
Speaker
Yeah, it's a good question. I think um if you look at the landscape 10 years ago, then we we didn't have so many customers on the public cloud, right? We we still had on-prem. And even nowadays, we still have customers using on-prem workloads. And we see the pendulum shifting back. Adobe, for example, is that a very on-prem shop for many reasons. Some of them is cost, some of them is compliance.
00:15:49
Speaker
But the the thing is when you move from on-prem to the cloud, everything becomes different, right? You can just lift and shift the on-prem solutions to the cloud and expect things to work. One of the reasons being is that the cloud is dynamic. We have ephemeral environments, workloads come and go, and you need somehow to have end-to-end visibility and keep up with what's happening. So the pets versus cattle is one of the the best examples of that.
00:16:17
Speaker
Yeah. And I think like just to wrap up your on-prem to cloud discussion, right? I think when when things were on-prem, you had a data center, you had an actual physical endpoint as well to stop malicious actors from getting in. And there was a network entry point as well as cloud. I think there are so many ways people can poke holes. I think, yeah, there is definitely a ah change in mindset needed.
00:16:40
Speaker
Yeah, and I feel like on top of that, ah cloud is also kind of ever changing, ever updating. And so you have to keep up. Speaking of speaking of keeping up, right this this is something I think I asked you when we met when we first met, which is like, where where the hell do people start, right? Organizations are often sort of daunted by the task of security for many reasons, right? Because security is often like thought of not first. I know you're probably trying to change that.
00:17:08
Speaker
um But also there's so many ah sort of attack vectors, whether it's on the developer machine or in running infrastructure and production. So I guess the question to you is, you know where do you see people sort of starting um or thinking about a strategy?
00:17:26
Speaker
Right, it's a good question. So the first thing you need to do is you need to to have a framework to evaluate your risk and prioritize them, right? You cannot fix what you cannot see and what you cannot measure. And then no, so you first need to have, if you want to evaluate the security posture, you need to have a threat model. And one of the ways to have a threat model is to use um and as well lean towards the The next topic is using your framework like a NIST, the CSF, the cybersecurity framework questionnaire.
00:17:59
Speaker
the the the questions The interesting questions that you want to ask yourself is, who has access to my data? right I get sued for hundreds of millions of dollars if I don't comply with um with the standards. and if i don't If I don't protect my workloads, if I don't protect my database and my buckets, what happens if the the social security numbers and the PI information of my customers gets stolen?
00:18:24
Speaker
and This is really scary. so You need to have and a framework that allows you to evaluate your risk and know where well the areas in which you're most vulnerable. Do you find that you know individual teams are doing these sort of evaluations, developing their own threat model, or is the business as a whole sort of developing that?
00:18:48
Speaker
Right. um For the most part, you would want tools and automation to to codify this knowledge because it's very hard to keep up. I mean, what would you do? Would you start a spreadsheet and then and like ah you know go over each of them and do it multi-cloud and at scale it?
00:19:07
Speaker
It just becomes so unmanageable. So yeah yeah, you need to have some kind of checklist. One of them is like CIS and NIST. So one of them is a community driven security controls and other is a federal one. You need to you need to agree on on some standard. If you're working with the Fed, you would go for NIST SP 800. It depends on your risk appetite and your risks.
00:19:33
Speaker
But should we, should organizations, right? Like if, if they didn't have a security practice and this is the first time they're building it, should this start from an open source standard or just ah try to cover your asses and like just make sure that they don't get sued by following the, whatever the government guidelines are. So where should, like, is it more community focused or vendor focused or just comply with government regulations and you're good? Usually the regulations and the amount of money that you can get sued for is the impetus.
00:20:03
Speaker
first part thing The law dictates how and how and what you should do. But if you are not protected by any law, then you have something like SOC 2. You want to sell your business, your B2B SAS to companies in the US, you have to be at least SOC 2 compliant, if not ISO 2701.
00:20:26
Speaker
And any if you are you or not, um you don't need to comply with any of these policies and compliance framework, then yeah ah very at the very least you could scan your scan your images, scan for vulnerabilit ah vulnerability management is like the first pillar. Identity and access management, multi-factor authentication, pass keys, these are like the, for the 80-20, this would cover a lot if you need to get started somewhere.
00:20:52
Speaker
that's ah That's a good term. I like i use that cover the 80% a lot, but I guess in security, that's a good way to start. Um, cause you have to start somewhere, right? You can't boil the ocean with all the securities. Um, I guess at once. So, um, I mean, I guess, you know, when you have a threat model and you've done sort of the legwork and figuring out where you want to start, where you want to build out your 40 80, um, you know, who is ultimately responsible for your security. And I know this, this answer is often, and it it depends, but I'm curious from your perspective and what you see. Right. Um,
00:21:27
Speaker
The short answer is everyone. Yeah, it's it's very unfortunate, right? I mean, times are changing and we need to be aware, both as platform teams and as developers, we need to be aware of the risks. One of the main challenges, if you think about software engineers in the big companies, they learn lead code and they learn distributed system design, they learn Go, and and now they have to figure out how container orchestrators work and how do I scale.
00:21:53
Speaker
And how do I do failover? and so So much stuff. And then they will they will be like, okay, we'll deal with the security thing later, right? um we will Everyone will get admin privileges, but don't worry. We we know that it's just go make it more important. and Exactly. yeah let And then your girlfriend leaves you and you need to go home and and you want to take a vacation. So you never yeah never fix this list privilege stuff, right?
00:22:23
Speaker
Yeah, exactly. yeah one of the weirdest season with you explicit good organization but okay I mean, it's a pretty good one. If you ask me, you don't really want to piss your girlfriend off.
00:22:35
Speaker
I think we're often thinking about and talking about on this show the idea of shifting left. and You mentioned like having to learn all these new things. um I think there's often, um you know when especially when you're learning a new skill to your point, there's often the assumption that security is taken care of. right Maybe they're given a platform or that kind of thing.
00:22:55
Speaker
um And they're saying, well, I'll just throw my application out of it. um And I don't really need to be responsible for it. So yeah, really interesting. And and kind of like, where, where do we think about, um you know, security? So I guess to that point, um since there's so many services, you you know, what is a developer, or maybe what you what you do at Wizz, like, where does the developer get involved in security?
00:23:20
Speaker
Right, um so in ways we like to say that cloud network security is a team sport. when When you look at the enterprise solution in the likes of Waze, Palo Alto, Quads, Cardstock, etc., the usually like um the CISO is the one driving the the policy and the decisions.
00:23:37
Speaker
But you want the platform where um you have point solutions that can detect vulnerabilities or can detect um critical data assets and the passwords and PII and things that you shouldn't have on your local disks. But the key is context. You want you want to be able to ah go from code to cloud and cloud to code. Recently, from time we launched with code that allows you to do just that. So the idea is that If I have a vulnerability in production, I don't want to just be aware that it exists. I want to know um what git commit introduced it. And I want to go back to this the git commit. And furthermore, I don't even want to think about what is the next version of the of the package, the JSON, the Node.js package or the Go package. I want a platform that will auto mediate it for me. I don't want to work.
00:24:32
Speaker
So imagine a place where they both developers and platform teams and the glc like the government's risk and compliance people and the see so everyone can just be one portal and have a single pane of glass or even different views and perspective of what are the risks in my environment prioritize and this is the key.
00:24:55
Speaker
I don't want to get the adult fatigue, right? Most platforms will say, hey, these are like 1000 find CSP and findings, go fix it. It's not it's not useful. I mean, maybe no one is is able to access my workload. Maybe there is no ingress access. So I want to know just what are the critical things that I need to fix in my environment on a daily basis. And I want the developers to have a self service portal. And I think this is where we want to be as an industry.
00:25:23
Speaker
No, I think it that like makes perfect sense to me. right I know we were talking about the 80-20 rule and ah you can fix 80% of the things easily. But if you're showing me all of those 100 things, the 20% that I'm not able to get to might be the reason I get hacked or somebody and gets remote execution capabilities and one in my environment. So having a tool that can give you a prioritized list from a severity perspective. I think that's definitely helpful. So of let let's talk about vulnerability management. right like You brought it up as like the first pillar. So when where do people start? like Is it something that a security team deploys and then they implement policies in terms of ah developers make sure you're scanning your images, scanning your code, or they're doing more of a real-time thing? like where How does an organization make progress on vulnerability management?
00:26:14
Speaker
Right, um so the first thing is that you you need that to have tools, and this is always true. You need to have platforms and tools that you can trust. right You don't want some tools that someone developed yesterday and consult some advisories. Advisories is a fancy word for a vulnerabilities database, one of them being an NVD, the national durability database.
00:26:38
Speaker
You want the tool that you can trust that uses the up-to-date advisories and can alert you on every given day what what are the risks. The second thing, you want to know if they came from ah your base image, if we're talking about containers. VMs also have a similar issue, but we want to be able to pinpoint what where the the problem came from and we want to do it on a daily basis and all across the pipeline. So DevSecOps is one of the fancy words for saying ingrained security in every step of the software development lifecycle.
00:27:14
Speaker
yeah Okay. So like how, what, what's our recommendation or what's business recommendations? I don't really have one, but in terms of ah to developers, like what should they pay attention to when they are pulling base images or starting to build on top of base images or even application code that they are copy pasting from either stack overflow or using gen AI to to generate some code for them.
00:27:37
Speaker
Yeah, so the first thing I would say if you shift mostly to the left is to have ID extension that you can scan before you even do git commit. a Why waste so much time on, hey, the CI is flagging my commit as having vulnerabilities and malware? No, I want to know it before I even do the git commit.
00:27:57
Speaker
ah So this is the ideal position and this is something that we recently launched the vs code extension for for which is that does exactly that. So you you see all your package is on there. You have a problem Spain and then you see all of the vulnerable package and you also get ah remediation. So how to fix that the next thing even if I was. um Even if I didn't notice or didn't install the extension, I want my CI-CD pipelines to have the guardrails in place. So I don't want any commit to be pushed unless it was scanned for vulnerabilities. Finally, we have the um if we're talking about Kubernetes, we have admission controllers. that
00:28:36
Speaker
there are the gatekeepers that sit at the front of the API server. Remember Kubernetes is an API driven machine or system. So we want to gate just one moment before it actually runs and deploys in production. We want to to know whether this container image was scanned for vulnerabilities. I mean, we have chat GPT today. Let's not kid ourselves. Everyone goes to LLMs and they're like, you know what? I'm lazy. I'm not going to IT MLs. Life is too short to IT ML files, right? yeah So um I go to a chat and I say, Hey, generate this young manifest for my and NGINX application and I might go back in. And it will do just that, but who God only knows who built this container image, right? yeah
00:29:21
Speaker
ah so like Yeah, and I like that idea of you know things, well, I should say things, automation moving closer to the developer. So it's one thing for us to say the developer should do more and shift left, but also awareness ah is only half the battle, right? And for asking them to do more is the hard thing. So having like automation, if I'm writing a YAML file and it to Text an image like go ahead and automatically go tell me what the cvs are and how to fix them that's very helpful versus you know you know we had a ah bullet point in here and and how to manage cvs and and you know i'm a big fan of murphy's law i believe in it. yeah meaning you know if you If you don't fix a CVA, it probably will be the thing that gets you hacked you know to your point before. so so I guess that's that's that's another question to use. How do you manage these from the developer's perspective, but also are there tools for catching these things when they're you know being pushed to an environment meaning if it if i scanned it and didn't do anything about it if i'm trying to push us into an environment can we like not to play the thing if we detect it later on the pipeline before you start i want to just add to this question so what about day two like maybe today the it's not a vulnerability that's being documented by nvd or it's not even a thing that can be exposed if it's running in production on the hundredth day how do we handle handle that scenario like who Is the developer supposed to look at the and NVD database or the tool every day and figure out, oh man, this code that I pushed has a CVE now? No, it's a great question and I like ah Ryan's point. Security is a continuous and ongoing effort. You may you may have scanned everything and everything was perfect at the time you pushed it, but as you said, there are zero day vulnerabilities and the environments are changing.
00:31:11
Speaker
the The data is changing, everything is changing. So what what you want to do is in addition to have the guardrails on your CI, CD and your ID, you want to have runtime, real-time monitoring and threat management. So threat detection. So the idea is if you could scan your environment, if you could take a snapshot, right? Because it's easy.
00:31:33
Speaker
it's a moving, um, you have lots of moving parts, so you need to have some kind of a snapshot to say, this is the moment. Um, I, I tried to ask what of the vulnerable workloads in my environment, then you would also get the zero day threats. And, um, if you use anomaly detection and behavioral analysis, then you can detect things that you didn't even know were explainable. Yeah.
00:32:00
Speaker
That's ah it' a valid point. Yeah. It's a defense in depth. It's never one tool. People always say like, oh yeah, I'll do vulnerability management. Oh yeah, I'll use Pasc is an MFA and I'm good to go. No, you should sleep with one eye open. You should go to bed and you should ask yourself what else did I miss? um So you need to have the ca address all along the way, not gates. You need to have tools that I'm just adding to my anxiety. I don't know what I was going to say. I was going to say, hopefully you're, you're sleeping and letting the automation sleep with both eyes open, right?
00:32:33
Speaker
what the question broke up back um Yeah. you're not getting enough sleep That's it. um um Cool. so we we had We had another one in terms of like workflows. you know Often, one of the first things we think about with our applications, at least, is how to manage secrets. so um and Often, we're like, oh, we'll just use Kubernetes Secrets Manager or like Vault. We'll just throw our stuff in there and usually like set it and forget it. That should be good enough. Tell us why that's not good enough.
00:33:04
Speaker
Yeah, this is one of actually my favorite topics and I gave a talk about it two months ago in San Francisco, the first CNCF meetup. So when you think about the secrets management, the first thing that developers would do, and i I did that when I graduated from computer science, I wrote my my first Node.js application.
00:33:27
Speaker
no application, you're like, okay, I just need to see that the database works. I promise. Like, uh, I know I will read it from environment, right? And then you do the git commit and, uh, the text that you have admin and password embedded in your code. And, um, we saw it. I mean, one of the most famous preachers was in the 29, I want to say 2019.
00:33:53
Speaker
um I cannot disclose the name of the company, but it's in the transportation industries. And because some developer left the public yeah left the keys in the public GitHub repo, applicants were able to exfiltrate tons of data, and they were sued for um hundreds of millions of dollars. I mean, you don't want to you don't want to go there, right? So this is the first thing a silly thing that you would do. You you would commit your credentials and identification tokens to the code.
00:34:21
Speaker
and The next thing you would say, okay, I'm going to use um i'm going to decouple the configuration and the the secrets from the app, and I'm going to delegate it into some secrets manager. I'm going to use environment variables. All the best part is very, very good. the The main problem with that is that um key management and secrets management is hard, right? um You need to be able to rotate them. You don't want long-lived tokens. They are the waterfall evil.
00:34:49
Speaker
What you want to do is you want to have a scoped and very short-lived token to reduce the um the chance of a threat actor that compromises your pod to be able to steal them and later use them. and So I'm not against Secrets Manager. I think Vault and Hashico Vault and AWS Secrets Manager and um external Secrets Operator for Kubernetes. They are all very good things except for the toil and the governance. and um all of the overhead around operationalizing it. So if you know what you're doing, sure, go ahead. The most modern approach to actually do that, and this is what I would recommend to do, is to use a concept called Workload Identities. You essentially create a trust between your S3 bucket or RDS database and your Kubernetes cluster.
00:35:36
Speaker
And you have a contract that does token exchange. So it's a lot of voodoo and magic under the hood, right? You have like a Kubernetes mutating admission webhook and a demon set and all kinds of these fancy buzzwords. But not sure you get from the cloud provider, um, short-lived and scoped access. And this is the best practice nowadays.
00:35:57
Speaker
how does that Go ahead. Does that also work, you know, between two applications, right? In the sense that, you know, say you're a developer, you're writing an API and some web server and you have some bearer tokens that need to be rotated, but you're doing it through Secrets today, so you kind of have to do that manually. Does that contract work? You described S3, does that also work sort of intra-cluster as well? Yeah, it's a good question. The whole identities, and this is a topic that I'm so passionate about, Kubernetes identities,
00:36:26
Speaker
and then and access management. So you have standards like SPFEE for the workload to workload authentication and authorization. And then you have job tokens and all for workload to cloud. So Kubernetes cloud, you need to be aware of the different ways that workload to workload can communicate with each other. It's a very complex topic. um We're getting there in terms of blocks and tools, but we're still, I feel like we're still not there.
00:36:55
Speaker
for For that scenario, right are people looking at security tools or they are looking at service meshes? like oh i can I can set those rules in inside a service mesh layer and that can ah avoid that communication or or drop that communication but if from an ah unauthorized apps. This is one approach, doing a service mesh. but When you go multi-clustered, then it becomes even more challenging.
00:37:19
Speaker
um right what That makes sense. I think multi-cluster is not something I thought about. So thank you. um My question originally was, ah if if my cluster has that identity capabilities to talk to S3 or talk to anything outside the cluster as well, if my cluster gets compromised, is that it? Like my entire estate is gone or vulnerable now, or is there a way to do it securely, right? That even if somebody got access to an API server or a pod was deployed that had some sort of remote execution capabilities, workload identities can
00:37:50
Speaker
kind of helped me in that scenario they would help you in the sense that even if the workload was compromised a few moments later say a few hours later or a few days later the attacker won't be able to use the credentials to access any of the assets okay so it is time limited and rotating as well Exactly. ah Okay, makes sense. Cool. ah So I know we spoke about a lot about challenges and shifting left and all these same workflows, right? ah We brought a whiz a couple of times and then I know like we have Ryan and I have covered like all the different funding rounds and acquisition rumors and everything that this has gone through over the past six to nine months. But can you give us a high level overview of what is ways and how is it trying to solve the the challenges that we discussed?
00:38:38
Speaker
Sure. So WISF was founded almost, well, a little bit more than four years ago. It was the fastest scoring startup in world history. Nice. You even come back to OpenAI? Come on.
00:38:53
Speaker
I mean, OpenAI is its own thing. and the so It's just killing all graphs. Like I saw that the graph, sorry to interrupt you all, but ah I saw a graph where the fastest to like 5 million MAU or something like that and OpenAI is like everything else was like months and years and OpenAI was like, I don't know, just a matter of days or weeks and like, yeah, they are operating at a different scale. Sorry, go ahead. Let's talk about Wizz though.
00:39:21
Speaker
OpenAI an alien technology which connects to your environments and it provides you with visibility. Performing inventorying and risk correlation costs all the pillars from all traditionally different silo domains. So usually When those talk about CSPM, cloud security posture management, like I have public S3 bucket, or I have my security group open to 0.0.0.0, et cetera, et cetera. But then there are so many other pillars. You have a data security posture management, and then you have CWPP, and the the idea is that
00:39:57
Speaker
Each pillar in its own is very important, but what you want to do is to get the insights and remediation guidance that you could seamlessly integrate into your day-to-day workflows, right? We started with with cloud where the idea was to protect the infrastructure and on a daily basis to have continuous security posture and compliance scanning. And then um we did that using agentless and and this was one of the most innovative approaches using the graph.
00:40:26
Speaker
modeling everything in a computer science-like way, so asking questions like, if a pod gets compromised on my Kubernetes cluster, what can the threat actor do? right can Can the threat actor access um my other resources, my GCP packets maybe? And this is a very hard question to answer, because doing it multi-cloud, you need to have knowledge expertise. You need to have researchers, and this is what we have in WIS. We have the the security research team that is exposing very noble things that no one would even imagine. right so You want the expertise across all the clouds. A lot of companies are multi-cloud, whether they want it or not. You have Belgian acquisitions. yeah vi um It's not like Facebook buys a company and and all of a sudden, everyone is on-prem. It doesn't work like that. yeah you You have production, you have business critical and money-making machines, and you want them to continue making money for you.
00:41:24
Speaker
so The challenge is huge. We call it the hybrid clouding list. So the first thing you need to do is do have you need to have visibility and inventory of what you have. And then you also want to be agent-lessly in order to to keep up with what's changing. If I have to install an agent on each machine, it will take me years. I have to open a ticket and this time when it's on vacation and we all well know how this movie ends.
00:41:47
Speaker
so on so I like the graph way of thinking about it. i like like Looking at social media companies and how how they actually build these social networks, that was their thing. like They may may build used a graph database and made it easier for a security company to take in the same principles and figure out like okay if this gets exposed, how how does it connect to other Okay. I didn't know that. So, and I do want to take my comment back about not being the fastest because I remember like open has been around for a while. I think we started in 2020 and going from there to close to a billionaire, man, that's awesome.
00:42:28
Speaker
whereas so ah So you mentioned security researcher, and this is sort of our off topic, but um Wiz has security researchers as like employees or externally you use them. Yeah. What what do they do? i'm i' you know i'm I'm just kind of fascinated. I'm really curious about that.
00:42:45
Speaker
Yeah, i'm I'm very fortunate and lucky to be working with them in the same building. They are ah super talented people that I really don't know. I ask myself, how do they come up with this stuff? Yeah.
00:43:00
Speaker
And it's not just like the the complex stuff that everyone thinking about it. Sometimes it's very silly that they recently gave a talk at a KubeFam ah by the security research team. I highly recommend that it's an episode by Eli Eloni. And people are just being sloppy. People are using Helm v2 that has the tiller component.
00:43:24
Speaker
yeah You have so much technical depth and legacy. If you haven't worked in any of the companies that you know that it's very hard to keep up with what with what's changed you. So they somehow have these instincts and insight to ask the right questions. Like if I inject this file on disk, um who is always going to access this file? How can I impact the controller, ah the cloud controller, that reads file from disk? oh I have access to this file system. Very good. Let's see what happens if I write. They come up with very innovative approaches and they also bring a ton of experience from the military service. sure yeah So yeah, Israel is known for a very good cyber security hub right and security approach team. So of it like is this the same team that kind of does red teaming as well for any new products that which is launching or these are focused on security and then there's like a separate red team as well?
00:44:19
Speaker
separate so okay okay that cool so Speaking of those researchers and what they look for, um the ones that are probably really easy to spot is ah when things are misconfigured or maybe they're not, maybe they're not but you know i you know we as humans are pretty terrible at using our big fat thumbs to screw things up.
00:44:38
Speaker
And now we have, you know like you said, JNIA bots that were just saying, how do I do this? And then probably taking it and not thinking about our security posture whatsoever. So, um you know how much of a problem is misconfiguration by either humans or automation and and kind of what do we do or what can we do to watch for these types of things?
00:44:59
Speaker
Right. So the interesting thing about misconfiguration is that if you look at Galata reports, this is one of the top five. yeah its it is It has been like that for years and it's only growing. And the reason is that the complexity. I mean, let's take an example. You take an AWS gateway with an AWS Lambda.
00:45:17
Speaker
There are so many different ways to configure and provision it. And I'm not just ah talking about the VPC and the security groups. like It's so hard to get it right and keep up with all the options and combinations. And you know we'll fix this security thing later. the So the defenders need to get it right all the time and attackers only need to build it once.
00:45:39
Speaker
know that fair you need that You need to be reactive and proactive about it. You need to continuously scan your environment for compliance and security posture and infrastructural score scanning. And ideally, you would prevent it um from being pushed to Git.
00:45:56
Speaker
Yeah. And I think the the gated things that you we mentioned earlier in the discussion, right like starting from the VS code extension to all the way to admission controllers and Kubernetes are a way to catch, hopefully giving us enough defense in depth to catch all of these things before they become an issue. oh I think I wanted to also ask about, oh I know we referred to single pane of glass before, but is that a thing like do we expect organizations to have a single security dashboard or an operation center view where everybody is logging in or we don't need to bother ah developers with ah and ah infrastructure side of things or maybe the tool itself has different views. So but it like where do you see this? like Is it a single pane of glass or multiple teams have their own views of of the dashboard?
00:46:44
Speaker
the The trend is obviously consolidation. If you're with any report and you look at the market, no CISO wants 500 tools. It's very, very hard to keep up with what's going on. That's why in ways we have hundreds of integration and it's going every year because customers um and organizations, they want to have one stop shop. I hate this term, but it is what it is. you don't want to Think about training your employees, telling your developers, oh You need to learn this UI and to platform teams and security engineering. You need to learn that UI and then and you you create a problem of vocabulary. I mean, how you have many people in the same room talking about issues and.
00:47:30
Speaker
and And then you you need to have so many meetings. We want to cut the middleman. This is the the goal. This is the i ideal. We want to have ah a ah SaaS platform or a SaaS host and it doesn't matter, but that everything is intuitive and you can navigate from production from cloud to code and code to cloud. I don't want six different platforms. I want a single source of truth and a single pane of glass and I don't have anything else with single.
00:47:56
Speaker
Nice. No, i know I get the point, right? Like how everybody talking the same language is definitely important. ah But do you see oh like with with platform engineering, right? Like with DevSecOps, like platform engineering is also a big thing in our ecosystem. Do you see integrations into IDPs as a thing? Like, oh, if the developers are using backstage or Cortex or some other IDP, they see the security things there, even though like under the covers, it's the same platform. Do you see different integrations with different tools or no? if you want to do anything with security go to this UI. Yeah so the in in platform engineering the term we use is golden path. You want to have a way to provision new projects and new micro services very easily and you want to have the the governance and compliance and security ah ingrained right so as soon as I generate the
00:48:49
Speaker
my telephone or I generate my my GitHub, I want to have all the scans for my port in place. the The main problem is that um these these tools, I mean, I like Backstage and I like Port.io, but I think we we are not there yet in terms of integrating IDPs with security compliance. It's a big target and it's very hard to keep up with the changes, but we will get there, I'm sure.
00:49:17
Speaker
No, thanks for that honest feedback, right? Like if people are evaluating tools, they need to have this information. So that helps. Makes sense. um ah Switching gears a little bit about you know um all these topics are also very important and and interesting, but sometimes it helps our listeners to get sort of some real-world use cases and examples. so Question for you of hers is, can you talk about any of your success stories when it comes to your customers or people you've worked with on sort of
00:49:48
Speaker
starting from a certain point and getting to a level of security that is, you know, proper, I guess I'll use. Yeah, um in ways we call it the log4j moment. When we started, we the the the four founders of Wizz are serial entrepreneurs. They founded another company called Adalum, which they sold to Microsoft for 350 million after like three years. So very, very ah impressive. When they launched Wizz, they had, you know, sometimes when you are a visionary, you think of as Steve Jobs and the iPhone and faster horses, people don't even know, um people can imagine how the future would look like. So you need to come with some innovative approach.
00:50:35
Speaker
And back in the Log4j days, ah I remember as after CEO, we said, who who lives Microsoft in the midst of the pandemic? The the world is coming to an end. I'm leaving Microsoft to start a new company. i mean What the hell am I doing? And it's so hard, even when you have the credentials and you have the network and you have the the the right talent, it's very hard to get the early customers and early adopters. But then you have things like Log4j.
00:51:03
Speaker
and um I remember that we wanted to um to get some very large profile customers in the automotive industry and they're like, no, no, no, this is cute. We'll talk to you later. We don't need you. And then log4j came and ah it was a mess. We didn't even have S-BOM back then, but we generated S-BOM even before it was called S-BOM. So we were able to do the inventorying and tell you in split seconds, less than like, um Less than a second, we were able to tell you, here are all the machines that are affected by the log4j exposure. and Here is how you fix it. It sounds simple. It sounds like a simple question to solve, but back then, you didn't have a standard that did that. and we did it like They were stoked. They saw it. and they like It's like technology living in the future. It's like, where did you get this alien technology from?
00:51:56
Speaker
yeah Yeah, this is the law for it was definitely like a pivotal moment, right? Like everybody was panicking. People didn't even know where those binaries or packages were being used. And yeah if a tool can give you like a complete overview, like, oh, go fix this, this, is this, this, this, you have an order of doing things for sure. Exactly.
00:52:14
Speaker
um I think one one more question before we wrap this up though. It is more around like and when when people think about ransomware and there is like a parallel industry in terms of ransomware insurance and ah if if you get hit, you collect insurance and you pay off the ransom and and you're good to go. Is there a similar industry being built for cloud native security for like cloud native insurance as cloud native security insurance? Sounds like a business idea. Yeah.
00:52:44
Speaker
Yeah, it sounds like an idea for a startup. Okay. Let's be in that that right now. No. Okay. Okay. Uh, about run somewhere. Lots of companies still don't fully implement infrastructure and code and get ups. And they do backups, but they never test the backup, right? You want to set RTO, um, and LPO, which are two KPIs for actually, um,
00:53:14
Speaker
you know knowing if your backup strategy is useful and working. I would say this is the first thing. The second thing is you want to have the backup in place. and um The third thing is that you need to have at threat detection and threat intelligence and real-time monitoring, and you need to be able to restore your environment in split seconds. I was ah under a crypto mining attack two years ago in a startup that I worked, and they just found my entire Azure tenant.
00:53:42
Speaker
Without notice, it was a zero-day attack, a supply chain attack. It happens all the time. And I didn't even know that I had it. they I just got an email. Hey, we shut down your environment. They didn't even explain. They didn't bother to explain it later. It took me a few days and then you become like so paranoid. It's like, okay, so how do I like what you don't trust? I was entering the room. I was running the engineering and I was like, okay, so who did that? you know looking at I'm looking at faces. I tried to make it a joke, but you it's it's very hard. yeah
00:54:17
Speaker
It's very hard to protect yourself against this kind of stuff. Oh, that's true. I think that brings us to I think our last question, right? This has been an amazing discussion of it. But if people want to learn more about ah the NIST frameworks or the different frameworks that that they can use to build their own set of guidelines, if they want to learn more about WIS, where can they get like go to to get started with all of these follow up questions?
00:54:42
Speaker
I would say um try pitch.p-e-a-c-h.wiz.io. it's ah It's a very interesting Kubernetes um and Cloud Native security frameworks to look at. With.io slash customers to see customer testimonials and um how more and more customers are leaving their existing security vendors and utilizing With. For example, With Image Trust is one, and we didn't get to talk about it, but this is one of the most noble innovations that we did in the space of software supply chain. So being able to to do keyless signing, I don't need to think about
00:55:17
Speaker
key management and key revocation, and I can also tie the scan um to certain policies so I know that I don't have secrets or CVs in my containers. Not only are they signed, but they also confirm to my policy. So this is innovative and noble and no other vendor, um as and say we currently know actually does that. Okay. That's awesome.
00:55:42
Speaker
Bob, I was thinking about how when Ofer was saying that people take backups and don't test them, it's kind of like the equivalent to our dagger episode with push and pray. It's like, just back up and pray. It's wasn't that a thing? It should be like praying at work. Come on, let's say everything.
00:56:08
Speaker
Oh, that's good. well um Over, I think this has been an awesome episode. Every time we talk about security on this show, I know we get a lot of great feedback because there's endless information of kind of ah you know the overwhelming amount of things that people need to think about with security. so It's always a welcome topic. and Thank you for helping us kind of walk through and explain some of what you've been kind of working on and and what you're up to at Wizz and we'll make sure and put all those show notes and links ah if for our our listeners and um yeah we'll have to have you back on the show in the future. I would love that. Thank you so much for having me.
00:56:47
Speaker
All right, Bobbin, that was another great security talk. I know we've had a number of them on the show. We'd probably have to go back and actually count how many, but I feel like I learn new things or think about different things each time. I know Ofer had a lot to talk about. so um you know Personally, I like the sort of idea that Wiz, where he works, kind of takes a holistic approach to security. right um I often am involved in like a very particular point of security when it comes to platform engineering, when thinking about the whole. um Obviously, there's so much. right so I know that's why I sort of asked a few different questions around, like where where the hell do you start? and I know though um those sort of frameworks that he talked about or NIST um is a great ah way to think about it. so if you you know As we were talking, I was thinking, if you took one of these frameworks, um
00:57:38
Speaker
or sort of papers or or guidelines and then kind of ran through those with the mindset of solving 80% or ah making a plan for 80% of your security. That's a great starting point. um It's probably a ton of work still. It is, but I think ah that like to to that end, right I think there are tools. I want to say CubeScape, the ARMO guys, the discussion that we had, where they have actually codified the NIST or NSC security framework. And you can actually like run your security tool against your app and infrastructure and they're like, check for the compliance with this set of recommendations. And it gives you specific things. So yeah.
00:58:19
Speaker
There are people in the community vendors and open source contributors that are making things easier. We just have to use them, you know, yeah just a small part. Yeah, I feel like every time we talk about this, right, we talk about shifting left, which has been a ah a very big topic probably over the last couple of years, right? um I don't personally, I don't love this idea of shifting left when we talk about it in that sense, because it I feel like it puts the necessity on the developer. to implement security. and and While that's great, I think it should be a parallel thing, meaning like this entire idea that a company like Wizz exists and can come into your company and and help you better your security posture doesn't mean we're not still shifting left. um But I feel like that security team and or vendor and or whatever um still need to exist because you know you have to have your tentacles in there to make sure you're meeting compliance and whatnot and having that single pane of glass. I mean, to go first point, like that's the ideal scenario is I have one place to look. and I think yeah the offer also brought up this interesting point, right? Where developers, when they're prepping for interviews or just building their skill set, they're going to places like lead code and like express still trying to code linked lists and reverse linked list. And I don't know, I haven't given a developer based interview in a long time, but like focused on algorithms and how to write code.
00:59:37
Speaker
I have rarely anecdotally heard scenarios where organizations are asking security-based questions. So if you expect that you're going to join your organization and take on these responsibilities, maybe add that as a step, maybe make that an industry practice. Like, oh, you have to think about security, even in in like an interview round.
00:59:56
Speaker
Yeah, and I'm sure maybe for platform or SRE or those kind of things, maybe security makes it away. But developer alone, you're right, um at least historically, have not thought about like a trick question around, you know, hey, implement this, what's wrong with it? Or what can you do security wise? And maybe they do, because we haven't done a developer interview in 20 years. 20 years, what's with you and Ridge today? like Just feeling it today. know i think I agree, right? Having that additional round definitely helps because new college grads, SE2s, SE3s, if you want everybody to be involved, I think it should be everybody's responsibility and not just your principal or your distribution engineers that are taking, like, approving PRs and going through your code. So, no, for sure.
01:00:45
Speaker
I mean, it's kind of like security is kind of like the diet to our lives, right? It's like, we always have to be aware of it, but we don't like to think about it or actually do it. I know, but I don't do a weird job on my diet. Exactly. Exactly. This is the point, right? The thing that has to happen, but we're all there. It has to be like top of mind. Every day, you have a weighing scale, you have a single bit of your eyes to tell you, you
01:01:13
Speaker
And on that note, when we're really picking ourselves up today and age and diet, but um I hope your diet's going better, Mr. and Mrs. Listener, or whoever you are. That being said, ah you know, we're gonna wrap it up now. Just a quick reminder, please ah join our Slack. I know we do get people trickling in there. I want to get back to using that some more. So, you know,
01:01:36
Speaker
Please join, um interact, introduce yourself, let us know about some episodes. ah Go check out our website, KubernetesBites.com, and of course join um our YouTube channel as a subscriber. we I think we hit like 400 recently, which seems like a milestone since we... Whoa, has it been a year or two? or I don't know. It's been a couple of seasons. um you know We're super popular.
01:01:58
Speaker
yeah Slowly and steadily, dude. I'm proud of those 400. Thank you to you. I know. Thank you to you, 400 people. For the audio listeners that haven't done that, come on. Get us started. Anyway, all right, Bobbin, that brings us to another end of episode. I'm Ryan. I'm Bobbin. Thanks for joining another episode of Kubernetes Bites.
01:02:24
Speaker
Thank you for listening to the Kubernetes Bites podcast.