Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Kubernetes Security 101 - 4C's of Cloud Native Security
516 Plays2 years ago
In this Episode of Kubernetes Bytes, Ryan and Bhavin talk about upcoming conferences and dig into the world of Kuberentes Security. Bhavin and Ryan talk about and dig into the various aspects of the 4C's of Cloud Native Security (Code, Container, Cluster and Cloud). Bhavin and Ryan dig in a foot deep from everything from encryption at rest, network policies, linux seccomp, software SBOM and ransomeware.
This episode had so many good resources in the show notes, we decided to create a community resource for everyone. Please see the below public google doc with all show notes, links and more. Feel free to comment and engage!
Recommended
Transcript
Introduction to Kubernetes Bites
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts. We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:28
Speaker
Good morning, good afternoon, and good evening wherever you are. We're coming to you from Boston, Massachusetts. Today is September 30th, 2022.
Hosts' Locations and Hurricane Concern
00:00:38
Speaker
I hope everyone is doing well and staying safe. Let's dive into it. I say Boston, Massachusetts, but you are not in Boston, Massachusetts. Sorry. I know. I was just going to bring that up. I'm in Sunnyvale, California.
00:00:50
Speaker
a little nicer over there. And I say staying safe. I want to say anyone that's, you know, knows anyone in Florida and everything going on with Ian, I hope everyone is actually safe. And, you know, we're thinking about everything happening down there, which is pretty wild. Yep, it is. What are you up to? Why are you in California?
Team Meetings and Conferences
00:01:09
Speaker
Oh, we had a team offsite. Like we hired a couple of new folks on the tech marketing team. So just wanted to like meet everyone in person, train them up. We have KubeCon coming in a month, right? So we need to make sure everybody is at the right game. Yeah. KubeCon's right around the corner. I can't believe it. We have one more show to record before KubeCon. So yeah, I was like, how, how is that so close? Please slow down.
00:01:38
Speaker
It did come up fast, didn't it? Yeah, that's awesome. I'm going to be going to Monctoberfest in Portland, Maine next week, Wednesday through Friday, which will be a lot of fun. So if you're a listener and you'll be there, let me know. Reach out to me. I'd love to talk to you and meet you. It'll be a good time, hopefully not too cold. I know Wednesday night, they're like doing a whole thing on the boat, which can get pretty chilly in October.
00:02:09
Speaker
Yeah, I don't know. Bring your coat, I think I'll say.
00:02:15
Speaker
So are you excited more about the conference or eating lobster rolls there? Like what's going on here? I think more about the conference. You know, we live on these coasts. I give a lobster roll, pretty good one, kind of, whenever I want. But Portland, I don't get to go to Portland that often, and it's a great city. You know, a lot of good food, a lot of, you know, good drink if you're into that kind of thing. And I like that city a lot. I just don't get there enough.
00:02:40
Speaker
No, so I have a recommendation for you. So there is a brewery called Bellflower Brewing in Portland. You have to go there. Like it's... Bellflower? Bellflower, yeah. So they don't pay me, but I'm just recommending this to you, not to the entire audience for my month. Couple of good beers there.
00:02:59
Speaker
Absolutely. I've visited a bunch in that area, so I will add that one to the list for sure. Yeah, it's going to be a good time. So yeah, anyway, if you're there, let me know. KubeCom's around the corner. Again, Bob and I are going to be doing this show in bytes. There's a little, you know, joke there if you get it. We're going to do little bite sized shows for about 10 minutes. So please reach out to us.
00:03:26
Speaker
If you're going to be at KubeCon, we want to talk to you. It'd be great to have everyone from the community that, you know, maybe listening to the show or has listened to the show or interested to just come on and give us a little what you think about the show and everything. So yeah, anyway, reach out to us.
00:03:41
Speaker
DMS, whatever it is. We have a fun show for you today.
Exploring Cloud Native Security
00:03:45
Speaker
We're going to go over a security in the cloud native Kubernetes space. And it's going to be sort of a one on one sort of mile wide, one foot deep kind of conversation. But there's so much to talk about in that space. So it should be a really fun episode for Bob and I to kind of dig into these spaces. And we're going to have other folks from the security industry on the show afterwards. But this is a good primer, so to speak.
00:04:11
Speaker
And before we get into that, we will cover a bit of news. So, Bob, why don't you kick us off? Okay, from a news perspective, I had a few.
AWS Outposts and EKS Local Clusters
00:04:19
Speaker
The one update I saw around Amazon EKS was they now have something called EKS Local Clusters on AWS Outposts. So, you could always run EKS clusters and EKS worker nodes on your Outpost system, but the control plane was still hosted in an Amazon region.
00:04:37
Speaker
This created a few issues like if you lost connectivity, the control plane would think the nodes went offline. So when the connectivity was restored, it will start deploying new nodes, start moving workloads, even though your original worker nodes were still there. And that created a lot of issues. So now with local clusters, quote unquote, you can also have your control plane running on Outpost.
00:04:57
Speaker
even if you lose connectivity to the AWS region, you have connected your Outpost 2. Your Kubernetes cluster is self-contained on that Outpost system, and you don't have to worry about spinning up new nodes and then creating all of that application turnover between nodes. So that's a cool new enhancement. Yeah. Yeah, that's super useful. Do you know if the control plane in the cloud is still aware of the control plane basically on your local data center?
00:05:21
Speaker
There's no control plane in the cloud now with local clusters. Everything is on its only local. Got it. Yeah. I didn't know if they communicated and it was like a sort of a backup of connection. Yeah. Got it. Yeah. Great.
00:05:32
Speaker
Nice.
Storage Migration Challenges in Kubernetes
00:05:33
Speaker
The second thing is around GlusterFS. So GlusterFS was part of one of, I think it was one of the entry plugins for storage when it came to Kubernetes. And now we all know we have been tracking this for the past few releases that we are moving everything away from entry to like a working CSI driver. With GlusterFS, there's no working CSI driver. So you will have to move or migrate your data onto some other storage system
00:05:58
Speaker
before you upgrade to the next release that comes out called 1.26. So last month or this month, I think we had 1.25. So it will take two or three months. But if you're using clusterFS today, start thinking about it. Start thinking about how you will migrate it off before you run into 1.26. So just an FIIZ.
Security Threats in Docker Hub
00:06:18
Speaker
And then we have, since we are doing the cloud native security episode, I thought I'll bring in some security news as well. So Sysdig, one of the Kubernetes security startups, I don't know if I can call them startups, they're big enough now.
00:06:33
Speaker
they did a 2022 threat report. And one of the key stats that popped out at me as they scanned over 250,000 images, so quarter million images, container images that were available on Docker Hub. And the results showed that like there were threat actors that were actively using Docker Hub images to spread malware. So basically, like most common was people installing or having crypto miners as part of those Docker images.
00:06:59
Speaker
which shouldn't be a surprise at this point. But there were also malicious websites, hacking tools, and unwanted software that were found in those images. So make sure that if you're using or if you're downloading a container image from a public repository, that you know what its components are and something that we'll cover in this podcast as well. But just a new thread report. This was just one of the things that I wanted to highlight. If you want to read through it, we'll have the link in the show notes as well.
00:07:26
Speaker
Yeah, super interesting report. I think not a surprise, a lot of the images that were, the majority of them were crypto mining, right? That were flagged. But there was a lot of other ones. And I think an interesting topic from that article was that these were disguised as other legitimate practice.
00:07:45
Speaker
So a couple that they I think mentioned is PyTorch and Drupal, right? So you might think it looks totally fine and it's disguised as that. So I mean, that's the whole kind of idea about using tools like this. So yeah, definitely go take a read out of it. Pretty insightful.
00:08:01
Speaker
Yes. And then next piece was also around the security front. So there was an update or a patch update to Rancher because they were storing secrets in plain text. So another no, no.
00:08:16
Speaker
So the reason I bring it up is not to desert rancher, but make sure that if you were using rancher, make sure you're updated to the latest release. You have applied all the latest patches to make sure that you're no longer vulnerable because this basically exposes your Kubernetes clusters to take over. Like if somebody even has read access to it, they can read all the secrets and basically take over your cluster. So make sure you're running the latest and greatest version when it comes to your rancher clusters. And we'll again have a link in the show notes to read more about how it happened or what
00:08:47
Speaker
Yeah. I feel like this is one of those like life rules for Kubernetes. You know, like if you go hiking, you just tell anybody that you're going there. If you're going alone, this is one of those things like never store anything in plain text, right?
00:09:03
Speaker
And I think next, I think we had three different things around for KubeCon. So as Ryan said earlier in this episode, right, we are maybe a month away or less than that.
KubeCon Highlights and Events
00:09:12
Speaker
Wow. Less than that. I think it's our 30th already today. We are doing like a few things. Ryan and I are actually doing a panel with previous Kubernetes by its guests. So we'll have Patrick from DataStacks, Gabriel from EDB, Ring from VMware, and the two of us.
00:09:29
Speaker
We'll do a panel for a day zero event called Data on Kubernetes. So again, if you are going to be at KubeCon, if you are looking for things to do on the Monday, day zero events, Data on Kubernetes is a great way to spend the day, learn about all different things. And bonus, you will maybe see a lightweight panel discussion between people who have already spoken to each other and not meeting everyone for the first time.
00:09:54
Speaker
Yeah, exactly. And the DoK community is such a great one. Bart does an awesome job with the rest of the team. And I think that talk, our panel, I should say is like one o'clock or something like that. So yeah, come check it out. Even if you get in Monday and you're in the morning and you want to come over, please head on over. Nice. And then I think, Ryan, you had something, right? And you didn't even disclose this in the news section, so go for it.
00:10:20
Speaker
I have a one bullet point here. I still have to get the link. But I've been kind of ramping up here at Dell. We're doing something fun at KubeCon called DevOps Debates, which is really just a fun debate forum that is no more than 8 or 10 minutes where we debate a fun topic.
00:10:39
Speaker
The one I'm going to be participating in and having a lot of fun with is sort of prove me wrong. YAML is the worst thing ever created. If you are a practitioner of Kubernetes, you've likely been using a lot of YAML, reading a lot of YAML, editing a lot of YAML, and maybe maybe you're not and maybe you're ahead of the game and a lot of automation is doing that. But I have a lot of things to say about, you know, how disgusting the sort of errors are when you have issues and how to find them. Anyway, prove me wrong.
00:11:07
Speaker
is the idea and it's friendly. There's no winners, but it's going to be a lot of fun. You might think you won, but yeah, it'll be a lot of fun. It's a friendly sort of community event to kind of meet and greet new folks and discuss fun topics. So we'll put the link in the show notes.
00:11:26
Speaker
Nice. And then one other thing I was also doing at KubeCon is on Tuesday, we have a whole different set of day zero events. We also have something called as a Kubernetes data workshop, which will be like a hands-on day spending time with Kubernetes clusters and how storage and backup and everything works. So if you have been listening to the podcast and are interested on those topics, check that day zero event out as well, something that we are running as part of Portworx, I guess.
00:11:54
Speaker
Cool. Yeah. Highly recommended that that workshop. It is a really good way to kind of get into that space and get your hands on a lot of hands on stuff. So that is the end of our news today. Why don't we jump into today's topic? Again, disclaimer, you know, Bob and I are no security experts. That's totally fine. The idea here is to really put everything in one place in a podcast forum format.
00:12:20
Speaker
You know,
Deep Dive into Cloud Security
00:12:22
Speaker
Bob and I have been in the industry for a while and I've run Kubernetes clusters not on the vendor side and had a lot of this sort of thinking around security, dealt with a lot of PII. So we have experience as well as just, you know, there's so much out there. And when you start to dig into it and realize, I mean, we were making notes for this show, Bob. And it was like,
00:12:44
Speaker
Oh, now we have three pages of notes, right? So hopefully we'll do justice of putting it all in sort of a forum for the show for you. And so let's dive into it. I think the first thing we wanted to start with, which is where do you start in this security journey, right? So, you know, I think a great place to start, especially since we're focusing on Kubernetes here, is we'll put a link in here, which is on the Kubernetes documentation site.
00:13:11
Speaker
They have a concept slash security slash overview, which does a really good job of putting an umbrella over what they call the four C's of cloud data security. And we'll talk about each C hopefully in this podcast. So what I'm actually looking at, I'll describe it because
00:13:33
Speaker
We are an audio-only podcast, but there's a layer cake of those four Cs, right? The innermost layer is code, and we're going to dive into really what all the security best practices and sort of things you can do around code. The next layer is container. So this is everything to do with, you know, how you run containers or runtime.
00:13:55
Speaker
building them, scanning them, everything to do with the container space and the application that's within them when it's running in a cluster, which leads me to that next layer, which is cluster. This is basically any cluster that runs any application that you want to secure.
00:14:17
Speaker
whether that's home built in your home, literally, or if it's on, you know, colo or on the cloud or on prem, whatever it may be, that cluster has to be sort of thought about very, very much in depth. It's probably, I'd say one of the more vulnerable places, but we can dive into about what that is. And then cloud, colo and data center, you know, those,
00:14:44
Speaker
you know, where the clusters actually are. Luckily, we've been doing security in those contexts for a long time. So there's a lot of security. And we're not going to dive into, you know, a ton of details. But we will reference sort of, you know, how you have to think about, you know, the actual infrastructure and cloud and where you're actually building those clusters and using containers and code. So
00:15:09
Speaker
No, I think the cloud security thing that you mentioned right is good because it's not that people are new to public cloud or running their own data centers. The only difference between running everything inside your own data center is that you own end-to-end security whereas in the cloud,
00:15:24
Speaker
We have always heard about the shared security model where the public cloud provider or the vendor that you're working with is responsible for making sure that physical access to the data center is secure and the drives are encrypted where your data has been stored and the access has been restricted. So all of that, again, that already exists in the ecosystem. So as Ryan said, we'll focus more and more time in this podcast on code container and cluster and then just cover a little bit around cloud and the data center level security that you need.
00:15:55
Speaker
Cool. So let's start with code. Let's start with that innermost sort of layer there and talk a little bit about what we should start thinking about when we think about code. So whenever we are thinking about running applications on Kubernetes, right?
00:16:11
Speaker
You have to start with application code and the way developers are putting together their code or assembling the logic of an application, they're using a mixture of open source packages and then obviously writing their own custom code. And open source packages should require or should do something called software composition analysis. So this helps you identify all the different
00:16:33
Speaker
open source packages and vulnerabilities that they might have. This analysis helps you prevent exposing things like secrets by scanning the configuration issues already or it helps you prevent a scenario where you are hard coding secrets or hard coding log locations and stuff like that in your code and then running it in production environment.
00:16:56
Speaker
As you're going through writing your code and building your code locally on your laptop maybe just think about what tools you can use to make sure that you identify what are the different components inside your inside all the different open source packages that you're using as part of the code so i think that's that's my one liner.
00:17:15
Speaker
And thinking about code a little bit more, there's a lot of things you can do when it comes to what Bob said, looking at the code, static code analysis, actually taking snippets of that code and being able to see if there's any vulnerabilities in the actual code itself versus just the packages that use. You can do things like dynamic probing attacks, which is actual running versions of those software that
00:17:39
Speaker
Target common well-known service attacks and those things to see if your code is hardened is a lot of things are laid out and then there's sort of You know
00:17:49
Speaker
cloud native container ways to think about your code base. If you're running a server or a service that exposes a certain port, make sure that build image and that actual software package only exposes that port. That's a little bit into the container space, which we'll get into next in terms of how you actually do that. And make sure your code is secure in terms of if it's opening connections to places, use TLS.
00:18:17
Speaker
do as much as you can there. There are ways to kind of enable containers that aren't ready to do those things. But if you're starting at the code, think about that first, right? Yep. And like, as you said, code basically morphs into the next layer, which is containers and how we can talk about securing these different container images. So
00:18:38
Speaker
Like even when we're talking about the 4c model or if you're a developer listening and you're thinking about your development life cycle like okay you wrote some code on your laptop and next is the build stage so how does that work right how do i make sure i'm using the cicd tools and generating.
00:18:54
Speaker
container images or building container images that are secure so both like when when you're writing code you're obviously committing it to some github repo repo or some github repo not github but this this will eventually kick kick off your build pipeline.
00:19:11
Speaker
And one of the jobs that you should have as part of your CICD workflow is making sure that you scan those images that are being generated. And these images will need to be scanned for any kind of vulnerability. And now, I think, as we are moving forward, and I know we'll talk about SBOM, our software bill of materials, and why that's maybe a needed component going forward, building an SBOM as part of your CICD pipeline gives you the full visibility into the risks of all the applicable third-party libraries
00:19:39
Speaker
and components in those libraries that you are using in your application, right? So it includes like, what are the different packages? What are the different versions of those packages? Like modern tools can also identify what are the vulnerabilities in those packages right now. So if you're using something that you didn't know had a vulnerability, you can fix it as part of the build job. But generating that S-BOM actually helps you
00:20:05
Speaker
make sure that you have a list of different packages in your application code. So if a new vulnerability comes online, you can check whether you are you are impacted by it and then fix it and and run those checks again. So I know I used as bomb as as if that was a very well known term. But let me give you a quick definition, right?
00:20:24
Speaker
SBOM, or Software Bill of Materials, is a complete and formally structured list of components, libraries, and modules that are required to build a given piece of software and the relationship between them. And yes, I did copy it from an official website, so that's why it sounded like one. But I just wanted to make sure people know what it means. And then there are two main standards for generating SBOMs. The first one is Software Packaged Data Exchange, or SPDX and SPDX Lite, or Cyclone DX.
00:20:52
Speaker
Again, going back to our original point that we are not experts, we just wanted to make sure that you are aware of these terms. So you can start looking at it on your own or we'll have future guests to talk about these details as well.
00:21:05
Speaker
Yeah, and I think when we talk about the actual container image as well as scanning, if we take a step back of who's building those and how those things are built, right, definitely dig into how you're building your containers and what that pipeline looks like and where you're kind of pulling
00:21:23
Speaker
other base images from. As we know, with Docker, at least in a Docker file, you choose a base image, you need to scan that base image too. You can't just scan the thing that you've built. This leads me back into the development piece of it because when you're building that image, don't just build an image that works because it works. Think about which I'm guilty of.
00:21:54
Speaker
But really think about what is the bare minimum that needs to be in that image to make your software work. Because that's going to reduce a lot of attack surface. It's going to reduce the size of that thing. A lot of good goes into thinking through that problem. And it boosts how you can easily scan and make sure that thing is secure.
00:22:18
Speaker
A lot of like I know Red Hat does this where they certify container images where they go through a process to make sure they're vulnerability free and kind of supported those kind of things. So, you know, that's something else you can look for if you're building applications.
00:22:34
Speaker
if you do need to pull from or use certain images, you know, see if they have a certified kind of, you know, background, or if it's a well-known image, it's not just like Joe Schmoe on Docker Hub built an image that seems like it could do the thing you want, right? That's probably a red flag. I'd probably put it in the bucket of red flags if you're thinking through that way. I think I like that point, right?
00:23:00
Speaker
making sure that you are only using up like good images that have been scanned by vendors is a great policy. But let's say like we have been talking about how developers are doing this, right? But if you're an operator or on the ops side,
00:23:15
Speaker
What we have seen is there are tools in the ecosystem that allow you to set a couple of things as part of the CI CD workflow or as part of this build pipeline. The first thing is having a list of approved base images. So basically whenever your developer submits any code and runs through the pipeline and they're running these scans against it.
00:23:34
Speaker
the container that are the base image that they're using for your container is not one of the approved page image based images obviously it will generate an error maybe the bill will fail so that the developer can go back change the base image that they're using and submit the task again so as as a security admin or a devops admin.
00:23:54
Speaker
Make sure that you're thinking about this, right? As Ryan said, there are vendors who are signing their images and making sure they have the official ones available. Have a list of approved base images that are available to all your developers. They know where to pull from. And this leads to the next point where there are two kinds of container registries or repositories.
00:24:13
Speaker
We obviously know of Docker Hub and we all know of GCR and ACR, all of the public cloud hosted container registries that are available for you to consume as a service. But if you are building applications and running them, right, think about running having a private registry and running
00:24:29
Speaker
continual scans on the registry. So we're not just scanning those images when there is an actual build job happening. We are continuously scanning the container registry and making sure that all the images that exist inside my container registry are always secure and there are no any new vulnerabilities that are affecting any of my existing images. So day zero, yes, but also from an ongoing or a day two basis, you should be scanning those images that you are using in your application code.
00:24:54
Speaker
And then you mentioned digital signatures on these images. That is becoming kind of a standard way to make sure things haven't been tampered with. You're not running an image that's been tampered with. A note on that is that it looks different depending on the container platform you're using. So there's not like one way to make sure it's signed a specific way. Do a little bit of research about how the platform you're using is making sure containers are signed and make sure you kind of
00:25:20
Speaker
Using those mechanisms is all I'll say I have to be broad about it because it's different for everyone. Oh, yeah No completely agree and then one last thing around CI CD that I had was setting assurance policy So obviously you'll have a list of approved base images for your container images But then you should also have rules in place as part of your pipeline that if people are using non-compliant images
00:25:41
Speaker
or if they are using packages, or if their code doesn't restrict privileges to the least privileged model, if they are embedding any secrets, if any malware pops up in that image, have something in place that continuously checks this as part of the CICD pipeline. Developers like to move fast and break things. I know it's a Facebook saying, but I think that's what they do.
00:26:06
Speaker
So make sure that as that security persona inside your organization, or even if you are a lead developer who is responsible for making sure that any code that comes out of your team is good and it's secure, have these policies in place so that there is always a guidance to fall back on. There is always automation to fall back on instead of having to rely on individual users doing the right thing. So that's it for CICD security for me.
00:26:32
Speaker
Cool, a couple more things of the container piece before we move on, which is, you know, the actual composition of users in your container images, right? When you build an image, you often assign, you know, there's an nginx user or, you know, memcache user, whatever you're building, that has privileges to use whatever is in that container. When you launch that thing, it has to be part of that user, and that's more of the runtime piece that we'll talk about.
00:26:58
Speaker
When you're building container images, think about this. Don't just build a container that has root. Exactly. That's the easy one to avoid. It can cause your images to be a little harder to run, but ultimately in the long run, figuring out to run that thing as a least privileged user, just keeping that concept of least privilege.
00:27:23
Speaker
In cloud container i think it's very important to definitely think about that and then one thing i actually learned as part of this part of this research which is really cool is the idea of run time classes and that that is actually something that was introduced in one dot twenty i believe or sorry it was stable in one dot twenty it's probably been around longer so learning about it but the idea is that different images have different levels of security that you need right
00:27:48
Speaker
So you may run one thing that needs more security and more isolation, and one thing that can use whatever default runtime. And runtime classes give you the ability to say, hey, all my nodes use this runtime, say it's container D. And that runtime can have different classes such as it can use a standard container runtime or something like Kata containers.
00:28:15
Speaker
to run your container. And that uses hardware virtualization to run your container in a more isolated manner. And so you can actually assign a runtime class to your container image when it actually gets run. And that's something that's, I think, probably overlooked a lot of the times. We build out these systems, we often just say like, oh, you just have one runtime and
00:28:38
Speaker
And then it gets launched, how it gets launched. But you do have some configurability there. So definitely take a look at the runtime classes. I was super interested in learning more about that when I was going through this. Next on our list here, since we're moving on from container, we might kind of bring back concepts of container because obviously we talk about containers in a lot of these different layers, right?
00:29:01
Speaker
whether that's in runtime in the cluster, when you're building during development. So we'll probably bring back some of these concepts. So the next part of this layer cake, right? We talked about code a little bit. We talked about container a little bit. The next one is cluster. And I think when people think about Kubernetes and security, they probably think about this layer most, or maybe not.
00:29:24
Speaker
Maybe that's just my personal opinion, given my background. But I think when it comes to Kubernetes, we think Kubernetes clusters, and then we think about how to secure that thing. And there's a lot of information here in this space that we can get into. But I think when we're sort of
00:29:42
Speaker
diving into securing a cluster, there's a really good resource. Again, the security documentation called securing a cluster. It's in docs, tasks, administering a cluster, securing clusters. There's a lot of good stuff in here. We're probably going to kind of scratch the surface here.
00:30:00
Speaker
But I think we'll start with, whether you're using a managed service or not, I think your life's definitely going to be a little bit easier using a managed service because you're letting that managed service or cloud provider, whatever it is, take on a lot of responsibility when it comes to a security container. Whether you feel good about that or not, some shops rather have their hands all over building their own.
00:30:25
Speaker
That's fine, too. But I think on a most basic level, Kubernetes and Cloud Native is all about sort of APIs and API driven, right? And we have to first think about who has access to those APIs, whether it's a brand new Kubernetes cluster or you've been running it for a while. Are my APIs secure? Can we have a bad actor kind of go ahead and attack those APIs, meaning
00:30:53
Speaker
you know, life rule, have some sort of authentication authorization around those APIs. Do not leave them wide open on the internet. That's how you get Bitcoin miners in your clusters.
00:31:08
Speaker
Which which we've seen before and it still happens all the time I don't remember we referenced a report on this podcast once of like How many miners were actually or or how many API's were on the internet right that we're all in our communities ones, right? I think we talked about that a lot
00:31:23
Speaker
Even though I think the report you're referring to had a really inflated number, millions of Kubernetes API servers are available. Eventually, that number was, again, still considerable. It was around the 700 to 1,000 mark, if I remember correctly. But it was not a million, but still, there are 1,000 clusters available. So if you have some Black Hat hackers listening to this podcast, please don't use that information. Or do, and they'll learn the hard lesson.
00:31:53
Speaker
And so that kind of leads us to sort of the authentication authorization mechanisms. And there's sort of a lot. I mean, we can talk about, I think, on this podcast, what Kubernetes gives you by default. We're not going to go into what every vendor can tack onto your cluster. But if you're kind of building or using Kubernetes, right, there's a lot of things that you can do by default. Hopefully, if you're building a Kubernetes cluster,
00:32:19
Speaker
API authentication is going to be an obvious one in terms of how you connect users to that. Whether that's local users with passwords and it's a dev cluster, do something. Or if it's a full-blown, larger cluster that's connected with OIDC or LDAP or those kinds of things that's tied to corporate groups.
00:32:40
Speaker
make sure all that is being authenticated and infrastructure has a way to do those things. I think a no brainer, but definitely something worth talking about. No, agreed. And I think when I was like, not for this podcast, when I was beginning my tech career, I always got confused between authentication and authorization. So I'm just going to
00:33:00
Speaker
Sure. Yeah. So authentication is the ability for you to allow users to log in. Like as Ryan said, right, you can have a set of user credentials if it's just a dev cluster or integrate with external identity management systems, such as a YDC or a lab or things like that. And make sure that only specific users or groups of users can access your cluster or access your resources. And authorization is comes after that.
00:33:26
Speaker
Like once your authentication is done, that's when authorization comes in and then a specific user can have different sets of permissions inside your Kubernetes cluster. So like one of the things that we have, I know like Ryan and I have worked on in the past is like once you have authenticated, you might have read only access to all the persistent volumes or you might have the ability to create persistent volumes. So all of that is falls under authorization. So again, something to make sure that you're using best practices. This is not something that's new in the ecosystem.
00:33:56
Speaker
authorization authentication have been around, but think about how you'll morph it or modify it for your Kubernetes clusters. Yeah, absolutely. That's a good way of thinking about it for sure. So authorization, Kubernetes gives you RBAC, basically, do this. And that stands for a rule-based access control. And that's integrated. It's a big part of what Kubernetes is. Yes, it can be a pain to work with, but it's an integral part of making sure
00:34:25
Speaker
sort of there's a separation of, you know, of what users can actually do as Bob mentioned before, it's actually what they're authorized to do in there. Meaning that, you know, do they have, you know, it's fairly, I think, straightforward in terms of the language it uses, right? Can you actually create things, delete things, get things?
00:34:46
Speaker
list things, right? All those things are configurable within RBAC policies tied to users and services accounts, right? Kubernetes pretty much has those two general high level roles, which is a user of Kubernetes, whether that's a admin or just a dev or that kind of thing.
00:35:03
Speaker
or read-only user, and then service counts, which typically are tied to some non-human thing that needs access to things that I think last a little longer, I think the details, but know the difference between those two.
00:35:19
Speaker
And then different objects can also have RBAC policies, not just users, right? A node can obviously be something that interacts with other components of the cluster. So dig into RBAC. It could be an entire podcast series going into RBAC. We're not going to do that here, but definitely dig into real-based access control. Make sure you're using the policies thoroughly. It's out of the box. It's there for you. It works really well.
00:35:48
Speaker
Definitely dive into that. Agreed. Kubernetes did have some things that it shipped with in terms of RBAC. If you're thinking about, maybe you've heard these terms, if you're just starting new with Kubernetes, if you are already using it, you might already be using these terms, but you have roles and you have role bindings that allow you to set a list of permissions to specific objects and give those permissions to specific users.
00:36:17
Speaker
or objects. So that's a role. And that happens at a namespace level. And then if you have also heard cluster roles and cluster role bindings, that happens at the cluster level. So even when you are configuring these set of permissions, if you're running a multi-tenant Kubernetes cluster, maybe think about roles and role bindings versus cluster roles and cluster role bindings. It's something to differentiate the level of permissions that any authenticated user might have against the resources that you have inside your cluster.
00:36:44
Speaker
And then I know we covered this in the GitOps episode that we did a few weeks back. If you are following the GitOps methodology, only that one piece of operator and custom resource should have access to your cluster. You shouldn't give access to any users that are trying to make any changes to your Kubernetes. So you can always follow different security methodologies when you're dealing with Kubernetes.
00:37:12
Speaker
Yeah, absolutely. And there's a lot more information here. We'll put a link into the actual authorization documentation that Kubernetes has. There's other methods that Kubernetes uses. RBAC's a very common one, probably the one you'll interact with more, but there's a back, there's no, there's a bunch of other ones. So dig into those for sure. I think
00:37:35
Speaker
Sorry, I think since we're still talking about cluster, I also wanted to talk about something that Ryan referred to earlier in this part, where you can either be deploying clusters on your own, regardless of the location or the infrastructure stack that you're using, or you might be using something that's offered as a managed service. But the way you're deploying this might be through using some sort of automation.
00:37:58
Speaker
You might be using some infrastructure as code template. You might be using some Terraform scripts, Ansible scripts to do these. Keep in mind that you also have to test those scripts out or run checks against those scripts, against those templates because you can introduce
00:38:16
Speaker
misconfiguration at scale in an automated way if your templates are messed up. So the ISE templates or infrastructure as code templates must also be scanned for any misconfiguration. So issues can be resolved earlier rather than fixing it on day two or fixing it live in production. You don't want to be changing the tires of an F1 car when it's going around the lab. You want to do it before the game has started. That was a really bad analogy, but hopefully it got the message through.
00:38:45
Speaker
I got the message. Don't worry. I've heard like changing something while the plane is in the air. I was like, they don't change tires off a plane. So I don't know where I'm going with this. I think, you know, basically building the plane while the. Yeah, I've heard. Yeah. Thank you. Thank you for helping.
00:39:11
Speaker
Yeah, so getting back to the cluster a bit, there's still a lot we can talk about, one of which is we've talked about the APIs around it, the users, the access, the actual nodes themselves that run containers, the kubelet is a very powerful piece of the cluster, it runs the actual containers, it has access to the node.
00:39:32
Speaker
The docs still say that by default, kubelets allow unauthenticated access to its API for whatever reason. I think for a lot of reasons is you don't need a lot of the complex setup if you're doing dev and things like that. But if you are thinking about security, make sure to do kubelet authentication and dig in there as well as taking the next step in your kubelet is
00:39:58
Speaker
controlling the capabilities that the workloads actually get to consume when they're run by the Kubelet. So this is a big part of what the Kubernetes and container security is all about.
00:40:13
Speaker
which is you're building it. So we've talked about code, you've written some code, you think it's secure, you build a container, you think it's secure. Now you need to run that thing in Kubernetes. So you've secured your cluster, you have the right access with users and our back and all that thing. You've deployed your containers to a kubelet. What can it actually do on that kubelet? There's so much that can be configured in that node in terms of how that container runs.
00:40:37
Speaker
And so there's everything from resource quotas, meaning how much CPU and memory an individual container at runtime can use, so you can limit those things. It may not seem like a security thing, but you don't want a container to be able to just run rampant and use up the whole node. You could continuously do that for sort of a
00:40:59
Speaker
DDoS kind of attack, as well as you can use limit ranges, which basically put a max and minimum of resources that a user can consume tied back to that user concept in
00:41:17
Speaker
in kubernetes and then there's there's a whole slew of pod based security that you can do so kubernetes uses pod based security context right pods can run one or more containers this is a very common sort of terminology
00:41:34
Speaker
But there's a lot of tools that you can use within that pod, such as the user ID that's being run by the container, the group ID, you can configure SELinux or configure the Linux capabilities through SecComp, and things like App Armor. There's a lot of things that we can dive into, but know that those things exist. I think SecComp and App Armor, they've been around for a while.
00:42:04
Speaker
And they've enabled you to be able to secure sort of the way that a compute node can be run with the Linux kernel. So giving limited permission to certain calls and defining a profile of what can actually run. You can tie those things to pods containers and there's a whole world of information
00:42:27
Speaker
in configuring those and understanding those, which we won't go into here. But they're very important and they're a big part of how a container actually runs and gets access. Because remember, in most container runtimes, we share access with the kernel for everything. So it's not as isolated as virtual machines and something like Kata. So we have to make sure we're thinking through those things because
00:42:55
Speaker
You know, a worst case scenario is a bad actor breaks out from container gets access access to the host, right? And then therefore everything on that host and probably maybe even your cluster is now at risk. So definitely dig into those.
00:43:10
Speaker
No, thank you for bringing up the container breakout attacks, right? I think the way Kubernetes was helping people enforce these policies was using the concept of pod security policies or PSPs. And it was a really important security feature in Kubernetes for a long time. It helped users set limits in place so that even if someone breaks out of that container, can't access the cluster node.
00:43:35
Speaker
There have been some changes in the way around PSP. PSP or POD security policies were deprecated in Kubernetes version 1.21. If you looked at the release notes for 1.25, they have been completely removed. There is a replacement for POD security policy and it's called POD security admission. This is an entry replacement.
00:43:57
Speaker
So if you're using PSP, now it's PSA or Pod Security Admission. This feature takes a different but simpler approach of applying these restrictions. So instead of the user having to configure a custom set of rules and of course
00:44:14
Speaker
on permissions for containers running on nodes. Now I think pod security admissions has three different levels of pod security standards or PSS, which can be applied as part of any pod on a namespace by namespace basis. So something to think about, right? Like PSP is no longer around. So if you're using it, make sure whenever you upgrade to 1.25, you change that up and use one of the three pod security standards using pod security admission.
00:44:41
Speaker
Yeah. And this is a really powerful tool, right? Again, you can do a lot of stuff like this, like make sure containers don't load kernel modules that are blacklisted, those kinds of things. Anyway, we won't get into those details. I don't know if others are guilty, but I'm definitely guilty of like, if you've ever built a container and you're dealing with security issues and maybe you're
00:45:04
Speaker
You just need to think to run because you're doing some development work. And so you just make it privileged. You just give the thing the full access to a privileged container. That's a perfect example of what not to do. In most scenarios, I would say if it's just for a super non-sensitive development thing, fine. But it's a perfect example of the opposite end of the spectrum of what you
00:45:30
Speaker
shouldn't do, right? That gives full permission. Privilege does solve a lot of problems when you run into trying to run a container that's, you know, hitting, you know, security policies, something like that. I don't know if you've ever used OpenShift, Bobbin, and, you know, you just want your container to run, but it takes you for, you know, for 40 minutes, I'll say, to plus to figure out the security policies. But it's really important to actually go through those, especially for production. I mean, a lot, a lot there.
00:45:57
Speaker
Okay, that's why I'm not running anything in production. I'm just doing dev tests. I'm just doing demos. Because I don't enforce all security policies all the time. So I should start doing that even for demos. Yeah, we all should. Yes. I think next let's go to our strong points, right? Like talk about persistent storage, talk about encryption, talk about ransomware production, I think.
00:46:21
Speaker
That's something that we can be kind of experts on. So I think just to start the discussion of whenever you're running anything on Kubernetes, make sure that you're using encryption at rest and encryption in transit. Encryption at rest basically means whenever your application is writing some data,
00:46:37
Speaker
storing it in persistent volumes. Those PVs are encrypted on the underlying storage. If you are transferring data between clusters, if you're creating backup jobs and shipping snapshots off to an S3 location, make sure those are encrypted as well. Make sure that any communication between your nodes is encrypted, so encryption in transit for those two scenarios. Encryption actually plays a really important role. Even if somebody gets access to your cluster, they can't decrypt your persistent volumes and
00:47:06
Speaker
Maybe look at secrets in plain text, which is something that we discussed earlier in the part Exactly, right crips at risk in transit definitely big things in the storage space I think you know just if you're using sort of a CSI mechanism or persistent storage provider I definitely look at making sure that That thing is also secure right you have Kubernetes itself has a bunch of API's but then you also have now the storage and
00:47:34
Speaker
system which could have its own sort of attack surface and how you interact with those APIs. That's something that's probably often overlooked. You kind of connect this thing, have the driver woohoo. You're ready to go. But make sure that it's a whole different attack surface that definitely can be
00:47:52
Speaker
restricted in terms of what kind of security is around those APIs and everything. There's some basic things in Kubernetes that also with storage, you can configure access modes to volumes, make sure that you have the right access mode configured for your use case. As far as file system group ID and those things, Kubernetes by default does a lot of things like kind of turn a bunch of files to the user based on those things and
00:48:21
Speaker
Um, if you, if you often, um, connect a volume, sometimes it's just everything's root. So, you know, things might not work. So pay attention to those mechanisms and Kubernetes, uh, in terms of.
00:48:33
Speaker
the user of the file system itself. Yep. Okay. And then talking about ransomware protection, immutability having using object lock enabled backups, all of that really helps out. Like even if you get attacked by ransomware attack, you might be vulnerable or your primary cluster might be vulnerable. But if you have an immutable copy of your application somewhere,
00:48:57
Speaker
you can restore from that copy and the reason it's like the reason you should be mutable and not just a standard backup is even if the hackers try to manipulate your backup snapshots which they do like if you look at reports out there they do try to mess with your backup jobs make sure that they are immutable so that if you get hit by one you can spin up new environments maybe you spend a new a w s account completely.
00:49:20
Speaker
It's been a very case clusters and i'm using it as just an example but you should you can restore from these immutable snapshots and bring your applications online so having that layer of protection definitely helps securing your organization's estate or securing it organizations infrastructure.
00:49:37
Speaker
Yeah, definitely something that you want to think about is having that backup for your cluster. Because as best of an effort we can do to secure all these clusters, sometimes there's holes that we don't know about. So having that, especially in certain scenarios where you have very
00:49:56
Speaker
sensitive types of data or a very large customer base that you'd lose lots of money for doing that. You want to make sure you offload a lot of your data and back it up. So certainly a very important piece of this puzzle.
00:50:15
Speaker
Well, we hit on a lot of these different layers. I think the one we didn't fully talk about, and granted we didn't yet get to things like networking security or what if you have PII information in there. We probably won't get to that today.
00:50:33
Speaker
But that last layer of the cake is the cloud colo and data center, right? So there's a ton of good resources, whether you're on AWS, Google, IBM, whatever cloud you're on, right? All those platforms.
00:50:48
Speaker
have sort of their own security center or trust center, whatever you want to call it, and best practices, right? This, again, you know, we're not going to dig a lot of focus a lot on sort of the different clouds and what you can do with there, but definitely follow those best practices. What we will focus on a little bit is in terms of like the infrastructure nodes that are sort of
00:51:12
Speaker
what you're running. And the one thing I definitely wanted to dive into is sort of that node and host level type of thing. So if you're running your own container platform or building it or running a Kubernetes cluster, there's a lot of things that a managed service will do for you that you might take for granted, like making sure your node operating system is up to date and your kernel is up to date. If you let those things go too long, that could open vulnerabilities.
00:51:43
Speaker
Container operating systems built for these types of things, like Bottle Rocket, or Photon, or Red Hat OS, which is Core OS, they have read-only root file systems and things that are more tuned to running container platforms. I think those are really interesting ways that enable less of an attack service than if you were to use a full-blown operating system.
00:52:09
Speaker
Uh, definitely something to look at as well. And, and then just, we mentioned it before things like lightweight container, uh, runtimes, you know, we mentioned cat up before there's a bunch of them out there, but this ultimately is something you want to think about if you really have high security concerns of what, you know, what your actual container, um,
00:52:30
Speaker
applications are running and you really want to secure it. Definitely look into those sort of other options that you have at that node and host level. But I think a lot of the node and host level things, Bob, and I don't know if you would agree with me, but these things, you know, and assist Adam in space have been around for a long time. And there's a lot of resources about, you know, what you can do to make sure your node
00:52:49
Speaker
and everything is secure first because that's obviously a good place to start. No, I completely agree. The reason we didn't spend as much time on that cloud layer was because people have been using it for a while. So again, if we hear feedback that, no, we need to spend more time, we'll obviously come back and do a detailed episode. But I think we have enough content for this podcast. I just wanted to bring one more thing after you finish talking about cloud security.
00:53:14
Speaker
Yeah, I'll end there because I know we've covered a lot already. Some things we didn't really dive into is networking. There's definitely a whole world there. So all I'll say is that there's a lot of tools and mechanisms you can use to secure your network in terms of specific network policies or making sure you use TLS or
00:53:36
Speaker
different network providers that provide different monitoring capabilities. Of course, something we also didn't mention is ongoing monitoring and auditing. Big piece of the puzzle here for security. Kubernetes doesn't do a whole ton here, but it does give you an auditing feature that you can definitely drive home and make sure you set up and continuously monitor.
00:54:01
Speaker
I'll leave it over to you, Bob. Yeah, and to close that point, I think Kubernetes does offer a basic set of capabilities, but you shouldn't stop there. If there is something that's missing in Kubernetes, go and look at the vendor ecosystem. The reason CNCF has a huge
00:54:17
Speaker
I know everybody likes to scare, whenever we are doing presentation, we show that CNCF landscape, show all the different open source projects, show all the different vendors that are participating. The reason people are around is because Kubernetes does give you that baseline of capability, but it doesn't do anything and everything that you might want to do inside your organization. So look at different open source projects, look at different paid enterprise offerings that can help you secure your applications better.
00:54:41
Speaker
One last thing that I wanted to do, it didn't fall into any of these pockets, but it's about implementing that red team, blue team philosophy inside your organization. Again, if you're a smaller org, you might not have enough people to form a whole team, but it might be just a red person, blue person.
00:54:56
Speaker
the ability to run penetration testing against your Kubernetes clusters. And there are tools out there. During my research, I found something called Cube Hunter that's available as an open source project that you can use. And again, they have disclaimers throughout their GitHub page that please don't use this tool against Kubernetes clusters that are not your own. Don't try and exploit different clusters using our tool. But it's a tool that's available that helps you run penetration testing across all the different layers that we just discussed.
00:55:26
Speaker
It runs in a couple of different modes so by default it runs in something called as passive hunters mode where it runs a series of tests identifies potential vulnerabilities inside your cluster but doesn't really exploit them if you are if you turn on the active hunting mode with my passing on the active parameter.
00:55:43
Speaker
This enables this cube hunter to run additional set of tests. So it will run the passive tests, identify the vulnerabilities, and try to use those and run additional tests using those vulnerabilities that are open, giving you an indication of what an attacker might be able to do. So this level of penetration testing is something that you should be doing on an ongoing basis, and it covers things like initial access.
00:56:07
Speaker
accessing your cloud credentials, accessing your Kubernetes cluster, accessing your kubeconfig file. Two things like if I get access to your cluster, can I exec into a specific container? Can I start a new container? All of those things are important. Can I create a backdoor container? Can I escalate permissions from a container and go to that node level? Things that we discussed in this part.
00:56:28
Speaker
How do I try? It also helps you perform penetration testing against your authorization and authentication policies. Can I access your Kubernetes dashboard? Again, there are so many things. They have a cool matrix that we'll put in our show notes or link to it in our show notes. But make sure that if you're just getting started with Kubernetes security, have a person in your team or a team of users
00:56:50
Speaker
That is putting on this hat and trying it out. You should be testing or pen testing your own systems to make sure that you are secure from external attackers and you're fixing things before it becomes an exploit that external audience can use against your clusters. And with that, I think I've covered my entire list of things. I know this has been a really long podcast of just me and Ryan talking, but that's with this specific space, right?
00:57:15
Speaker
It is such a huge ecosystem and different things that you can do. It's really important to at least just bring these things out so you start thinking about it. And then as Ryan said in the beginning of this part, we'll definitely try to get more Kubernetes security guests on the part as well so we can focus on individual layers as well.
00:57:33
Speaker
Absolutely. We'll end it here and I want to say there's so much in this space. I think what we'll actually end up doing is something a little different. There's so much information we have in our show notes. We might just post an external Google Doc where if the community wants to check it out or add to it, those kind of things might be
00:57:55
Speaker
Something we can do so look for that again will be a cubecon in a couple weeks. It's it's right around the corner. Definitely let us know if you're going to be there. We'll be talking to the folks from a stack OS. If you're into sort of distributed blockchain stuff and how they use Kubernetes in a couple weeks will have that episode coming.
00:58:15
Speaker
And I will say that, you know, in summary, definitely start with the four C's of cognitive security and that will let you branch out into all these different things. And a lot of things we didn't touch on today. We know we didn't touch on every aspect of it, but hopefully this podcast gives you sort of a sense of everything that you need to start thinking about.
00:58:35
Speaker
And Ryan said, if you are at KubeCon, come say hi, grab a few stickers. Yes, we have stickers. Yes, we'll have stickers. Come say hi, grab a few stickers. And then if you have interesting stories to tell, we'll be more than happy to have you on the podcast for that bite section. But I just wanted to make sure that people know where to find us. So we'll be definitely doing the data on Kubernetes Day. We'll have a panel there.
00:58:59
Speaker
But then during the show, we'll be at our respective vendors. So I work for Portworx. So I'll be hanging out at the Portworx booth. I know Ryan now works at Dell. So I'm pretty sure he'll be hanging out at the Dell booth as well. So come and find us. Or if you just see a couple of people in a corner somewhere with a couple of microphones trying to record something, more likely than not, it's just the two of us trying to record something in a corner.
00:59:24
Speaker
Awesome. Yeah. All right. Well, that brings us to the end of today's episode. I'm Ryan. I'm Bobbin. And thanks for joining another episode of Kubernetes Bites. Thank you for listening to the Kubernetes Bites podcast.