Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Kubernetes Cluster API 101 with Scott Lowe
520 Plays2 years ago
In this episode, Bhavin interviews Scott Lowe, a Principal Field Engineer at Kong. The discussion dives into what is Cluster API, how it works, and how it helps users deploy Kubernetes clusters at scale using simple YAML files. We also talk about the different components of the Cluster API project and discuss the new features introduced by the community.
Show Notes:
1. Scott Lowe -
https://twitter.com/scott_lowe
Blog - https://blog.scottlowe.org/
Podcast - https://packetpushers.net/series/full-stack-journey/
2. Cluster API - https://cluster-api.sigs.k8s.io/
3. Kustomize with Cluster API - https://blog.scottlowe.org/2021/11/01/using-kustomize-components-with-cluster-api/ https://blog.scottlowe.org/2021/10/11/kustomize-transformer-configurations-for-cluster-api-v1beta1/
News:
- Grafana lab adds K8s monitoring for Grafana Cloud: https://www.dbta.com/Editorial/News-Flashes/Grafana-Labs-Releases-Kubernetes-Monitoring-for-Grafana-Cloud-153953.aspx
- Kubeshop acquires majority stake in InfraCloud’s OSS product BotKube: https://www.cxotoday.com/press-release/kubeshop-acquires-majority-stake-in-infraclouds-oss-product-botkube/
- Kyverno moves to the CNCF Incubator: https://www.cncf.io/blog/2022/07/12/kyverno-moves-to-the-cncf-incubator/
Recommended
Transcript
Introduction to Kubernetes Bites
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts. We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:30
Speaker
Good morning, good afternoon, and good evening wherever you are. We are coming to you from Boston, Massachusetts, and today is July 20th, 2022. Hope everyone is doing well and staying safe with the heatwave that's going around different parts of the US and even Europe.
Introduction to Clustered API Discussion
00:00:48
Speaker
For today's discussion, it's going to be me, but we have a great
00:00:54
Speaker
a guest to talk about an interesting topic, which is Clustered API. Before we dive into today's topic and introduce our guest, let's see what's happening in the Kubernetes ecosystem.
Industry News: Kubernetes Monitoring and Acquisitions
00:01:05
Speaker
A few news to discuss today, Grafana Labs, people in the Kubernetes ecosystem or even outside are familiar with Grafana as a tool that they have been using for monitoring and using the dashboard functionality.
00:01:18
Speaker
In the past couple of weeks, Grafana Labs has added Kubernetes monitoring support for Grafana Cloud. So now, all you need to do is if you want to leverage the dashboarding and monitoring functionality from Grafana, you need to install your Grafana agent on your Kubernetes cluster. And in minutes, the kubestate metrics will be shipped to the Prometheus and Grafana Cloud instance. And from there, you have out of box access to your Kubernetes infrastructure metrics, logs, and events
00:01:46
Speaker
which are along with the pre-built dashboards that Grafana Cloud has for you. So monitoring your Kubernetes clusters and the apps that are running on top of it just got easier by leveraging this Kubernetes monitoring capability from Grafana Cloud.
00:02:01
Speaker
In addition to this, a couple of updates around open source projects. So CubeShop, which is an accelerator for open source startups, has acquired a majority stake in a very popular Kubernetes product or project called BotCube. So BotCube is an open source.
00:02:17
Speaker
chat ops based product that's developed by a vendor called InfraCloud. And chat ops is basically helping our bot cube is basically helping customers monitor their Kubernetes cluster and then help them with interactive debugging. Again, we'll have links in the show notes where you can learn more about what they do. And if it's an interesting use case for you to try out in your Kubernetes clusters.
Kyverno's CNCF Incubation Graduation
00:02:42
Speaker
And the third piece of news that we had was around Kyverno. And again, I know we have discussed Kyverno when it was added to the CNCF sandbox projects today. The news came out that Kyverno has now been moved to the CNCF incubator section. So I really like Kyverno because since it's open source, we have heard it from a lot of customers that
00:03:05
Speaker
They already use it for enforcing security policies across your Kubernetes clusters. But the three features that I really like about Kiberno is, one is the admission controller webhook, which basically because of this integration, any admission requests that come from the Kubernetes API server, it monitors these and applies configuration best practices and policies, and even assigns certificates and web configurations.
00:03:30
Speaker
Next is PolicyReporter, which is a neat UI, graphical user interface for policy reports, where you can just look at all the different policies that exist and see how your cluster compares against those. And then they have a library of more than 180 ready-to-use policies that include popular ecosystem tools and platforms. So Kiberno is now graduated to the CNCF incubation phase, and then we'll obviously keep monitoring this project to see as it moves up.
Guest Introduction: Scott Lowe from Kong
00:03:58
Speaker
But that's it for the news this week. Let me introduce my guest before we have Scott join the podcast. Scott has been in the IT field for more than 25 years and currently works as the principal field engineer for Kong. If you have ever worked with VMware, I'm sure you have come across as blogs, books, podcasts, conference talks. I know I've personally read most like a lot of his blogs and I've read the Mastering vSphere book as well. Scott has been doing these
00:04:29
Speaker
I've been following Scott's blog for at least five years, learning more about software-defined networking, Kubernetes, and the topic for today, which is Clustered API. So without much delay, let us get Scott on the podcast.
Scott Lowe's Role and Professional Journey
00:04:43
Speaker
Hello, Scott, and welcome to Kubernetes Bites. It's great to have you on the show to talk about Clustered API. Let's start by having you introduce yourself to our listeners and talk about what you do at Kong.
00:04:57
Speaker
Yeah, sure. So thank you for having me on the show. I appreciate it. My name is Scott Lowe. I live in the Denver, Colorado area. So been here probably over a decade or so. Been in the IT industry for a while, had the opportunity to do a lot of different things. I've been fortunate enough to publish some books, wrote a couple of very popular books on vSphere back in the day.
00:05:23
Speaker
working on another book now that we can talk about later if you want. Been blogging for a very long time. A lot of people know my website, blog.scotler.org.
00:05:34
Speaker
I've had the opportunity to speak at a number of different conferences or user group meetings, that kind of thing. I've always enjoyed doing that and getting a chance to meet folks. And I run my own podcast too. So I'm part of the Packet Pusher's Networker podcast. I run a podcast called the Full Stack Journey podcast, which is all about helping people learn the, let's see what's our catchphrase, the full stack of technologies that are present in today's data centers, I think is the catchphrase. So anyway,
00:06:02
Speaker
And as you mentioned, I'm currently working at Kong. I'm a field engineer there, which just means that I help customers with product implementation focusing on service mesh.
00:06:15
Speaker
Like I said, I've done a bunch of different things. So I've done networking. I helped bootstrap the NSX business at VMware as part of the Nicira acquisition. I worked with a lot of great people there. Left there, went to a small company called Heptio, which then got reacquired by VMware. And then we helped bootstrap parts of the Tanzu business, which was great. And so now I'm on to another startup, this time working on service management.
00:06:40
Speaker
Awesome. Thank you for that introduction. And personally, I've been following your blog, and I think it was 2015 or 2016 when you did the Charlotte UserCon keynote. So I saw you there in person. That was great. And I've always liked the technology
00:06:59
Speaker
I forgot what it's called. Tech shots or something like that on your blog, right? Right, right. The technology short takes. Yes, short takes. Yes, thank you. Short takes are always useful and I've gone and discovered other bloggers because of that. I'm glad that I was able to get you on the podcast to talk about cluster API.
Understanding Cluster API
00:07:20
Speaker
Let's just talk about what is cluster API and how did you get started with it, right?
00:07:25
Speaker
Yeah, sure. So I think about cluster API as the answer to the question, what if we were using Kubernetes to manage the cluster lifecycle for Kubernetes? At the heart of Kubernetes, there's this core reconciliation loop, which is responsible for evaluating the world as it is, i.e. the actual state, and the world as the operator or the user wants it to be, which is the desired state.
00:07:55
Speaker
And it uses that core reconciliation loop for everything that happens in Kubernetes. And so if you think about it, well, we could take that core reconciliation loop and we could apply that to the creation of other Kubernetes clusters. So we could say, OK, my desired state is that I want a cluster with three control plane nodes and three worker nodes running this version. And then apply that, give that to the reconciliation loop and say, OK, go make this happen.
00:08:25
Speaker
And at a very high level, that's essentially what cluster API does, right? There's a set of custom resource definitions and associated controllers that are responsible for managing the objects that are used by cluster API.
00:08:39
Speaker
And as a user or as an operator, you define what you want the desired state of your new cluster to be, and it will go off and it will make that happen. So it's actually really cool. I first got started using cluster API at Heptio. This was prior to the VMware acquisition, and Heptio was doing a bunch of work in the cluster API space early on, and that continued after the VMware acquisition.
00:09:06
Speaker
So I got involved early, was testing it, using it, providing feedback, that kind of thing. A little bit later on, I had the opportunity to do some documentation contributions to some of the cluster API docs and primarily been focused on using cluster API on AWS. So most of my work is there. So yeah, that's it.
00:09:27
Speaker
Cluster API has been a cool project. I think I got introduced to it when I was working with Google Cloud Anthos and when it was officially launched, it only supported vSphere as the provider. That's how they did things. They had their management cluster and then using cluster API for VMware or vSphere and deploying additional VMs on top of your vSphere cluster and
00:09:52
Speaker
provisioning the actual workload cluster that Anthos has and connecting it back to their cloud control plane. So a cluster API has been a cool project. And I think if I remember correctly late last year or maybe early this year, that's when they graduated and even released the 1.0 version. So even to get to 1.0, they had so many different companies already using it internally. And then having a mature project out in the ecosystem is great.
00:10:17
Speaker
So my next question is like, okay, cluster API is great. How does it work? How do all of these different things, how do we append the functionality that Kubernetes already provides and provide this reconciliation loop for deploying and managing Kubernetes cluster itself?
00:10:35
Speaker
Yeah, yeah, that's a great question. So some of it we kind of touched on already, right? We're like, we have this idea of a custom resource definition or CRD, which we know is the way that you extend the Kubernetes API to be aware of new kinds of things, right? And then along with those custom resource definitions, there have to be associated controllers
00:10:56
Speaker
which manage the reconciliation loop for that type of object because the life cycle for a pod might be very different than the life cycle for a cluster, for example, right? And so they kind of need to know, um, the intricate details of that particular type of object. Um, what, one of the things that the cluster API, um, sort of contributors and designers and architects, and I, I'm coming at this just by the way, strictly as a user, like I'm, I am not a programmer. I'd love to be one day, but I'm not right now.
00:11:24
Speaker
But I had a lot of interactions with folks who were sort of leading this charge and kind of architecting it. One of the things they really thought about was, OK, how can we make this extensible? Like, how do we give you basic functionality that you can use, like right out of the box, right? But how do we also make it extensible so that we aren't sort of locked into any one thing of way of doing things? And one of the things that I saw emerge, and this was
00:11:48
Speaker
know, sort of very early on when the APIs were like B1 alpha 1. So the very initial release of the APIs going from B1 alpha 1 to B1 alpha 2, we saw them begin to implement things that helped them maintain the separation. And so what they have now is this
00:12:04
Speaker
the set of logical objects that sit at the core of cluster API. So we have things like a cluster or a machine or a machine deployment. And these are logical objects like they just described at a high level. This is what I want this thing to be or to do.
00:12:23
Speaker
And then each of those things is then connected in some way to an implementation specific mirror of it. So I'll talk about the AWS version, but it's the same if you're looking at vSphere or if you're looking at Azure, one of the others.
00:12:39
Speaker
So you have this machine and that machine is the high level logical construct. And so whether you're running on AWS or Azure or vSphere, you're always going to have a machine, right? But underneath that machine, you'll have a provider and we'll come back to the term provider in a moment, but an infrastructure or platform specific version. So you'll have like an AWS machine or a vSphere machine or an Azure machine, right?
00:13:01
Speaker
And that begins to capture some of the implementation-specific details, like on AWS, what is the instance type? Are there specific parts of the AMI, like the AMI selection, are we influencing the AMI selection in some way to use a different AMI than when it might choose by default? What region or availability zone is it running in? Those sort of things, right? Because those details vary from platform to platform.
00:13:30
Speaker
Then you have this cluster API core, you have a set of infrastructure components which we call providers, and these are the things that interact with the specific platforms. We have a set of bootstrap pieces, bootstrap providers that allow us to change the way we bootstrap a node. It's the default, if you will, is to use a tool called kube-adm.
00:13:55
Speaker
to bootstrap a Kubernetes node, right? And so the bootstrap controllers will write all the QBDM code and then pass all it off and run QBDM on the node to bring it into the cluster.
00:14:06
Speaker
And then as the APIs evolved, we also saw the emergence of a control plane section where we actually manage the control plane as a separate entity, which allows us to say, oh, I want to describe the properties of the control plane. So I can say, I want the control plane to have one node, and it'll create a control plane with one node. Or I want a control plane to have three nodes, and then it'll scale the control plane to three nodes, right? That part in and of itself is almost
00:14:31
Speaker
For people who have been in Kubernetes for a while, it's almost like magic, right? Like being able to scale that control plane from one node to three nodes sort of on the fly is like mind blown because if you did that in the past, there was all kinds of hoops you had to jump through and all kinds of things. Anyway, so you have these different components that come together and they all tie together. And what you end up with is this big, long YAML manifest that describes everything. So it says, here's the cluster and here's the associated AWS cluster. And the AWS cluster says,
00:14:58
Speaker
here's all the infrastructure pieces that you have. And then, oh, I have a control plane with three nodes and there'll be a template for how the nodes are created. Oh, and I have a machine that looks like this and so on and so forth. And all of these, again, are represented by custom resource definitions on what we call the management cluster, which is a Kubernetes cluster itself. And we'll talk about the sort of like inception loop of like using a Kubernetes cluster to manage a Kubernetes cluster.
00:15:27
Speaker
At some point you have to start that loop, yes. Right, right. We've got to start the loop. We'll talk about that in a moment. But all of these things exist as custom resource definitions on the management cluster. And that's where the controllers run. And so when you apply the YAML against the management cluster, it'll say, oh, I'm going to go create these objects. And then
00:15:42
Speaker
And that's where it kicks in that reconciliation loop. And so it starts talking to the underlying cloud provider, whatever that is, like whether it's AWS or Azure or talking to your vSphere hardware or your vSphere presence, right? All that kind of stuff. And then it just runs through that reconciliation loop until the actual state of having a cluster as you described it matches the desired state, which is what I described, right?
00:16:08
Speaker
Um, so it's, it's, it's, it's, it's actually really, really cool. Um, if, if I do say so myself, like the idea of being able to say, you know, I want, I want to just describe what I want the cluster to look like, and then it will go do that for you. I know. And I think.
00:16:22
Speaker
I really like Clustered API because I don't have to keep up with the manual instructions on how to do all of these things. I know I can build that YAML file which has all of these different sections, and Clustered API will work with the individual providers and deploy these Kubernetes clusters for me. So talking about providers, can we talk more about EKS Anywhere or Google Cloud Anthos, or even for that matter, VMware Tanzu Kubernetes Grid? I know all of these different distributions rely on Clustered API to provision the underlying infrastructure.
00:16:53
Speaker
Yeah, yeah, yeah. So providers, again, they come in sort of a couple different flavors. There's the bootstrap providers. The default one is Kube ADM. But the idea there is like, if you needed to do your bootstrapping via some other mechanism, then you could put in an appropriate bootstrap provider there. So if you wanted to bootstrap the nodes using something other than Kube ADM, it is sort of modular in that way. We have the infrastructure providers.
00:17:19
Speaker
which again are responsible for talking sort of down to the underlying cloud provider, whether it be AWS or Azure or vSphere or Google or whatever. And that's sort of where you see these
00:17:29
Speaker
these distributions like EKS Anywhere and Anthos and others sort of step in. They're leveraging, in both these cases, the vSphere provider, which allows them to then deploy vSphere machines, which are the objects that you use. And those vSphere machines may come from another, maybe created based on another cluster API object known as the vSphere machine template, which sort of provides the,
00:17:56
Speaker
well, the template for what a vSphere machine look like, right? And interacts with vCenter and all of the underlying infrastructure to
00:18:05
Speaker
create all of those things and then bootstrap them, right? And then there is a provider for the control plane as well, and it's called the QBADM control plane. Theoretically, we could have other control plane providers in as well. I haven't seen any. I don't know if EKS, Anywhere, or Anthos use a different control plane provider. But again, the idea being that as long as you are
00:18:28
Speaker
speaking sort of the high-level logical objects that the core of Cluster API expects, then a provider can then take and implement sort of a provider-specific feature, right? It allows them to extend the Cluster API in ways to either work on new platforms or others. And I hang out in the Cluster API channel on the Kubernetes Slack a pretty fair amount. And so I see in there people working on Cluster API providers for Oracle Cloud Infrastructure, for example.
00:18:57
Speaker
and others. I think that's one of the real values of the very pluggable and very modular approach that the maintainers took is that it does allow us to plug in these other pieces and extend the functionality of the core project.
00:19:12
Speaker
Yeah, and it's great, right? Because let's say I got started with using Clustered API on-prem on a vSphere cluster. I don't need to understand how my AWS EC2 instances should be deployed. I can just leverage a different provider and Clustered API will do that thing for me. So it adds that scalability component and while reducing the amount of relearning that has to happen just because I want to deploy Kubernetes against a different infrastructure platform.
Initiating Cluster API Loop
00:19:36
Speaker
I want to go back to the how do we start the loop question, like what's the first step, and then let's say how we can kick that reconciliation loop into action.
00:19:46
Speaker
Yeah, yeah, sure. So typically what happens is you have to start out with a cluster because we need the Kubernetes APIs and that reconciliation loop to then create the other clusters. And that first cluster is typically referred to as your management cluster. Now how you get that management cluster, it can be done in a variety of ways, right? So a really common way is to use something called kind, Kubernetes in Docker. And what you'll do is you'll create a local kind cluster
00:20:15
Speaker
you will initialize that local kind cluster as a management cluster. And this just means installing the cluster API, CRDs and controllers into that kind of cluster. And then you will create the first, what we call the workload cluster. And then it will go create that workload cluster on whatever platform you are. So here you are, you're sitting on your Mac, you've got Docker desktop installed or some equivalent. You run kind, it creates a local Kubernetes cluster.
00:20:42
Speaker
You install cluster API into that and you give it a manifest to say, go create a cluster on AWS. So it goes and it creates a cluster on AWS. So you've got your kind cluster and now you've got this cluster on AWS. Well, we don't want to keep this kind cluster around forever, right? So we then will move the cluster API resources from that kind cluster up to that first cluster you created on AWS, right? And there's a command to do this that the cluster API community provides, right?
00:21:09
Speaker
Then once you've got all the components sort of moved up to that cluster, that one is now your management cluster. That's where the cluster IPA controllers and CRDs and all that reside. You can then decommission the kind cluster, okay, go away, right? And then work strictly against that long-lived management cluster that's hitting AWS. And then from there, you can apply manifest to create other clusters as needed, right? That's one way of doing it. You could also, and this is actually what I do in my lab environments,
00:21:38
Speaker
is you could also create infrastructure on the cloud provider of your choice, right? So I use a tool called Pulumi. I go out and I instantiate all my AWS infrastructure, including some EC2 nodes, and then I will manually bootstrap those nodes using kube-adm. So I say, I say manually. kube-adm is not all that manual, to be honest with you. It's not like I'm, you know, writing system defiles, but I'll use kube-adm, I'll, you know, intentionally invoke kube-adm myself,
00:22:05
Speaker
to bootstrap those nodes into a Kubernetes cluster and then make that the management cluster. So you could also do that. There's a couple of different ways, but you can take sort of the easy path and using kind and cluster CTL or cluster cuddle or cluster control, whatever term you want to use, right? To do all of that, or you could go bootstrap it and then initialize it. Either one works. And then once you've got that initial management cluster up, then you can begin to spawn other clusters and go from there.
00:22:35
Speaker
Yeah, and I think whenever I talk to people about cluster API, the transition from kind to the management cluster is always that blows their mind because they are like, how do we even get that first cluster up and then having something like kind and the ability to spin up something local and then the ability to transfer all of the custom resources and the roles over to the actual management cluster is great.
Cluster Class Concepts in Cluster API
00:23:00
Speaker
Okay, so that's a great foundation for what Clustered API is and how it works with providers and helps you provision Kubernetes cluster, right? I was catching up on talks from KubeCon and they were talking about a few newer things and one of the things that caught my eye was the definition of a cluster class. Can you talk more about what a cluster class is and how it helps organizations scale?
00:23:23
Speaker
Yeah, sure. So I'll preface all this by saying I've kind of held off on the cluster class stuff so far because I was giving it time to mature. But the idea behind a cluster class, as I understand it at least, is if you look at a cluster API manifest, sort of the full manifest that's generated when you do a cluster CTL generate cluster or whatever mechanism you use.
00:23:51
Speaker
There's a lot of repetition as you go through there. You'll say, okay, I'm going to have, let's say, a machine deployment. A machine deployment is like a deployment for pods, but it's for machines. You give the deployment, you say, hey, I want three machines and it goes and manages and make sure there's always three machines.
00:24:10
Speaker
Those machine deployments will reference a machine template. They'll reference a kubaidm-config template. And you'll just see this stuff keep getting repeated over and over again as you move through there. So the idea of a cluster class is that you can move all this stuff up to a higher level abstraction to say, here's the class of things that I want to look for. And then you just reference that cluster class in your cluster manifest, which allows you to reduce the amount of boilerplate
00:24:38
Speaker
text or configuration that's found in the manifest for a particular cluster.
00:24:44
Speaker
So, you know, I mean, that's, that's pretty cool. Um, there's other ways to go about doing it as well. Um, they aren't quite as elegant, I guess you could say, right. Um, but one of the things that I've been doing for quite a while is I've actually been using, um, customize, which is, you know, the Kubernetes tool for sort of like declaratively modifying, um, manifests. I've actually been using customize with cluster API manifests to build, uh, all the cluster API manifests for the clusters that I, um, ended up generating. So.
00:25:14
Speaker
I leveraged a piece out of customize a relatively new piece that have customized called components and then, um, building, uh, an actual workload cluster. Then it's just a matter of like writing a customized file that says, here's the base. And here are the components that I want added. Like I want to use this version of Kubernetes in this AWS region with this instance type, um, using, you know, this thing, and those are all customized components. And now I just do a customized build pipe into kubectl boom. Right.
00:25:41
Speaker
And it just makes it super easy for me to sort of mix and match components based on what it is I'm trying to do. So that's another way you could do it. Again, it's not quite as elegant perhaps as a cluster class, but, you know.
00:25:54
Speaker
Yeah. Like personally, I didn't know like customize and cluster API work together to deliver such a solution. So that's something that I have to check out as well. So thank you for sharing that today. I think there's a little bit of a, there's a little bit of a, of a, of an extra piece. So there's a couple of blog posts that I can, I can give you.
00:26:13
Speaker
to include in the show notes that talk about how to, this is kind of weird, how to customize, customize. One with a C, one with a K. How you can kind of make it aware of cluster API, right? So that it knows the links between the objects and where fields are found. And it knows how to rename things appropriately and sort of do that. But once you do that, you can use customize completely
00:26:36
Speaker
sort of natively, if you will, with cluster API manifest. And it's actually, I really like sort of the workflow. It's super handy. Okay. Yeah, definitely. Like we'll include that in the show notes so people can check it out. Another new thing that I found around the cluster API project is the concept of machine pools and machine health checks.
Machine Pools and Deployments in Cluster API
00:26:55
Speaker
How are machine pools different from what you mentioned earlier around machine deployments? And can we talk about that?
00:27:01
Speaker
Yeah. Well, let's talk about machine health checks real quick. They've been in the product for quite a while in the project for quite a while. And the idea behind a machine health check is just like a health check that you would define for like a pod, right? In that we want to make sure that the machine and the machine is linked up to an actual Kubernetes node in the cluster. So you have a Kubernetes node in the cluster, which points to a machine, which points to an infrastructure variant, right? AWS machine or vSphere machine or whatever.
00:27:28
Speaker
And that infrastructure variant represents the actual infrastructure that was provisioned on whatever platform. So the machine health check is just a way of saying, hey, I want to ensure that machines are up and responding, and if they aren't, then I want to mark them as unhealthy so that in conjunction with a machine deployment, then I can decommission a machine and spin up a new one to replace it, right? It sort of removes some of that human toil of like, oh, this machine's down, let me delete it, let me recreate it, blah, blah, blah, right?
00:28:00
Speaker
With machine health checks, you said it's similar to the readiness and liveness probe that we have from our pod level. This translates to the liveness probe. Is there anything that matches the readiness probe as well? How do we know a machine is ready to be entered into the cluster?
00:28:16
Speaker
Uh, so that's just handled actually as part of the normal, uh, like node bootstrapping process, right? The node will, the cubelet on the node will, will respond with a status of saying it's ready, um, or it's not ready. And you can actually see that when you do like a cube CTL, get nodes and you'll see status is ready. Yeah. It's sort of the readiness check, right? So like really, um, the machine health check is more of an ongoing, um,
00:28:39
Speaker
you know, are you operating, more like a liveness check, right? Are you operating? Are you live? Are you doing what you're supposed to do? If you aren't, then, you know, I'm gonna mark you as unhealthy. And then it'll kick it back to the machine deployment controller for remediation, which typically means delete and recreate a new one. The difference, by the way, between machine, like a machine deployment, as I mentioned earlier, sort of the machine equivalent for a deployment for pods, right? Machine pools, on the other hand, if you think about it, like,
00:29:10
Speaker
Machine deployment says, I want three machines, right? And then it will go create three machines, which will then create three infrastructure variants, right? But if we think about it, a lot of these cloud providers already have mechanisms for running groups of compute capacity. So on AWS, we have like an auto scaling group, right? And we go to find the properties for the auto scaling group, and then the platform manages making sure that they exist and all that, right? And that's the distinction between a machine pool and a machine deployment.
00:29:39
Speaker
In a machine pool, we're leveraging provider specific functionality. So we're leveraging like on AWS and auto scaling group or the equivalent, I think it's called a virtual machine set on Azure, I believe is what it's called or virtual machine scaling set, something like that. I think scaling set, yes. Yeah, probably that's what it is. Thank you.
00:29:59
Speaker
So we're leveraging that cod provider functionality, uh, to manage a group of machines instead of saying as a, as a management cluster itself, I won't manage the machines, right? Instead I'm going to tell provider to manage machines and get the provider all the detail it needs rather than manage your machines directly. And that's, that's the key difference between machine deployment and machine pool. Okay. So like thinking about scaling, right? That brings me to my next question, actually, like since this is the,
00:30:26
Speaker
focusing on the reconciliation loop, I can edit, I'm assuming I can edit the YAML file, increase the number of nodes and cluster API will automatically scale my cluster for me. But is there a way to use cluster API for setting some auto scaling rules? Like if I'm running out of capacity, can cluster API help me there? So for that, you're going to have to rely on the existing cluster auto scaler functionality, which will integrate with cluster API. I'll be honest, I think
00:30:55
Speaker
The relationship between cluster API and cluster autoscaler is a little weird because you've got this reconciliation loop, right? But then you've got the cluster autoscaler, which is sort of like automatically scaling things. And so you're like, I don't know. But I mean, this is the way it works right now. So until as a community, we come up with a more elegant solution.
00:31:17
Speaker
But yeah, you're right. One of the big benefits is that you can simply modify the desired state, i.e. the YAML, which ideally you'd keep in some sort of source of truth, like a Git repository or something like that, and then use a controller GitOps-type workflow to then apply that to your actual infrastructure.
00:31:37
Speaker
And it will then go and scale, you know, machine deployments for you from, you know, one machine to five, or it'll add a new machine deployment for you, that kind of thing, right? You might end up with multiple machine deployments, for example, if you wanted to distribute your Kubernetes cluster across different failure domains. So let's say on AWS, you want to have some workers in, you know, different availability zones, right? Well, a machine deployment can only deploy
00:32:05
Speaker
identical machines, and the failure domain is part of that spec, so it can't distribute across zones. If you want to distribute across zones, you use multiple machine deployments. And then you specify the failure domain as part of the machine deployment. And then all the machines within the machine deployment are identical. They all fall in that zone.
00:32:22
Speaker
The control plane then knows, from a Kubernetes perspective, it just sees nodes. It knows about that topology stuff because of annotations and labels on the nodes, of course, but the details are being managed by cluster API. Gotcha. This is great. I can have different machine deployments for different types of instances as well.
00:32:40
Speaker
Absolutely. Can this help me? Like if I want to use like reserved instances for majority of my worker nodes, but then I want to leverage something like AWS spot instances, right? For that on-demand capacity component. Can I have that as part of my machine deployment, a second machine deployment, and then just update that parameter whenever I need more capacity?
00:32:59
Speaker
Yeah, I'm pretty sure that you can. I haven't done a lot of work of spot instances in clusters. Just I haven't had a use case to sort of drive that functionality, right? But I know that the AWS community has worked really hard on building the mechanisms necessary to allow people to use spot instances because they can be more cost effective. So, yeah.
00:33:22
Speaker
This has been a great discussion.
Resources for Learning Cluster API
00:33:24
Speaker
I do want to wrap it up, but the next question I had was around how do people learn more about cluster API? This is the first time they're hearing about such a project, which is, I think, difficult if you have been following the Kubernetes ecosystem, but if this is the first step, what's the next step? How can we learn more?
00:33:41
Speaker
Yeah. So I actually think that the cluster API website, and you can include a link to that in the show notes and stuff, but it's basically cluster-api.sigs for special interest groups.kates.io is a really great source of information. I know that the contributors for the project have worked really hard on the documentation. And as I mentioned earlier, I've been fortunate enough to be able to make a few contributions to documentation.
00:34:06
Speaker
there as well based on my own work. But they'll give you a quick start guide to sort of like on the different cloud providers. Oh, you want to get a quick start on AWS, want to get a quick start on Azure, whatever. They'll walk you through what's required, what other commands you need to run, what information you need to have.
00:34:21
Speaker
And then you can actually just spin up or you can use kind as we mentioned earlier to create what I call the bootstrap cluster, right? To create traverse management cluster and then go from there. And then you mentioned you were watching cluster API presentations from KubeCon. So there's been a number of those as well that have been really great. But I think for a new year's or just getting started, I would probably start with the website and the quick start guide. I would tell folks, I always like to kind of talk about
00:34:50
Speaker
learning paths, because I do this in my own podcast as well. But make sure you're familiar with Kubernetes basics first. If you aren't familiar with Kubernetes already, make sure you take some time to become familiar with Kubernetes. And there's lots of resources out there for that. And then once you're familiar with that core, how Kubernetes operates, then you can begin to expand in a cluster API.
00:35:13
Speaker
Oh, I'll also mention, too, that I mentioned that I used to be in the Tanzu group at VMware. While we were there, we bootstrapped this site called Cube Academy, which has a bunch of video lessons. And there's some video lessons there for folks, if you're getting started with Kubernetes or if you're getting started with Cluster API.
00:35:28
Speaker
Yeah, it's a fantastic resource, right? We had an intern join our team this year for the summer and he wanted like that Kubernetes 101 content. I was like, just go and create a free account on VM at Cube Academy and that's the place to start with the basics. So thank you for reminding us of that great resource.
00:35:48
Speaker
Yeah, yeah, yeah, absolutely.
How to Follow Scott Lowe's Work
00:35:50
Speaker
Absolutely. So that's how people can start with Clustered API. How can people find out more about what you are doing? What are you up to? And I know you already have spoken about your podcast, but any anything that you want, you might want to plug.
00:36:02
Speaker
Yeah, yeah, sure. So as we've talked about earlier, I do a fair amount of blogging. So you can visit me at blog.scotlow.org. And I've got some cluster API content there as well, various things that I've written about cluster API as the project has evolved over time. So there's some stuff there. If you want to get started with cluster API, I'm pretty active on Twitter. So if you want to hit me up on Twitter, my Twitter handle is at Scott underscore low. And I try to
00:36:33
Speaker
keep content appropriate there for mainly technical stuff. And then I'm present in a number of different Slack communities. Pursuant to sort of this discussion, you can find me on the Kubernetes Slack. I hang out in the cluster API, cluster API, AWS, and a few other channels. So you could just find me and DM me if you were interested in that.
00:36:57
Speaker
Perfect. Yeah. Thank you so much, Scott, for joining us today. I was so glad that you accepted and were able to join. Thank you so much for the discussion and all your expertise in Clustered API. We look forward to inviting you back again if I find another topic that works. Yeah, sure. I enjoyed being on, and I'd be happy to come back and talk again sometime in the future. So thank you. Appreciate it. Yeah. Thank you so much, Scott.
00:37:21
Speaker
Okay, so that was a great discussion with Scott. Hopefully you guys learned a lot about Cluster API, what the project is about, and how it can help organizations that still want to deploy Kubernetes clusters, but they still want that flexibility in terms of the underlying infrastructure. So just to recap and talk about key takeaways, Cluster API is the answer to the question, what if we used Kubernetes to bootstrap Kubernetes?
00:37:44
Speaker
And we spoke about how you can have your kind clusters on your local machine or local workstation, and then have that spin up a management cluster, and then eventually spin up a workload cluster where you can deploy your applications. And we also spoke about the custom resources and the different providers that exist today, right? So you have a different provider for vSphere, which is what AWS leverages for EKS anywhere on virtual machines.
00:38:07
Speaker
Google Cloud Anthos or VMware Tensor Kubernetes grid leverages for deploying those virtual machines or Kubernetes worker nodes and bootstrapping that cluster. We also spoke about how customers can use these providers but also the fact that there are innovation or there are
00:38:25
Speaker
There have been new features like cluster class and machine pools that help organizations scale and use cluster API as that resource to deploy and manage and operate Kubernetes clusters at scale. For learning more about it, we'll obviously share the links that Scott has in our show notes. So that's it for this podcast.
00:38:47
Speaker
Thank you for watching. Again, we only have one ask for you. If you like this podcast, give us a review, reach out to us if you like to see us cover more topics, and please share it with your friends. It helps us grow our audience, which again, for our existing listeners, you have been doing a great job at sharing our podcast, but again, share it more, bring more people into this community so we can talk and educate everyone around communities and stateful applications and basically anything around communities.
00:39:16
Speaker
With that, that brings us to the end of today's episode. I'm Bhawan and thank you for joining another episode of the Kubernetes Pights. Thank you for listening to the Kubernetes Pights podcast.