Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Scaling with Kubernetes as a team of one
400 Plays3 years ago
In this episode, Bhavin and Ryan interview Kai Davenport who has been running Kubernetes in GKE to host various projects. Kai is a developer and an operator rolled into one and tells us about his lessons learned and experience running Kubernetes as a team of one. The discussion dives into Kubernetes as an abstraction, how to manage backups of applications and data, when to use quotas and more.
Show Links
Recommended
Transcript
Introduction to Kubernetes Bites
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts.
Cloud Native News and Insights
00:00:14
Speaker
We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:28
Speaker
Good morning, good afternoon, and good evening wherever you are.
Trip Experiences and Fall Plans
00:00:32
Speaker
We're coming to you from Boston, Massachusetts. Today is October 8th, 2021, and I hope everyone is doing well and staying safe. So let's dive into it. Bhavan, how are you? What have you been up to? I'm doing good. I've been kind of busy. So last week, we had an impromptu New York City trip.
00:00:51
Speaker
trying out a lot of new bars, speakeasies, and restaurants. So that's always fun. But I'm looking forward to this weekend, because I just want to take a break. Last weekend, we drove on Friday for six hours, and then Sunday for four hours. And it was like, this is too much work to do on a weekend, even though it was for fun. But yeah, that's what I've been up to. Yeah, and you're heading out to KubeCon, right? So might as well take a break before you do that, right? Yes, next week is KubeCon.
00:01:20
Speaker
I have not been up to a whole lot. As you can tell, I'm coming off of a cold, a little bit of a cold, COVID negative. Don't worry about everybody. Okay. But yeah, you know, having a cold is no fun, no matter what type of bug it is. So I haven't done much lately, other than trying to get my work done and rest up. But trying to enjoy this fall weather, you know, here in New England, there's massive amounts of things to do. So various October fests and those type of things, which is, you know, always fun.
00:01:49
Speaker
Yeah i'm waiting for peak foliage like i want to drive up to the white mountains but i was like i don't know if that's worth it yet maybe this maybe not next weekend so we'll see. You can get peak foliage and really good views if you don't drive up there too trees are everywhere.
00:02:06
Speaker
It's more of a warning that it gets very busy up there. I spend some time in Lincoln, New Hampshire in the fall and in the spring. I prefer the spring because there's just so many people there. Leaf peeping, right? It is beautiful though. So if you're going to just go for it, we'll see how it is.
00:02:24
Speaker
I love the team.
Meet Kai Davenport
00:02:27
Speaker
Okay. So today's show is really exciting. We do have another guest on the show, Kai Davenport. I'm really excited to have Kai on the show. I've had the pleasure of working with him in the past when I was at cluster HQ and Kai is a bit of a jack of all trades. So he's full of knowledge, full of experience. And I'm really excited to see what he has to say about running Kubernetes as a team. He does a lot of,
00:02:51
Speaker
development and various projects for customers that he works directly with and runs them on Kubernetes himself. So we're really interested to know how do you do it and what does it really do for you. So this is a really exciting topic. That should be fun.
00:03:06
Speaker
Like he's a developer and operator wrapped into one. Exactly. Jack of all trades. Great. So before we dive into that, let's cover the news.
VMware Innovations and Discussions
00:03:18
Speaker
I know you've been tracking a lot lately, so why don't you kick it off?
00:03:21
Speaker
Yeah, so this week was VMworld. Even though it was like a virtual event this year, they still had a lot of announcements around the Tanzu ecosystem in different keynotes and solution keynotes and some deep dive technical sessions. One of the key things that I really liked, and actually this was announced a day before VMworld as part of the DevOps Loop one day event that they had was VMworld Tanzu Community Edition.
00:03:46
Speaker
This is a distribution that's free to download. You can run it locally on your PC or Mac on Docker, or you can run it on AWS or Azure. It gives you all the different things that are included in the Tanzu project. So like if you are looking for it, like it uses cluster API to deploy EKG clusters or Tanzu clusters.
00:04:05
Speaker
but then it also gives you the option of deploying chromates and krafana it uses valero and sono boy user cert manager and open policy agents all of those things are packaged into this one distribution that's free download you don't even need to provide an email address you can just download it and off you go.
00:04:22
Speaker
Another interesting thing was around Tanzu Mission Control, which is their SaaS-based management portal for Kubernetes clusters, not just Tanzu, but also EKS and EKS and GKE. They now have a starter edition for it, so it's still SaaS-based, but I think it will be a limited set of features where you can maybe attach your clusters. There is a registration page for it. I did register for it. I'm waiting on getting access, so I'll let the community know when we try it out and see what features are included.
00:04:52
Speaker
Other things on my list are just VMware cloud with Tanzu services. The VMware cloud has been VMware's infrastructure as a service offering with different public cloud vendors like AWS and Azure and Google Cloud.
00:05:07
Speaker
This is just providing or enhancing that service with the ability to run tanzu so if you're running vSphere with tanzu on prem inside your own data center you can now consume tanzu services on your VMware cloud environments as well and it comes with tkg and tanzu mission control essentials here so again some different form of management.
00:05:27
Speaker
And then one last thing that I wanted to highlight was vSAN 7 update 3. I know we hinted at that in the last episode, but we got more details around it. It does support Kubernetes topology parameters for stretch clusters. But one gotcha there is it only works for upstream Kubernetes right now. No real support for VMware Tanzu. So if you are running VMware Tanzu, you might have to wait for the next update. But again, still a cool feature.
00:05:56
Speaker
And then one thing that I'm looking forward to learning more about is Project Cascade. So Cascade allows VMware to build a control plane for deploying infrastructure as a service and container as a service platforms. Again, I'm still looking for more details. If anybody out there knows more details about Cascade, please reach out and I would love to talk more about this. But yeah, that's it for me. That's it for VMworld.
00:06:19
Speaker
Wow, lots of things going on there. It sounds like Community Edition along the same lines of EKS Anywhere and those kind of distributions that come out. It really helps onboard folks to the platform. So yeah, lots to learn there, lots to unpack. It sounds like I have a lot of reading to do.
00:06:40
Speaker
And we have KubeCon next week, so yeah, it's just going to get exponential. Yes.
CNCF Presentations and Rebranding News
00:06:45
Speaker
So there's three things that were on my list I wanted to mention this week, which two of them were CNCF presentations, one on Canister, which is a framework actually from the Cast and Veeam folks around sort of solving operational requirements between applications and Kubernetes when it comes to backup and recovery
00:07:07
Speaker
So, you know, this is really about understanding the individual workloads when it comes to using tools that allow you to do things that may be a little more custom for your environment, such as run specific, you know,
00:07:25
Speaker
commands, or freeze operations, or database dumps, and those kind of things. All very important stuff when you're running stateful workloads. So really cool presentation. I enjoyed learning a lot there. And then the other one was about OpenEBS 3.0. So a lot of new stuff around GVAC store, local PVs, and Maya store, as well as some glimpses into 4.0. Both of those presentations were great. Definitely go check those out. And then the last one was
00:07:54
Speaker
From sort of big news, you know, we follow storage OS quite a bit, you know, since they're very close to, you know, what we do day to day at our at our jobs and they rebranded to on that. I hope I'm saying that correctly. But that's a whole new sort of branding and focus for storage OS. So really interested to see where that goes in the community and.
00:08:17
Speaker
That's really all I had for the news. One thing we did want to mention was this episode will come out the week of KubeCon. So we won't be discussing KubeCon until the following episode. So if you're listening to this episode during KubeCon around KubeCon, we will have a KubeCon update post this episode. And I'm pretty sure that will be just like a news episode, like A to Z, like what's going on with all the different vendors.
00:08:44
Speaker
And I'll be there in person. So again, if you're listening to this, during the week of KubeCon, if you're there, just say hi. I'll be at the Portworx booth. And I'll let you know if the StorageOS booth says StorageOS or on that.
KubeCon Anticipation
00:08:57
Speaker
There you go. You never know. I will be in the virtual booth. So if you do stop by virtually to KubeCon, come by the booth there and come say hello. I have a couple office hours where you can see me in person if you want. That's like full coverage right there.
00:09:13
Speaker
Great.
Kai's Kubernetes Journey
00:09:14
Speaker
So let's dive into our topic, which is, as a reminder, scaling Kubernetes as a team of one. Let's get Kai on the show. All right, Kai, welcome to the show. It's great to have you here on Kubernetes Bites. Thanks for coming.
00:09:30
Speaker
It's great to be here and thank you for inviting me. Yeah, so today's topic as we sort of introed was scaling Kubernetes as a team of one. But before we dive into that, let's learn a little bit about you and your background with Kubernetes or just as a person.
00:09:49
Speaker
Sure. I suppose I'm what you'd call a jack-of-all-trades master of non-engineer. I love learning the new things up to a point where I can be dangerous with them and then put them down and learn something else. I'm not what you'd call an expert in any one area, but I suppose Kubernetes has been the thing for the last five years that I've just
00:10:12
Speaker
constantly been using, I didn't set out to think I want to really master Kubernetes. It just became something I kept hitting from different angles and coming across. That kind of experience through exposure might be how you call that. But before then, before Kubernetes or BK, I suppose, 2015, I suppose in my head is that time, right? Software engineering on various different projects like
00:10:42
Speaker
e-learning software was one of the big things i spent a long time doing back in the days of flash i'm sure people remember that um and desktop publishing company where i did a whole kind of multi-user server thing for um a company that did desktop publishing software that was moving their kind of windows software onto a web version um and various other kind of my my just before we moved to kubernetes era my proudest project before
00:11:09
Speaker
was a Perl system that ran an internet for an events company. I wrote that in 2001 in Perl and it's still running. And I know this because when it doesn't work, I know about it. I'd say I put about maybe an hour a year
00:11:27
Speaker
maintenance into that thing, and it just sits there running. But since 2015, and containers arrived, and I've been working with various companies, mainly in the stateful, anything that has anything to do with saving stuff to disk and containers has been a pattern, it seems, but also lots of marketing departments.
00:11:50
Speaker
as well as software engineering roles, a combination of the two. I think really in this world, if you're in a marketing department, you kind of have to be a software engineer. You can't wave
Managing Projects on GKE
00:12:00
Speaker
your hands. It's more than just marketing speak. You have to speak the language and therefore have done the things that you're speaking about. But I kind of enjoy the marketing roles because you get to talk about things and have
00:12:12
Speaker
have a bit more fun than sat there coding on a computer all day. So I've enjoyed all the conferences and that kind of stuff. And yes, I think the subject of what we're speaking about today is like lots of various side projects that I'm involved with doing or side gigs as I like to call them. And all of these are running on a single GKE cluster.
00:12:35
Speaker
and various different things. One, I'll go through some of them. One of them is a website publishing system that builds websites from Google Docs. Another is a franchise management system. It's like an intranet for franchises and various other both exciting and boring projects that all need to run somewhere.
00:13:03
Speaker
All these applications like you started from scratch and running on Kubernetes or something you had to morph into containers or modernize them basically.
00:13:12
Speaker
Good question. I mean, I'll say the dreaded words WordPress is one of them. So that definitely, definitely counts as a morph. And yes, we've got WordPress running on Kubernetes and no, it was not designed to do that. So all the demos that we as TMEs do actually have an impact in the real world, like the most common application is WordPress.
00:13:34
Speaker
Totally right. In fact, I'll be honest, I'm fairly sure I've used some of these how to run WordPress on Kubernetes demos when I was trying to do it for real. I love that moment that you chat with somebody, oh, I've got this thing going on, but how are we going to run it? How are we going to build it? I'm just like, well, I could handle that for you.
00:13:58
Speaker
I think is the lead on to the thread of this conversation is, what I'm really saying is I've got this magic tool that you don't know about. I can just throw some configuration out and then forget about it. Five years later, it's all still there.
00:14:17
Speaker
So you mentioned this was a single GKE cluster. Yes. And that to me is already a very interesting topic, just because you have many projects running in the same Kubernetes cluster, assuming they're all in different namespaces. So how does Kubernetes itself allow you to do this when you don't have a full DevOps team or engineering team behind you?
Standardizing APIs
00:14:42
Speaker
Yeah, good question. I think the most important thing is probably the fact that it just standardizes the API for all the primitives. Something I've always enjoyed explaining to people who are less technical, because a lot of the people I collaborate with on all these projects are
00:14:58
Speaker
There may be technical people, like they are engineers and they're coders, but they're more of the kind of like, I know how to make the thing appear on the screen type coders as opposed to where does it actually happen in the real world. As in like, all we're doing is tricking rocks into thinking with lightning world, right? I love that quote, that's not mine.
00:15:20
Speaker
but the standardized API for these three primitives, storage, networking, and compute. When you explain to people, look, all of this cloud stuff, all this complicated arrays of different services like AWS, hundreds of things that you can do, and it all boils down to some arrangement of those three primitives in the end of the day. Let's call TPUs compute and there's some slight variances on these things, but it's just, where do I save the data?
00:15:46
Speaker
how do i get that data to somewhere to do some calculations and rince and repeat is all of these things when she boiled down red and kubernetes gives you those primitives in the easiest form as in once you learn about deployment services ingress and persistent volumes just like cool i can now put together all of these different stacks in a way that i
00:16:07
Speaker
literally did the same thing to another project yesterday this other project was completely different languages and that's another thing to mention like all of these projects are different languages so okay yeah i've got a way to put this together like why don't you just run it on vms if you're just a small outfit
00:16:25
Speaker
is the kind of counter argument to scaling with Kubernetes as a team of one, right? And the answer is like, show me the way that I'm going to run Go, Node, Python, Perl, PHP, three types of different databases on one VM when I don't worry about dependencies. And the answer is obviously Docker and containers.
00:16:45
Speaker
That's why containers are brilliant. You just get to go, I don't need to worry about the environment that this app needs to run in anymore. That is a godsend. Then you say, great, I've got containers. I can run lots of things on one machine. Then you go, but one machine isn't enough.
00:17:01
Speaker
So what do we do now? And, you know, back going back to 2018, there was a little phase where the answer was swarm or Kubernetes or Mesos, you know, there was three things in there. And then Docker basically said, okay, fine, we lose Kubernetes.
00:17:17
Speaker
we're going to bundle Kubernetes with our thing now, and so it standardized it. The real answer to that whole world is if you just want to run things and you don't have the time, because that's the point, is you don't have the time to really think about it or dare I say in some cases care, and you just want to run it. You need it to be more than one machine because vertical scaling always has a limit and with this, you can just keep going.
00:17:44
Speaker
what other tool can coordinate containers across more than one machine and it's kubernetes in a nutshell so the fact that it's the go to choice for that solution and the fact that it standardizes the api's for these primitives i think that's the really low level answer the high level answer is i just don't have to think about it.
00:18:07
Speaker
Now we know the benefits of why running on Kubernetes makes sense and all the different languages that you can run on the same cluster. But how do you talk about your development pipeline? How do you get things developed and pushed into Kubernetes? And how do you operate and manage your Kubernetes GKE clusters?
00:18:24
Speaker
Sure. What tools do you use? Yeah, good question. So yeah, each project will have its own namespace. There's potentially a stateful component in it. I run a centralized Postgres on the Kubernetes cluster, it's worth mentioning. Basically, because I'm a cheapskate and GKE
00:18:46
Speaker
RDS, like wherever the GKE, the Google Cloud RDS thing is called, like Managed Postgres is expensive, right? So I'm just like, well, I'll just, I'll make a PV for Postgres, I'll say on an hourly backup schedule. And I've, you know, the Postgres is fixed on a node, a node 10 and a node selector. So there's like one day of base node in the cluster. And there's three other worker nodes, right?
00:19:12
Speaker
So let's just push that to one side, like the Postgres angle. Most of the apps use Postgres. Some of them use MySQL. There's a Mongo in there because why? I don't know. I didn't choose it. It's like we need to run this app for someone. But I suppose what I'm saying is there's a stateful node just because then in my head, I'm like, that's the one amount of separation I want to keep between. I can scale up the stateless stuff. The stateful thing is a case of like all the databases run on there and nothing else.
00:19:41
Speaker
else. But if that node goes away, everything would come back because they're all on PVs on an hourly backup schedule, and I don't need to worry about that node. It feels a bit like a pet, but with insurance.
00:19:57
Speaker
Because it is a single thing, it's right. So let's put like, if the thing needs Postgres, I've already got that running in the Postgres namespace. So I wouldn't need to instantiate a new Postgres server, I would just create a new database on my existing Postgres database and then put the access credentials like the username and password into a secret in the namespace. Once you've done that, the question is, how do I build the containers and deploy them, right? So literally only three things do I ever use. It's deployment,
00:20:26
Speaker
its service and its ingress. With those three things, almost all of the apps use those. I mentioned secrets. Four things. There's secrets as well for the environment to go into the deployment. GitLab is the main workhorse of getting this stuff done because push to GitLab, run the tests. In GitLab, we've got Docker Compose stacks to run the tests because it's integration level test. I'm going to go on a slight divergence for a second.
00:20:53
Speaker
I'm a bigger fan of integration and acceptance tests than I am of unit tests. Yes, I write unit tests, but the big important question for me is if the app is a box with a button on it, and the point of the app is when you press the button, the light turns on. The test that really matters for me is if I push that button, does the light come on?
00:21:14
Speaker
So anyway, I'm not going to go down a testing rabbit hole. This is not this podcast, but run those tests on GitLab. Then they pass. The next step would be having built the images. I missed that step. Build the images, run the tests. Now what do we do? Push the images to GCR, the container registry on Google. Last step is just patch the image now in the deployments. So I'll manually set up the services, the ingress,
00:21:40
Speaker
the namespace and the initial deployment such that something's running, I can see that it's working. Then from that moment, every push to GitLab after the test pass, all it's really doing is replacing the image tag in there. I think in some cases, of course, I'll be updating some environment variables, in which case it's just pushed the environment variables there. But in terms of 99 percent of the how do I get new code out there is pushed to GitLab,
00:22:08
Speaker
build the images after the test pass, update the image tag in the existing deployment. I almost went down GitOps land and I'm a fan of GitOps. I like GitOps as a concept. It's great. But once like what am I trying to say here? One core reason for using Kubernetes as a team of one is because it has to be simple because you don't have the time or resources to go down rabbit
Simple Tools and Workflows
00:22:34
Speaker
holes.
00:22:34
Speaker
So I'm tempted to use the simplest tools, often the most primitive tools, when possible. And GitOps, to me, feels like an awesome idea when you've got the kind of theoretical future of needing to unpick the history and understand what happens. In my world, I'm just like, right, push the new one then.
00:22:54
Speaker
So now, in no way am I having an opinion over what should larger teams do. I'm just staying there. I've found that to be an excellent system because there's very few moving parts. Yeah, as a team of one, you don't need to have overhead processes to just slow you down. You got it. Exactly.
00:23:10
Speaker
That's exactly what I was saying about it. I have two brief follow-up questions. Before we get too far away from where I've tracked them in my head, you said you have a backup schedule for once every hour.
00:23:25
Speaker
I'm curious how you came to the decision that every one hour was good enough for A, your customers, or your processes. And then the second question was around, a lot of this seems to be bundled in Google Cloud. So your PVs, your data, your backups likely, and your images. Are you ever concerned about needing to move to another cloud? Or is it not even in a thought right now?
00:23:55
Speaker
No, those are two really good questions. So on the first question, the fact is, like, if, you know, okay, no, I'm just going to say it. The fact is, if I was Facebook, losing an hour's date, it wouldn't be very good. We speak on this podcast the day after they were down for six hours. Yeah, exactly right. So
00:24:14
Speaker
I actually didn't think of saying that. I was just going to say Facebook and then it's obvious. But for me, saying to a customer, look, we've had a terrible outage. We're really, really sorry. Sometimes things go wrong, guys. You have lost the last hour's worth of data. That's the worst case scenario.
00:24:35
Speaker
That would be okay for me with the most valuable customers that we run stuff. And people pay money for these things. It's not all just like hobby stuff, right? But none of them are like mission critical. Oh my God, everything went... Lives are not in danger and livelihoods are not in danger if an hour's worth of data is lost. Yeah, Facebook lost like billions of dollars. I doubt you are running those workloads.
00:25:00
Speaker
It's a risk of cost and reality. Exactly. It's a risk reward analysis comes to the conclusion that one hour is a good amount of time. And of course, let's bear in mind, one hour is the worst case scenario here. Let's average it out. On average 30 minutes, it's really not starting to sound that terrible. And the sorts of apps that we run are mainly content publishing apps.
00:25:24
Speaker
So now what we have to do is look at the right profile of these things, right? And it's not every second there's tens of thousands of rows coming in. It's literally every hour there's maybe one or none, right? So I think we're okay on that basis. And again, dare I say, the whole theme of this thing comes with a little sprinkle of pragmatism. Otherwise you can very easily go down to Cap Theorem on that question and you're not going to have a nice life.
00:25:50
Speaker
If you're worrying as a team of one thinking about cap serum on backups, then I'll know how we can ever, you know, if that makes sense. Does that make sense? Do you just backup your persistent volumes or do you backup like the whole application, like the deployments and the ingress and the services that you mentioned? Good question. So I know there are tools around.
00:26:08
Speaker
that backup whole namespaces of everything. I don't have that much worry because everything that an app needs to deploy itself to Kubernetes, it has a deploy folder within the repo. The only thing missing there would be secrets. What I do is every time I update the secrets, I push an encrypted version of those secrets up to a storage bucket such that if the worst happened and the whole cluster went away,
00:26:35
Speaker
We've got the volumes backed up in, I think it's object store, and I think I've got it set so it's multi-region. If the worst happened to a region, then we've not actually lost any data. We've got the disks themselves. We've got the secrets that we need to put back in there. Everything else is in the GitHub repo because that's just configuration in the form of Kubernetes manifest. What I've not thought about is time to recovery.
00:27:02
Speaker
So it is true that, you know, oh, no, disaster strikes, other people would be faster to recover using clever tools. That is true. But in my world, I'm just like, okay, that's A, that's never happened. B, if it did happen, it would be a matter of a few hours. And, you know, last time we say this, as we've just been demonstrated, like, that's okay, apparently, even if you look very big.
00:27:27
Speaker
platform in the world. No, I'm really joking there. We'll stop there. But yes, no, it's a good question. The answer is no. There's no automatic restoring of everything. It would be a manual process of restoring things. And again, it's a bit like, okay, we could easily probably go down a few days at least of going, I want that automatic restoring feature. I've just not got around to it yet. And I think potentially there's an effort reward ratio analysis going on there.
00:27:53
Speaker
At the same time, you still want to keep things simple. You are a team of one. So adding yet another tool and needing to possibly maintain it and make sure it's working right is just another option. This also helps people who are looking to just get started. They don't need to boil the ocean. They just need to like...
00:28:12
Speaker
Yeah, start with something simple and make sure it works for their application. And then if they want to scale their application or their team, they can bring in these additional tools. Totally correct. And I think that is the exact pragmatism that you need when you start off because otherwise it's so easy to get lost and you almost getting capacitated with all of the options. So you end up doing nothing. It's like, I don't know where to go.
00:28:33
Speaker
too much to think about as well as just create a simple path, go down that path. Oh, now I'm cold. Okay. Now I might think about getting a coat. Whereas before you even think about walking on the path, Amazon coats, there's 10,000. Oh, I'm just going to give up. Ryan, what was your second question? Because it was a good question. We talked about hourly backup schedule of why an hour? What was the other one?
00:28:55
Speaker
Yeah, the second one was really about putting all your eggs in one basket around GKE, Google. Yes. Yeah, I think the answer there is there's nothing like... It's a really good question. If I was using TPUs, for example, right? Oh, I'm going to use all their really clever stuff or BigQuery or all of these sorts of things, then I would be maybe a bit worried about vendor lock-in, which I think we can call this concept cloud lock
Cloud Independence and Resource Management
00:29:21
Speaker
-in. Cloud lock-in. More than vendor, right?
00:29:24
Speaker
Because then, yeah, like, oh, Google are known to switch stuff off. You know, it's like, it's not, it's not an alien concept that they just turn around and go, yeah, we're losing the cloud wars. We're just going to switch off the whole thing that could happen. Let's face it.
00:29:39
Speaker
There was an article right like five years like Google Cloud gave themselves five years if they don't track it higher in the list They'll just go yeah, I just switch it off like why not? So if that happens And I'm locked in I'd be worried right now I would read that article and I'd be thinking as a team of one Oh, no, I've made the wrong decisions fact is everything standard, right? So everything is just Kubernetes API plus like any cloud worth its salt will give you PVS
00:30:10
Speaker
And some kind of backup schedule for those pvs and some kind of object store and that's it there's nothing else i'm using in google that is the specific to the cloud exactly and i think for that reason it kinda comes back to the,
00:30:26
Speaker
What happens if disaster occurs and how quickly would we restore? It would be a certain amount of manual. Okay, let's move this to there and let's plug that into this now, but it would be okay. And it wouldn't be a horrendous exercise. It would just be a bit annoying.
00:30:42
Speaker
I have to say Google Cloud is excellent in terms of UX for a single person, and compared to all the other clouds, it just shines. I've spoken to a number of people about this, and AWS is much better than Google Cloud. Okay, cool, but where are you coming from there? Oh, I'm a kind of business guy. Oh, yeah, okay, that makes sense, because it is. AWS really nails it for the person spending the money.
00:31:05
Speaker
But for the engineer, the person that's actually got to use the thing, I mean, AWS is great. I don't think I'm comparing them. It's just as an engineer, you can tell Google Cloud was written by real engineers for engineers. And so from my perspective, I just prefer that experience.
00:31:21
Speaker
Yeah, I've had every time I've used GK, it's always just like to the point of being simple, like things just work. Things just get started and you're like, oh, I'm already running ahead of time, right? With everything you try to do. It's a good lead in, I think to the next question is, is, you know, have you had any lessons learned developing this scenario? Did you start with other technologies, other clouds? You know, I heard you say you run a single Postgres instead of multiple Postgres is like, are there any lessons learned around just
00:31:51
Speaker
Yeah, I mean, the first thing would be that keep it simple, stupid, because it sounds all enticing. Oh, but what if your Postgres goes down? And the fact is Kubernetes will just bring it back up again. So yes, there's 30 seconds to a couple of minutes where Postgres isn't there. And yes, all your apps will blow up. And yes, your monitoring will tell you that.
00:32:09
Speaker
So, second thing, have monitoring, even if Kubernetes says, oh, I'm going to look after it, like, no, that it's doing that job. I mean, Kubernetes comes in so much out-of-the-box Prometheus metrics. Like, you don't basically need anything else than the standard, hey, Kubernetes node exporter plus Grafana, boom, you know everything that's going on on your app now, right?
00:32:36
Speaker
install that and use it. And I always do. And the reason there is another lesson. Oh my God, this is really important. Like always add resource limits to your manifests. Don't do the cowboy thing that I started off like, ah, I don't even know what memory or CPU profile this thing's going to use. So I'm just going to like, yeah, just do what you want kind of thing. And then let that build up and it will become a problem.
00:33:02
Speaker
So because then you're essentially looking at the griffana which is why to have griffana why is everything slow oh it's because that thing is eating everything why is he eating everything cuz i didn't put any limits on it and so all of a sudden essentially think of that like kubernetes without resource limits like
Learning Kubernetes
00:33:19
Speaker
a navigator without a map you know i could do an excellent job for you but if you don't tell me where.
00:33:24
Speaker
I have no idea what to do here. So resource limits essentially is very, very important. That's definitely a lesson I learned. Yeah, okay. I think I had this note to tell you. Read Kubernetes the Hard Way by Kelsey Hightower, right?
00:33:41
Speaker
and then use a cloud provider to do it for you. That's definitely a lesson, because it's really good to understand everything that's going on under the hood. Like, oh, there's a control server, and there's an API server, and there's a kubelet, and they will communicate in this way. Great. It's kind of interesting, if no other reason. Just read it, because it's fun stuff. Well, you're supporting this platform at the same time you are trying to just run it, right? Exactly. That's right. And then, yes, once you've read that, just click a button on Google Cloud.
00:34:12
Speaker
That's a great way to get started. How can people who are just hearing about Kubernetes for the first time get to where you are and what are the other tools that they should look out for or other training modules that are available?
00:34:25
Speaker
Yeah. So I would, I would go and hit up catacoda step one, because like, I know, I would think we will probably all know Ben who runs catacoda is great guy. And also it's just such a great resource. Like go and use the thing without having to worry about how to get to the first stage of typing commands in. So there's nothing better than going, I had to not do anything other than sign up for a catacoda session. And now I can type Qt CTL a thing and it's just there. And all of a sudden the explorer,
00:34:53
Speaker
part of you takes over and you start poking around, and at which point then you're going to want to know, well, kubectl what? And then a catecoder should guide you through that. Next thing, there's an infinite number of resources on Google for, hey, here is the basics of Kubernetes. Here's how you interact with deployments and services and ingress and PVs and et cetera. So the most important thing is to not have to worry about provisioning yourself a cluster in order to interact with one.
00:35:20
Speaker
I am once you got to there then the next stage should be what do i actually want to do with it now it might just be i want to learn how it all works there might not be an end goal of i want to run something in which case knock yourself out like just install all the operators there are and install everything and see what happens if there is a case of all but i need actually to do something with this tool then.
00:35:41
Speaker
just get the thing that you need to happen done. Don't follow all of the other rabbit holes. Another thing I wanted to say is don't go down the operator and CRD rabbit hole if you're trying to run something. A lot of them are brilliant, like crunchy data postgres operator, for example. Excellent.
00:35:59
Speaker
code quality out of this world. If you're really serious about, we've got teams of hundreds of developers and we need them all just to have Postgres on demand, then that kind of tool is going to really be the way that you achieve that. But if you're a single person looking to just get a Postgres set up,
00:36:16
Speaker
But you choose, therefore, I need the operator and the CRD just to spin up a deployment. That's going to hurt you because there's so many abstractions now away from what's actually going on. So first, in terms of tools to get started, things like Catacoda, things like, dare I say GKE because it will give you $300 to start off. Just spin a cluster up and interact with it and poke it and see what happens. Then you should move to, now I actually want to get something set up.
00:36:45
Speaker
strip it all back and do not go down the all of the shiny lights past cuz there's so many lovely tools there are really useful and absolutely essential when you get to the stage of needing them but if you've not thought to yourself i really need that tool because this is the problem i have that it solves.
00:37:02
Speaker
Just ignore it for a minute because you don't need it. That's, I think, from my personal perspective from this angle, having also worked in teams that manage large clusters where it's totally different to what I'm saying to be clear. But when my neck is on the line, when that tool doesn't work, and I don't know how it works because I just went with the fact that it would just work, it's going to be really horrible. I have been there installing various operators for a thing where actually you just need to install the thing.
00:37:32
Speaker
Yeah, so I think my advice there would be... Another thing is to really solidify that.
Cert Manager and Community Connections
00:37:40
Speaker
Basically, every single one of the deployments on my Kubernetes stack is just a deployment service plus ingress, plus potentially the PV. But as I've said, most of them just speak to the existing Postgres. So in the namespace the app runs, it doesn't have any staple thing at all.
00:37:56
Speaker
It's just the deployment for front end and API, the service so the networking can reach those things in the ingress for the world getting to them. A tool, this is like almost an advert just because I love it so much, right? cert manager, install cert manager. Absolutely. That thing is just incredible. The idea that you have to pay somebody to get a TLS certificate sent to you by email just sounds so archaic now cert manager exists.
00:38:26
Speaker
The idea that you can just throw an ingress record at the cluster and 30 seconds later TLS is working, it's just brilliant. It's worth saying actually for a single person just going from zero to running app in about five minutes is that sort of stack where you just go, I'm going to copy a deployment manifest from somewhere else, I'm going to copy the service manifest from somewhere else, and I'm going to grab the ingress record from somewhere else, change the domain name,
00:38:53
Speaker
Basically not have to change anything else and throw those three manifests at a cluster and your app is up and running and it's an incredible feeling when that works and then four years later it's still doing that and all you've ever had to do is get pushed to get lab and the whole thing works.
00:39:09
Speaker
Now, for a team of one for small apps, there's the caveat. Just saying. We're going to have to collect all of these. We'll include in the show notes all the links to the things that you're referencing so that folks that are listening, if they're interested in getting started with any of these things, we'll have those listed. Maybe even if you want to throw some others that we didn't even talk about today, I think that would be great.
00:39:36
Speaker
As the last question we'll ask you is, you have a lot of great experience here and you've done a lot of cool work. Is there any way people can get a hold of you or look at the work you've done? Is it GitHub, Twitter? Sure. I've got GitHub, Bino Carlos, B-I-N-O-C-A-R-L-O-S. People often ask me why that name sounds like binoculars. It's not that.
00:40:02
Speaker
I once was doing a job with some, it's a funny story, I was in New York for a summer painting Madonna's house after she just moved out with some of my friends from South America and they could not say Kai, right? It's like, Kaki, and he says, ah, you can be Carlos for the summer.
00:40:24
Speaker
So I kind of enjoyed that and I've just used that as my internet handle. Why Bino? I don't know. I just kind of made up a name for it. I didn't even talk about this in the intro. What a tidbit, right? Or at the end there. It was a cool summer. It sounds it.
00:40:44
Speaker
I was just on holiday board, right? And Twitter is Kai underscore Davenport. I don't really do LinkedIn. I don't really tweet, to be honest, but yes. So GitHub is the way to go if you want to check out some stuff.
00:40:59
Speaker
Well, Kai, I will say I've learned a lot from you today. And this is this is saying a lot. I feel like I've I've worked with you at various gigs, too. And I always learn something new every time I talk to you. So it's, you know, maybe you are the most interesting man in the world for me. I don't know. It's been
00:41:19
Speaker
Wonderful to have you on the show and maybe in the future we can have you back to talk about some of those other topics such as testing that could be a whole other thing. Yeah, I stalked myself. Well, it's been a pleasure, Ryan Bevin. Thank you so much. And it's really great you're running this podcast. It's been a lot of fun. Thank you for coming on and sharing your thoughts. Of course. Awesome, guys.
00:41:48
Speaker
Well, I don't know about you, Bobbin, but I feel like I learned a lot from Kai just now.
Reflecting on Kai's Lessons
00:41:53
Speaker
I haven't actually run my own endeavors on Kubernetes myself, but I think if I were to start today, I'd probably take out a lot of what we learned today. I don't know about you.
00:42:05
Speaker
I know. I have only met Guy once before this recording, and I was like, man, this guy knows so many things and has to be on top of everything to make sure that he's writing code for different customers, deploying their applications, operating them, managing them. It's all just one person. We definitely had good or great key takeaways from this recording, so I'm really happy.
00:42:30
Speaker
Yeah, agreed. So I think to recap those, the obvious one is using that kiss methodology, keep it simple, stupid. And I really like that term, just because the reality is, what we kept hearing is, when you're only one person, you can really only take on that much. So keeping it simple actually makes you more productive in the long run.
00:42:55
Speaker
Yeah, definitely. He didn't have any complex CICD or GitOps workflows. Since he was the only one responsible for deploying or pushing code to production, he knew where things were and was ready for all scenarios. Instead of just getting drowned by all the marketing around how you can optimize developer or DevOps workflows, he just kept it simple. And he has been running things for like five years now on GKE. Isn't that what he said? Yeah.
00:43:23
Speaker
Yeah, and some of the lessons learned, I think we're good to hear, right? Use quotas. I know I'm personally guilty of doing this when we work on demos and things like that. I'll just launch anything. I won't really pay attention to the quotas. I mean, given that our stuff isn't up for that long and we tear it down, it's probably okay. But yeah, if you run the stuff in production, you're going to want to use these things. And really, the goal there is to save yourself a headache down the road. You are only one person yet again.
00:43:52
Speaker
And then like to tie with quotas, he also highlighted you should have monitoring systems. So if you don't have quotas for certain services, you can identify them and change things if needed.
00:44:01
Speaker
Yeah, or both, right? And then some of the other ones I took out from this was his use of Postgres as sort of a central
Kubernetes for Small Teams
00:44:11
Speaker
component. Even though he ran it on Kubernetes, he wasn't running a bunch of Postgres service. It was still one, even though it was deployed as a container around backups and sort of his RPO and RTO techniques. I think as a team of one, some of those things that he worked out with his customers, like taking backups every hour made a lot of sense.
00:44:30
Speaker
his take on expectations of his customers for RTO times, there is a certain sense of like, yes, it's paid for, but at the same time, I am a person of one running this stuff. So it really sort of ties into, you work on those things with your customers based on the type of application, the type of workload. He didn't say that these were transactional applications happening, millions of records every minute or something. It was a pretty small amount of data.
00:44:59
Speaker
And then multi-cloud wise, that adds a lot of complexity. So I think his strategy of, let me use these great abstractions from Kubernetes and focus on those abstractions rather than the specific cloud so that if I needed to, I could. Not that it would be dead simple where he's at right now, but that was actually an interesting take on it, which made sense.
00:45:26
Speaker
Yeah, I completely agree. Doing that episode with him, there's a quote that I listen to. Again, it's a lame quote, but if you look around all your parks and cities, there are no statues for committees. A single person is really powerful and can do a lot of things if you follow a great workflow and have some structure in place. I feel that listeners can really learn many important aspects around
00:45:55
Speaker
deploying and learning about Kubernetes from this episode. I'm really pumped. Agreed. Well, I think that wraps it up. As a reminder, anyone listening to this episode, please take the time to review us. It helps us in the long run or send us a message on Anchor.
00:46:14
Speaker
and review anywhere you can and where you listen to your podcast. Exciting stuff. On our next episode though, we did mention you were headed off to KubeCon in person and I'll be at the virtual booth. Our next episode, which will be two weeks from now, two or three weeks from now, will all be about KubeCon. So pretty much an entire KubeCon recap. Really exciting stuff. I know. I'm looking forward to that one.
00:46:39
Speaker
All right. Well, that is all for today, folks. And until next time, take care. Thank you for listening to the Kubernetes Bites podcast.