Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
What is GitOps?
708 Plays2 years ago
In this Episode of Kubernetes Bytes, Bhavin Shah and Ryan Wallner dive into GitOps. You will learn about how GitOps came to be, what GitOps is, what tools are used to accomplish GitOps and what the benefits of adopting GitOps on your team will be.
Where to get started
- https://argo-cd.readthedocs.io/en/stable/getting_started/
- https://docs.gitops.weave.works/docs/getting-started/
- https://github.com/weaveworks/awesome-gitops
Links / Background:
- https://about.gitlab.com/topics/gitops/
- https://www.weave.works/technologies/gitops/
- https://developers.redhat.com/topics/gitops
- https://www.youtube.com/watch?v=P2W3dJUxpR0
- https://www.youtube.com/c/DevOpsToolkit/videos
- https://www.youtube.com/watch?v=f5EpcWp0THw
- https://github.com/weaveworks/awesome-gitops
- https://www.gitops.tech/
- https://www.weave.works/technologies/gitops/
- https://about.gitlab.com/topics/gitops/
News
- SBOM 101 - https://sysdig.com/blog/sbom-101-software-bill-of-materials/
- CSI inline volumes - https://kubernetes.io/blog/2022/08/29/csi-inline-volumes-ga/
- The ultimate list of open source cloud native tools - https://containerjournal.com/features/the-ultimate-list-of-open-source-cloud-native-tools/
- K8s 1.25 is out - https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release
- Registry.k8s.io https://groups.google.com/a/kubernetes.io/g/dev/c/DYZYNQ_A6_c/m/oD9_Q8Q9AAAJ?pli=1
- SIG Storage Spotlight - https://www.kubernetes.dev/blog/2022/08/22/sig-storage-spotlight-2022/
- GitLab Release 15.3 - GitOps features free - https://about.gitlab.com/releases/2022/08/22/gitlab-15-3-released
Recommended
Transcript
Introduction to Kubernetes Bites
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts. We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:28
Speaker
Good morning, good afternoon and good evening wherever you are. We're coming to you from Boston, Massachusetts. Today is August 31st, 2022. Hope everyone is doing well and staying safe.
Seasonal Changes & Personal Plans
00:00:40
Speaker
Let's dive into it. Last day of August. I almost said summer. It feels like the last day of summer, but it's still plenty of sun. I'm not ready for summer to be over. That's for sure.
00:00:55
Speaker
I'm in denial right now. Yeah, I don't know. Although the although the pumpkin spice lattes are coming out in full storm, you know, I'm just not quite there yet. Yeah, like full disclosure, I am not a big fan of pumpkin spice. Like, yeah, I don't know. Like, OK, go ahead and judge me people. I don't think you're alone there. I think it's kind of overrated. It can be good. But yeah, no pumpkin pie. All for it. Everything. Yeah. Flavored. I don't know. Yeah.
00:01:24
Speaker
I have a, I have a trip coming up, um, uh, going to Vermont for like five or six days, doing a bunch of camping up in the green mountains, um, off-roading motorcycle, uh, adventure riding kind of a thing, um, with like 10 or so people just for like three or four nights. And it's going to be a blast. I feel like that's my last like trip of my summer. And then I'm going to dive back into things, you know, come fall season when it comes to,
00:01:53
Speaker
When it comes to this world, do we know how fall conference season really comes to the course? What about you?
00:02:01
Speaker
I just came back from a trip, right? Like last week I was out at Glacier National Park. It is so beautiful. Like, man, it's just next level. Like it became my favorite national park. Like Yosemite was there, like, and it stood the competition of like Zion and others. But Glacier was like, nope, nope, this is it. Like, yeah, like the hikes, the mountains and the weather was perfect. It was not too sunny. It was supposed to rain, but it didn't. And it was just that
00:02:29
Speaker
hazy cloudy weather you can see the mountains you can do your hikes in cold cool weather and it was perfect man yeah i'm jealous i'm jealous so top of the list for you huh yes yeah i'll i'll share some pictures uh with you and and maybe on twitter as well but yeah it was a lot yeah i mean you're traveling right now so
00:02:47
Speaker
Yep, in San Francisco. Back on the West Coast, I don't know why I went back to the East Coast after Montana, but that's how I think PTO and conference planning works out for me. I'm not that good at making those continuous trips. Don't worry. Sometimes, even if it's a few days at home, it makes a difference. You can do your laundry, for example. Exactly.
00:03:10
Speaker
All right.
Deep Dive into GitOps
00:03:11
Speaker
Well, today's show is going to be all about GitOps. It's just Bhavan and I, we're going to be digging into sort of what is it and sort of going through some of the tools and sort of philosophy behind it. But before we do that, let's go through our cloud native news. Why don't you kick us off, Bhavan?
00:03:27
Speaker
Yeah, so I think for me, this last couple of weeks, last week was really low on news. I know we had 1.25 when you want to talk about it, but for me, I found a couple of interesting articles that people can read through and maybe bookmark. The first one of them was from a container journal.
00:03:44
Speaker
and it just lists out all the different open source cloud native tools. It goes from CICD tools, it goes from security and compliance tools, observability tools, service mesh tools, just everything, all the options that you have available to you. If you are looking to add to your
00:04:05
Speaker
different projects that you already use inside your enterprise or inside your organization. You have a list that's ready to go. It spans everything from storage to databases to container run times to streaming and messaging. So it's a full list. Again, they don't have details, but at least it gives you all the different options that are available for you.
00:04:25
Speaker
And then, following along the trend of one-on-one, good to know information. S-bomb, Lexistic did a good S-bomb one-on-one or software bill of materials one-on-one. What is it? How does it work? How you should be building one or how you should be asking one from a vendor? Since a lot of us are using open source projects or using container images that we're just pulling down from some
00:04:48
Speaker
from some registry, we don't even know. So it's always good to have an S-bomb just in case anything gets attacked or any CVs or vulnerabilities come out and you need to verify that you're not impacted.
Kubernetes 1.25 Updates
00:04:59
Speaker
So if you're trying to understand more about this new buzzword, which is S-bomb, just go to read through it. It's a good article. It's at least a 10 minute read for you. But yeah, that's a good link to have.
00:05:13
Speaker
And then the final feature was in the 1.25 release. Spoiler alert Ryan, I think 1.25 is out now.
00:05:23
Speaker
I know you'll talk more about it, but the thing that I wanted to highlight was just CSI inline volumes. That became a thing that reached or graduated to general availability in the 1.25 release. I know the CSI inline volumes are something that was added as an alpha feature in 1.15 and beta in 1.16, but it allows you to have another option for ephemeral volumes. I know we have used
00:05:46
Speaker
config map volumes and secret volumes and stuff like that. CSI in line volumes just allows you to create an ephemeral storage that follows your pod lifecycle but from a CSI driver so it's not your Kubernetes ecosystem like it is your storage vendor but it's just an ephemeral volume so you can in the link that we'll share in the show notes.
00:06:09
Speaker
It does explain when to use this feature and when not to use this feature. If you want to persist your data, don't get confused that, oh, this is CSI, so this will be persistent across my pod restarts. No, this is ephemeral still, but it's an option for ephemeral volumes. But yeah, that's my list.
00:06:27
Speaker
Absolutely. Yeah. That link you shared with a bunch of the cloud native tools, I think is super interesting. Right. And it's not even a long read, but it sort of takes the topic, um, high level sort of parts you're going to look at and sort of building a cloud native sort of stack anyway, you know, so for those who aren't reading it and listening.
00:06:45
Speaker
scheduling and orchestration, observability, security compliance, CICD, and then sort of dives into each one, which I think is sort of the way that you sort of come into looking at all the different spaces and sort of the persistent storage and cloud native database tools is where we live most of the time. And, you know, there's always new things kind of cropping up. So I think a really useful article there and an easy read as well.
00:07:12
Speaker
Yeah, you know, 125 is out and I put it in here because I feel like we've been talking about it for the last three or four shows. Yep. And we've covered, I think, what's coming out in 125 and sort of what's relevant to community storage. So if you want to know those things, go listen to the last few episodes, or we'll put a bunch of show links into a bunch of sort of recaps. A lot of people do blogs and sort of shows about getting
00:07:41
Speaker
all you need to know in new Kubernetes versions. We'll link those here. But I do want to put it out because it is officially out now that we've been talking about it forever. One thing I think worth noting is they have in there the change of where you pull images from. That is now becoming a registry.kdis.io. Not to say that the old kdis.gcr.io link will stop working. Of course, they wouldn't just cold turkey everybody.
00:08:10
Speaker
The release date, that's going to still work, but they are encouraging people to switch over. And there's a couple links I'll put in the show here is why they went about doing that. And part of that is spreading the load across cloud providers. They're starting with AWS. Obviously, the GCR link is Google Cloud.
00:08:29
Speaker
um, Lincoln. So that's an effort to, to spread that load. Um, and they've officially put that out, which I think, um, was maybe, um, missed in a lot of the, what you need to know, uh, podcasts and, and even ours, I don't think we've talked about it yet. So worth noting and, and there's a lot of automation tools and scripting. I know that goes around the Kubernetes that may have those types of URLs in it for where you put images. So go take a look at that.
GitOps Principles & Benefits
00:08:56
Speaker
the second thing i put in here was a spotlight on six storage so this is an interview with shing yang um and previous communities rights guest yeah and frederico so we've had shing yang on the show and and it sort of takes you through uh a conversation about
00:09:12
Speaker
the state of the sort of, um, storage and Kubernetes space, as well as the data protection working group. Um, it talks all about things like, you know, the CRDs and everything. It's a conversational, you can read it out. Um, Xing Yang was a great guest and has lost to say, and is, you know, uh, always been very involved in this space. So go take a look at that spotlight on SIG storage.
00:09:37
Speaker
And the last thing which I thought was very appropriate for this show is GitLab released 15.3, and other GitOps features are free. So we'll be talking about what GitOps is and maybe how you can go check that out and maybe start using them if you don't know what GitOps is. So that I'll put in there a little quick plug to GitLab, I mean, at least.
00:10:01
Speaker
So that is it for Cloud Native News that we had. We'll put all those links in the show notes and now we can dive into our main topic. So the topic again is GitOps. I guess the natural place for me to start a show about topic like this is
00:10:21
Speaker
what does GitOps mean and what is GitOps, right? And I'll throw this to you, I guess, first, and then we can both maybe give our own point of view on what GitOps is and maybe how we feel about it.
00:10:35
Speaker
Yep. Yeah. So, GitOps, right? I think there's a lot of noise. It's difficult to find the signal. Like, if you look at the core definition, the principles, benefits, things that we'll talk about in this pod, it is a very specific thing. But the way people talk about it, there is a lot of confusion between infrastructure as code and GitOps and CICD.
00:10:56
Speaker
just a one-liner. GitOps is a way to do Kubernetes cluster management and application delivery, or it is a way to use Git as your single source of truth for declarative infrastructure and your application stack. So it's just a way for you to use Git
00:11:13
Speaker
as that single source of truth and push your application and infrastructure code, store it there, and then have your communities clusters run the latest version. So I don't know, that's maybe a three-liner, but yeah, that's my thing.
00:11:28
Speaker
Three-liners are okay, I feel like. Yeah, you know, you make a good point in the fact that I think there's a lot of information about what GitOps is, and maybe that can be confusing, I think, going into GitOps from the start. And we're gonna try to do a little bit to help that notion of confusion in sort of discussing this in this podcast. But I think, you know, I'll start with sort of my take. I view GitOps as sort of a framework that takes
00:11:57
Speaker
best practices into play. And we'll talk about what those best practices are and uses those to sort of really develop sort of infrastructure automation and correctness using version control, right? And Git being the version control because it's in the name, you know, Git is a big part of what this is. And we'll talk about, you know, if you're not familiar with Git, it's a big part of it. Really can't do Git apps without it.
00:12:26
Speaker
But I think it's more of a framework of best practices that we've developed over a period of time to deliver applications and increase developer experience and things like that. So for me, it really falls into the category of framework because you can use all sorts of tools to make up what GitOps means to you. And we also will talk about the different tooling, but I think at a core
00:12:51
Speaker
right? That's what it really means to me. And maybe that's part of the confusion is that it can be a little nebulous of a term like most terms. I mean, so let's dive into sort of how they relate to what we've done historically, right?
00:13:09
Speaker
Before GitOps came along in 2017 with Kubernetes, we had a huge DevOps movement. And just to show that these terms are so similar, it's because they are very related. DevOps is a big part of how we got to a place where we could really dig into something like GitOps.
00:13:33
Speaker
And DevOps is more of a philosophy, a culture, again, best practices with tools, right? Things like making sure that you use CI CD pipelines and you're focusing on the delivery pipeline of building, testing and releasing with different tools and automations and getting that feedback loop right back to the customers. And so these are sort of
00:14:02
Speaker
guiding principles and philosophies that an organization really needs to understand before they can really, I think, adopt something like GitOps. I mean, maybe you could do it. If you weren't a DevOps shop, do you think you could just come in and just do GitOps? I think that's the question.
00:14:23
Speaker
No, I think you're right. DevOps is even another fuzzy term, but it's the way to build modern applications and make sure that you are releasing continuously. As you said, the customer feedback loop is important. You're pushing code out there.
00:14:39
Speaker
You're using constructs like containers communities microservices to make sure that you can iterate faster on your applications get them in front of your customer and then if there are any issues with it or if you want to add more features just keep working on it right it's a continuous development loop rather than just following like a the older waterfall model that we had where it took like nine months to get any release out so.
00:15:00
Speaker
DevOps is the first step. I think infrastructure as code and GitOps and all of these things are just enablers. DevOps is a vision. You want to get there, and then how do you actually push code to production? That's where things like infrastructure as code and GitOps and everything can help you.
00:15:19
Speaker
Exactly. Yeah. I mean, and historically, right. The reason we call it DevOps is because developers and operations teams were historically two siloed separate things, right? And it was very hard to communicate between them. Therefore, it took a lot of time to get things into production.
00:15:35
Speaker
um, and deliver on an application, right? Historically, that's why we, you know, I guess coined DevOps and why it's been successful. It's really that speed and, and, and, um, sort of reliability of having that type of framework and allowing companies to scale when they adopt those sorts of, um, continuous delivery and integration mechanisms. Right. Um, and then as you said, infrastructure as code really sort of grew out of, Hey, you know, we're, we're, um,
00:16:04
Speaker
Focusing on a DevOps philosophy here in our organization, we're changing the culture to be DevOps. Well, how do we do that? Well, we do a lot of integration and automation. Well, how do we automate? Well, we adopt something like infrastructure as code. And really, infrastructure as code is codifying exactly sort of
00:16:23
Speaker
what the state of something that you want to be. So, you know, a lot of times the examples I use is things like Ansible and Puppet, which define, you know, what a server looks like and how it's configured in code, right? And that gets applied to sort of a resource such as a server and it gets changed into that desired state. And that's a really important piece of
00:16:50
Speaker
understanding GitOps is that core concept of infrastructure as code, right? That core piece of there's a declarative sort of programming language that shows what your infrastructure looks like and you can apply that to your infrastructure and it becomes that desired state. That's a big piece of what GitOps is.
00:17:15
Speaker
Yeah, and infrastructure as code was a thing even before like people started using Kubernetes, right? Like things like puppet and chef. I remember like 2013 2014 people were already using those in production. There were these puppet cons. I think I attended one of those in North Carolina. But yeah, like the there are there have been these tools that are available and people were using them to set up like
00:17:37
Speaker
physical servers, virtual machines, cloud instances. There were so many different providers for Ansible and Terraform and Puppet that can actually help you bring your infrastructure up. And since you wanted to add some agility even outside Kubernetes,
00:17:54
Speaker
you use these modules or templates or code to make it repeatable. It is very difficult for humans to do a specific task over and over and over again without introducing errors. Using something like infrastructure as code definitely helped you streamline some processes. You knew the end state of executing that command. You knew that once I run this playbook,
00:18:16
Speaker
This is what i'll get and then i can deploy my application on top of it so that's where infrastructure as code help a lot of organizations modernize the way they deploy infrastructure stacks it help a lot of operations team out because it covered everything from storage infrastructure to compute infrastructure even Cisco had a module i if i remember correctly in my previous jobs.
00:18:36
Speaker
where we could set up these infrastructure stacks and have them ready for developers to come and deploy their application. But that's where it stopped. That's where it was. It wasn't using Git as that single source of truth that we want to talk about in GitOps.
00:18:53
Speaker
You still use version control for your infrastructure as code scripts, but there wasn't a controller or something that was always checking and making sure that your desired state matches the current state. It was a one-time operation like you can run it, and then if you run it again, it is item put in. If it changed, it will fix it, or if it's not, it won't do anything. But that's where infrastructure as code stopped. Next came GitOps.
00:19:20
Speaker
Exactly. Right. That's, that's the, I think a key point is that we had all these great tools, but a lot of times you still had sort of DevOps teams still running these things by hand or, or maybe on a cron schedule or something like that, where the sort of full life cycle and automation process of what the infrastructure as code was and to that desired state and how to keep it that way, that piece of the puzzle was the thing that's missing. So, and that does lead us to get up. So please.
00:19:48
Speaker
The reason why GitOps came along and why it works is because Kubernetes is declarative as well. Kubernetes is that desired state engine. Once you ask Kubernetes to do something, you don't have to tell it how to do certain things. You don't have to tell us, okay, this is how you deploy your deployments or pods. You just list out, okay, this is how I want my application to be. This is the container image I want to use, and Kubernetes will make it so.
00:20:16
Speaker
Since Kubernetes had that declarative benefits, GitOps became a thing that, okay, now, if we look at a very high-level workflow, I can treat my application code and my infrastructure stack as code, and I can push both of these to different Git repositories, and I can have hooks from... There are pull models and there are push models, and we'll dive into those in a few minutes.
00:20:42
Speaker
Using either of these terminologies or methodologies, you can make sure that Kubernetes is always running the latest and greatest of your infrastructure and your application code. That's what GitOps enables. Just by doing a simple Git push into your registry, running that CI pipeline, running that CD pipeline maybe,
00:21:00
Speaker
You can make sure that whatever is running on your communities cluster is always the latest and greatest or what you want it to be and there are some caveats and gotchas like things that you shouldn't do when you're using get ups but in an ideal world i would say like that's it like get push to having your infrastructure at the version that you wanted to be that's get ups.
00:21:22
Speaker
Yeah, I think so. And, you know, we'll dive into sort of what the principles of GitOps is. I think that'll probably help you take it even a step further to break it down for those who are still wondering, well, I still don't fully understand it. But before we do that, GitOps really sort of came into this sort of market along with Kubernetes. And that's, I think, a big reason why we're talking about it is because a lot of people who are managing, you know, Kubernetes
00:21:52
Speaker
clusters or on a DevOps team that really has a stack built around cloud native technologies and Kubernetes are probably either thinking about GitOps or using GitOps these days and or making sure that they're moving towards that direction just because it's becoming sort of
00:22:13
Speaker
a standard way to be able to manage your Kubernetes infrastructure. And that's because everything in Kubernetes is very codified. There's a lot of YAML, there's a lot of tools that easily let you spin up Kubernetes and all that can be
00:22:29
Speaker
defined in code and files YAML files lots of YAML files and put in a place in a version control like get to really Maintain that single source of truth and that's the biggest right part and and I think why we're spending a lot of time on here and I think a lot of the reasons because it's it's it's one of the first two Principles that we're going to be talking about. So why don't we dive into the get ups principles, Bob?
00:22:57
Speaker
Yeah, so the very first principle that GitOps has is declarative infrastructure. So having everything as code. I was like, for this podcast, I was listening to a different talk. And then the person was like, in 100 years, I'm hoping we have something different from YAML. But YAML is the best tool that we have right now. So we have to deal with YAML. That was funny.
00:23:21
Speaker
But YAML is the way we define our Kubernetes cluster. For example, the cluster API episode that we did, YAML is how we deploy Kubernetes clusters as well. YAML is how we define our applications and we deploy it against our Kubernetes cluster. So everything is kind of, as you said, codified and represented as a declarative infrastructure or declarative stack.
00:23:47
Speaker
And I can push that into, which leads me to the second principle, version control system. And since we're talking about GitOps, it's Git. That's where we store all of this information, all of the code. So let's say you have all your code that you are maybe writing and updating on your laptop. You have a Git repo somewhere. Once you are done and you're happy, you did some local unit tests, you want to push it out, you do a Git push.
00:24:13
Speaker
You update your source control repository or the version control repository and get and then you run kick off a bunch of ci pipelines like make sure that you do want to pause you there and say. For those of you maybe who might not be familiar with get right i just want to do a quick.
00:24:28
Speaker
What is Git thing? Git is version control. And what we mean by that is it tracks changes in any set of files on a local computer and you can push those to a service or a server where those files are continually tracked on the server side. So it coordinates sort of work among people. So if you're all working on a code base, you can track those files, submit those changes, merge them together.
00:24:54
Speaker
and it's a collaborative tool based around tracking changes in source code. So just wanted to let us know. Oh, thank you. Yeah, that was really helpful. And then now you have things in version control, you have tested it out, and that's when you have the way to push your changes to your cluster. So GitHub's principle, three and four.
00:25:16
Speaker
Third is approve changes that can be automatically applied to your system. And then the fourth principle is software agents to ensure correctness and alert on diversion. So what this means is these are push approaches and pull approaches. Push approach. What does that mean? You have, in addition to your continuous integration, a CI, you have your continuous delivery or CD pipelines as well, where whenever a new branch is
00:25:41
Speaker
or whenever a new code is merged into your main branch or whatever you're using for your connecting to your Kubernetes cluster, something kicks out like Jenkins X or something like that will push the new changes to your Kubernetes cluster and apply it against your Kubernetes cluster.
00:25:58
Speaker
Are the other way around the spool changes you have an agent that this can be a flux agent i go cd you can have that running on your communities cluster and it's continuously monitoring your get repo so whenever you have a new push to your good repository.
00:26:16
Speaker
it will basically download those changes, apply it against your Kubernetes cluster, and make sure that whatever is running on your cluster exactly matches what you have in your Git repository. So that's a continuous loop that these agents do on your Kubernetes cluster itself. So those are kind of the four principles, like entire system described declaratively. Git is used as a single source of truth. Your approved changes are automatically applied to the system, or software agents are used to ensure correctness and alert on diversions.
00:26:44
Speaker
Yeah, absolutely. And I think for those listening, if you go and look up the principles of GitOps, you're going to find essentially these four across the board. And it's generally agreed on what these things are. And I do want to note here personally, and I think it's useful to think about is the first two, declarative infrastructure and version control.
00:27:09
Speaker
These are things we've been doing a long time, right? These aren't new. GitOps didn't make them up, right? GitOps to me is sort of those second two.
00:27:16
Speaker
automated changes and change management using polar push-based automation. That's definitely a big part of what GitOps is. And then the monitoring and reconciliation using the agents to make sure that whatever is declared is remaining that way. And if it changes, it's to let people know. Because ultimately, those couple of things at the end there are the root, I think, to me, what GitOps is. And they solve real problems.
00:27:46
Speaker
Like the problem of, you know, I've deployed something to, you know, in the past without GitOps and even maybe for DevOps, you deploy something to a server and you're like, yeah, it's running there.
00:27:58
Speaker
After some time, you have no idea if that thing's changed. You have nothing keeping an eye on about what state it should be in, even before infrastructure is code, tools like Ansible, where you could at least run every now and then to make sure things remain the same. But that server can mutate and things can change. The cloud provider could
00:28:21
Speaker
Um, you know, do updates under you and things could go wrong. And, and I think that was a real pain that you find being solved with these, you know, last few principles. And I think, you know, if I were to focus on, you know, any two of these, the first two are sort of necessities. The last two are really what make it.
00:28:40
Speaker
Yep. And as you said, right, things can change even with GitOps, like users have to be really careful. Like you, you, you cannot have, or you cannot claim that you're using GitOps. If after you do a Git push and your infrastructure has been reconciled, you go back through the UI and make some changes because that just breaks everything. Like that's those changes are not in your Git repository. Those are things that you modified on your own. So if you are using this, this methodology or this framework, as Ryan said, you have to use it
00:29:10
Speaker
You can't go and then make some manual changes through a UI because that won't be reflected in your Git repo and then in the future, let's say two months down the line, you're like, oh, let me see what's running in my production environment and you open up the Git repo. Your repo won't match what's running on your Kubernetes cluster. So if you're using this, make sure you keep away from making any manual changes on your cluster on your own.
00:29:34
Speaker
Yeah, and in a way, the infrastructure as code, what's defined and get is sort of a type of documentation, right? We said the term single source of truth a lot, but it solves a problem of not having things documented. Maybe you had a run book in the past that someone were
00:29:51
Speaker
to deploy an application and maybe they're an expert in it. So if they're on vacation and you go to that run book, you better hope it's accurate, right? And there's no missing documentation. But in a GitOps world, what's in the production is in code, right? You know that that sort of consistency will always be there. Whether anyone's on vacation or forgot to document something, it's literally running what's there.
00:30:13
Speaker
Again, just a real problem, I think, that we've seen in the past with just things like documentation, which is great. And I know we said we'd talk about tools and frameworks next, Bobbin, but I think it probably makes sense to talk about the benefit of why do this, right? So let's jump there.
00:30:30
Speaker
No, you clearly listed out the first one. You know what's in your repo is the single source of truth. But I just want to add to that. This allows you to make sure that you can go back in your history. You can see what was the state over the last couple of years. You can see what changes were merged. You can look at that code changes. It becomes your thing where everything that changed in production
00:30:58
Speaker
You have a history for it. If you're not making any manual changes, all the changes are forced through that Git repo. So you can, looking at your history or looking at your logs, you can figure out, okay, what changed? Maybe you joined as a new employee and you just wanted to see why things are the way they are. You can go back through your repo history and then you can find out what was merged and what changed. So single source of truth, we can't emphasize it enough. That's a really important thing for you.
00:31:22
Speaker
People can come and go you can always rely on a repository to check what's running and what was running before.
00:31:30
Speaker
Absolutely. And I think, and I think a next one that not a lot of people think about is sort of the improved security model. Right. Git itself uses sort of cartography to manage changes and verify, you know, who made that change. That's something built into Git. When you do a Git push, it happens, right? You're, you're authenticated and, and, and, and so Git provides these sort of strong sort of I guess guarantees on what's correct and what isn't.
00:31:59
Speaker
Um, and so that overall, I think when you have those guiding principles of get built in, it sort of dramatically makes those CI systems, um, uh, harder to, to be seen as sort of a big gaping hole of, of, um, uh, as a threat vector. Right.
00:32:19
Speaker
And I think that improved security, I think maybe isn't the first thing I even thought about with GitOps. It was everything else, but I put it as the second one here because it is a big benefit.
00:32:30
Speaker
Yeah, and to add to that, right, there is a different perspective to improving your security posture or security model. Like definitely Git has everything, but what this means is what GitOps enables you is you don't have to share access to your Kubernetes cluster with all the developers. You can give them access to your repository and you can use the security mechanisms built into Git.
00:32:50
Speaker
But this means that you don't have to open up your permissions on your actual Kubernetes cluster. You can only give permissions to that repo and any changes that can be made comes from that repo. But other than that, your Kubernetes cluster is secure. It is its own thing. You handle all your permissions. You can handle all your permissions in the Git repo and who can push changes and what changes can be pushed. And the best part is you can approve these changes, right? Like if you don't like something, as the owner of the repo, you're like, nope, no decline or whatever.
00:33:22
Speaker
So it does help improve your organization's security posture because Git is the only thing that's supposed to make changes to your Kubernetes cluster. That way you can follow the principle of least privileges and have your Git Reaper do all the work for you.
00:33:37
Speaker
Yeah, and that also acts as a giant audit, right? Everyone pushing, everyone pushing, all your developers pushing to that instead of your Kubernetes cluster, you have an audit trail, which also I think leads me to, it simplifies application deployment for developers, right? Because they have a standard process for making changes and when they've
00:33:56
Speaker
When they've updated their code base and run through the CI to build their image and changed anything about how the application actually runs, they push it to Git, it gets reviewed, and if all is well,
00:34:12
Speaker
it deploys to the clusters, the target cluster, whether that be dev, test, sort of in a very consistent and common way. And I think for developers, it makes a big difference when they kind of know a standard way that they have to interact. And I know this from experience of
00:34:33
Speaker
working with many different Kubernetes clusters directly, maybe they're different versions of that kind of thing. If you're interacting with them specifically, well, maybe this YAML doesn't work with that cluster, right? Because it's running a different version, or if that cloud doesn't support that thing, well, their interaction should obfuscate a lot of that, and CI-CD systems and pipelines to get it deployed should manage those types of things and or the review process in general.
00:35:01
Speaker
Yeah, and like simplified application development, right, like it's simple to say, like, okay, the process is easy, but it has real benefits, like it increases your developers productivity, like it increases the speed with which you can push out code, it enhances the overall developer experience that you have inside your organization. So, GitOps has those
00:35:19
Speaker
unrealized benefits that you might only see in like, maybe six months or a year, you won't see it day one. But like, once you build that philosophy, or you build or adopt that framework inside your organization, you will see these real benefits show up. So and I want to go back to your audit log, like in the security model discussion, like, yes, like we want to do blameless posttops, postmortems, but that gives me a list of like, okay, who messed up?
00:35:47
Speaker
or what messed up. Sorry. Who should I be blaming in a blameless postmortem? Like who should I be giving that look? Hopefully a group of people that can look back at what went wrong, right? You know, from who delivered it to who reviewed it. I mean, there's, there's always multiple hands in play, but yeah, it's a big part of it. And
00:36:08
Speaker
Ultimately, all these things should lead to, you know, done right should lead to faster things in production, meaning going from development to production when you have these things in play. I know it sounds easy to say, right? There are a lot of moving pieces and it does take some getting used to, you know, making your, you know, having your developers interact with your systems this way, especially if they're, you know,
00:36:31
Speaker
I'm used to interacting directly with clusters. It can be a change, but again, this is why I think going back to a changing culture and org is definitely part of that education process. It's not just rolling out new technology. That never really works. You have to definitely adopt the mindset. Being in a DevOps model already helps and being a Kubernetes shop and adopting GitOps is
00:36:57
Speaker
is always going to be, I think, a good place to start. And a majority of developers, if not all, 100% should be familiar with Git. That's how they should be writing code. I think that's what they teach us in school. This is how you maintain...
00:37:15
Speaker
This shouldn't be something really different or really drastic for them. They should be familiar with how Git works. This is just taking that to the next step and actually taking it to production. It does help with making sure the process is overall simpler.
00:37:31
Speaker
Absolutely. And from an ops standpoint, the correctness and sort of consistency of that code base is very, very helpful, right? Not having to rely on things that used to work in the past, but may or may not work. It is what's working right now, right?
00:37:47
Speaker
and having that sort of ability to make sure that that is consistent across the board and having everyone do it that way. I think this works the best on this topic. This works the best that everyone uses this model, right? Not like, hey, we're going to tell this group of developers to do it this way, but then we have these special DevOps-y people who have admin access and touch the cluster directly. It doesn't work as well that way, right? We definitely have to think about it from the point of view.
00:38:17
Speaker
I think we covered, I think the benefits we wanted to talk about with GitOps. Maybe it makes sense to talk about push-pull model and then we'll dive into tools before we tell people where to get started.
00:38:30
Speaker
Yeah, and so like, just one thing on the benefits part, right, as you said, like, you shouldn't have people who have access to your Kubernetes cluster and make changes, you shouldn't have super users. Because the whole point of GitOps, the whole point of DevOps is to move towards a model where you're not figuring out
00:38:48
Speaker
what exact kernel modules and what exact packages run on your production environment and then try to troubleshoot that either in real time or you are trying to mimic the environment and replicate it in a testing environment to update or fix things or do root cause analysis. You should just rely on Git to push changes to your Kubernetes cluster and if the latest change broke something, there is a Git rollback.
00:39:15
Speaker
You can roll it back. Whatever worked in the past, whatever worked an hour before, should work again. Again, I'm not selling it enough. This will bring you real value, even though I don't work for a vendor that sells GitOps. This is something that can help the ecosystem a lot by just adopting these principles and getting these benefits.
00:39:38
Speaker
Absolutely. Couldn't agree more. So push-pull, we've used these terms and I know you explained it a little bit, but I think it's probably worth sort of diving into what push-pull really means and what the differences are. So I guess I'll give a stab at sort of the push-based model and push-based, pull-based, and then you can kind of add to it.
00:40:03
Speaker
So with push-based, and the name helps, right, in this terminology. And if you know Git, push and pull should also be very familiar terminologies. It's usually a deployment strategy that's sort of implemented in CI CD, right? So with tools like Jenkins, Travis CI, CircleCI.
00:40:20
Speaker
source code, sort of where the application lives there, all the YAML to deploy applications or configure what you need to configure lives there. And that build pipeline that we're used to with CICD is triggered, which ultimately builds container images and finally deploys the actual
00:40:42
Speaker
YAML files to the clusters from the CICD tooling, meaning that it's pushing the changes from the Git repo and CICD to the cluster. It's applying
Push vs Pull Models in GitOps
00:40:57
Speaker
them. If you're really with kubectl, it's kind of like saying kubectl apply.
00:41:03
Speaker
And this happens sort of in that sort of build pipeline. And so that's my, I guess, my definition. And I'll let you add to it, because I know you probably have some to add to it. And then sort of the push-based model is a different strategy, meaning that it's the same concepts that are sort of, I just talked about with push-based, but it's different in how that pipeline actually works, meaning that, you know, traditional CICD pipeline is triggered externally.
00:41:31
Speaker
Whereas a poll-based environment is sort of an agent that lives and observes changes to a repo, meaning that when a change is approved and put into Git, it'll say, oh, look, the state of this thing needs to change. I've recognized that.
00:41:48
Speaker
And as my agent in the environment and say in a Kubernetes cluster, it will take that and make sure that change is then matched, sort of that desired state is matching actual state in the cluster, meaning it pulls that changed into the cluster.
00:42:05
Speaker
Um, and that it's, it's, again, a lot of the same tools are going to be used, but, um, the end state of that first one, the push model pushes it to the cluster and applies it versus the end state of the pull model is you, you've made the change and get, and your agent takes over.
00:42:21
Speaker
Yep. Yeah. Yeah. I think I can't do any, I can't add anything to that, but just one thing, right? Like it will be different set of tools. Like you might be using something like Jenkins X and CircleCI as Ryan mentioned to use the push model, but the pull model will be something like
00:42:38
Speaker
a Flux CD or an Oracle CD thing where you have an operator and a custom resource running on your Kubernetes cluster. That will be the first step. You will need that agent to be running on your Kubernetes cluster, and then once it's up and running, you have linked it to your Git repo. It will follow that reconciliation loop and make sure that whatever changes are approved and merged are reflected on your Kubernetes cluster as well. Different set of tools. Personally, we don't have any preference.
00:43:08
Speaker
you want to use in your organization, go for it. Just follow the methodology or the framework, and you will get these benefits here. Yeah, and I think a key differentiator with the push and pull is that push only applies things when that external event happens that the plan is triggered. Versus when you have an agent looking at a repo, it'll always make sure that desired state. A lot like it worked with things like Puppet and Ansible, you had agents looking at
00:43:36
Speaker
Yeah, things that applying into the servers, everything ever changed, like if NTP gets changed, or if someone does it, it'll go fix that. And in pull based, it does the same thing. And I think one of the things like before we talk about the tools, right, one of the things that we didn't cover is
00:43:53
Speaker
You shouldn't be pushing actual secrets or you shouldn't be storing credentials in your git depot. That is a gotcha. So make sure that while you are looking at GitHub site, use something like a hash got vault or some key management system or some credential management system.
00:44:08
Speaker
where you can store your credentials and store your secrets. So whenever your change gets approved or merged and your flux operator maybe is or flux resources agent is trying to reconcile things, it's not like actually pulling plain text credentials. It is using something like a key management system to have that on your Kubernetes cluster. So that is something that's not covered by GitOps today. So make sure you have a strategy for it.
00:44:36
Speaker
Yeah, absolutely. All right, let's switch gears and talk about some of the tools and frameworks that are probably, if you look GitOps up, you're going to find. The first one I want to talk about is WeaveWorks, their sort of agent and framework is based around Flux and sort of their, they were very early on in the GitOps days. It's going to be one of the getting started guides that we suggest, right? WeaveWorks is definitely one of those ones where
00:45:01
Speaker
Is definitely a great place for information and education around what get ups is and how to you know Really get started and get out your feet wet with the whole process Flux is actually part of sort of what you'll find in the we works information Trying to find a good Sort of definition for flux. Let me I didn't write one down here
00:45:28
Speaker
Okay. But like just a list of tools, right? Well, you do that. Like we works, I think that they were the ones that coined the term GitOps and they run these GitOps cons on a, on a, I think annual basis or maybe a six month basis. So you will always find a good amount of information, good amount of customer use cases that are coming and talking at those conferences. So in addition to the tools, they also have a lot of resources that you can go and learn more about this topic and see how it
00:45:55
Speaker
you can actually implement it in your Kubernetes cluster. But they have things like Veev GitHub Score, which is a free and open source continuous delivery product that you can use to run your apps in Kubernetes. They also have a cloud offering called Veev Cloud, which gives you that experience, it gives you workflows, it gives you a dashboard. Argo CD, right? That was a really successful open source project. I know Intuit supported a lot of it or had people dedicated working on that project.
00:46:25
Speaker
I know a couple of people who were working in Intuit, they basically left Intuit and started their own company called Acuity. They are basically offering an enterprise version of ArgoCD but still contributing back. There is a startup now in the ecosystem that is maintaining ArgoCD as a project with some venture funding.
00:46:43
Speaker
Absolutely. And switching back to Flux, both Flux and Argo CD, and I don't know about any of the other ones we're talking about, are CNCF projects. I think they're both incubating. But Flux is that sort of observability portion that we talked about earlier in sort of
00:47:01
Speaker
Um, how get ups works, right? It's that thing that sits in your environment that observes your get repo. And essentially you make changes to the repo and flux will, um, look at that, make changes all based on that. And there's a whole bunch of sort of secondary tools that, um, are sort of involved with the flux projects around, you know,
00:47:22
Speaker
Git and Security and Canary deployments those kind of things kind of dive into but there's a lot out there We're not going to dive into each one of them code fresh has a bunch of tools and frameworks and things you can do with get get ops one I mentioned earlier in the show was get lab get lab has a whole bunch of
00:47:40
Speaker
Features that have been sort of behind their paywall, but until today I guess or 15.3 Some of them are free. So that pole based model I believe is now free With it harness. I haven't done much of that with my with that myself But I know it's another tool out there and we works has a really good repo at github.com slash we works awesome dash get ops which has a whole bunch of
00:48:07
Speaker
tools and information around a lot of what we talked about today and is a great place to start. So that's a great next step, I guess. Yeah. And then tying it back, right? Like something that we covered in the news section of this podcast was an article that lists all the different tools. There is a CICD section, like they list Argo and then other CICD tools that are hosted by CNCF include things like Flux, which we spoke about, Brigade, Captain, OpenGitOps, OpenCruise. So there are different options available.
00:48:36
Speaker
You can learn more about these, try it out, see if it works for you and get started.
00:48:41
Speaker
Yeah, exactly. And the links that we'll put into where to get started with GitOps are really all around Argo CD and WeWorks. There are going to be others out there if you look into this. These are just ones that I've gone through the WeWorks myself, so I know that that's a good experience, so I can recommend it. I haven't done the Argo CD one, but I know it being involved with CNCF, and I've heard a lot about it, a lot of good things about it. Argo-cd.readthedocs.io. There's a really good getting started there.
00:49:11
Speaker
as well. So we'll include all those links where you can get your hands on, um, uh, you know, uh, started, uh, with get up. So yeah, there is like, uh, uh, uh, while I was doing my research as well, right? Just making sure that I'm representing what's latest and greatest. I also came across this cool YouTube channel called the DevOps toolkit. I know I think it's run by Victor.
Conclusion & Resources
00:49:32
Speaker
Yeah. He does a great job. Like he has a, like he, he does a lot of communities things, but he has a specific playlist for get ups.
00:49:37
Speaker
And he talks about Argo, he talks about Flux, he talks about his experience with them as well. So if you're looking for more resources in addition to what Ryan mentioned, I would recommend that YouTube channel as well. Yeah, absolutely. So, you know, we're going to wind the show down here. I think, you know, my big takeaways for this, you know, GitOps topic is really trying to understand those principles, right? Everything is described declaratively. Everything is inversioned in Git.
00:50:05
Speaker
which shows the desired state, which gives you a whole bunch of benefits, right? Those things, two things, again, we've been doing for a long time. Number three is that approved changes can be automatically applied to the system. Again, through push or pull mechanisms, both work in GitOps. I think pull is definitely, I think, becoming more popular.
00:50:25
Speaker
things like operators and Kubernetes and things like that. And then lastly, right, those software agents that make a big difference that sort of run and keep an eye on that desired state is the actual state to make sure it matches the desired state, right? Really kind of understanding those four principles and where, where in sort of the stack, each thing sort of has a play in your, your infrastructure. And once you kind of understand that, I think it makes a lot of sense why this makes a lot of sense.
00:50:54
Speaker
Just to add my takeaways, if you follow these principles, benefits will include things like you have a single source of truth, you have a log, you have an audit log, you can see all the different changes that were made to your Kubernetes cluster or your application. So that's one key benefit.
00:51:11
Speaker
you get an improved security model so you can follow least privileged model for your communities cluster you can use permissions inside git hook and merge code who can approve code and that's what gets pushed to your communities cluster it increases your developer experience or enhances your developer experience increases productivity simplifies overall process and it makes sure that
00:51:32
Speaker
there is consistency, right? Like what is in your repo is what's supposed to be on your Kubernetes cluster. So I think that's me, right? Like that's my four key benefits that I get from using GitOps as a framework.
00:51:43
Speaker
Absolutely. And for those listening, hopefully you found this useful. If you're new to GitOps, hopefully we covered enough to really have you a little more comfortable with the topic of what GitOps is and how it became to be. But that's it for today's show. And I do want to encourage our listeners to reach out to us, review us. We had a listener reach out on LinkedIn the other day and kind of
00:52:08
Speaker
you know, appreciate the show and also suggest a topic that's perfect. That's exactly what we're sort of the type of feedback we're looking for. You know, we're excited to hear about what you want to hear about, and hopefully we can find the right people to talk about those things. So please, encouraging you to message and or view us wherever you consume your podcast and or on social media.
00:52:29
Speaker
Um, the next podcast we have coming up is going to be talking about Kafka. So lots of cool stuff. Uh, if you're kind of involved with Kafka use message cues, this will probably good episode for you. Um, but with that, that brings us to the end of today's episode. I'm Ryan and thanks for joining another episode of Kubernetes Bites. Thank you for listening to the Kubernetes Bites podcast.