Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Open Policy Agent (OPA) 101
1.4k Plays8 months ago
In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin talk to Charlie Egan, Sr. Developer Advocate at Styra about all things Open Policy Agent or OPA. This episode is meant to be a 101 level episode, where we will learn what OPA is and how it can help improve the security posture for your applications running on Kubernetes. The discussion dives into some of the use cases for OPA and how it helps with Kubernetes admission control, auditing, etc.
Check out our website at https://kubernetesbytes.com/
Episode Sponsor: Elotl
Links:
- https://elotl.co/luna
- https://www.elotl.co/luna-free-trial
Timestamps:
- 07:46 Interview with Charlie
- 01:02:00 Key takeaways
Show Links:
- OPA Ecosystem https://www.openpolicyagent.org/ecosystem/
- OPA Envoy Integration https://github.com/open-policy-agent/opa-envoy-plugin
- Charlie's site: https://charlieegan3.com
- OPA Docs: https://www.openpolicyagent.org/docs/latest/
- OPA Playground: https://play.openpolicyagent.org
- OPA Slack: https://communityinviter.com/apps/openpolicyagent/signup
Recommended
Transcript
Introduction and Hosts
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts. We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:30
Speaker
Good morning, good afternoon, and good evening wherever you are. We're coming to you from Boston, Massachusetts. Today is April 2nd, 2024. Hope everyone is doing well and staying safe. Can you believe we're already in April, Bhavan? I know. Q1 is done, at least for people that use calendar year and fiscal year. Yeah, are you? CY or FY? Those terms bothered me so much until I kind of got used to them. Just stick to one.
00:01:01
Speaker
So first quarter is done. I'm not sure how you are doing on your goals, but, uh, the most part.
Travel Experiences and Stories
00:01:08
Speaker
Yeah. I am learning Japanese though, like through Duolingo, like I'm planning a trip to Japan. Yeah. So I'm keeping that streak alive. You know, let's go. Can you say something in Japanese? No, please. Konnichiwa. Let's go. Konnichiwa, Rainsan.
00:01:28
Speaker
Nice. No, that'll be a fun trip. I really want to get to sort of that part of the world. I was just talking to a buddy of mine how we have a like in the next three years to try to travel to like Thailand and Vietnam because we've always wanted to and he's a birder. So I don't know if you know, but Thailand has the most species of birds or whatever, like any country.
00:01:48
Speaker
Nice. Yeah. Well, I know that because basically because of him, right? Okay. There's no other reason for me to know that question. I mean, we've had birders in our lives before, right? Our previous boss and stuff. So he's pretty pumped. I'm mostly going for like the landscape and the food and the food. Yeah. But I haven't been to the country, sorry, world at all. Right. So it's
00:02:09
Speaker
It's always been sort of something I want to do. So it's exciting, man. I'm definitely not a bird. I mean, they're great. I have nothing against birds. The reason I know this is like we were in Patagonia and we were on a hike and Manji was like, Oh, that bird is awesome. And I just kept walking. Birds exist here. Yeah.
00:02:35
Speaker
I'm tired. I just want to finish this. I'm not looking up. Did we ever talk about Patagonia on this trip? No, I don't. Besides the birds, because we know how you feel about them. No, Patagonia was awesome, dude. I have some tips if you ever want to go to South America and not just Southeast Asia. Ideally, I would love to travel more in general, but we'll see where I get to first.
00:03:02
Speaker
No, Patagonia was super cool. I can't believe we haven't spoken about this. Right? I don't know what happened. We blacked out last month. No, but it was supposedly like early summer, but it was still cold. I had my jacket on for all the hikes. As we had discussed before I went to the trip, we were planning on doing Chile and Argentina. We did both of those. We had some ups and downs because we were just planning the whole thing on our own and just renting out a car. There were some challenges there too.
00:03:32
Speaker
The Chile part was really crowded and more touristy I felt because it was like an official national park and there were tours. And it was their season to do that. But the Argentina side, I love the town that's near the Fitzroy Mountains. It's called El Chaltén. Dude, there are like 20 restaurants and just lodges or hostels or hotels and that's it. There's nothing else.
00:03:58
Speaker
Uh, it's, it's a perfect place to stay. Uh, we got, got some good pizza and good, good local food. Uh, traveling, you can learn so much just by the food or even just the people, you know, at these places with food. Yeah. Like in one of the restaurants, I know we are going down the top at all, but at, uh, at one of the restaurants, I was sitting beside like, it was a communal, communal table. So I was sitting with a guy from, uh,
00:04:28
Speaker
What's the country opposite Argentina? Come on. You got me. I don't know much about that. Let me pull up a map. Yeah, it starts with an. OK, I need to pull that up to South American countries. You know, here's here's the Uruguay. OK, I got it.
00:04:46
Speaker
So we had somebody from Uruguay, we had somebody from Chile and we had somebody from Peru. Like those three were friends. I don't know if they knew each other before or they just met that week, but they were hanging out, having a good time. We were talking about like politics in different countries. Like Argentina was definitely going through the presidential elections and the change of power there. So there was a fun discussion, just meeting different people and looking at or talking about different perspectives. Again, best part of travel, right? You get to meet people.
00:05:13
Speaker
Oh, absolutely. Finding excuses to travel is always a good thing. Yes.
Lack of Recent News and KubeCon Recap
00:05:22
Speaker
Since our KubeCon episode had all the news that we had to bring, I think this is going to be another no news episode. Yeah, another not the news segment. You only get two to four a year around the KubeCon.
00:05:39
Speaker
No, I think post-CubeCon vendors are done with all the marketing. Nobody announces anything after CubeCon. It's slower for sure. I mean, news trickles out and things like that. I'm ready to get our guest on and talk about an interesting topic.
00:05:55
Speaker
Yeah, we will, we will link to that news episode. So if you do, if you're itching for, for news and you're, and you're crying a little bit, because there's not another, not the news segment, you can go and listen to that episode. But yeah, our guest today is Charlie Egan, who is a, uh, senior, now a developer advocate at STIRA.
Introduction to Charlie Egan and OPA
00:06:14
Speaker
STIRA. I forget which one, uh,
00:06:19
Speaker
I'm sure he will tell us, right? We're going to be talking about open policy agent, OPA. And if you don't know what it is, good, because now you're going to find out with us because we don't know a lot about it either. But it certainly is some pretty interesting pieces of content here. So without further ado, let's get Charlie on the show.
00:06:42
Speaker
This episode is brought to you by our friends from Ilotl. Ilotl Luna is an intelligent Kubernetes cluster autoscaler that provisions just-in-time, right-sized, and cost-effective compute for your Kubernetes apps. The compute is scaled both up and down as your workloads demand change, thereby reducing operational complexity and preventing wasted spend.
00:07:07
Speaker
Luna is ideally suited for dynamic and bursty workloads such as dev test workloads, machine learning jobs, stream processing workloads as well as workloads that need special resources such as GPUs or ARM-based instances.
00:07:24
Speaker
Luna is generally available on Amazon EKS, Google Cloud GKE, Azure AKS, and Oracle OKE. Learn more about Luna at elotel.co.luna and download their free trial at elotel.co.luna-free-trial.
00:07:45
Speaker
All right, welcome to the show, Charlie. Welcome to Kubernetes Spites. It's good to have you here. I'm excited to learn a lot today, but why don't you give yourself an introduction for our listeners and a little bit about what you do.
00:07:56
Speaker
Okay. Great. Yeah. Um, so yeah, thanks for the invite and thanks for having me. Um, so yeah, my name's Charlie. Uh, I work on the developer relations team at styra, uh, and with the, with the open pro with the open project on the open project. Uh, my focus really is around making open easier to learn and easier to use. I spend a lot of my time working with.
00:08:17
Speaker
new users of the project, new maintainers or new contributors, people who are interested in doing their first open source contribution often, and also taking part in events and representing the open project at various events. We've got a coupon next week, so that's something that's coming up. With those different things, I find plenty to do.
00:08:39
Speaker
That's awesome. No, I think, like, thank you for taking the time out, right? I know we are recording this pre-cube console. Appreciate that. And since we got connected, I saw you also got promoted. So congratulations, Charlie. Like, now you are, I guess, a senior developer advocate. So that's awesome. Yes, that's right. Yeah, that's my title. So yeah, thanks very much and very observant. Very observant of you. Yeah.
00:09:02
Speaker
Let's start by talking about OPA. OPA or Open Policy Agent. I want to learn more about it, as Ryan said. This is going to be a learning experience for us. If you have listened to our previous podcast, we have done Kubernetes Security 101 episodes and we have had vendors
00:09:20
Speaker
like armosec and like mondo talk about like the scanning piece like once your workloads are already deployed on your communities cluster. I want to learn like if you can give us like a high level overview and then we can dive deeper around what opa is and how it helps improve security postures for an organization.
00:09:37
Speaker
Sure. So OPA is a general purpose policy engine. And I suppose very much part of OPA is the policy language, which is called Rego. So I think of OPA really as a whole as those things. So the language and the policy engine to evaluate policy in that language in the context of some data
00:10:00
Speaker
There are lots of other parts to it, I suppose as well. We have ways of integrating with OPA with various different languages. There's the community and the support and the ecosystem of tools built around OPA, some of which we'll touch on today. But at its core, OPA is a general purpose policy engine and policy language for making policy decisions.
00:10:23
Speaker
I think that's hopefully a good overview. Yeah, that works for me. Can we talk more about some of the use cases? I know last time we spoke, we were talking about how it helps you do two or three different things instead of just one thing that relates to admission control. Yeah, I suppose you're interested around security, how it helps people make their platforms and Kubernetes classes more secure. Absolutely.
OPA Use Cases and Deployment Scenarios
00:10:51
Speaker
So like, or where it fits in in that in that stack, like as a runtime security piece, is it something that you run when you're building your applications and whatever. So, so yeah, really, it's the most common use cases for us is authorization, that's mostly
00:11:08
Speaker
the use case that we tend to focus on and a lot of the work that we do and the content we produce is with that as the focus. So anywhere that you are doing authorization is a kind of top tier use case for us that would often be perhaps applications within your platform talking to each other, communicating with each other, sending each other messages and whether those messages are allowed or not.
00:11:33
Speaker
And you can think of potentially different ways in which that might happen. So that might be happening in your service mesh, where it's an integration with the service mesh proxy and the service mesh proxy is consulting OPA to make decisions around what messages between services can be sent. But it could also be the application itself is integrated with OPA and making calls to OPA using the REST API.
00:11:56
Speaker
It could be it's an API gateway or some other central piece of your infrastructure where policy decisions are being enforced for a whole range of applications and so on. I suppose those are the kind of application authorization use cases that are kind of the most prominent for us, but there are sort of Kubernetes specific authorization
00:12:19
Speaker
or authorization-like things as well around who can deploy what into which namespaces, what operations can they perform on existing Kubernetes state. And Kubernetes provides, perhaps we can get into this in a bit more detail later, but good integrations to control what's going on within a cluster as well. So I'd say that those are, when I think about
00:12:46
Speaker
I think about the use cases where that's probably where I would start. Like I said in my introduction, OPA is a general purpose policy engine, and you can use it for more than just authorization. You can use it anywhere. When I'm talking to people, as I'm expecting to do lots of next week, I'm always encouraging people to think about
00:13:08
Speaker
that they can often think of one use case, and that's why they've come to speak to you, but try and encourage people to think about the other use cases, other places they may be running policy.
00:13:18
Speaker
And particularly, Kubernetes people are very familiar with this Kubernetes submission use case. But often, because their team isn't responsible for it, maybe they don't really think so much about application authorization. Or maybe because their team, again, isn't responsible for it, they're not thinking about the permissions on accessing data stores, or controlling what happens in large monorepos, or other infrastructure as code stacks that are unrelated from Kubernetes, perhaps.
00:13:46
Speaker
Yeah, really, I mean, we have a range of out of the box or kind of open source part of open core integrations with different projects, but there's also the open ecosystem, whereas a whole range of different things that people are building with open. So yeah, I mean, as I said,
00:14:04
Speaker
To summarize, authorization is our tier one thing that we generally focus on and make sure we have a very good story for, but it is intended to be a very general purpose tool and therefore you to evaluate policy in a standard way in a whole range of different use cases.
00:14:21
Speaker
Yeah, it makes sense. And I'm just looking at some of the documentation and things like that in the background here. It was actually really... I didn't know how I felt about myself seeing so much YAML until I saw what the declarative policy would look like in Python. And I was like, oh, that's actually quite nice. It's not just more YAML, I should say. So is this declarative policy mostly done in Python or I guess it's... Oh, Python.
00:14:48
Speaker
I'm not sure exactly which page you're looking at, but Rego is the language. It looks a bit like various other languages, but it is its own language. It is its own thing. Okay. It looks very much like Python if I was, you know. Yeah. It takes, I suppose it's, I mean, it's
00:15:04
Speaker
Most, I suppose like this, the syntax has, is, you know, it's evolving, but, um, is mostly set now with Rego V1 and Opa V1 on the horizon. But, um, I, when I first started using it, I always thought it looked quite like Ruby, but then the assignment operator is quite like Go, but, but it's, I suppose it's kind of most sort of, um,
00:15:28
Speaker
natural ancestor or inspiration would be something like data log. Depending on how you look at it, maybe SQL is a query language. Even though it maybe looks like an imperative language or looks like Python, I think when people start to use it, they often find it works very differently. It does work very differently from an imperative language.
00:15:54
Speaker
Got it. Okay. That explains a lot. Good to know. So, you know, I know it's a it's a general purpose policy tool, and it works with more things than just Kubernetes, right? I see some
00:16:09
Speaker
WebAssembly stuff, some AWS CloudFormation stuff. We're going to focus a little more on the Kubernetes side of things, but maybe before we jump into the integration and how it works with Kubernetes, maybe talk a little bit about where Opa sits in the various use cases. So if you're using with CloudFormation or if you're using it with that kind of... Does it usually sit with the application? When you're writing policy, does it
00:16:37
Speaker
sit with the application or is this more of like a sysadmin sort of tool? Yeah, no, it's a good question. I suppose like the use case really defines where Opa is running. Okay, so when you
00:16:53
Speaker
So let's imagine a few different use cases and think about where Open runs. So you're running, you're building an application and you want to run some policy over that application's dependencies, for example, which is something that people do with Open.
00:17:11
Speaker
In that case, OPA is probably going to be available as a binary or running as a container stage within a build pipeline. So you would pull the OPA container or download the binary and run some OPA command based on some structured data about the application's dependencies and fail the step if the policy violated.
00:17:32
Speaker
So that's one example. Another example would be the Kubernetes admission example. So you would run a replicated deployment of a number of OPA pods on a Kubernetes cluster. The control plane would have validating webhook configuration to point to the OPA service within that cluster. So the control plane would reach out and OPA would respond over an admission review saying, is this
00:17:59
Speaker
resource allowed or denied. And so in that case, Oprah is running very much like any other Kubernetes workload. Thinking about another use case would be like, we have an integration with the Envoy proxy, where you can run an Oprah sidecar adjacent to your Envoy sidecar alongside your application. And in that case, Oprah is running in the sidecar model next to each application. And
00:18:27
Speaker
So yeah, I suppose like there's three different use cases, three very different ways in which Opa is used and invoked, I suppose. And really, it's kind of like, I suppose to think about one other use case, and you made a brief reference to the WebAssembly.
00:18:44
Speaker
WebAssembly way of building Rego and evaluating it as well. Like that was a feature that was added to OPA's capabilities in order to allow people to run OPA where you can't run normal binaries or you don't have access to run containers and things. Examples being kind of cloud, sorry, CDNs, edge locations and things like that.
00:19:12
Speaker
you know there with those four there are others but like that that gives you a pretty good spectrum of like these are the different ways in which you um in which you you might run open very cool and yeah we'll we'll put a link to uh i guess the ecosystem page or something because there are a lot of integrations that i i'm realizing now that uh you know it being general purpose there might be a use case we don't cover here on this show so if you're listening uh we'll put that in the show notes where you can kind of learn a whole bunch more too
00:19:39
Speaker
So, you know, let's double click on the Kubernetes integration, right? So, with OPA and the, I guess, the emission control use case, you know, how does that integration work? I think it's called Gatekeeper. Can you explain a little bit more about that project and kind of what the capability is? Sure. So, Gatekeeper is a sub-project within the wider open policy agent project. Okay.
00:20:06
Speaker
Gatekeeper is, I suppose, an extension on OPA for the Kubernetes admission, validating admission, mutating admission use case very much specifically. And Gatekeeper has been
00:20:24
Speaker
adopted by various cloud providers in their managed offerings to offer this functionality in a managed way, admission review and mutating functionality as part of the managed Kubernetes offering.
00:20:42
Speaker
So, but often people are running it themselves and it works much the same way that I, as well as any service which backs validating web configuration in that you run a deployment of Gatekeeper and it can respond to admission review requests as new, typically as new things or changes are landing in the Kubernetes API. The Gatekeeper project have
00:21:12
Speaker
have a few extensions, I suppose, that make it, which people find valuable when using it with Kubernetes. One of the main things that people use Gatekeeper for is to allow them to define their policies in custom resources. So if you are one of those teams who define everything in Kubernetes state, you make heavy use of controllers and so on, Gatekeeper is quite a good fit.
00:21:38
Speaker
In particular, if your main policy use case is actually Kubernetes, is only really Kubernetes. You don't have an additional state store for your policy to hand and you want to just make use of the Kubernetes API to store the policies.
00:22:00
Speaker
Gatekeeper is there for people in that use case. They also have a CLI tool to allow you to run the same policy against resources prior to deployment. So you can do a kind of pre-flight check. And Gatekeeper also maintain a library of policies for common use cases.
00:22:21
Speaker
common Kubernetes use cases. So for example, there's like pod security policy. When I started using Kubernetes, there was this pod security policy, but now they have this implemented as admission control checks instead. So I'd say that those are the
00:22:38
Speaker
There are a few other features that Gatekeeper has, but I think those are the kind of core things that people find. People who are kind of working with OPA but exclusively in the Kubernetes admission use case, they're the things that they find useful.
00:22:56
Speaker
Charlie i have like a basic question here so communities out of the box has some sort of role based access control right it does its own authorization sorry authentication and then authorization so are there like additional features that a user would get by running opa and gatekeeper i like the pre flight check functionality but like,
00:23:15
Speaker
Are there more knobs that you can turn by running this project on your Kubernetes cluster than just relying on what comes out of the box? Yes. This has been a moving target within Kubernetes for a while. I suppose with the common expression language and integration, native policy functionality landing in the
00:23:41
Speaker
in the API server itself, like this continues to be something which is evolving, but I suppose the way that
00:23:53
Speaker
the way that I came, I started using OPA in this use case. I was working on Kubernetes platform and we had a requirement which was that particular teams could only create internal load balancers or services with a particular cloud provider annotation that meant that their services load balancer would be provisioned as an internal load balancer rather than an external load balancer. Some teams were allowed to provision external ones and some were allowed to only provision internal ones. And this is something that at the time you couldn't
00:24:20
Speaker
I think you could do this now with CEL in the API server, but at the time, this was an example of
00:24:30
Speaker
something which you couldn't do with Kubernetes native permissions. And I think even though you can do that with CellNow, I think both OPA and OPA Gatekeeper have ways of gathering up other Kubernetes resources to allow you to make decisions based on the state of other resources. And that's, I think, something that you can't still can't do without an admission validating admission check. So for example,
00:24:58
Speaker
Another policy we had at the time when I was, when I was first working with Opa was around making sure that teams were using unique host names so they couldn't create duplicates across different names, spaces and clusters. So we would have all of the other Ingress resources and the state of all of the other Ingress resources loaded in to Opa. And then you could run a check and say like, okay, well, based on the existing Ingress resources, is this name unique? And if it wasn't unique,
00:25:28
Speaker
then it would reject the request. I guess we've sold now in the API server, it's increasingly a kind of break glass sort of, oh, I have a business requirement for this particular policy and I need something a little bit more custom.
00:25:48
Speaker
I think that in particular is, or at a high level like that's where OPA, but also any kind of custom validating admission tool fits in. Both OPA and Gatekeeper have also ways of bringing in external data in addition to external Kubernetes states. So I talked about bringing in the list of ingress resources, but you might want to also bring in data from another system.
00:26:14
Speaker
And OPA and to an extent Gatekeeper have a good ways to do this as well. So sometimes you might be basing it on permissions that are defined elsewhere or data which is defined elsewhere. So yeah, I think that's the main thing.
00:26:31
Speaker
Gotcha. Thank you, Charlie. So, like, since you were talking about authorization, right?
OPA and System Auditing
00:26:37
Speaker
That's great that OPA helps me authorize different aspects of my application stack. But as an access admin persona or the operator persona, can it also help me like audit, like which parts of my applications were trying to do some things or what requests got approved or reject? Like, is there a UI or a CRD based resource that I can query to look at what happened in my cluster?
00:27:00
Speaker
Yeah, so good question. Like, auditing is very important.
00:27:06
Speaker
feature for us, it's a feature of Gatekeeper as well. It works slightly, it works slightly differently from in OpenCore versus how it works in OpenGatekeeper. And again, there are, you know, the Styrus product has a UI, there is, so to allow you to view auditing data about decisions that have been made, but just to roll back to,
00:27:34
Speaker
like what's going on there like when the resource arrives or when a decision authorization or any decision is made by opa or gatekeeper there is a decision that's being made and in addition to responding to the caller so
00:27:50
Speaker
Just to bring in some terminology, there's a policy enforcement point and a policy decision point. A policy enforcement point in Kubernetes is the API server, but a policy decision point is OPA or gatekeeper or any validating mission integration. In the service mesh example, the policy enforcement point is the proxy, the policy decision point is the OPA sidecar running in the same pod.
00:28:19
Speaker
That policy decision point, it responds to the policy enforcement point, but it also is able to log in various different ways a decision that's been made. So OPA produces these batched decision logs, which it can send to an HTTP endpoint. It just so happens that the styroproduct has a way of gathering these up and displaying them in the UI.
00:28:40
Speaker
For Gatekeeper, it works slightly differently. They send audit data to a PubSub sync that you can configure. I think they also support some way of storing recent audits or recent decisions in Kubernetes state as well. And at least state that used to do that.
00:29:02
Speaker
I know you said. But the PubSub is like, they're kind of, I think the main way that they encourage people to look at all the data that they engage in. Okay. So HTTP and PubSub, like, is there any way to feed all of this into Prometheus and Grafana? Or this is not really like, it's not generating metrics information, right? That are time series based information.
00:29:24
Speaker
Yeah, so you can see, OPA does expose Prometheus metrics and things, but I suppose the information that you're, well, I suppose we're a fan of these data includes log-based metrics, but typically, I suppose in the way that I slightly outdated, thinking on the matter goes in that decision log entry,
00:29:53
Speaker
It's to me very important that the decision log entry contains enough information in order for you to replay the decision. So what you need to do that is you need to know what the message that came in was and what the policy version and data version was at that time. And then you need to log all of that information so that
00:30:17
Speaker
three months, a year down the line, you're asked to say, why was this allowed? You can go and get the policy at that version. You can go and you've got the information that was provided to the policy decision point in your log. And you can then say, OK, well, it happened for this reason. There was a bug in the policy or some of the permissions hadn't been updated in the permission store yet and so on. And I think, like,
00:30:47
Speaker
the you can you can get like performance metrics and stuff out of open about like what's your your query decision time and this kind of stuff which is you know operationally important data but when it comes to the the you know same same for gate when it comes for
00:31:04
Speaker
The decision log data like i think the use case is slightly different the use cases around responding to like an auditing type challenge or a security incident type challenge and the information that you have there is. Structured i suppose in a or kind of there for a different purpose may be as i see it.
00:31:24
Speaker
Okay. Thank you. Yeah. And I imagine, you know, while you guys are talking, I'm imagining this use case where like you would have some policy that says, you know, you can't pull from, you know, uh, some registries that aren't in these, uh, pool of registries or they must have a certain label or that kind of thing. I imagine that historical data could be something you could look back at or even monitor, right? To say if all of a sudden we're seeing a lot of violations of this policy, right? Um,
00:31:52
Speaker
that could be something that a team could look at or even have in historical data to evaluate for whatever reason.
00:32:05
Speaker
Yeah, you could do all sorts of analysis on the decision log data. In the star product, there's a kind of representation of different response codes and things presented that people could use, could access the raw data and like you say, do whatever you, whatever kind of after the fact analysis you were interested in.
00:32:27
Speaker
Sure, sure. All right, so I'm going to switch gears a little bit and talk to you about sort of service message. I can speak today. I didn't do my vocal warm-up. So, Opa is or isn't a replacement for a service mesh, or where does it sort of fit in
00:32:50
Speaker
into the way service meshes work. Because we often hear about these tools, service meshes in general, also having sort of policy between microservices and things like that. So explain a little bit of the difference or similarities.
00:33:07
Speaker
Yeah, sure. So, Opa is not a replacement for a service mesh. You might find some of the policy functionality that any given service mesh provides could be replaced by Opa if you wanted to. But there are additional features of the service mesh which Opa has no
00:33:30
Speaker
it doesn't solve or has no intention of solving. I suppose one of the main features of a service mesh is providing service identity, which is, that's a kind of like
00:33:47
Speaker
infrastructural service provided to tenants of a platform to enable service to service authentication. And so like, that's, I suppose, two definite things that OPA is not doing. OPA is not provisioning identities, and it is not authenticating workloads or users.
00:34:11
Speaker
it is you can write policy to authorize requests from services from users based on the identities that they have which may or may not come from the service mesh but it's fundamentally not providing identities and not.
00:34:30
Speaker
are not authenticating users and services. So I think at a high level, that's how I think about it. They are different things. The Venn diagram overlaps where, as you were right to highlight, service messages have some amount of policy functionality, similar to the Kubernetes admission use case where the Kubernetes API server has some policy functionality overlaps with that part, but only that part.
00:34:56
Speaker
It's not a cluster administration, et cetera, tool.
00:35:02
Speaker
So yeah, like people use OPA alongside service meshes. A common way that people do this is we have, you know, for Envoy service meshes in particular, we have the OPA Envoy flavor of OPA, which is built and released in case with the OPA releases, which, so Envoy has an API called the external authorization API, which is GRPC API, which is not supported in the
00:35:31
Speaker
sort of mainline OPA container or OPA distribution, but the OPA Envoy Proxy implements this API. It's also implemented by Staira's Enterprise Policy Engine, but this gRPC API allows the Envoy Proxy to send information about the request it's received, where it's come from, and headers.
00:35:51
Speaker
so on and you can run policy over it and expects similar to the kubernetes admission review expects a particular response back to the proxy in the proxy will respond accordingly or add headers remove headers accordingly there are various different operations you can do.
00:36:07
Speaker
So suppose like, it's open allows you to extend the policy controls of your of your mesh. So, so yeah, like that, I think that's how I how I think about it. Yeah, that makes a lot of sense. You know, one thing I was sort of thinking about while you were talking about this rate, meshes typically add sidecars and
00:36:29
Speaker
Uh, you know, they enforce things like that. And then you have this other container, which is essentially what Opa is doing and kind of, I guess, you know, Envoy could get something and then, you know, basically consult Opa to, to make sure things are.
00:36:43
Speaker
you know, enforced in that policy and then kind of comes back that, you know, does the concept of, um, sort of bloat ever come up or like, what's the performance of like adding these layers? It's a, it's a question we hear often, especially talking about meshes. So I'm curious of, you know, if you get that question as well. Yeah, it's, it's a, it's a very valid question. Um, I think, yeah, so people, people have been talking about bloat since
00:37:10
Speaker
when we first started talking about sidecar-based service meshes for as long as service meshes have been around, right? So the OPA container isn't for small amounts of data. If you load loads and loads of data into OPA, surprise, surprise, it consumes more data and memory. But if you aren't doing that, it's not
00:37:33
Speaker
It's not a particularly heavyweight workload to run alongside all of your application instances. So for certain types of policy use cases, that's totally fine if you're not using lots and lots of data.
00:37:49
Speaker
Okay, however, if you are using lots and lots of data, you can imagine how if you had like a each open was consuming a gigabyte in memory, and you're running this for every single instance, it would be quite expensive when it came down to it, in terms of the resources that were provisioned just to run sidecar containers. So this was
00:38:14
Speaker
I suppose like OPA, we always thought about OPA running close to the application so that you can have quick response times. But that use case where you have lots and lots of data is one of the main reasons that one of the
00:38:29
Speaker
Ideas for a different deployment pattern that we had with enterprise where you would run a larger well provisioned replicated service separately in a quick enough or a low latency enough area of the platform that would allow you to have larger amounts of data loaded in that you would.
00:38:50
Speaker
centric to call to centrally from you could run open in the same way. It just happens enterprise over has some ways of more efficiently working with very large data sets. But that that deployment model isn't invalid. I suppose what I would encourage people to do is, you know, as with all good engineering problems, it's a trade off.
00:39:11
Speaker
What do you care the most about? What is most important? Is it that you are able to make authorization or is it that you're able to make policy decisions over a large data set and you're willing for it to be a little bit slower? Or would you rather have it such that you have many, many policy engines very close to your applications in order to make very quick policy decisions for those applications?
00:39:36
Speaker
I think trade-off is the key term there, right? I mean, there's no one-size-fits-all, there's no one answer. It really depends on the use case. And again, like what you were saying, like if you have security components and mesh components and Opa in there, well, you might be trading off on the fact that you know things are enforced and they're secure and all these things. And yeah, you might spend a little bit more money, right?
00:39:59
Speaker
Uh, I think it might be an okay thing. Yeah. Yeah. So Charlie, like I know you, you were talking about like OPA running as a sidecar container and then working with on Y proxy, uh, at least in the service mesh ecosystem, right?
Centralization and Policy Standardization
00:40:15
Speaker
We see a transition or.
00:40:16
Speaker
at least a push towards more of that ambient mesh capabilities where you don't have to run these sidecars. Is OPA going to have like a similar solution where instead of relying on sidecars is like a centralized component that does all the policy requests? And just curious to know if that's the direction that we are headed into.
00:40:32
Speaker
Yeah, so I suppose in this use case, Opa, if you wanted to use an ambient mesh and you still wanted to get low latency policy decisions, the way to do that would
00:40:48
Speaker
potentially be to run open as an instance on each node, you could continue to run it as a sidecar. And it would again be be up to you really as to like what you wanted. So there is some simplicity in running as a sidecar still. But but you might want to also manage it as a kind of
00:41:11
Speaker
cluster service where there is a local instance for applications on that node to speak to and so on. Like, you know, OPA isn't going to get bundled any lower in the stack. So like, if you want OPA functionality, you're still going to need to run OPA somewhere. So like, I suppose in the same way that you still need to run your service mesh components.
00:41:37
Speaker
on the, on the nodes as well. Um, you're maybe not running sidecars anymore, but, um, so you still want to run open somewhere. So if you want open, if you want, if you want the fastest possible response times from open, you're still going to need to run open on the node somewhere, either as a separate deployment or.
00:41:55
Speaker
set running on that node or as a sidecar. So yeah, I think it just comes back to what people want to do again. The possibilities are there. The deployment modes are supported. It's just like best practice versus something that's special for us.
00:42:11
Speaker
Yeah, but I feel like this is quite an evolving area and it's kind of TBD as to what the patterns will be. But there are some things that if you're interested in using OPA to standardize how you run policy across a whole platform, like you can't get away from running OPA somewhere and so on. But I can imagine people switching up how they run OPA,
00:42:39
Speaker
as sidecars become potentially even less trendy, it might be the trend that people start to run them in other ways. Cool. Well, I know this is like my favorite term so far. I'm going to switch gears again a little bit. You know, something Bhavan and I have often heard from our guests is more and more teams are worried about, not worried about, but are getting into multi-cluster, right? So this might mean,
00:43:07
Speaker
You have on-prem and cloud. You might have multiple clouds. You might have just multiple clusters in a single cloud. And there's various sort of gateways between these or networking implementations. Does OPA live in this sort of multi-cluster management when it comes to policy at all? Yeah.
00:43:25
Speaker
Yeah, I mean, I think I actually think this is a really good use case for open, because I think what when you get to a certain scale, like multi cluster, sometimes means we've got the US and the EU and Asia Pacific cluster. And then that's it. But sometimes multi cluster means we have, you know, this entire platform over here, which is managed by this department, this entire platform over there, which is managed by an entirely different department.
00:43:54
Speaker
they're kind of adversarial to an extent, but they work for the same company. So like multi cluster, I suppose, is a very broad definition of, of like, sort of infrastructure sprawl. And I think, particularly in an enterprise size businesses is quite common scenario where the sprawl, the sprawl level is quite high. And like, multi cluster is actually multi platform and like multi team multi division and so on. And I think
00:44:22
Speaker
where OPA, this is actually a really good environment for OPA to run because what you can do, I think can be really powerful about OPA is actually giving you a way to standardization. So applications running over here might be in these languages or tend to be using these frameworks and languages applications running over there, working in a slightly different way. But
00:44:52
Speaker
you know as a security team you or a kind of
00:44:56
Speaker
wider scope, a platform team with a wider scope to the whole estate, you might be interested in making sure that policies are consistent and consistently enforced across a very large area. And, you know, like I said, they could be could be quite different in lots of different ways in areas. And I think Oprah is great in that in that case, because you can, you know, if you want to make changes to
00:45:24
Speaker
want to make changes to the policy running in lots of different places, you can centrally manage the policy and have Opus running in different, potentially very different environments, still referring to a single source of truth for your policy distribution. Now we have good APIs within Opus to reload policy bundles, bring in new data sets and unload them across
00:45:46
Speaker
potentially thousands or hundreds of thousands even of open instances. And I think that combined with the standardized auditing functionality to gather all of those decisions that are being made in different environments together and being able to audit that data, that decision data and distribute new policy out of band from the other deployment mechanisms which are changing the state in these environments.
00:46:13
Speaker
I think that that is what is really powerful about open the multi cluster, multi platform use case is that
00:46:22
Speaker
you as the auditors don't care about platform by platform or cluster by cluster, like they care about the business and where the customer data is and so on. And, you know, when, when you're trying to respond to the problems set by, by audit requirements, that standardization or having, you know, having a tool to standardize that functionality, I think is really important and very valuable. And I think that's what overfits.
00:46:49
Speaker
Yeah, and the one use case that comes to mind, right, especially multi cluster, and you gave the example of sort of us and you write GDPR comes up a lot is, are there ways to sort of realize, you know, policies or enforcement for something like that, you know, if, if maybe you're making a call to store some data, but you want to make sure it's a certain type or can't be, you know, there's some limit on how long it gets stored, like, where does open fit there in that, you know, there's just one example, like,
OPA and GDPR Compliance
00:47:16
Speaker
Yeah. Um, so I suppose like there are, there are kind of, there are sort of two parts to that. Like, um, it depends how you, like Oprah can't magically find out if you're sending data to the world. Um, but, but, you know, at the same time, like, uh, Oprah is well equipped, uh, and offers you the tools that you would need to implement such policy. So if you have, uh,
00:47:44
Speaker
for the sake of simplicity, a cluster which stores data and runs applications in the states. You have a cluster which stores data and runs applications in the EU. You have two open instances running in those two clusters, just again for the sake of simplicity. You can have those open instances labeled such that they load differing policy versions.
00:48:06
Speaker
So you can still define standard policy in one place, but the policy which is realized and enforced in those two different locations is controlled by the environment that that instance is running in. And you can, again, based on labels, describe the instance's environment. And then I suppose as to what that policy might look like, it might be that you
00:48:33
Speaker
You have a storage service which has a host name, which is something like uswest.example.com, storage.uswest.example.com, and you might write policy that only applies to services in the EU that if they attempt to connect to within a service mesh use case maybe, they're trying to call out to a US
00:48:57
Speaker
host name, they'd be like, Oh, no. Or if it's a post request, but not, you can, you can get data from the US, but you can't post data to the US or something. Like, that's how, how you might enforce that kind of policy, at least in an HTTP use case. So yeah, I don't know. Hopefully that answers the question. I think that's kind of how I think about it. Yeah, absolutely. And I think that strikes a follow up question for me, right? So
00:49:18
Speaker
In that example, you laid out a cluster in US, cluster in Europe. Is there a way, if as an admin I have a GitOps pipeline or something that is deploying new clusters, can there be a way that OPA becomes a part of that deployment and then it automatically pulls things based on labels? Something gets triggered, a new cluster is deployed, somehow OPA is deployed on it, and then it uses its EU tag and pulls down the policies. All of this possible, is all of this automated?
00:49:47
Speaker
Yeah, so we have the concept of a discovery. The feature is called Discovery in Hope, which allows you, as you described, to label new instances, unhave instances.
00:50:03
Speaker
will load a particular discovery configuration and then discovery configuration based on some rego, which runs when the instance starts will define what policy is eventually loaded. So and that is not as long as the policy bundle server is network reachable from the instance which is deployed, then it can pull down and load ready for evaluation the policy that it needs. So you could
00:50:31
Speaker
have the deployment, the deployment of open of an open instance, just again, for the sake of simplicity, but it could be a sidecar or whatever open service in a given cluster could actually be very, very minimal. It could be like, okay, I need to run open and I need it to be labeled as this is an EU deployment of open in one of our EU clusters. And then it, you know, it can be that the
00:51:00
Speaker
that once it knows all the open instance knows it's like i'm an open instance and running in the u and this is where i go to get my policy okay and then once it knows the information it can then go and fetch the policies that's there available or expected to be loaded for an instance meeting those those specifications at that time so yeah it's not like in order to update policy versions and things i think it's
00:51:26
Speaker
You know, it's, it's interesting because it's kind of like, it is slightly, it's slightly different from the Kubernetes sort of default mindset. It's like, I need to make any update involves an update to the Kubernetes API, but I actually think, you know, I'd like people to consider, uh, whether that is the correct way to can think about updating of policies. And, uh, especially when you think about it from this like higher level, um, you know, multi cluster, multi-platform use cases, like, well, actually we might want to roll out some new policy.
00:51:56
Speaker
and gather policy separately and separately, especially if people are running hundreds of clusters. It's like, okay, so I'm about to update hundreds of clusters in order to update policy in hundreds of clusters. And it's kind of like, that doesn't necessarily match what I'd like. Yeah, you still want to release policy in a reliable way that you can be confident about once it goes out, that it works, but we have lots of tools for that too. You can write open
00:52:22
Speaker
Rego tests, you can deploy policy to a separate environment or still run things through dev stage prod if you do that kind of thing. So I'm not necessarily saying, oh, this is always going to work. Test in prod, you'll be fine. But just, I think, thinking about policy being a function which isn't necessarily always defined in the Kubernetes API for the multi-cluster use case can allow you to be quite reactive.
00:52:52
Speaker
give you a lot of benefits of standardization at that scale. And one more follow-up. I think I might have missed this in that explanation, but I just want to confirm. If I add a new policy or change an existing policy, and if I have those hundreds of clusters, I don't have to trigger anything manually. If you have a CD system, continuous deployment, there's an agent running that will automatically pull these new policies.
00:53:18
Speaker
And then force them? Yes, so the way that that would work is you don't need to, like, crucially, I said, you don't need to go around and, you know, update all of those open deployments. They, you can have, so it's kind of just based on HTTP. And if the ETAG header is set different, it will load a new policy version. So like, that's typically how it's communicated. Like, this is where you should load your policy from.
00:53:46
Speaker
download it and load it load it one time and it will know what the e tag on that policy was set as unlike you know it's like on if you're thinking like all now i've got to like make sure my server is setting the head and stuff but yeah you do but also like common object stores will do this for you.
00:54:03
Speaker
So if you just push to a new path within object storage, the e-tag header will be updated for you. So it could be that you just bundle up your policy and push it to S3, and you get this functionality for free. And you can configure how OPA polls. You can make it poll once a second if you want. And the eventual consistency model, I think, generally works pretty well. There are some ways. There are various ways about bringing in data and policy updates more quickly.
00:54:32
Speaker
data updates more quickly but for policy I think like that kind of resolution is usually fine for people. Yeah and I imagine there's also other ways you can do this with like GitOps tools like Argo maybe since it's you know policy is driven at the you know rega language right.
00:54:51
Speaker
Is that a fair statement to say as well? This would be around rolling out updates to your policy using a sort of standard CI CD tool. Yeah. Yeah. So I guess in that case, it would be like, oh, well, there's a Git repo and it contains all of the policy or contains the policy. You know, the CI CD tool might pick up that and it would run some tests against it. It could run
00:55:16
Speaker
some unit tests. It could also spin up an open instance and load in that policy and run some end-to-end tests, making some example HTTP requests, for example, do all sorts of things. Then it could be like, okay, well, I'm going to push this policy bundle to a pre-prod or some kind of canary setup and monitor existing deployment of open. It's like, okay, they haven't all
00:55:42
Speaker
you know the no error in like in the status of open you can and the like health check you can check whether the opens is a successfully loaded a policy bundle and then you can monitor to you could do something like i will make sure that all of a sudden not everything is being denied or whatever check you needed and you could proceed and
00:56:02
Speaker
do like a multi-stage roll out codified in your pipeline in some way. Was that the question? Absolutely was the question. We've talked a lot about that on the show too, so it might be something.
00:56:18
Speaker
Yeah, that's how I would imagine it. You can go as deep as you want, I suppose, on the multi-stage rollout of different policy versions, but at a minimum, you could have Argo run your policy unit tests and deploy it to pre-prod environment and then you could do some manual testing and then you could continue or whatever. But yeah, as long as it can update some HTTP server that serves policy bundles in the end, then you're fine.
00:56:48
Speaker
Right, right. All right, makes sense. Well, I know we're coming sort of to the end of our time here, so I want to make sure you have a chance to... I know we didn't get to talk about Terraform or validating Terraform plans or anything like that, but we'll make sure to put all these links in here. But if there's anywhere someone listening might be interested in learning, getting started, learning more, how do they find you, this would be the time to give you a shout-out.
Learning Resources for OPA
00:57:13
Speaker
Yeah, sure. I'm Charlie Egan III most places, so if anybody wants to reach out to me, they can do that. I have CharlieEgan3.com. Thanks. I have a very predictable styro email, but I'm pretty easy to find myself.
00:57:34
Speaker
Just on the Terraform thing, the tool is called Conf Test. We have a community project in the open ecosystem that's quite good for that particular use case, but I'll provide you guys some links of different bits and bobs. But yeah, like at the very beginning I said,
00:57:50
Speaker
that I was interested in making over easy to use and helping people who are learning. I think it's really fun and I really enjoy writing Rego and that's why I'm here. But I'm keen to help other people go on that journey. So yeah, I mean, just to like list a few things that I think would be really good for people to start with.
00:58:11
Speaker
It's obvious that there's the open docs, but if you're interested in just getting a quick start with the language, I've also worked on this Rego cheat sheet recently, which is a kind of one pager. There's even a PDF that you can print out if you wanted to, to sort of highlight some common or sort of key contact points with the Rego language.
00:58:37
Speaker
I think the language is often the beginning of learning OPA and I hope that this is a useful tool. For people who don't know about it, there's a playground, an online interactive playground at play.openpolicyagent.org. I'll send the link for that too.
00:58:53
Speaker
This is a way that you can provide information in the way that OPA operates where you have some input information, some data which has been loaded in prior to the policy evaluation time on your policy code, and you can click evaluate in your browser and you don't need to worry about
00:59:12
Speaker
running containers or installing binaries or anything like that. I still use it a lot. I don't think I use it less than I used to. I think I probably use it more. I think it is great for beginners and anybody playing around with OpenRigo.
00:59:29
Speaker
I'm trying to get their head around things that work. It's also a really good way to ask for help because there's a publish link. You can share some policies and data, some input, and try and see why with a colleague or in the open slack, which is where we do a lot of community support.
00:59:48
Speaker
In the playground, there's also the linting output. So in the last year, my colleague Anders and I have been working on a new project called Regal, which is a linter for the rego language, which has a large array of different recommendations to rego code. The output from the linter is displayed in the playground, but you can run the linter in
01:00:13
Speaker
on your laptop, you can run it in GitHub actions on some, if you have a Rego policy code base, you could run it too. So yeah, I'll share a link to that too. We're excited to have people try that and give us feedback, but it's, we're very happy with where it is at the moment. Recently, we've added support for the linter and VS code as well. So if you're a VS code user, that would be a great thing to check out.
01:00:37
Speaker
And finally, like the OPA errors, we've been working on this OPA errors section of our doc site, which I'll share a link to as well. It's a kind of common errors that you might encounter when working with, particularly with Rego.
01:00:56
Speaker
I think because of the rego works in a slightly different way from what people are often used to and familiar with, some of the errors can use a little bit of extra information to get to the bottom of what's wrong. So yeah, we've got that too. I'll share the links for the slacks. Awesome resources. Yeah, for sure. Cool. Yeah, that's great. And remember, Charlie Egan 3, not the first or second. The third one.
01:01:23
Speaker
Does it have something to do with British royalty or just that's the username you got? When I was very young, my lucky number was three. Charlie Egan wasn't, I think it was when I signed up for a Gmail account way back. So I was like, oh, I'll have Charlie Egan and my lucky number stuck. Even better coincidence, right? Well, Charlie, thank you for coming on Kubernetes Bites. I learned a lot. I think there's a lot more people could learn.
01:01:52
Speaker
as well, so appreciate it once again, and it's been a pleasure. Great stuff. Thanks both.
01:01:59
Speaker
Okay, Bhavan, that was, I know, quite a learning experience for me, having known nothing about OPA before going into this conversation. But that's the beauty, I guess, of having a podcast and learning in public. So yeah, I'd be interested in hearing what your takeaways were. Yeah. And Ryan, I was in the same boat, right? Like I had done some research. I always liked their Vikings, Viking based stickers and things like that. And the logos that they have, but
01:02:26
Speaker
apart from knowing that they help you with admission control, I didn't really know what everything, like what are the different use cases for OPA. So I was glad that we got Charlie on. Yeah, for sure. Whose day job is to talk to people about what OPA is and how it can help. And he did a great job. Yeah.
01:02:41
Speaker
So again, this was another experience where we learned a lot from the actual episode. We do try to do our research, but even in research, when we read text, sometimes it doesn't really add up to what you can get from a person who deals with something on a day-to-day basis. Absolutely, yeah.
01:02:57
Speaker
So the discussion around admission control and gatekeeper and how they can add to existing RBAC that Kubernetes offers out of the box, I think that is really helpful. I like the rego integration so that you don't have to rely on after your apps are deployed, you can include it as part of your application as well. So when it gets deployed, it has some of those admission control things or it has some of those rules where
01:03:18
Speaker
It doesn't have to take in requests from a specific part, a different microservice that might be part of your application. I think that was really helpful. And I'll just end with auditing. As we all know, it's super important. First of all, you don't want anybody unauthorized to access your clusters and resources. But then if somebody tried to do that, you need a solution that you can go back and look at historical data and how it can help you improve your security posture. So I think those were important takeaways for me. Yeah, for sure.
01:03:48
Speaker
I came away from this conversation really understanding more about how flexible the tool is as a framework, right? So OPA is not just something you can use.
01:03:58
Speaker
in the Kubernetes use case with the Mission Control. But even within Kubernetes, right, I was kind of looking while we were talking and you can do some basic stuff like, you know, everything that gets deployed and goes through these policy checks, you know, must contain this label or the specific host name. Yeah, that kind of thing. Right. There's so many things you can do and kind of
01:04:19
Speaker
back and related to a lot of the IDP stuff we've been doing, you could easily see how an IDP could kind of build this into their golden path where it's like a lot of this stuff kind of just happens as you deploy and let alone the sort of infrastructure and tooling beyond OPA. So that was, I think, enlightening in the fact that you can use it with CloudFormation and Terraform and Envoy and all these things like being able to produce a Terraform
01:04:46
Speaker
plan and send it through sort of OPA to do some policy checks on is this plan compliant before I go ahead and you know.
01:04:56
Speaker
send it. Again, to just add to your statement, like the multi cluster management, how OPA can have that single store where you have all their policies and then each new cluster or each new OPA discovery agent that comes online can pull down those policies. I think that's super important. Like things have to be automated to that extent that it doesn't become a chore. And like security by default, I think is a good place to be in. So OPA definitely is a cool project that I would like to like
01:05:26
Speaker
Learn more about and hopefully you're sure right cuz I feel like open is one of those tools that you find as a as a solution to an issue right where it's like, you know, maybe you're building a platform and then all of a sudden you have the need for these other compliance related or sort of
01:05:43
Speaker
you know, policies, and you're like, Oh, open might be the tool for that. It's hard to for me to think of, like, I'll think from open from day one. So yeah, that's, yeah, maybe that's the challenge for just me being not familiar with the tool, which is probably the reality. Cool. Well, hopefully, you listeners have also
01:06:03
Speaker
Got something out of what OPA is and if you aren't familiar and you learn something then great. We accomplished our goal for today Just as a reminder again May 22nd use the code Kubernetes bytes for a 10% discount off your registration fee for the Kubernetes community days in New York City and this time became prepared like May 22nd is actually a Wednesday, dude
01:06:31
Speaker
Yeah, I didn't even say it this time. It's eight to Wednesday. So middle of the week should give you time to get there and take off. Or if you're already in the city, then even better. But yeah, please use the code. I don't think it needs to be in all caps, although that's how we have it here. But if it doesn't work the first time and you have it in all lowercase, try to milk out. We'll see. I'm not sure that's a real thing.
01:06:56
Speaker
Um, anything else Bob before we wrap it up here? No, I think we had any to wrap this one up. All right. Well, that brings us to the end of today's episode. I'm Ryan. Thanks for listening to another episode of Kubernetes Bites. Thank you for listening to the Kubernetes Bites podcast.