Transcript
Speaker: You are listening to Kubernetes Bytes, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Wallner and I'm joined by Babin Shah coming to you from Boston, Massachusetts.
Speaker: We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
Speaker: Good morning, good afternoon, and good evening wherever you are. we are coming to you from Boston, Massachusetts. Today is October 26th, 2026. Hope everyone is doing well and staying safe.
Speaker: We have a great episode lined up for you today. ah we are going to talk about how Kubernetes is being used inside of Future Play Games. like We have Burke, who's a staff software engineer at futurepe Future Play Games, talking about how they have adopted Kubernetes, what does the decision matrix or decision journal look like whenever they are making any new architectural decisions. And then obviously we have to talk about AI. So we'll talk about how they are embracing AI inside their organization.
Speaker: It should be a great conversation and I'm hoping a learning one too for all of our listeners. So without further delay, delay delay let's get Barky on the pod. Hey Barky, welcome to the Kubernetes Bytes podcast. Can you please take a couple of minutes to talk about introduce yourself and FuturePlay to our listeners and what you do at FuturePlay?
Speaker: Yeah, absolutely. Thanks for having me. It's a pleasure to be here. ah My name is Berakai. I'm a staff software engineer and server leader at FuturePlay. um We are a mobile gaming company out of Helsinki, Finland.
Speaker: ah We're up here at the north a bit. um So we do mobile games. That's what we do. and And my responsibilities are anything and everything related to servers, infrastructure, game backends, and and anything that falls underneath that.
Speaker: Gotcha. Awesome. No, thank you for that. And like today it's a bit cooler in in Boston, but again, I don't think I can complain in front of you that the the fall season is starting into winter. So I'll keep you stay off of that. Okay. So um like you are on the Kubernetes Bytes podcast. So we have to ask you about Kubernetes, right? Like, can can you please tell us what your current Kubernetes architecture looks like, how it has evolved and then what components or what open source projects do you use?
Speaker: Yeah, absolutely. Well, I can give a brief maybe introduction about why do we use Kubernetes in the first place as well. um so game So in terms of mobile games or gaming in general, there's a lot of aspects that we think about back-end. So this can be multiplayer features that players are maybe accustomed to, the listeners, or this could be more about product management life cycles. So we are talking about campaigns and experimentations and segmentations, targetings, A-B testing. So there's a lot of also overlap with, I would say, traditional tech there as well. yeah So this is this is what we built. And then well, the journey that how we got here was that, I guess, four or five years ago, we started to build our in-house stack.
Speaker: um and And during this period, well, you like to host a summer, right? Like how do you orchestrate all this? um and And I guess during my talk in KubeCon as as well, where I mentioned the keep it lean, keep it minimal attitude.
Speaker: um There is, of course, a lot of assumptions about Kubernetes, like its complications and complex ways of doing it. But we actually found the difference between like what it means to be basic versus what it means to be simple. And we didn't really want to reinvent the wheel again for a lot of the aspects that Kubernetes ah serves us. So thus we chose the Kubernetes as our base. We are mainly a GCP shop.
Speaker: So we manage the Kubernetes clusters to control playing on GKE. um um Otherwise, how does it look like? Well, we do have multiple clusters for multiple games and different environments.
Speaker: We do have this singular monorepo with our applications. Our applications are mainly written in Go. um and And they get fan out essentially into all these different clusters ah with different values and um secrets and and um anything that you need for it to run the services. issue but So that that is roughly how it looks like just from Kubernetes perspective and and Kubernetes storage itself.
Speaker: Okay, so you said you have ah one cluster per game, kind of, right? and But then how do you handle the geo-distributed nature of mobile games? like i'm Assuming your games are available in all different countries and geographies, do you have an instance of the game running across like in each of these regions? Or is it just one cluster with, i don't know, CDNs or something upfront? like yeah How do you handle the latency issue? like When playing games, obviously, you don't want any lag.
Speaker: Absolutely, that that is true. um Maybe one thing about our games is that because we're in the mobile gaming industry, um it is not as real-time as what you might expect from a competitive PC game. So we don't so be still value real-time and the latency. So are I think...
Speaker: P99 is like still 150 ms or below worldwide for every single endpoint we have. But we are not trying to run this thing as in like a competitive mode where every millisecond life kind of counts. So it's it's it's not exactly geo-distributed yet, but of course in the future if we need to. um Roughly what you say is correct. um you know We would have different instances, different relays, different edge networks, um essentially for different set of, let's say, lobbies where the players might be located at. That is a different set of challenges. in indeed
Speaker: Gotcha. Okay. And you you brought up one one interesting thing right in your initial answer that you have one principle like, hey, let's keep things simple. ah but Like, are there other principles? i know from your famous, from your talk, you do talk about, or sorry, from from your KubeCon presentation, you do talk about having a decision journal. Like, can you talk a bit more about what, what are those criteria that help you ah figure out or whenever a new trade-off decision has to be made, what are the criteria that you go through?
Speaker: Yeah, absolutely. ah To maybe support that answer a bit, so our current team structure is we're pretty small and lean team.
Speaker: um You don't have a dedicated SRA team or we don't have a dedicated DevOps team. we We are in the mentality of like, you build it and you run it. Yeah, um and yeah but I mean, I think this goes for a lot of small teams as well and small organizations. um So does that mean that every piece of open source or vendor-related product or software library that we introduce into the stack.
Speaker: We don't really want to do it just because it's cool. That is fun. We are all engineers. We like to have fun with new and shiny toys, but we still got to keep in mind like how how much it ah of an effort that it takes to get there. um So thus, in the the decision journal was a system that and still is a system that we've been using from the start.
Speaker: Okay. And the mis ah it like the main points from this journal is, so the one thing is ah how hard or how easy is it to make a change when you're making a decision.
Speaker: That is the the biggest thing. So for example, the Go decision that we made, like why don't we go go, that is a hard one. But for Kubernetes, it's actually, even though it seems difficult, it's bit on the medium side because technically we can still containerize and then run it somewhere else.
Speaker: um Or data structure-wise, for example, we use document-based database. Technically, we can convert this into a table database as well and then running on a Postgres or something else, not Mongo. um But then that will be like, okay, how do we convert 10 plus million players' is data? like And while keeping the real time. and So that becomes hard. So these are kind of the things that we go through when we're trying to make a decision.
Speaker: yeah and this, at least the the last thing that you mentioned, kind of feels like the Amazon, I don't know, principle of one-way doors versus two-way doors. It did definitely feels like that's one of the top things in your mind, right? Like, hey, we let's choose Kubernetes. But in the future, if we had to move to serverless, we can just package up those containers and and run them in a serverless as well. ah So nothing is is set in stone. Like, it it gives you the flexibility and portability that you would want from any of these decisions. Okay. You mentioned document database, right? So is that like something that you run on Kubernetes or as part of this decision journal, right? Like you guys figured out, hey, it might be easier for us because we are the ones that have to run it consume things as a service. And then how does that play into other decisions? are Like do you, in your architecture, is there a mix of ah hosted services versus self-deployed and self-managed services using operators? Like how do you go about that? Can you please talk about it?
Speaker: Yeah, 100%. It's mixed. So we started with MongoDB in managed and it still is. So it's from Atlas service that they call it. um So we do have a couple clusters there for again, different environments and different games, different shards, capacities and different replica sets and different machine types, whatever you want to imagine there.
Speaker: um But, for example, we do also have a ClickHouse instance where we store, it's not our main data lake, but it is a secondary data lake for different purposes.
Speaker: And that is in-house you know in a Kubernetes cluster. um For example, Kafka is also from Confluent, like as a managed solution, um but that might also be changing, like some Again, these are like really flexible things for us, and depending on the workload bandwidth that we have, do we have time to maintain the thing that we are using itself? That is a bit of a business decision as well, right?
Speaker: ah Because it's like, do you really want to buy it or do you want to build it and maintain it yourself? Of course, we're not going to build a database, but just even the maintaining part is a challenge. No, no, no, 100%, right? And especially given the lean size of of your team, right? Like every decision has to go through that buy versus build ah criteria. um Okay, so like given that it's a lean team, how do you support your developers, right? Like how much handholding or self-service exists in in your organization? um
Speaker: What falls under your team's responsibility versus and what you have to deliver as services to your game developers, maybe? can Yeah. Yeah, yeah. um I would say when when it comes to game developers definition, it depends on the studio, of course. But but being vain when we say a game developer in a studio, our usually customers are not just like client engineers.
Speaker: okay that That is also, let's say, product managers and designers and and um player experience people, marketing people, everybody, especially these days, it can touch to everything when it comes to a central tech platform of a gaming studio.
Speaker: um so So we try to cater to, of course, everybody. ah that That's our goal. It is a challenge. It's like a different priority challenge every single week. But um I would say starting with maybe, for example, the product manager side isn't like developer. So we try to build you know scheduling tools, targeting tools, and I think I mentioned earlier, like experimentation tools. For designers, this can be also also overlapping with with the client engineers.
Speaker: We might be building game features itself. so So the server team, again, it's both the application side and the infrastructure side. So we might ourselves be be building these features such as tournaments or friends or social features or I don't know, mailboxes, but you know, in anything that you can imagine in a like let's say a mobile game.
Speaker: And then of course there's the client engineering side of things. And there I think it is a bit about, okay, how do we implement it and how do we make the um client engineers happy. um And there we do have an SDK where they can implement into the game engine to to utilize the the systems that we have. We do have open API specs that gets updated automatically and deployed, and then they can just access it from the tool of their choice um and then essentially the build on top of it um however they like to. So this is like the main platform services, I guess, that we try to ah provide.
Speaker: And for ourselves, of course, it is also, ah you you like to you know ah improve the developer experience of ourselves as server engineers too. And I think the the main tool that we have there is, for example,
Speaker: we can, like every developer gets their own tenant. um so So local development is a really great thing, but we've also known that, you know, yes, Docker exists, but it's, you know, there's always this question about, know,
Speaker: you know it works on my machine. so so so so So yeah, that that that kind of thing. um So we still found, like for example, being able to test it on actual infrastructure, a good thing. So you can just type in, you know build a tenant and then the service that you are building locally and in 10 seconds, it just gets deployed there. um you're Isolated from everything else, you can just experiment on a real thing however you like to. So these kind of things also, like when it comes to, ah i would say, self-service that developer experience category is an example.
Speaker: No, that's super cool, right? Because if i if as a developer, I can go from my machine to an actual tenant, or I'm assuming that's a namespace on a Kubernetes cluster in 10 seconds and get my app deployed. I think that's ah that accelerates the the testing timeline by a lot.
Speaker: And the ability to interact with close to real world data. likes I'm assuming that you use, I don't know, clones or read-only copies of your MongoDB instance to make sure that Whenever it gets to that scale testing, things are tested with with real enough data, real-world data to to make sure everything works before it actually gets pushed to production.
Speaker: Absolutely. yeah okay Okay, nice. And again, being a product manager, when you said you also serve product managers, like is this like ah we see on LinkedIn, right like like a lot of suddenly after AI, everybody's an influencer because AI likes them, helps them write.
Speaker: LinkedIn posts. But one thing that we have seen is, ah hey, the PMs are now supposed to come up with prototypes. Is that something that happens at future play? Like, hey, it's not just about writing PRDs and Jira tickets and documents, but it's also about, hey, can you show me something very basic that to help show the value of the feature that you're recommending and then devs can pick it up? Is that what you're supporting? Or can you talk but more about how you support PMs?
Speaker: Absolutely. that That is also one of fun part of it ah these days. um So in the past, like you said, it was bit more traditional. They were, hey, we want this feature, we need this feature. We break it down and then yeah implement it using, I guess, whatever the the team's workflow is, agile or something else. um And these days we get a lot of these prototypes, um and and And I i think I found, at least personally, I found this really valuable because you know communication is really hard, I guess. um you know However the language barrier is is there or not there, there's always things that get lost while trying to communicate a requirement or a feature. And then there's all these iteration cycles that happens back and forth.
Speaker: um I found that, you know, especially with the OpenAPI platform that we have, um that everybody can extend upon, of course, within some limitations, ah they can build their small tools.
Speaker: ah This of course opens lapa like a different, completely different door onto, okay, should we now have 50 to 60 different tools of the same thing. That is a different challenge. I think everybody's trying to figure out like how do we go from there. um But certainly it is something that we are valuing, at least. Gotcha. Okay, interesting. So like the the conversation so far has been like, yeah how how do we use Kubernetes? But one of the things that that keep coming up, even in the feedback that we get from our listeners is, that's great. right like it' It's not a solved problem, but it's still it's great to learn from each other. But the the brand new or the blue ocean is AI on on Kubernetes or AI inside real customers, not just vendors talking about it. So let me switch gears and then talk about like AI now. right Can you talk about um how your team uses AI today to manage Kubernetes and the rest of the infrastructure, right like the MongoDB, the Atlas thing that you mentioned, or the Confluent Kafka thing that you mentioned? like How does AI play a role in and and be it be that teammate or submit something something in your in your team?
Speaker: yeah Yeah, absolutely. I think I can quickly recap the overall sentiment that we had over the years because we've been trying to experiment with LLMs and AI for the last two to three years, I guess, and and we still do.
Speaker: um So it started with, I guess, everybody's favorite, let me just write this small script. And if you remember, or if the listeners are remembering the initial iterations, it couldn't even write the proper syntax, right? You would just like get a copy and paste, and then you would try to refactor the whole thing, and then you would ask yourself, why am I even doing this? Then kind of the the copilot came, and the first moment like every software engineer hit the tab, it was like a magical thing. oh yeah. or Like, oh, wow. um It was like a mind-blowing thing, but then that kind of also slowly normalizes and, you know, kind of died down. And now we're in like new, fast, organically shifting environments, right? ah Where something new comes up every quarter. So,
Speaker: Whatever I say now, it might not be what we're doing next quarter. I don't know. that that That is certainly something that I wanted to also like ah set up here before I explain our AI usage, because it is constantly changing and it's always good to experiment and iterate like as fast as possible.
Speaker: yeah especially around this topic, I would say. yeah um When it comes to what is being used within the team is, of course, again, we are main software engineers that they are trying to run the infrastructure. So from software engineering perspective, it is pretty typical agentic workloads.
Speaker: development cycles of fighting what everybody knows these days. So using cloud codes and codexes, um um especially around the again the tooling side where the impact radius is, of course, it's still important, but it's not like, hey, we wrote one wrong code and then it just blew up 11 million people's games. That is maybe not a good thing. so So especially around the internal tooling area, we've been taking a lot of advantage of this faster iteration speeds.
Speaker: And also when it comes to the infrastructure side, side I think like, you know, Going back to your original question about how do we use AI into the infrastructure, I think the main thing is exploring our familiar stuff for us, like reading about the documentation and, hey, the ingress is dead now. We need to move to Gateway. Okay, how do we do this?
Speaker: um And then we can maybe try to prototype something real quickly. i don't in gateway way to you know see how would that works for us, rather than spending like days to read line by line. Of course, we still read everything. like That is a debate, I know, like whether you read code or not these days. um But we still read the infrastructure side, I can tell you that.
Speaker: um so Yeah, it's be you know that is, I think, one um primary thing that we use for or the infrastructure side. Then there's also the routine work. So for example, there's a new service that is coming up, or there is a new, let's say, a game or some kind of a new prototype, you need to deploy something. It's not that hard, you just copy and paste and find and replace, but It became routine things that you know you don't even want to spend an hour or two on. yeah So you can just prompt it and and it does it in the background and sets all the values and you know different namespaces and different types and different keys and different values and then we just fill whatever that is needed. So that goes for Helm charts, that goes for Terraform files, that goes for CICD configurations. um So those are kind of like you know I have this thing, but I want to multiply it. So it just follows the pattern, essentially.
Speaker: um ah how How do you validate, right like let's say you are going from version 3.3 to 3.4 of some random container image, of how do you make sure that all the instances of it were updated? like I'm assuming this goes and makes changes to like a a CI-CD platform like or or something that you host in in GitOps, right? Like something in Git that gets triggered. um Is there a validation step before it goes and executes these things and makes changes in your production? Or it doesn't do you allow your AI-based modifications to make changes in in production environment or not?
Speaker: ah not Not in production. So what happens is we use GitHub Actions these days um for for our CI CD. We don't have Argo, we don't have other complex layers. If you use it, all the best for you. You don't you know judge. a we Again, it goes into this keep it lean, keep it simple mentality. yeah So ah there we do have different layered approach to deploying an application even on a normal day, let alone AI or LLM. So it goes into, of course, automated tests, and end tests, smoke tests, um and then this happens and then it gets deployed to the development environment. And that's where we do the smoke tests for every service that we have.
Speaker: So if there is an issue with the container that it doesn't run or some endpoint is not working, you're going to see it there. Or the system is going to notice that and say that this response didn't come the way that we want it. And thus it just holds the whole pipeline. um And then we go check it out what happened. um Plus, we still, like I mentioned, the reading code part, the infrastructure part, I believe everybody's reading in the team, at least I am. um But yeah, I trust my team and I'm sure they're also reading and they're making sure that like the LLM changes for the infrastructure part is like set up correctly. No, no, no. Agreed, right? Because I still remember like 10 years back, um the employer that I was working for had like an engineering support team, right? They were not the IT team, but like supporting internal developers. yeah Whenever a new VMware vSphere upgrade came in, I think they spent
Speaker: maybe three months just like um ah doing those upgrades or testing those upgrades over and over again in like a test bed and then running workloads, but making sure all of the features don't don't break anything or there are no breaks um when when they actually go through this process in production. It's funny that that timeline, based on what you shared, it has accelerated so much that now you don't have to spend a quarter just to, obviously we can't, right? Like we can't afford to do that anymore, but it's funny that that's the point where we have reached now.
Speaker: um Okay. That's great from an infrastructure side, but let's go back and and talk about developers, right? Like we, we went from that tab completion of a, of a, a line of code to an entire function to now so much code being written by AI. um ah how How have your developers adopted it? like What does that workflow look like? I know you were talking something about the Slack ah integration. Can you talk a bit more about that?
Speaker: Yeah, um so i would say it's still a, like I mentioned maybe earlier, the iteration still continues. I don't think we have reached a like a conclusion, and I don't know if we will ever reach a conclusion about this.
Speaker: um the The current state is the take usage of live codecs and and cloud code to do our application development. um So it's of course different from live coding.
Speaker: It is, let's say, agenting development, I guess is is a term where still software engineers at the end of the day is responsible for what goes out. So there is no like, I didn't do it, AI did it. We don't really accept that. I mean, that that is like a conclusion that we came with the team as well. So we are like, ah you know, you of course, all blameless retros and, you know, um incident reports, but um AI is not an excuse still. it Might be one day, I don't know. Otherwise, otherwise yeah, definitely um just building applications um and
Speaker: An example, again, with how easy to kind of let it follow the previous patterns is in Goal, we use hexagonal architecture that is pretty standard for a lot of applications.
Speaker: um And there we, for example, have a lot of boilerplate code that needs to be handled even before you start writing the business logic. um Normally, you would just again copy paste, find and replace, spend half an hour, an hour, you can just like, just follows the whole pattern immediately. And then you can get to prompting or LLM assist is writing and the actual business logic on a new service or an existing service as well.
Speaker: ah So those are kind of the main points about the developer side. The Slack um side is a bit of, ah I guess, this year's developments where we are trying to take it to our next steps. And it also ties into, I would say, the the question that you brought up earlier about like product manager side and how do we interact and that prototyping phase.
Speaker: um And it it is an attempt ah that we have made to solve that issue. I don't know solve is the right term, but at least like try to clarify yeah because we cannot really stop people from trying out new things and we shouldn't. um But it brings the issue of what if we have, let's say, an internal platform dashboard. Do we want like eight or seven to different instances of the same thing? Most likely no. so what this Slack bot currently does is it takes the request and it goes through the the loops and and and the flows charts that we created. um and And it might ask clarifying questions about the feature, the PM or anyone else like the software engineers as well requested.
Speaker: And then it will come up with a solution within our boundaries on the code base that we have for that internal dashboard. um And then open up a pull request and a preview. It gives like a preview link to a sandbox environment so you can go there and check it live, click on things and see if it's to your liking. If not, you can give feedback. It goes back.
Speaker: goes through the same chart flow again and you know until you're satisfied, I guess, as an end user there. um and And once you're happy, you just press approve, and then it comes to us for now. Again, it goes back to the values of we still haven't automated not reading part.
Speaker: you know We still check the code of the end pull request for now. But in the future, I don't know, this might e evolve into something like ah more automated self-learning systems where we can give examples what is accepted and what is not. And if it is a power threshold or a blower threshold, then it might have a chance of being like auto-merged, etc. So that is, I think, a brief introduction to the that board as well.
Speaker: Yeah, I know. And you brought up a great point, right? Like the eval thing. Right now, when we talk about evals, it's more about like, i don't know, me being in the PM world, like PMs writing evals for features that they are asking AI to do. But yeah, there are no evals for code base, right? Hey, don't use this. Don't use this. There's some flavor in... like spec driven development and things like that but not really like 100 trust that let's just push this ai generated code to to to prod but because it has passed specific checks that were predefined interesting so we brought up cloud code and codex multiple times i i'm curious to understand the decision journal criteria of choosing to go down that route versus
Speaker: maybe self-hosting Kimi models on your own infrastructure, and then getting GPUs and setting all of that up, and then using those for internal AI usage. Can you talk about how you made that choice?
Speaker: Yeah, it is a hot topic and I personally love self-hosting things. I have a surprise, surprise, a home lab and a server rack at my home, like where I do all my experimentations and nerd out on things, including I guess the lens.
Speaker: When it comes to business settings, of course, the tone and and the approach needs to change. um Here, I think the why we did not go with self-hosting. It's mainly how fast things are changing.
Speaker: um That is, ah I think, a real concern um in terms of the investment of of this size needs to be made for a um company like ours. So if you're, in um I don't know, a hyperscaler or a tech company whose business is this,
Speaker: the question might be completely different to But us as end users, it is again and and a different question where ah should we miss, let's say, i don't know, half a million, a million, and then go buy GPUs. If you can't find it, that then where you buy it and then you sell for something, you start using it six months later,
Speaker: either the hardware is now 2x or the model is held in size of the same performance and the output, and now you have all this hardware and the investment that you made hoping that it will outlast next three to five years is now all obsolete. um I don't know if we have some sort of more slow equivalent of LLMs.
Speaker: It could be a new GP out of the oven. um Yeah, 100%. So that that is, I think, the main thing why we still don't run our models. And it's it's not about, I think, technicality-wise, it's always, again,
Speaker: possible to do. So it's it's just about aligning the business goals here a bit. um and And also, I guess everybody knows the cloud and Codex, like two big AI labs, has a lot of subsidized pricing currently, guess, compared to the cost that it it has. So I guess that's like you know everybody else, um you know myself included in my personal life,
Speaker: It's really cheap to just get a subscription and um you know use that. No, no, no, 100%. ah If you look at the Enthropic S1 filings, you will see that the amount of like loss quarter over quarter they have to generate the amount of revenue, all of that is just cost that they are subsidizing for us.
Speaker: um Interesting. So ah is there any part of this AI... cycle or AI end-to-end orchestration that happens inside Kubernetes and say does something run on Kubernetes or everything is consumed as a service and then then you just yeah use that to build your apps? like like Can you talk a bit more about that?
Speaker: Yeah, I would say both. Of course, um us as engineers and rest of the organization and teammates and everybody else in the industry, I think, is just has some kind of subscription they use on their personal machines. But then um for the Kubernetes side, I think going back to the Slack bot that I mentioned, ah that runs on Kubernetes itself. So so we have a Slack socket essentially that is tied to the Slack channels or threads where people can interact with the orchestrator and then the orchestrator will spawn up um
Speaker: essentially workers of different types. So this can be a builder worker, an assessment worker, or verifying worker. And then then we'll orchestrate this via Kubernetes jobs, essentially. um And then then manage their lifecycle and isolation, etc.
Speaker: And then that will be using the codex or cloud depending on what is the context. So the bot doesn't run on local machines. um it it It's managed the whole workloads. They are managed through the Kubernetes side.
Speaker: Gotcha. Okay. And these these you you said the orchestrator, right? Is this some custom piece of software or were you talking about like the Kubernetes API server, for example? um Of course it uses the Kubernetes API, so I guess like to create a Kubernetes job and kill it, etc. But then the orchestrator itself is written in Go. Okay. So it's it's like some kind of an in-house thing. It is really not really complicated. It handles basic like state management um in case of things like, you know,
Speaker: redundancies um and queues and how the work is of being or like assigned to different node pools based on the resources that it needs um and and the job. like it's For example, a question job to follow up a question takes much more less resources than what it takes to build a ah front-end dashboard because it would need to run end-to-end tests, etc. So it That orchestrator is kind of the brain between, like an interfacing between the the request and the database and also um how how the workloads are kind of managed. look And then in end because it it integrates with Slack, it somehow carries forward the identity of the requester? Like how does that isolation work?
Speaker: um Yeah, so I'll say, so the isolation, of course, there there is the... it's It's a huge topic. like There's a name that I see in somewhere where the like the Kubernetes namespace is like ah small chips like trying to protect the door and they just kind of just open the door and the chip just cracks.
Speaker: um So it's it's of course there's namespaces, but it's not just that. So there is some amount of, like on a high level, you know which worker can access which repository, which credential,
Speaker: How can they talk with each other? um How can it run the test? What is it using to run the test? um What is it using to review? Which model is it using? um so And also, for example, scope checking, like even though it is like not necessarily related to Kubernetes,
Speaker: We don't, for example, want people to um change authentication, security rules, or other important system-related stuff within the code base. Even if it comes through Slack or Product Manager or someone else, yeah there are some like non-negotiables where it like tries to scope the thing out first.
Speaker: um And otherwise, the technical side, it Definitely runs its own cluster. At first, I was like, for right like a half a second, I thought about running this in one of the clusters that we have. And then I was like, yeah, maybe to no.
Speaker: um So it got its own cluster. There is different network policies. ah that is attached to it, like for the network, like who can reach what and how it interacts with the outside world or what comes inside. Because it's not ah it doesn't have to be a public cluster because there is no 10 million users reaching out to it, right?
Speaker: um the so The jobs are, of course, like the cleaning upside and the orchestration happens through the Kubernetes jobs lifecycle itself. So I don't have to create something magical for that. So again, it goes back to trying to use whatever we can.
Speaker: um There are things of, yeah, there there is also like small amount of like, I haven't still dug too deep into it ah because it's again, i would say for a project of this size where we are trying to build the platform that builds the platform. We are quite early stages. um So there is also like secure computing involved from Linux kernel as well. how You can isolate different hardware pieces and like the kernels, modules itself, I guess,
Speaker: um on on ah like i made that I can mention on a high level here without going into too much details. Gotcha. No, I think like the the thing that really like interests me is the whole integration and the backend scaffolding that that is needed. right like All of those things you had to design and and implement first before enabling these workflows through Slack. right Because yeah it's a dedicated cluster, but even in in that, whenever a new request comes in, what happens, right? A new namespace gets deployed. These are the network policies get that get attached.
Speaker: So building all of that intelligence into that orchestrator is the thing that that I really wanted to talk about. So thank you for for sharing those details. ah One last question, right? um Let's say you have three different users, ah John, Tom, and i don't know, Kevin. When they are submitting these requests through Slack, does Slack know that this request is coming from John or does it treat just each request as its own tenant or ah an entity inside your Kubernetes cluster. And then it doesn't matter who the request is coming from. It's just isolated and and and secure.
Speaker: No, we care about who John, Tom, and Kevin is. so Because ah again, there's a lot of, I would say, compliance and regulation layers we need to think about, right? um and And there's a lot of, you know, we trust our teammates, but we also verify like by audit logs essentially to keep track of who's doing what and what they're trying to change. So we essentially, each request gets its own identity and we keep track of it.
Speaker: We also have our internal different, like I guess, like permissioning system for different internal usage purposes. Because someone might need something more sensitive and someone might need something less sensitive, do we want to have the same level of permission there? Most likely no. So it kind of follows the same trend here as well, where we are trying to be as accommodating as possible. So there is no true limitations, except a few parts, but the identity gets tracked for other purposes.
Speaker: So in in this scenario, maybe you use something external, like an entry ID or something that helps you keep track of permissions. Okay. Okay. Interesting. Okay. Got it. No, thank you for that additional level of detail. no ah How do you, like across all of these projects, right? Like, obviously we have the decision journal in the first place to decide how how things should happen, but like so many different projects that your team and and you are leading,
Speaker: how do you measure outcomes? ah like How do you measure success? and And what are the metrics that you use for any of these projects? like How do you come up with that? And then obviously, how do you try? yeah I mean, I think the main KPI is how much money are we making, right? yeah that that that that is like you know Because as I guess listeners of this podcast, good people will also know that infrastructure and server and backend side for a lot of end user organizations is a bit of a cost center.
Speaker: We are here to make sure that we can enable rest of the company and there's no blocker and things get moved from A to B as fast as possible and as robust as possible. um and And then they can, for example, schedule the right thing and build the right feature. So I don't think there is a one-to-one correlation we can make. but you know essentially align ourselves with the greater organizational goals there. um I don't want to really track, for example, how many lines of code we are producing. that that is like Especially today with LLMs, that was worse back in the day, even now. Although I'm starting to think, for example, maybe we can track how many the lines of code has been removed.
Speaker: And it's like a success metric because like then we can maybe see who's actually reading the code or not. But but yeah still we haven't done that. Otherwise, another point that we are of course making sure that we track is the uptime and something that we've been proud as well for such a small team.
Speaker: um Over the last three years, I think we are almost at 5.9 of uptime. Like all these years, yeah not not not quite, I'll be honest. It's like four nines and eight nines. So they're just there. They're just there, but it's unlike every month like I'm recapping the uptimes. I'm hoping that we'll hit that magical 5.9 mark.
Speaker: um so So that's that's also the thing that we constantly want to keep in mind. And also players' feedback important to us, like what kind of a performance they are getting. like Of course, there might be a lot of things involved, like client features, e etc. but We still keep those in mind and and and their experiences to make sure that like, is there something we can help with? Or um they reach out for support. How can we make our support team's life better so they can help the players better, etc. So these are, I think I would say, what it means to be successful for us. And also how much are we spending? That is also important. But then the final thing, like is it infrastructure cost citing for every infrastructure thing.
Speaker: Oh, yeah. And like, I want to go back on the five nines, right? Like I know sometimes people have difficulty understanding how they translate into uptime, but getting to five nines means that over the span of one year, you have a downtime of five minutes max, like across 365 days, it's five minutes and 15 seconds. I just looked it up. So kudos, man. Like even if you are at four nines and eight, I think that still keeps you in the 10 to 15 minute mark a year. Yeah. but and Yeah, and we've been doing this for three years, I think like you know this week. I think 4.9 is 52 minutes. yeah And when I was i guess starting this platform first, that was the goal. And and I think still, officially, I would say that is a reasonable goal for everybody to have. But now that we are here, we don't really want to give up like this because it's like so good to say that we have 5.9.
Speaker: That's awesome. ah One last question for you. like What's next? like what What are the other areas that you're looking into? like Obviously, we spoke a lot about AI. um like What's next for you as a team? If you can share anything around around technologies that you are evaluating? And then, obviously, I do want to ask you, like where can people learn more about you, connect with you, and ask more questions if they have.
Speaker: um I would say today we still are experimenting with the LLM side a lot. I know we spent a lot of time discussing today, but I think it has a good reason to be discussed, unlike everybody else. So I think that is still the main focus. Like, what can we do more or can we do more, you know, to rethink some stuff.
Speaker: um What is the good? What is the bad? um And then it come to, and then there is no rule book for this, right? Every team is exploring for their own um platform and for stem themselves. So I think that is still the main thing.
Speaker: um That's why I appreciate you coming on the pod and talking about your experience, right? Like, because as you said, there's no rule book, there's no defined best practices yet. Everybody's just trying to learn from each other and each other's experiences. So without giving away the the secrets of how your organization works, at least talking about, hey, this is how we tried things out. This is how we use it. I think it it will help a lot of our listeners for sure. And a call to our listeners, right? If you have similar experiences, please come on the pod because yes, this is how we we learn together.
Speaker: Absolutely. I think it's one beautiful thing that we have still is the open source community, right? Everybody just sharing their opinions. If you don't agree, you can say why don't you agree, of course, kindly and at least personally. andm I'm sure a lot of people will be happy to discuss. And then maybe we learn something from each other and then you know move on to what is next.
Speaker: ah Maybe my mind is going to change. Maybe your mind is going to change. It's not about like who wins. it's It's about, I guess, being a community. So 100% I agree with you there. So I think that is the main thing. Of course, we'll still keep the Kubernetes upright, we'll still keep the finance, we'll still keep the um new features rolling. Those are like kind of standard things.
Speaker: Otherwise, people can find us well playing our games if it interests everybody. um you know its its You can just free to download. So ah you just go do that. um Or about We don't really have, I think, currently a technical blog, but I do have my personal blog, like barcai.fi. They can just go there and mail LinkedIn. We can chat on LinkedIn or email or things like that. Happy to chat, happy to you know discuss about things, these topics or others.
Speaker: so Awesome. Thank you so much. We'll make sure that we include a link to your personal blog ah in the show notes. Thank you so much, Barcai, for joining me today. This has been a great episode and I do appreciate your time today.
Speaker: thanks for having me. Thank you so much for listening to the episode. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review or sharing it with your friends and colleagues, as that really helps us grow the podcast.
Speaker: You can find all past episodes or learn more about the show at kubernetesbytes.com. See you in the next episode.
Speaker: Thank you for listening to the Kubernetes Bytes Podcast.
