Transcript
Speaker: You are listening to Kubernetes Bytes, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Wallner and I'm joined by Babin Shah coming to you from Boston, Massachusetts.
Speaker: We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
Speaker: Good morning, good afternoon, and good evening wherever you are. We are coming to you from Boston, Massachusetts. Today is November 9th, 2026. Hope everyone is doing well and staying safe. We have another great episode planned for you guys today around AI inferencing on Kubernetes. We have Cedric Clyburn, a senior developer advocate at Red Hat, joining us to talk all about model weights and KVCache,
Speaker: and different types of memory across CPU and GPUs, and how organizations can implement ah inference and model optimizations as they are thinking about running their own models on Kubernetes.
Speaker: Should be a great conversation. So, without further delay, let's get Cedric on the board. Hey, Cedric. Welcome back to the Kubernetes Bytes podcast. Thank you for joining us again. um For people that didn't um hear you talk the last time you were on the podcast, can you please take a minute and and introduce yourself and what you do at RADHAT?
Speaker: Yeah, thanks so much for having me back. I really appreciate it. i I was last on the the show talking about MongoDB and Kubernetes and and storage.
Speaker: Now I'm honored to be back. I'm a developer advocate and an engineer at Red Hat. And the past few years, I've been working on ai inference. So specifically the part where the model is actually doing something, which now has become really popular in the world of agents, code assistants, all these chatbots now where they're using, you know, an incredible amount of tokens, right? um And so I've been working and and helping developers to run their own models. And once they're running models and it works for POC, then it's another challenge to actually run that on Kubernetes.
Speaker: and scale that up. So it's a it's an honor to be back. I i yeah and i'm excited for am excited for this episode, right? Because those are the things that we want answers for. Like, hey, how do i do a POC or how do I run my own models? And especially on on Kubernetes. so So let's talk about AI workloads, right? Obviously, you said that the the recent search that we see with with agents and coding assistants definitely has brought um inferencing front and center. But the new cycle, right, is still dominated by the new model releases. Like, hey Mythos comes along and everybody panics and then Fable comes along. So like that's where majority of the mindshare is. But can you talk about what you see when you talk to customers who want to leverage AI for actual, like,
Speaker: adding value to your business or or running it inside your organization. Yeah, yeah. You see like these new model drops and I'm actually happy to see it, especially in the open source community.
Speaker: um I track hugging face as a good metric of like how many open models are out there. And so, you know, when I first got in this space a few years back, it was maybe half a million.
Speaker: models were out there and the majority of them were kind of BERT style predictor models that could analyze, you know, an Amazon review, right? See if the customer is happy or sad. And now it's like four thousand four million models are out there. Holy shit.
Speaker: You know, the the leading deep seek type of models. And then there's also stuff like fine tunes for specific use cases. So I'm happy to see like these new model releases and I'm happy to see it, especially in the open source world, because there really wasn't that much of a a community a few years back and now everyone's either running their own model locally, like with Olama or LM studio or, or ah with a lot of people I work with, they're trying to say, how do we run our own code assistant for our team? um So the thing is right when you're running it on your laptop, ah you have you run Olama or one of these models yourself? yeah
Speaker: Yeah, so it's really nice if you're just one user um or one or two users and you're sending one request at a time. um But the the majority of these use cases for teams is they have, you know, thousands of developers who need access to a tool. And then now you're starting to deal with all of these Kubernetes um features and concepts like, hey, how do we do access control? How do we do authentication?
Speaker: load balancing, observability, um all of all of these extra components. So you know while it is fun to try it out locally, now it's a challenge to get this running on Kubernetes. And it's kind of a very evolving space right now oh yeah ah in the past few years. And then yeah well like, I know I did run it in the past, like maybe six, like six months back, but then the model sizes just kept getting bigger and bigger. And then it was just so slow on my workstation. was like, yeah, maybe I do need to like get a subscription somewhere because yeah, I don't have a, I don't know, a ah spark system or something. And then really a machine on my desk to run my models locally. was like, okay, figure out how do I keep up? So yeah. Yeah.
Speaker: I can only imagine organizations wanting to do this at scale for their multi-tenant environment and obviously keeping everything secure. Exactly. But how do you answer these questions? right So the last episode that we did was with a staff engineer from Future Play Games. And he said they they do choose to go with plot code and OpenAI Codex because they feel that with the with the pricing, obviously it's difficult to get gpus they cost a lot of money but then even from a pricing perspective it feels like these companies are subsidizing the hosting costs for the users is that true or how like where where do you draw the line and what are some of the benefits that organizations can see when they are running their own models
Speaker: Yeah, yeah. And I think that's a very common story, even even within my own organization at Red Hat, right? We're a very open source company, but we're still using proprietary tools like Codex or, you know, Cloud Code to do this work, right? But the thing is, um because because not everyone should run their own models, right? Maybe you don't have access to hardware. Like these old GPUs I used to use for gaming, right?
Speaker: are now being resold on eBay for more money than I bought it like years ago. So it's kind of crazy to see this. um But it's ah it's definitely like a like a an economics type of thing, right?
Speaker: ah If you're just trying to do a POC or as a developer get started, much easier to just call a frontier model um because these model sizes have really, they've gone from, you know, like...
Speaker: hundreds of billions of parameters for the GPT-4s to now GPT-5 and 6, which are trillions of parameters. And you can't run that on typical hardware. But we'll talk about a little later in the episode, like distribute distribution of of weights and KVCache. It's going to be a lot of things.
Speaker: um But um the subsidization of inference is a really interesting conversation, both for like developers, ah but also... from an economic standpoint. um There's a really good source that I read up on called Semi Analysis. Have you heard of it before?
Speaker: Nope, not me. um it's ah yeah It's a very like ah you know nerdy ah type of ah publication. where They talk everything about inference and everything about model cost and tiers.
Speaker: And they estimated i think six months ago that your 200 subscription like the claude max would give you as a personal user about two uh twenty thousand dollars of api tokens okay yeah so you could say they're subsidizing yeah they definitely are um more recently it's ah it's about to be knocked ah november 2026 it's about fourteen thousand dollars so you're kind of starting to see The amount of tokens you get for a specific plan go down.
Speaker: um But remember, this is the API cost that they're charging external customers, right? Like anybody off the street. You have companies like Cursor who are re reselling, reselling tokes sitting tokens. Reselling tokens, yeah. Yeah, you know, man at the the middle man, who I'm sure get like a pretty hefty discount.
Speaker: um But it just shows you, yes, these are definitely subsidized, but I think the subsidization is going down. And a lot of developers now when they use coding assistants, They're hitting the maximum usage.
Speaker: um And if you're really, you know, in a flow state and you hit that max ah usage, then it's it really impedes what you were trying to do. So um they definitely got us on ah on a hook, I would say. You know, like the the meme of like the have the fishing pole and then there's a carrot.
Speaker: david yeah you know carrot interesting yeah the carrot is on the thing right now yeah eventually we'll get the stick at some point in time once they have i don't know anthropic goes public and they have their first quarterly earnings report and the stock tank they're like okay we need to bump up the prices yeah i can't wait to see that well i don't hope to see that but i can't wait to see that too right true so ah it's an interesting time Yeah, for sure. OK, so now let's say I've gone through this economic analysis and figured out that I want to run my own models. What are some of the SLOs that I need to think about before like going and deploying something on Kubernetes on on day one? like How do I plan for this? Can you talk about that a bit?
Speaker: Yeah, yeah. The SLOs for generative AI apps, um like these coding assistants, are are really different than traditional apps or APIs that we're used to monitoring.
Speaker: um Because, you know, it's it's it's very simple. You know, it's a quick response back. You'll get some type of, you know, 200 or whatever response from an API, right, if you're calling it. um With the language model, because they work with this streaming pattern that you typically see on chatbots where it's one token at time, um well, then you know it really depends on um a few different metrics. So when you're getting a response stream back from Cloud Code or whatever your AI application is, there's the time to first token.
Speaker: And a lot of teams prioritize for that because that makes it feel snappy. As soon as you get that first response, like the, right? It and doesn't really matter how long the next ones are.
Speaker: You're like, okay, it's working. um And so you'll see a lot of different applications, ah you know, prioritize this time to first token differently. um One example is Google Maps.
Speaker: They have a chatbot in the the Maps application. Yeah. Yeah. And what they do is they they wait for the entire response to be generated, then give you that information versus um Amazon's chatbot it gives you token by token. um And once you receive that first token, everything after is called the inner token latency.
Speaker: So think of each character being given back as each token. We'll talk about how that works in a little bit, but okay that's that's the time in between each token. um And then the total the total speed of tokens coming back is the throughput.
Speaker: um So that's how many tokens or or requests can the system process. um And so once you're actually handling 100 users or 1,000 users, that's also really important um because you know everything else is affected ah by the the total load on the system.
Speaker: um But if you're a single user, the first two, the time to first token, inner token latency are the most important. But those are the the three main SLOs that that anyone has to deal with, even if you're not running the models yourself.
Speaker: Because systems like ChatGPT, if you're using the web application, are about 40 tokens per second. um So there are different inference providers, inference being that step of, you know, getting the response back, like Cerebris or Grok, that are really popular because it's like 500 tokens per second.
Speaker: Oh, wow. Okay. Right. yeah um And we'll talk later about the the tricks they do in order to do that. Yeah. And and that that was like one of the questions I had as well, right? Like what's a good metric, right? Like if I'm starting from scratch, is 40 a good number for internal token latency or it should be in like 400 or four? Like, yeah. So I thank you for setting the that, right? Like what is the industry standard or norm? So if I'm designing something, I know what to aim for. Okay. um But let's Let's start by, like I know we we started by talking about Ulama. right like I was running model locally on my machine. um Obviously, there is a way to package up a model as a container. And I think a few years a couple of years back, we had somebody from Microsoft talk about talk about how a model can be a container and it just runs on Kubernetes. And that's about it.
Speaker: right And you could talk to it using the service endpoint. But then why do you need the inference server? right like if if if What is the additional capability that the inference server provides that a model running in a container doesn't.
Speaker: Yeah, yeah, yeah the yeah. The topic of inference and inference engineering is is kind of like a new like a new industry, a new job title that's going to be really important in the next few years.
Speaker: okay And so i'm glad I'm glad you asked that question because when When models first came out, um like the the first llamas, right? What you would do is you would use the Hugging Face Transformers library, and you would load the model weights into whatever GPU you were connected to. And then you would just send a request um using the the library and get that response back.
Speaker: um But you, in your Jupyter Notebook, in your Python ah program, whatever it is, you are kind of handling the inference. um You were scheduling the request, you hit ah you know run, sell, or you ran the Python file, you got that response back. um But you know it's kind of like um like an Apache server, you you query a certain page and you get that result back in HTML.
Speaker: But to be able to handle multiple requests, it's like kind of what you were talking about is they they package it up in a container and then they directed a Kubernetes service so that the traffic would go to that container.
Speaker: um But for these for these these models nowadays, it's like, we're not sending just one request. We're sending an agentic prompt, which is going to say, i want you to inspect this this specific file. I want you to run some tests, run this linter. And then I want you to take a screenshot of this application to see if the code change worked, right? yeah So think about like, you know, a hundred different request going back and forth with the model just for one user.
Speaker: And you start to see how big the demand can can grow for just one prompt from a user using ah a coding agent, for example. um So an inference server, whether it's Lama C++ plus plus with Ollama, whether it's VLLM or SGLang or any of these inference servers, essentially act as a scheduler.
Speaker: So they can schedule requests maybe together. They will be managing the GPU memory. So a very finite resource that we have to take care of. um And as those requests are processed, we generate short term memory called KV cache. um And so if I send the same request again, it'll be much snappier because that's already been processed um by ah by our inference engine. So Essentially, we've come a long way from the Transformers library just running um ah loading a model and running requests to running maybe OLAMA in a pod as ah as a service endpoint to now these very Kubernetes native integrations um where it uses projects like KSERV,
Speaker: to embed with CRDs and you know custom resources into your cluster all of the components needed in order to stand up a gateway, stand up a router, and stand up an inference server.
Speaker: okay So it's that it's it's becoming more mature as we go. And I assume in the next few years, um Kubernetes is is going to change, not change, but you know,
Speaker: it will grow into kind of like an agent native platform where you can host not just the models, but also the the agents on Kubernetes. And it's going to be really cool for us for us all.
Speaker: Okay. So the inference server is there to extract away a lot of things, provide these router and gateway capabilities at scale, right? Like, so it's not just one user with one prompt, it's one user with hundred prompts and then thousand of those users, right? So it's it's built for operating at at that scale. Okay. Okay. Makes sense. Exactly. Well said. Well said.
Speaker: I was just summarizing, Cedric. Come on. Thank you for being so kind. No, no, It's important, yeah, because it's like, at at the basis, you're just, as a developer, you're calling an endpoint. You're calling a standardized open AI endpoint, right?
Speaker: But behind the scenes, there's... ah so many different things happening low level with the GPU ah that normally you don't have to worry about. You can you can really go deep on Twitter or an X and and and all of these, there's a lot of conferences like PyTorch conference that that will talk about every part of the kernel to to to the highest level. um But you know this hopefully serves as your introduction for a lot of people joining us like what an inference engine engineer is what a scheduler is, um and you can really geek out about that, which is cool Do you have specific subreddits that you monitor, like that you want to give a shout out to? Yes, I spend a lot of my time on Hacker News and Reddit. The Local Llama, so local and then LLAMA is my favorite source of information for if you want to run your own models, see what people are doing with models.
Speaker: um And ah it's it's just really cool. Good vibe over there. I recommend it. we'll make sure we include these links as, as in when we, we share those on the episode in our show notes as well. um You already spoke about KV cash, right? So let's, let's talk about, like, let's peel the onion and talk about the ah the technical aspects as well. So what are model weights and what's the KV cash and what does it stand for?
Speaker: Yeah, yeah, yeah. That's a good question because everyone talks about it. um But for me, i wasn't ah I wasn't an AI researcher. I'm just a regular developer. And um and yeah you know this is quite an interesting topic because We think of the model weights as what's released on Hugging Face or what you can download yourself and run on your own machine. It's just a kind of a big binary blob and of the of the learned kind of behavior to predict the next token, right?
Speaker: So it's pre-trained data and and refined kind of tuning, right? And then you have a little bit of a configuration file. so you'll see like ah specific parameters for how the model behaves and ah the architecture.
Speaker: And so this is what would predict the next token, right? But the thing is, ah you know, that's a static file. When you're actually running it in an inference engine, what happens is you auto-regressively generate the next token.
Speaker: So imagine, Robin, you have a prompt. He's like, ah I want you to add new test cases for this new function that I just wrote up, right? Right. All of that input um then is embedded into vectors which are tokenized and can be used by the model to generate the next ah probabilities for what the answer should be. ah So it would start, you know, let's say, oh, it's that test ah test prompt.
Speaker: Well, then the model is going to say, okay, we need to go to this ah class or whatever and and look at these test cases, et cetera. um So that's that's the probabilities of like what will happen. and And when you actually run it through inference, you generate the cache, the key and value cache for all the previous tokens.
Speaker: okay So your input tokens are are computed and converted into embeddings. um And that's just a fancy way to say ah like specific numbers that represent your input. Yeah.
Speaker: But to generate that response, the inference engine ah goes through token by token ah for all that input and generates ah you know what will be the output. But this takes a lot of space um because of how many layers it has to go through in a model.
Speaker: So the model has these... different layers in the transformer architecture that it keeps on going through to figure out what the response will be. And each of that is a matrix multiplication, which then adds up to so much data.
Speaker: And I'm talking about for a standard agentic workload, one prompt that you would give, it could generate 20 gigabytes of space. Oh, wow. In order to generate that. Right. so And this is space of of memory on a GPU.
Speaker: is very scarce. um So the inference engine handles this, but no matter what you use, you're generating that KV cache. And so if you ask the same question again, and you use some maybe the same tools and look at the same file, that's already stored.
Speaker: You don't have to recompute that. So you'll get faster time to first token, enter token latency. um But it really makes, the GPU scarcity so real.
Speaker: um Because if you don't have GPUs in order to hold this cache as you're generating responses, um then you can't you can't do inference. So that's why everyone wants their hands on GPUs, because you're not just loading the model weights, but you're generating KB cache as you send requests to the model.
Speaker: Okay, so the KVCache, like, is it a shared resource between different users? Or is it like, I don't know, each user gets their own cache and and it it maintains some sort of history or cache for the conversation that I'm having?
Speaker: yeah yeah Yeah, that's a good the good question because this is something that's being worked on in the space of gateways. um Because ah by by default with naive you know loading a model into a GPU and and running inference, it's all going the same space.
Speaker: okay um But let's say that you exceed that space, you don't have any more memory, and you spawn in a replica. Well, the thing is, how do you make sure that if I have that same request as you, that it goes to the same GPU that already has the the hot or sticky KV cache, right? yeah And so as you have multiple users, how you know how do you make sure you have those efficiency gains from the caching as you replicate the same model on multiple nodes or pods?
Speaker: So this is a big challenge right now for distributed... Inference is like, okay hey, how do we make sure that requests are going where they need to be? Because by default, our two requests will go to the same GPU unless it's maxed out.
Speaker: um So, you know, with GPUs very scarce and KVCache, a very, you know, on-demand dynamic type of resource, um this is like a really big engineering challenge to to kind of be able to scale models up because everyone wants to, you know, send requests to these models, use RAG, use chatbots, code assistance.
Speaker: um But not all of us have the hardware to do it. And if you're renting hardware, It might be a few dollars an hour, but it really adds up once you have to scale to 16 GPUs or 32 or something.
Speaker: OK, makes sense. It's a great question. So this this work is is actively being done right now. So there is no definite answer. right like if If somebody is interested, they can follow some of those four same forums and participate in those projects and then get to that outcome. But this is still an unsolved problem for our ecosystem.
Speaker: Yeah, it's it's a work in progress. A lot of this work is being done in the CNCF through projects like Envoy and API Gateway and LLMD, which is something that our company, Red Hat and IBM, NVIDIA, CoreWeave, Hugging Face, we're all we we all have the same challenge.
Speaker: It's like once once you try to load a really big model, to effectively route all these requests based on priority. So for example, like ah ah a customer using the chatbot should have a higher priority than an internal user, ah developer, right?
Speaker: yeah So there's a lot of metrics that you can rate in the routing aspects ah through Kubernetes that can then efficiently and intelligently ah send that to the right inference server or the right model.
Speaker: Okay, okay. So okay yeah it's it's all open source work that's being done in the CNCF. And every KubeCon, they make a really big announcement um about either the inference or the agents.
Speaker: And so when you see those keynotes, it kind of, you know, it puts together all these pieces of like, yes, everyone wants an agent that they want to run, but to do that, you have to be able to run inference at scale. and And if you go with a paid provider, you might exceed the budget that you have for AI.
Speaker: So it's, you know, we're very early in the ecosystem, but okay I think really important projects like this, like think of the operating system, right? You know, you had the Unix wars and eventually Linux won because it was open source.
Speaker: And then think about Kubernetes and you know containers and how you know Kubernetes became the de facto. And I think you know open source projects like this will end up also being the de facto because can everyone can learn and use and contribute to the the same problem.
Speaker: Yeah. and And all of this doesn't force me to become an AI data scientist or an AI model developer, right? Like I don't need to understand how to train a model and and build that transformer set. I can just focus on how can I best serve these models. So they are two different challenges, even though oh,
Speaker: oh Even if I don't want to become in that model trainer, this is still something important that I would i have to do as as something in the next, I don't know, two to three years or maybe five years. And then organizations will have ah a leg up if I can deliver a better response to my users in a faster way and then make keep keep a track of those SLOs. Okay. Okay. This is... this is So much fun to this point. Let's, let's I don't know, talk a bit more about the types of memory. So to to prep for this episode, i did watch your deep learning.ai course around inferencing. So huge shout out there. Thank you for for creating that with with Andrew. um you You had a great pyramid explaining the different types of memories, everything from CPU to GPU and different types of GPU memory. Can you talk about what those are and who's responsible for what kind of functions and operations?
Speaker: Yeah, yeah. Thank you for for taking the course or watching the videos. I'm really happy because um Andrew Ng for me has always been an inspiration for learning about what's next and what the future looks like for AI because I've always experienced with Google and Baidu and stuff.
Speaker: um So it was really cool to to to see what he thinks is next. And um he was like, thank God you guys finally did a course on VLLM. was like, yeah. I'm happy we did it too. um But the course, yeah, we dive into the different types of memory because a lot of us are used to just talking, you know, thinking about the CPU and the RAM that you have, right? 32 gigabytes of RAM. um But when you're doing inference,
Speaker: You can do it on a CPU. It's a little bit, c I won't say a little bit, it's definitely slower. um But, you know, the revolution of GPUs is, at least remember when it when it was just for gaming and Bitcoin, it's like you could do so many matrix multiply like multiplications a second, right?
Speaker: We call those flops. um So how many how many multiplications can be done and how much compute can be done in a certain second, right? So... So CPU inference was possible, and it still is possible for things like image ah vision detection to just run that on the CPU itself.
Speaker: But for LLMs, you definitely need a a certain speed, and right? That only comes from GPUs. And the GPU memory is kind of split up. You've got the main GPU memory, which is the HBM, high bandwidth memory. So something like an A100 or H100 will 20 or 80 gigabytes of this.
Speaker: twenty or eighty gigabytes of this And it's going to hold the model weights and the majority of the KV cache. um So it's high bandwidth. It's very fast. But to actually do the compute, um information and specifically data of ah of these requests is moving to the SRAM. So this is the on-chip, the memory that is like very fast, but we only have a small amount of it. Okay.
Speaker: So this is where the compute happens, the weights, and the activations are moved here in order to do the multiplication you going to generate the next probability of what the request will be.
Speaker: And then that's moved back into the HBM, the main 80 gigabytes or whatever of of the the model. um So you've got a lot of data movement here. um And that's why there's a lot of tricks that you can do beyond just running an inference server in order to um speed this up.
Speaker: And we'll probably talk about this later in the episode. And because this you have this certain relationship with you know a high amount of of main memory and then a smaller amount of really fast memory, you've got these Cerebus or Grok type of chips, which have a lot more of thisra so this what this really means is that Your initial prompt was have to be a little bit slower, but you can decode it much faster. Decode is the the stage of generation yeah of of a model's request.
Speaker: um And so, yeah, those are the two stages of pre-fill where you're kind of taking in an entire prompt, getting that ready to be converted and decoded into cat to the cache.
Speaker: And that's compute bound, right? So you need a lot of compute for that step. And then the memory transfer is what you need to decode. So that's why we talk about it a lot of like of this data movement going back and forth from the GPU.
Speaker: And when you connect multiple GPUs together to run a really big model, that might be a trillion parameters, you have technologies like NVLink that make it possible to do data transfer between not just one GPU, but multiple.
Speaker: at ah at at one time. So we're talking about terabytes per second. And, you know, just, you know, the capabilities of GPUs, but they're definitely, you know, in demand. And that's why people are still buying these older GPUs.
Speaker: um Because you can really people found a lot of tricks to maximize their hardware. um And so it's it's really cool with that. You know, the only thought that is in my mind right now is, shit, I should have done this episode last year so I could have bought SK Hynix and those stocks. Obviously, we knew that yeah yeah the Jensen is going to South Korea to to um ah announce partnerships with HBM vendors. But without this, like, hey, HBM is such a critical part of of the entire architecture. Damn, we missed out. I know. I know. I'm sure there's going to be something next.
Speaker: Yeah. Oh yeah. under this is something deep freak after Find that out in this episode. Okay. Let's, let's talk about next year. You'll be a billionaire. ah 100%. Okay, so let's talk about the different ways I can optimize these, right? Like, um ah as part of the same course, I did hear and then learn about terms like continuous matching, prefix caching, things like that. Can you please lay ah lay these out in terms of of both model optimization, inference optimization, whatever categories these fall in, and what are things that users can control ah when they are running inferencing?
Speaker: workloads? Yeah, that's a great question. um Because now that you've learned about what's on the GPU and and why that data movement matters, then you can kind of understand about these different optimizations for the step of inference.
Speaker: um Because you have a couple key kind of concepts like continuous batching that are now part of most inference engines that you use.
Speaker: um But a lot of this came from 2023-esque research in ah and a paper that VLM published. um So one of these is the continuous batching because...
Speaker: Typically, when you had models that were predictive AI, um they were all about the same size of a request. And you could kind of just statically load them, wait for that response to come back, load a batch of 10, wait for that response to come back.
Speaker: But... with LLMs, your request might need 1,000 tokens, my request might need 5,000 tokens. um And so you can't really take this static approach to different sized requests coming in. And so this is dynamic batching and continuous batching. That means as my request ends at 1,000 tokens, another one ah is slotted in to fill its place continuously.
Speaker: And so the slots in a GPU can be used for these new requests without having to wait for this entire batch to finish. Interesting. So that was a big change between predictive AI with the YOLO models and, you know, predicting what's going to happen versus generative AI, which is you give a prompt, we see how many tokens are are generated.
Speaker: So that's that's the biggest first and inference optimization. okay But that's that's just for scheduling, right? um As as i was we were talking about, the GPU only has 40, 80 gigabytes of of onboard memory, right?
Speaker: um And because of fragmentation of requests and sometimes what would happen is we would we would save space just in case a request was big enough.
Speaker: but maybe the request is much smaller, half the size that we that we that we typically allotted for, then we waste that space. um And so before VLLM came around, there was um a lot of this fragmentation happening of wasted space in the GPU. So we we had 80 gigabytes, we weren't using, but maybe half.
Speaker: okay So ah what they did is they came up with this new algorithm called page detention. which ah instead of saving one big block for a request, you would treat it like virtual memory in an operating system, where you would essentially have smaller blocks that can be managed independently, and then you have a table that would reference where those blocks are.
Speaker: So instead of having to continuously have one sequential block saved for your request to the LLM, we split that up and piece it together where it's appropriate and where there's space on the GPU. And this was like a huge improvement. This was saving almost or utilizing like 99, 98% the GPU versus before like 40, 50% was being utilized.
Speaker: where versus before was like forty fifty percent being utilized Okay. And this this translates into real dollars, right? Like I know we are talking about like very nitty gritty details, but each page or blog that gets empty or not being utilized has a real impact on the amount of money that you are spending on on renting that GPU or depreciating that GPU once you bought it outright. You need to make sure that you're getting the most usage out of it. And that's why these things are super important. And the research that's being done in these areas is super important.
Speaker: Oh, 100%. You think about maybe like, ah you know, SpaceX, right? a A big reason of why they were making money and IPO is because they were making like $3 billion dollars a month from renting out their GPUs to Google and to Anthropic. Anthropic, yeah.
Speaker: Right. And so it's crazy. Like if it's, it's $3 billion dollars just for one company. And if you have those resources and you're paying billions of dollars, if you can just get 1% more out of those GPUs, that translates to hundreds of millions of dollars.
Speaker: So like these optimizations are so important as you, as you run things at scale and for a business, it's like, you got to maximize every, every percent you can. So it's, it's really important.
Speaker: So yeah, like SLOs are important for from a customer experience perspective, but this is what's actually important from a business perspective, like TCO perspective, I guess. um And then getting that that percentages up and and something that you meter.
Speaker: But one question, Cedric, here, right? um Is this something that gets turned on by default these days? Like when we were talking about this and in 2026, are these defaults and I don't have to worry about it? Or these are still things that I need to like... keep in mind and and configure correctly as I'm deploying these inferencing servers.
Speaker: Yeah, yeah. ah Thankfully, it's by default. So if you're running Lama C++ plus plus on your on your Mac or MLX or something like that, or SGLang or VLLM on a server, VM or Kubernetes, it's by default. It's the scheduling and memory management that has become the back end of all of these different inference engines. But then later, we'll talk about other optimizations you can do for the model.
Speaker: Yeah, um that gets you a little bit further. But by default, you don't have to worry. It's a part of everything now. um Maybe a few years ago, it was more more work. But now, once you run a container, that container, ah you're good to go.
Speaker: Okay, got it. Anything else you want to cover in inferencing, like optimizing inference workflows? Or or do you want to switch to model optimization? um the The last thing I'll say about inference workloads is like, um especially with with the topic of Kubernetes, is that a lot of teams I talk to have different types of resources. It's distributed across different clouds where maybe some is on AWS or some is on GCP, some is on-prem.
Speaker: um And so these projects that distribute up the the different stages of of inference, the the pre-fill, which is very compute intensive, and the decode, which is very memory intensive, allow you to kind of mix and match ah those those hardware resources to what fits best.
Speaker: um So, for example, if you have a really big input prompt, like maybe it's a batch of 100,000 reviews, right?
Speaker: Then you can you can specify which hardware that should run on. And maybe it it does it shouldn't be the same piece of hardware for both stages. Like maybe you can use CPUs for some part of this and then GPUs just for decode. So there's a lot of work being done with LLMD, um which is the CNCF project in order to one, route the request, but then also two, to split these stages up.
Speaker: and And this isn't the model, but this is just the... um the the distribution of of of compute and an incoming requests that you can do to get like big gains um while still being able to do multi-cluster and and add in all these additional guardrails and stuff.
Speaker: But not from the model, but from like the setup that you have. So for the platform people, for the ops people, um there's a lot of ah of optimizations that you can look into that are really cool.
Speaker: Do you have KubeCon North America session to talk about all of these things? Or do you know if there is one that we can push and tell our listeners to go attend? There is one. last Last time I did a ah workshop showing LLMD at KubeCon EU. This year, I'm staying home with the kids. i Just kidding. No kids. I'm going to ask you that question. Like, oh, do you already have kids? That's awesome.
Speaker: making up excuses. Um, but, um, yeah, uh, there will be sessions on LLMD. My good friend, uh, Alexa Griffith has a ah visual explainer okay where she's going to do a session on that and there'll be keynotes and stuff.
Speaker: Um, it's a big, it's a big effort in the CNCF right now. And they're putting a lot of engineers on this task um because of the the potential for for savings and for the future. Like ah in in in countries like ah you know France or Germany, they have these really strict regulations on if you can use public model vendors.
Speaker: yeah um And so this AI Sovereignty Act is going to really change things. for a lot of organizations who want to use ai right maybe they've been using it so far but these regulations are going to change how you can process data where you can send it to and so people are increasingly looking to hey i want to run this myself not just because of the cost standpoint because but because of um compliance reasons yeah compliance reasons yeah um so we'll see what happens in the future but yeah you definitely check out the sessions kube cons in what two weeks yep
Speaker: Are you going? Nope, me neither. I have kids at home. I'm just kidding. No there' no kids. We both use that excuse. It's too good. no Yeah, for sure. Okay. Let's talk about model optimization.
Speaker: ah Who needs to worry about it? Right? Like if if I'm just downloading a model from Hugging Face, do I have to worry about it as part of my organization and my day-to-day job or I should write like and do some optimization of my own?
Speaker: Yeah, yeah, yeah. Because if you've ran LLM Studio, Olama, OMLX, you actually have used ah model optimization without even knowing, which is really cool.
Speaker: Because when ah something like DeepSeek or Kimi or GLM comes out on Hugging Face, they release it at the full precision. So the the the most capable model that they can, of course, for the benchmarks and everything.
Speaker: But that model is going to take i a terabyte of memory. right So you would need to have 16, 32 GPUs in order to load that in. um And so what we've what I spend a lot of time doing is helping people to compress models.
Speaker: okay So compression really, it stands for quantization. um Mainly is where you take those model weights that are released at a really high precision.
Speaker: So brain float 16 is the precision of how many bits in each parameter. So think about it like pi, you know, 3.14159 blah blah blah.
Speaker: If you can, you know, shorten it to 3.14, you still get the same type of... you're still, you know, gonna be able to draw a circle, right? yeah Of course it might not be to the precision you want. So you can reduce down to integer 4 format um or integer 8.
Speaker: and you still have the same behavior responses of the model. um Maybe you lose a few percent of accuracy in terms of what the model remembers or can do, but you go from a model that might take up about a terabyte of memory to 500 gigabytes of memory.
Speaker: And for a lot of teams, that's a big enough, you know, you know That's the number of GPUs I have anyways. Right, yeah. It's a big enough reason to do it that they can sacrifice a little bit of ah model performance and um take take advantage of these ah low precision cores in a GPU to accelerate inference. So not only do you have a smaller size of model, but you can do more operations per second so you get a faster throughput or tokens per second.
Speaker: Okay. So this is... yeah In this scenario, right? Like, am I downloading the model, like the 100% precision model, downloading locally downloading it locally, running it on a GPU, running the tools like LLM compressor on it, and then re-uploading to a hugging face in my own repository? Or am I doing this every time I'm i'm downloading a model and running it locally?
Speaker: It's a one-time. one second okay okay One-time operation. Good. Yeah. And a lot of people just, you know, go to Hugging Face. They look up Red Hat AI. They download our models or other teams' models, and it's good to go. So a lot of, you you don't have to worry about doing it yourself unless you have a custom fine-tuned model.
Speaker: For most use cases, the model's already been compressed. You load it one time, you're good to go. Okay, perfect. Awesome. yeah And then how do I measure performance or or this precision? right You said it can come down from 100 to some percentage that's not 100.
Speaker: How do I make sure the tools available to to help me benchmark my model performance or do i need to still go and run those um MMLU and other other benchmark tests? Yeah, there's a lot of published benchmarks out there for the model providers who ran their own benchmarks. Of course, if you run five benchmarks, you can pick the best one. yeah So you can run the benchmarks yourself. Yes, there's a tool called LMEval by Eleuther AI. okay It's just open source tool. There's many more out there, but that can allow you to run these benchmarks. um And there's both static benchmarks of, hey, what is the right answer a to
Speaker: A, B, C, or D. And then there's these agentic benchmarks like TerminalBench that evaluate a model's ability to use tools and accomplish a task.
Speaker: And so you'll notice certain models and certain harnesses like Cloud Code versus Codex take different amounts of time and use different amount of tokens, even if it's the same model.
Speaker: Yeah. So it's a very interesting dynamic for the benchmarking. um But then you can also measure like tokens per second, the time to first token, the P99 or P95 latency for the slowest requests.
Speaker: We have a tool called Guide LLM, which is a part of the VLM ecosystem. a lot of people do to do these benchmarks. And it just, you call your local server.
Speaker: it sends a predefined request or a certain dataset, and then it gives you that result back in JSON. Um, so there's a lot of tools out there and open source projects to both evaluate the accuracy of the model, but also the performance, uh, depending on what hardware you have.
Speaker: Gotcha. Okay. Now all of these, i think we need to link these models, as sorry, these tools, uh, that, that you have described so far, because yeah, all of this is brand new information to me and I'm sure for most of our listeners. So thank you for sharing these, um,
Speaker: You also brought up LLMD few times. Can we talk about like what's the difference between VLLM and LLMD? Is there one? And what should I know about which one to choose? Yeah, that's a good question. um Because l o VLLM, very popular popular project, like 90,000 GitHub stars, um it's it supports almost every model the the day it's released.
Speaker: um But the way that people run on Kubernetes is maybe run it as a deployment, right? Apply like 100 lines of YAML with the model and the location of a PVC for the for the actual files and then a service right create ingress boom good to go um but if you have to scale this up to accommodate more users or of other models and end up creating replicas then as we talked about before you have to make sure that um each replica is ah is getting sent the right traffic so that we don't regenerate the kb cache so that's what llmd does okay is it's uh inference aware routing so it's
Speaker: uses something like Envoy and the Gateway in order to distribute the inference across clusters on Kubernetes. And so we can tell using a combination of incoming requests, but also ah things like Prometheus, like how ah how loaded or you know full a GPU is.
Speaker: And if that GPU is maxed out 100%, it'll go ahead and make a smart model decision, ah smart decision to send that request to the next model. Okay. So it's kind of the Kubernetes way to run inference servers. Interesting. And this works on anything or just OpenShift?
Speaker: any Anything. GKE, AWS, OpenShift. We want to make it very neutral to where you can run any inference server and across any Kubernetes service.
Speaker: um And it's it's a pretty easy way to like take advantage of benefits for... ah situations where you have multiple models or you just have a lot of traffic um so that you don't just create new replicas and and lose a lot of speed and and say you lose a lot of money at the end of the day.
Speaker: um And it works with KServ. It works with um Envoy. it's ah It does disaggregation, what I talked about before. yeah And so it's yeah just an open source project that that kind of fits how we see the future of of AI running on Kubernetes.
Speaker: Got it. OK, no. Awesome. Thank you. Thank you for that. um I think just a last couple of questions. One is let's talk some about day two operations, right? Like, yu I got a set of models running through LLMD. How do I perform? Like, do I have to worry about scale up, scale down, scale out, um ah upgrading the versions of models? Like if there's a new version of model, ah how do I do all of these things or are they abstracted away through LLMD as you were just describing?
Speaker: Yeah, yeah, yeah, because it seems like every day, new model, right? Yep. And you want to be the first to to to take advantage of it, if especially if it can make your your applications ah you know run better, customers happy.
Speaker: um So day two is is really interesting because you can use a lot of your existing experience with Kubernetes and operations and platform to handle these rolling updates.
Speaker: um So VLM, for example, will support day one operation of of a new model, right? um But this means a new architecture. So starting to migrate to a new version and and having a gateway that can handle...
Speaker: these rolling updates, right? um But at the same time, what if your application doesn't perform as well, right, with this new model? So you can use things like um pipelines with Argo or something like that, where you run a request that hits the model server,
Speaker: and judge with ah maybe a model or maybe with a human that yeah that response back. You can automate that or have that as an event-driven action for every time you do a model upgrade. Because we see sometimes model drift where the model gradually doesn't perform as well and you don't really notice it till it's too late. yeahp um So you can use a lot of Kubernetes existing ah projects and concepts in order to do this upgrade, but also this verification of like, hey, we're doing a to V2 of this model.
Speaker: Is it gonna work as well? um Let's test this in a distributed fashion and let's automate this ah using you know compostable yeah and Kubernetes native patterns. So it's pretty cool Okay, no, this is this is perfect. right and I'm sure for us to go into all of these kinds of details, we need like two additional episodes to talk about operations. And we are running close to time here. So let's let's just try to wrap this up. Let's talk about ah what's next for you. I know you are at a conference right now. So I do appreciate you. I don't jumping on a call from the expo floor and doing this podcast and sharing your knowledge. But what's next ah for you? Where can people connect? and then follow what you're doing. If you have any links, blogs, people, people, how how can people reach out to you? Please share those as well.
Speaker: Thank you. Yeah, and apologies if there's any background audio, but yeah. Well, thanks, Bhavan, for having me on the show. um What's next? Ooh, let's see. i would say Kupam. Kids. Kids. Yeah, well, I'd say first is like check out these resources we've talked about. We'll link them below, but LLMD, so Running AI in Kubernetes, the VLM Project, because you can even run that on your Mac or your CPUs, which is really cool.
Speaker: um And I would say there's there's a lot of interesting work going on to running models on CPUs, because all of us have a CPU, all of us have a Raspberry Pi, all of us have a mobile phone.
Speaker: yeah And these days you can run a small language model like Quinn on your phone. So I think the future of onboard ai to where you don't have to send your data anywhere, you can run things locally and you can take advantage of ai without having to send that data to someone else.
Speaker: I think that's huge. And I think that's a big opportunity for open source. So that's kind of my closing words. I'll i'll drop my LinkedIn below. I do YouTube videos on a channel called IBM Technology. Mm-hmm. So if you're interested in distributed inference and AI engineering, that type of thing, check out the videos. We just explain concepts.
Speaker: And ah yeah, man, keep tuning into the Kubernetes Bytes show. Let's go. A lot of cool people on the show. And the previous episodes, I just listened to them before on you know AI agents and security. right These topics are not only important now, but they're going to be more important this year next year after.
Speaker: Awesome. No, thank you for doing going on and off the podcast in the podcast. That's the best way to better I do appreciate you you spending some time with me. Thank you so much. And yeah, let's talk again next year when these projects have made even more progress.
Speaker: Sounds good. Well, thanks for having me. Thank you so much for listening to the episode. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review or sharing it with your friends and colleagues, as that really helps us grow the podcast.
Speaker: You can find all past episodes or learn more about the show at kubernetesbytes.com. See you in the next episode.
Speaker: Thank you for listening to the Kubernetes Bytes Podcast.
