Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Kubernetes for VMware Admins: Understanding KubeVirt
557 Plays2 days ago
In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin talk to Janakiram MSV about all things KubeVirt. The discussion starts off by talking about the need for KubeVirt and then dives into the details of the KubeVirt architecture, and what you need to do when deploying virtual machines on Kubernetes, including commercial solutions like Red Hat OpenShift Virtualization and SUSE Virtualization.
Check out our website at https://kubernetesbytes.com/
Show Notes:
- https://www.youtube.com/watch?v=DDwfPuWlsl0
- https://janakiram.com/resources/
- https://kubevirt.io/
Recommended
Transcript
Introduction to Kubernetes Bytes Podcast
00:00:03
Speaker
You are listening to Kubernetes Bytes, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Wallner and I'm joined by Babin Shah coming to you from Boston, Massachusetts.
00:00:14
Speaker
We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:30
Speaker
Good morning, good afternoon, and good evening, wherever
Weather Chat in Boston
00:00:32
Speaker
you are. We're coming to you from Boston, Massachusetts. Today is March 6th, 2026. Hope everyone is doing well and staying safe.
00:00:41
Speaker
Bhavan, let's dive into it. How you doing? I'm doing good. Yeah. ah The weather is finally improving. I think I'm done with winter. ah I don't know, for a month i at this point, but we did get that snow big snowstorm in February.
00:00:54
Speaker
I was complaining all to all my friends here in California, in in India. And then I saw your text and you said, yeah, you got twice or thrice the amount of snow I did. I was like, okay, i need to shut up now. and Yeah.
00:01:08
Speaker
Yeah, we were. We were near the epicenter, right? I'm i'm a little southwest of Boston and basically right on the Rhode Island border. But we got 27 inches of snow, over two feet of snow, a lot of snow. And just south of Warwick, it was like 31. And then in Providence, I think 37.1. So three feet of snow. And that's a half hour away from me, right?
00:01:32
Speaker
um Yeah, a lot of snow. And speak for yourself because it snowed and iced this morning out here. Yeah, it did. I think we got like a couple of inches. But a couple of inches, it's so funny that I'm in and this mental state that I don't even think about it. Yeah, that's easy to
KubeCon India and Kubernetes Evolution
00:01:50
Speaker
shovel. Like, it will take me 20 minutes, but it I can get it done. When we're talking about, don't know, 12 inches, ah more than 15. Yeah, like, whew.
00:01:59
Speaker
i Yeah, I don't know. I'm very thankful to my neighbors, dude. Like, they have snow blowers and I didn't get one in time this year. I underestimated it. Like, like most of the winters haven't been that harsh. yeah that And I was like, no, I can manage with a shovel. It's okay. I've been shoveling for the last three years. And no, this time it was a really wrong decision. I think when when we got 18 inches.
00:02:22
Speaker
Yeah. Yeah. The neighbors saw me clear just one line in 10 minutes and just the amount of work. And he's like, wait. And he just brought a snowboard over. And like five minutes to clear my entire driveway. I was like, thank you. and like like What a homeowner this guy is. doesn't even know how clear his driveway.
00:02:40
Speaker
not yeah no. You know what's funny is I read a bunch of statistics about how the amount of snow this year was actually more on par of what a New England winter should be. Meaning, this is how a New England winter is. um But we've been... What is all that global warming, man? Come on.
00:02:57
Speaker
Right. we have We've had so many mild winters, especially since, I think, since like 2015 or so, that we've just grown accustomed to like, oh, it's, you know, we can see the brown grass all winter.
00:03:11
Speaker
We haven't yet used to all this. And this is just more like what what it is supposed to be. Yeah. I'm... I'm ready though. I think Tuesday is supposed to be like 60 and I can't wait. And people, I know like folks in California are probably just laughing, you know, but I love the season still. I wouldn't trade it for anything. and ah That being said, ah we do have to deal with, you know, the snow, but everybody's got something to deal with in terms of. No, no, no. Next year when I'm prepared, I'll be there with you this year. I'm just in in that complaining mood that a this is not what I'm, I signed up for kind of thing. Not an ad, but I love the Husqvarna blowers. Okay, send me a link after this. ah yeah I want to bookmark it. And if it is it is in stock right now, order it right now. Just so that you can get like second winter in in March and April and we get another storm.
Exploring VMs on Kubernetes with Johnny Kiram
00:03:57
Speaker
I want to be ready for it.
00:03:58
Speaker
Yeah, Husqvarna or Aaron's are like top two brands, I would say. Which is ironic because I have a Troy built that I got for $200 off of someone on an estate sale. Because I didn't want to spend a lot of money and my neighbor plows and i was just like, I just need something to like. Yeah, sure. yeah That being said, it just it started spewing oil on the last snowstorm. Oh, OK. So you get what you pay for, Bobbin, you know.
00:04:26
Speaker
All right, well, um we do have a fun episode today. We're diving into the the whole world of VMs on Kubernetes and sort of what probably is top of mind for a lot of ah VM admins out there asking themselves, you know, what is it? How does it work? What skills do I need to learn? How mature is it? um You know, Broadcom is going through a lot of changes and my company's, you know, not renewing or only renewing for so
Kubernetes as a Universal Control Plane
00:04:54
Speaker
long. You know, we're we're goingnna We're going to figure out what's next. And Johnny Kiram, MSV, who is an architect, an analyst, a previous guest of Kubernetes Bytes, and overall just a Kubernetes guru and expert in my my own words there, um is going to come talk to us about this topic. All right, welcome back, Johnny. I say welcome back. It's been probably a few years, I think, since being on the show. I'm i'm not sure exactly the ah date of that show, Bobbin, but I'll have to look that up. It's great to have you back. i know we have an exciting topic talking about sort of the world of virtual machines and the ah in the context of Kubernetes and and you know where that's come in the last year is is is really from... you know down here to way up here. So we're excited for you to kind of dig into that. But why don't you reintroduce yourself for our our new listeners?
00:05:48
Speaker
Well, thanks for having me. It's great to be back into the show. I think the last thing we we discussed in in Kubernetes Bytes was about AI, ML, worth. He was running on top of Kubernetes. right It's been a couple of years and now it's it's all about LLMs and fine-trimming and running inference on top of Kubernetes. well We are going to talk more about VMs, but let me start by introducing myself. My name is Janikiram MSV, this sort of Hyderabad, India.
00:06:15
Speaker
ah Been working on Kubernetes since 2014, 2015, deploying the guestbook, The Hello World of Kubernetes on various flavors of Kubernetes. And I am a CNC of Kube Astronaut, spoke at multiple Kubecons, very passionate about microservices, Kubernetes, and running AI workloads on Kubernetes.
00:06:36
Speaker
Oh yeah, you were one of the main steering committee members for KubeCon India as well, right? If I remember correctly? And i'm participating in the KubeCon ah India keynotes?
00:06:47
Speaker
Yeah, I did the two keynotes. so The first two KubeCons in India. ah One was in Delhi last year and ah more recently the KubeCon in Hyderabad in my home city.
00:06:59
Speaker
Gotcha. Oh, that's awesome. That means less travel for you. I know you have we have seen you at multiple of these conferences across North America and Europe. If a conference happens in your city, that's always great. like we we We can vouch for that when Red Hat Summit happens in Boston.
00:07:14
Speaker
That's right. Absolutely. Okay, so Jani, I'll kick off the questions. So for listeners, I knew or old, as Ryan said, that are coming from the traditional VMware background, right? How would you like describe Kubernetes to them through a VM admins lens, right? Because they have spent maybe 15 years, 20 years using vCenter, using all the VMware tools as tools and tooling around it. How do you describe Kubernetes to them?
00:07:41
Speaker
Yeah, the best way to to map what you already know with Kubernetes is thinking of Kubernetes as vCenter, vImagine, but for Cloud Native Era.
KubeVert's Role in Virtualization
00:07:51
Speaker
Yeah.
00:07:53
Speaker
So yeah vCenter is at the front and center of VM orchestration and that is the go-to console all the VMware admins. ah Whereas when it comes to cloud-native microservices and containerization era, it's Kubernetes. So it's primarily Kubernetes that orchestrates containers, pods, and basically we compute. so instead Instead of managing VMs on ESXi hosts, you are essentially managing containers on pods on nodes.
00:08:22
Speaker
ah So the best way is to compare and contrast vCenter with the Kubernetes cluster and the control plane together. So that is the best way of mapping.
00:08:34
Speaker
But fundamentally, I think the deeper shift is transitioning from imperative to decorative management. you know Most of the vSphere and vCenter folks are very comfortable with teacups.
00:08:47
Speaker
They click here and there and get the job done. Whereas what will put them slightly off the gear is the reason to deal with YAML files. So you and you have to basically understand how YAML functions and then taking it one level above GitOps. So you basically commit YAML files to Git repo and then the controller will reconcile. So the the shift is basically from the metal model of ah using a GUI to using YAML files and then eventually GitOps.
00:09:24
Speaker
But What remains the same is the layered mental model. you know You still have compute, you still have storage, you still have networking, management. Those four layers are exactly how ah things are structured even in Kubernetes world. So they are not very different.
00:09:44
Speaker
Yeah, that's a good point that there's a lot that maps ah sort of higher level conceptually when you start to dig in and start to look at how it's implemented and get into the YAML, i think is where a lot of the differences come in. But those those higher level kind of mappings do, I think will help a lot of vm administrators coming to Kubernetes. um Now, there there is sort of a ah main project that's backing a lot of this movement in the Kubernetes space, right? This is Qvert. And you know the the question I want to ask you is sort of in your opinion, you know why did we see Qvert emerge and and sort of what's what's it kind of, um what what gap is it really solving for ah for us in the Kubernetes space and why is it important to those VM methods?
00:10:34
Speaker
Right, right. So there are two key factors now that that led us to KubeWord. The first is Kubernetes started to transition from orchestrating just containers and ports to orchestrating a lot of things, including cloud-based VMs through control plane and sorry, cross-plane, cloud resources, and then Vaizen and edge clusters, and even clusters using Cappy, right? So...
00:11:07
Speaker
There has always been this thought process that why not Kubernetes orchestrate VMs? So since 2019, Qubet has been um an incubating project under CNCF Umbrella.
00:11:18
Speaker
So it's been in the works for a long time. that is That is one. So the community and the ecosystem always wanted some convergence between VMs and containers. So that was in in the works.
00:11:32
Speaker
sure What actually accelerated that was Broadcom's acquisition of VMware in 2023. Then the dramatic restructuring of the licensing model and all the conversations around the new cost structure only accelerated the maturity of KubeWord.
00:11:50
Speaker
And what was moving along slowly... ah took up a faster pace and became much more mainstream and much more mature. So these two factors, right one, the the intention of Kubernetes becoming the control plane, not just for ports and containers, but for VNs on one side. And then the factor that actually accelerated this whole process is Broadcom's acquisition of VMware. So these two basically made KubeVert what it is today.
00:12:20
Speaker
And KubeVert is basically the bridge between the Cloud Native ecosystem and the VM ecosystem. Yeah. And KubeVert, it's not like the community just built all new software to make KubeVert happen in 2019 since it's been incubating, right? It's based on existing technologies that have been around a long time, right?
00:12:43
Speaker
Yeah, absolutely. It builds upon the primitives of Kubernetes. It is it is not a ah foreign entity to Kubernetes. It is very well integrated and it takes advantage of various primitives, which we'll discuss in our conversation.
00:12:58
Speaker
ah Various primitives that are already proven in the Kubernetes architecture and the ecosystem. So it it just... relies on some of these developments and some of the innovations that took place in the Kubernetes ah world, Kubernetes ecosystem.
00:13:14
Speaker
Yeah, and as far as virtualization technology under under the covers, right yeah I'm sure vm admins are familiar with things like KVM and QMU and those kind of things, which are you know have been in the Linux ecosystem for a very long time. And actually, Kuber builds upon those, not just building upon the primitives in Kubernetes, but hey, we're not rewriting a new hypervisor here. This is not what we're trying to do, necessarily. Absolutely.
00:13:41
Speaker
Yeah. Okay, so I think um more on the hypervisor front, right? um I wanted to ask you, Jani, that um is is Kubernetes kind of the only solution?
00:13:54
Speaker
i mean We always see in the industry, obviously, when you're scrolling through LinkedIn, right? ah There are so many different vendor solutions out there, like Kubernetes slash kubeword is one of them. But then there's obviously Nutanix, there's Hyper-V, there is different flavors of KVM. So I know Brian and I are but are biased because of Kubernetes Bytes. We would want Kubernetes as the next stack. But what advice would you have ah in terms of both short-term planning and long-term planning to people that are going through these evaluations right now and figuring out is Kubernetes replacing their virtualization stack or is it just becoming the new control plane to orchestrate things? like ah
Managing VMs on Kubernetes
00:14:33
Speaker
how How would you advise some of the ah some of our listeners here?
00:14:37
Speaker
I'm actually as biased as you. yes
00:14:44
Speaker
exciting more. Get like a Hyper-V guy on as well. the As long as we're verbally saying we're biased here. Yeah. So, yeah, after all, this is the Kubernetes podcast and all of us ah no acquired those stripes of Kubernetes over a period of time.
00:15:02
Speaker
So so i have I have been advocating Kubernetes as the universal control plane for a long time. you know I've been one of the biggest advocates saying, let Kubernetes become your universal control plane, orchestrating every entity that is running in a distributed mode.
00:15:18
Speaker
And my most recent... ah at KubeCon Hyderabad in the most recent KubeCon India was Kubernetes as the orchestrator for AI agents. So I fundamentally believe Kubernetes has matured and reached inflection point where it can pretty much orchestrate um any resource.
00:15:40
Speaker
Now that resource is defined by ah the ecosystem, the developer, the DevOps engineer, but VMs are very well structured o to become a subset of the Kubernetes control plane. So it is the new control plane. you know The hypervisor isn't going away.
00:15:58
Speaker
ah But the KVMs of the world will still do the heavy lifting underneath. But what is changing is the management layer on top. you know Kubernetes is absorbing and kind of subsuming the responsibility and unifying how ah VMs are typically dealt with the lifecycle and and how you manage everything else.
00:16:17
Speaker
so And it's not it's not the first time. you know When Kubernetes was launched, everyone thought it is typically meant for stateless workloads. yeah right But then we have seen how Kubernetes has become the platform for stateful workloads. And I'm talking to two advocates and evangelists who are representing NetApp and Workworks. We agree.
00:16:41
Speaker
so You know, you completely agree with me that you know stateless workloads are first-class citizens on top of Kubernetes. And ah in a way, Kubernetes subsumed that functionality.
00:16:52
Speaker
and And today is all about virtual machines. And the universal control plane definition just logically and naturally extends to virtual machines. So the bottom line, Kubernetes is not replacing virtualization.
00:17:04
Speaker
i It in is just becoming the control plane. Virtualization is a parallel technology. I would actually visualize... the Run-C, which she' is at the bottom most layer of the stack with KVM sitting on the same node. So you have Run-C to basically handle the lifecycle of containers. You have KVM to handle the lifecycle of virtual machines.
00:17:27
Speaker
And just like... RunC talks to the ah container runtime interface and surfaces containers and pods all the way up through deployments and stateful cells.
00:17:39
Speaker
You have KVM's cells, which are virtual machines, surfacing through all the layers of Kubernetes and becoming available as deployment objects, which are like virtual machine and virtual machine instance types.
00:17:52
Speaker
Yeah. Yeah. I think that one of the questions that I do get, right, is obviously VMware, because of its history, has had so many years of headstart on Kubernetes as the VM orchestration platform, right? So there are features that are not there. So I think I really like the way that you put it.
00:18:11
Speaker
that it's it's a control plane. Like virtualization is one of the tenants, right? Like, yes, virtualization is one of the things that we'll orchestrate. But if you look to the future, nothing is stopping you from adopting containers if you're coming from VMware. Nothing is stopping you from using the same orchestration layer for deploying those AI agents. So we'll definitely link that keynote that ah talk that you did, Jani, as part of our...
00:18:32
Speaker
ah show notes but yeah I think that's that's the key essence right it is the control plane and it doesn't really matter what the workload is today like i know two years back we were talking about how AI is running on Kubernetes now we're talking about how VM is running on Kubernetes I'm sure we'll have you back in two years to talk about some some other new workload that's running on Kubernetes so I think yeah that that really helps um no thank you for sharing that
00:18:55
Speaker
Right. So I do want to dive in a little bit to how things actually work right for the practitioner that might be listening. um And so let's dive right in and talk about how a VM actually runs inside of Kubernetes. We mentioned KubeVert, but maybe from a high level, let's let's talk through in in sort of your ah mind map here, Johnny, ah how a VM actually runs inside Kubernetes with KubeVert.
00:19:22
Speaker
Absolutely. So basically, Qubet wraps a KVM virtual machine inside a pod. Now, that is the... fundamental unit of deployment or the unit that you can deal with.
00:19:36
Speaker
So KVM-DVM running inside a pod, you know that is the first way of visualizing it. and And then there's Word Launcher, you know the pod that basically owns the VM process and the KVM hypervisor, ah where the actual action takes place. So the Word Launcher pod is responsible for owning the VM process.
00:19:57
Speaker
And KVM still does the heavy lifting. So from Kubernetes perspective, it just sees a fault. But from the guest OS perspective, it's just a VM. So that is the beauty of this architecture. right So when you take a node, and as I said earlier, if you peel off all the layers of Kubernetes, you will actually hit run C at some point. Run C is the bottom most layer of this architecture, right? And similarly, you have the KVM as the bottom-most layer of this kubeword stack.
00:20:27
Speaker
Now, all the heavy lifting will be still done there. But the KVM virtual machine is now wrapped inside a pod. And for all practical purposes, that is still a pod. But you can actually do kubectl, get virtual machines and get virtual machine instances, and you can still see them alongside pods.
00:20:47
Speaker
So double-clicking on this, you know the architecture basically has three main components which you need to know. And it it's very difficult to explain this without a slide. I would typically wide-vote this. But let me attempt explaining this in a podcast. right so The Word API server handles incoming API requests and validates the VM object. So that is basically the external-facing API layer responsible for accepting the e request so to launch a VM.
00:21:17
Speaker
Then the Word controller right is is responsible for watching the virtual machine objects and it reconciles them. now and Think of this like a controller. It's it's the loop that basically manages a lifecycle of a VM wrapped inside a pod.
00:21:32
Speaker
And then the word handler, ah it's it's a demo set you know that runs on every node and uses sleep word ah to talk to the underlying hypervisor. So these are the three important things. So every node has a demo set.
00:21:46
Speaker
ah which is negotiating underlying virtualization layer which is KVM. And ah it it actually communicates with another tiny layer called libwork which is responsible for actual processes.
00:22:01
Speaker
As I said, it's little bit challenging to explain this without a whiteboard, without a slide. like you know this These are basically the essential building blocks ah that you can you can think of. And then there is a crucial difference difference or distinction between a virtual machine object, which is the persistent desired state like a VM.
00:22:23
Speaker
And then there's a virtual machine instance, which is the runtime of the virtual machine. So you have a virtual machine that defines the state, how it actually should look like. And then there is virtual machine instance, which is the runtime of the actual virtual machine, which represents a running instance.
Challenges in VM Migration to Kubernetes
00:22:40
Speaker
So, you know, this entire separation enables graceful restarts, live migration, declarative power off, everything that you typically do with a VM, but it goes via the pod, and which is wrapping the VM within Kubernetes.
00:22:59
Speaker
Interesting. So I think like um ah one follow up on that front, right? It's very like because it has so many moving parts, right? You described a pod, a virtual machine, a virtual machine instance. if i talk if If I'm coming from a VMware world, right? I deploy a VM and then i see the VM and I see the VMDK files that are attached to it.
00:23:22
Speaker
How do we map that, like, hey, this is what a VM encapsulates to a Kubernetes ah environment? So you started with the three, but are there other things that form the storage and networking layers that are needed by a VM to two function as well?
00:23:39
Speaker
Yeah, definitely. but But before we get there, you know, ah you You actually open a blank editor and you start writing an YAML file to represent your virtual machine.
00:23:52
Speaker
And that's going to define your VM. ah This is like a template. Now think of ah the template in the VM world and here the YAML file essentially defines the characteristics of the virtual machine.
00:24:03
Speaker
and the kind is going to be virtual machine. So you start with that and then you apply that to your Kubernetes control plane like the way you apply any other YAML file, right a stack.
00:24:16
Speaker
And then it goes through these layers that I explained earlier to turn that into a virtual machine instance where you are turning on the definition that you have declared in the YAML file into a running instance.
00:24:32
Speaker
Gotcha. So I just have to define the VM spec. Everything else, Kubernetes takes care of takes care for me. Exactly. Now let's talk about how it takes care of it. right So now Kubernetes has these CRDs or Password Resource Definitions.
00:24:48
Speaker
ah If you want to bring any new type into Kubernetes, like deployments, faithful set, daemon set, or a JAW, you can define your own resource definition. And they are typically called as CRDs, custom resource definitions.
00:25:04
Speaker
So your VM is just a CRD called virtual machine.
00:25:11
Speaker
And then it cleans some parts for actual execution because parts of the fine-grained deployment units within Kubernetes. And then these pods have persistent wall-room for storage and services for networking. So it's not reinventing anything. It is basically composing what you already had into Kubernetes primitives.
00:25:34
Speaker
right So now, if you if you want if you want to go further and talk about the actual technology behind this, for networking, QBert is going to rely on multiple CNI to attach multiple network interfaces to the VM pods.
00:25:48
Speaker
right Because this is This is how you you basically replicate the multi-nit pattern. It's very common in um enterprise VMware environments. right So you can attach multiple networks to the same VM and you simulate that in Kubernetes through specialized CNI, the container a network interface.
00:26:11
Speaker
called Maltus. Now that is going to help you map your network topology when you are deploying this in Kubernetes. And for storage, the most important aspect to understand is containerized data importer or CDI.
00:26:26
Speaker
Now this component hand is importing VM disk images from yeah URLs, right? And from PVCs and container registries, and it basically creates persistent volume claims that the VM can consume.
00:26:40
Speaker
So this is going to create the persistence layer, you know, just like the VMDK that you you attach to your VM to make it stateful. So, CDI is essential for anyone managing VM storage because at least now you efficiently define the data path you know for VM provisioning and um all the performance characteristics that you typically define and spin up ah VMs in the VMware environment. So to summarize, there are CRDs that bring you new types called virtual machines and virtual machine instances.
00:27:13
Speaker
Then there is Multis CNI that maps you to how you perform networking ah in the VMware environment. And then you have the a CDI, which is the containerized data importer for managing ah the state and the storage for the VM.
00:27:32
Speaker
And then finally you have the ah CDI, which is also managing your data park you know for VM storage.
00:27:42
Speaker
So... so Yeah, on the TDI front, right? Is it just for my boot disk, quote unquote, or is it also for my data disk? So if I'm deploying a VM from, let's say, ah an Ubuntu template, right? Is it ah just going to create my boot disk, a small PVC, maybe 10, 20 gigs, or can I customize it and use it for a 200 gig PVC or terabyte PVC, and there will be partitioning somehow underneath it? Is all of that controlled by CDI? What if I need a second PVC? Does that also need to be a second CDI instance?
00:28:17
Speaker
Yeah, so ah every time you want to bring a stateful VM into Kubernetes, CDI is the way to map that.
00:28:28
Speaker
okay So it is not just for the board disk. Any additional stateful disk for storage that you have attached to the VM will also go through the path of CDI.
00:28:40
Speaker
Gotcha. Okay. Okay.
00:28:43
Speaker
All right. So i want to i want to look a little bit and double click on how VMs are actually sort of managed and scheduled into Kubernetes. We talked about sort of how they run and some of the some of the concepts like pods and and how data gets into them. But, you know, VM admins coming from the Broadcom and ah sort of world might be familiar with DRS or Storage DRS or vMotion, those kind of technologies. Now, how do these things sort of map to how Kubernetes schedules schedules VMs?
00:29:19
Speaker
Well, um VMs are definitely more constrained than containers. because an containers are stateless and they are extremely lightweight. So the scheduler has a lot of freedom on placing the containers wrapped as pods into any of the nodes.
00:29:35
Speaker
Now, the fundamental difference is VMs carry stake. you know they are always stateful. or If you are bringing a SQL server or a Postgres SQL VM or an Oracle VM, and which is a decade old, they carry lot of state and they have CPU pinning requirements. right You might actually want to pin ah the VM to a specific core or certain number of cores of CPU because of the licensing.
00:30:00
Speaker
ah Remember, most of these old software vendors, old school vendors still sell per CPU licensing, right? yeah So you cannot randomly allocate number of CPUs and remove
KubeVert vs. Commercial Solutions
00:30:13
Speaker
them at runtime. So CPU pinning is a very important requirement.
00:30:16
Speaker
And then there is something called non-uniform memory access or Pneuma awareness, um which again is very specific to the way the virtual machine is done on a on a host, right?
00:30:27
Speaker
Now that needs to be considered. And more recently, the GPU pass-through has to be addressed. Now, sure All these requirements ah make scheduling of VMs within the Kubernetes environment very, very complex than throwing a container as a pod and ask them to schedule on any available nodes that meets the requirement.
00:30:46
Speaker
So this is very different. So ah the scheduler now needs a lot of fins. right So if it it needs to handle node selectors, affinity roles, and requires a lot of deliberate configuration that will unpin the compute storage network needs of VMs than containers.
00:31:08
Speaker
Gotcha. So, Jani, you brought him NUMA, right? Those were the sessions at VMworlds that I attended, that I always skipped. Like, I went to one of those, and it it was just too detailed. And, yeah, I was like, I don't care about this. I just need my VM. So, and my my question to you is,
00:31:26
Speaker
Does CubeWort as the project, open source project from the community handle these advanced scheduling decisions? Or is it like flavors on top, right? Like Red Hat OpenShift virtualization or Suze virtualization, these guys are solving it. Like where is it where is this problem being solved?
00:31:44
Speaker
So um let me let me answer this with a disclaimer. right Now, this is not something that I'm extremely comfortable or confident of answering, but my understanding is the core Qubewort primitive, Sandil, the scheduling aspect of it, they are baked into it.
00:32:02
Speaker
But some of the commercial implementations like Red Hat, OpenShift Virtualization, SUSE, enhance that further. But Qubewort fundamentally has these scheduling ah enhancements baked into its controllers and operators. So it is part of core to work stack, but it is certainly enhanced by commercial implementations.
00:32:25
Speaker
Gotcha. And one one more follow-up. So Ryan, when he asked the original question, right, he was talking about DRS. And DRS was a really cool technology in VMware, right? Like it figured out the resources and then it moved VMs around, things like that.
00:32:39
Speaker
Does that capability exist today? I'm just trying for our listeners, right, to draw some mappings like, yeah, DRS box check. Like does DRS, something like DRS ah ah exist on a Kubernetes environment with KubeWatt?
00:32:55
Speaker
It's a very tricky um ah use case to run on QVirt.
00:33:02
Speaker
You know, Vsperia's DRS essentially rebalances workloads across hosts based on their utilization. Now, unfortunately, QWERT has no e equivalent you know when it comes to automatic load balancing and automatic scheduling of VMs across multiple nodes based on resource utilization.
00:33:21
Speaker
That is capability maybe expected from the commercial implementations at some point, but plain vanilla QWERT has no one-on-one mapping between DRS and the Kubernetes implementation.
00:33:38
Speaker
Yeah, that makes sense. And and there's some there's some differences in actual implementation, right? the The real-time data, the VMs sort of being acted upon versus how that works in Kubernetes. I think there'd be, you know, a third party that would have to essentially manipulate as it a schedule extension to constantly do that. And this is where I think, you know, some of the some of the exact like A to B mappings, I'm thinking about those like us in grade school, like match this this one to this one. It's not entirely clear. It might be a little bit of a squiggy line. um But like you said, a lot of the ah the the primitives to support doing those things are there. um And you know we're we're definitely early early days in this transition. right So um I think maybe this leads me a little bit to my next question is sort of,
00:34:29
Speaker
ah As a VM admin who's you know maybe taking the leap or taking the step with their with their organization to start implementing VMs on Kubernetes, are there workloads that still shouldn't move to Kubernetes or
Future of VMs on Kubernetes
00:34:45
Speaker
Kubernetes? Or should they be kind of considering anything they have in their their v their VM a estate capable of moving to Kubernetes at this point?
00:34:55
Speaker
So that's an interesting question. ah That's why I'm asking you that question. but
00:35:04
Speaker
So, you know, anything that has very tight hardware dependency ah is not an ideal candidate to move to keyword yet.
00:35:15
Speaker
And again, i'm I'm not an expert on this, but ah the SRIOE, the Single Route I-O Virtualization Networking, you know, which which delivers extreme throughput.
00:35:26
Speaker
ah Now, those workloads cannot be ported as is to Kubernetes. They are... they they They have quality of service, they have SLAs in terms of throughput, and those workloads cannot be easily migrated to Kubernetes. that is That is the second constraint. right So with hardware dependencies is one, ah then SRIOV networking limitations is the second, and the third one is Microsoft licensing for Windows Server. like That's a very nuanced complication that we, a maintenance, would have to deal with.
00:36:01
Speaker
ah So if you are running Windows-heavy estate, it is a real compliance conversation, right? So moving Windows workloads to KubeWord as is, will open up a Pandora's box.
00:36:16
Speaker
So you have to... I do have to understand how the licensing UIAMS has worked, what complications you're going to deal with. So to summarize, um ah three types types of workloads are still better running VMware as of now. One is workloads with very tight hardware dependencies and hardware constraints.
00:36:40
Speaker
ah Workloads that demand high network throughput and a demand real multi-nick environment with dedicated network control planes and so on.
00:36:53
Speaker
Third one is Windows workloads that are running on VMware. These are better left to VMware environments as of now ah because we don't have a straight answer or a or a mechanism to run them within QWord and Kubernetes world.
00:37:11
Speaker
Gotcha. No, i think ah it's a fair point, right? And I know we are putting you on the spot with a lot of like blanket questions and expecting a black or white answer. So no, these these are obviously like things that we have seen or you might have seen in with the customers that you're working with in the community that you are involved with. So obviously, ah this is not like a, I love when when I listen to financial podcasts, right? They say, hey, and mutual funds are subject to market risk kind of a thing. Yeah.
00:37:39
Speaker
If there are workloads that you want to move... We're hand-wavy. Move to open ah Kubernetes and OpenShift virtualization, right? Like, do ask those questions, right? Like, do ask those questions to the vendors that you want to work with or to the community. do do some research.
00:37:54
Speaker
There might be things that have changed since the time we do this podcast, and you might be listening to this in three months, and there is a new release, and some of these things are fixed. So, obviously, as as Ryan mentioned, right? Like...
00:38:05
Speaker
Early innings, a lot of work being done in the ecosystem to move these things along. And talking about one of such projects and features, I wanted to ask about live migration. Like I think one of the coolest demos when I started, think I was an intern at NetApp. I saw the live like vMotion of a VM from one node to another node in a vCenter and that was super cool. Again, I know it had been around for a while, but I was looking at it for the first time.
00:38:29
Speaker
So is there live migration in KubeVert and how does it work? Where does it fall short? Can you talk more about that cool feature as well? Yeah, this is this is one area where there is a lot of work that needs to be done. Because what most of the network folks underestimate in the Kubernetes environment is the bandwidth requirement for live migration.
00:38:54
Speaker
Because ah It's a very common corpology in Kubernetes to see two networks. One is the control plane network that basically connects the control plane master nodes for traffic.
00:39:08
Speaker
And then you have a workload ah network, which is typically meant to channelize the traffic for the workloads. So you you very clearly separate and differentiate between the control plane traffic and the gate-of-plane traffic.
00:39:22
Speaker
you know The traffic that is used for Kubernetes administration, the traffic that is used for the network that is used for the workload traffic. Now, these two are very clearly defined.
00:39:33
Speaker
Now, when you actually bring in something like live migration, you need a dedicated network pipe just to perform VM migrations in real time.
00:39:44
Speaker
ah Now, this is not what Kubernetes is ideally designed for. It doesn't really have that kind of a network backhaul to really perform live migration in real time.
00:39:58
Speaker
There will be a lot of hiccups, there will be a lot of friction when you attempt to do this, just because the network will choke and that will become the biggest bottleneck. But having said that, um you know The KMU Live Microsion under the hood, ah which can trigger through Kubernetes APIs, you can kind of simulate the vMotion, but it is not going to be smooth. It is not going to be as smooth as a Live VM moving from one ESXi host
00:40:28
Speaker
all the way to the other while keeping all the clients connected. It's not going to be that seamless, but you can achieve this with some tweaking. But the disclaimer here is watch out for network performance and network bottleneck ah because Kubernetes doesn't have that dedicated network for live migration.
00:40:48
Speaker
Gotcha. No, I think that's super important, right? Because, yeah, just showing demos of live migration in, in KubeWord is great. But yeah, have when we are thinking about production environment, ah all of these nuances become definitely, definitely become important. So, Jani, a follow up question, right? Is it also because, again, I'm just trying to ask the most basic questions. um in In VMware, right we had a a VMFS datastore where all the VMDKs were hosted. And then if your VM object is moving from one host to another, we we didn't really have to move the storage as well. right it was still ah part of the same datastore.
00:41:27
Speaker
In Cube World, we have different PVCs and if it's redried once, redried many, like especially for redried once, there is a lot of mounting and unmounting that needs to happen for the for it to reattach to the new VM. Is that also some plays a part or not really?
00:41:43
Speaker
ah That's an edge use case. you know that's like ah And again, in the context of migration, I'm at to see this in action. To be very honest, this is something that I'm at to see in action i and actually configure myself. So that's a very interesting a scenario on what actually happens to the attached drives.
00:42:08
Speaker
Gotcha. Okay. Makes sense. Yeah, and I mean, to your to your point, Bob and Johnny, right? Like, live migration is part of it. And um realistically, you you know, adopting Kubeware, adopting VMs on Kubernetes, you're going to want to do some testing, right? ah To your point, Johnny, get your hands on it. This goes for the same this this goes the same for anyone listening who's working on this, right? Get your hands on it. um You know, there's tools out there where you can you can do some sort of, like, benchmarking of things like bootstorms or VM creations and and all sorts of things, or you can develop your own, but you know these are the things that you're going to want to know that you'd never, you you know you really didn't have to think twice about, given your your current VMware estate, because like it's just been around for so long. So um we encourage ah you to try to break things right in this in and this ecosystem. um Although it's come a long way. so um
00:42:59
Speaker
Speaking of that, I think you know the skill set that VM admins need ah does change a bit. right A lot of the skills that VM admins are transferable, meaning how we think about VMs, how we do planning, um you know how we manage things, but underneath the covers, how we you know implement doing those things might ah change how they used to do it. So the question for you, Johnny, is sort of like from operational skills for VM admins, you know what things do what things should they learn first and what they look can they save for later?
00:43:38
Speaker
Yeah, absolutely. So I have seen a lot of VM admins jump to CKA and CCAD exams and burning my fingers.
00:43:49
Speaker
ah Sorry for using that term. Yes. The problem is you know Kubernetes has grown in complexity from where it started. And even for folks who are cloud native and cloud native first,
00:44:05
Speaker
Kubernetes is not easy to understand. It is it is complex. It has a lot of jargon, terminology, and lot of ah nuances to deal with. So my suggestion to VMware admins, if they are transitioning to cloud native and Kubernetes, is to do two things. Number one, get your mental model right.
00:44:25
Speaker
In the sense, map out your compute, storage, network, management layers to Kubernetes. Get that right. And when it comes to compute, you know make sure you understand the basic control plane architecture. you need to You need to be able to define what is a controller, what is a scheduler, what is HD for, and what is the API server. How do they talk to each other? Now, this will take about a day or two to to understand well And then you can start mapping the basic layers of your software-defined data center, which is now VCF, to the Kubernetes ecosystem.
00:45:06
Speaker
Now, you need to be conceptually aligned in terms of how these are benignured and how they work together. That is very architectural and strategic way of transitioning.
00:45:17
Speaker
Now, when it comes to hands-on and being very tactical, you've got to get familiar with KubeCuttle. There is no way... Getting it out of that. g work look No more clip quick click ops, Johnny? oh No.
Kubernetes as a Universal Platform
00:45:32
Speaker
Unfortunately, you know, QCAD is your friend and you have to master how QCAD works. So you have to understand various features and various aspects of QCAD and get familiar with it.
00:45:42
Speaker
Then the third stage, slightly advanced stage is GitOps. Because GitOps is a very essential production environment and you need to understand the the the combination of deep workflows with Kubernetes it is and how these controllers actually work with reconciliation and desired configuration state and all of that. So that's slightly advanced. But summarizing what I said,
00:46:08
Speaker
get your mental model right, number one. Number two, master the tools like KubeKettle and make a couple of other CLIs that you are dealing with, particularly for using CodeWorks, PixieCutail.
00:46:21
Speaker
So things like that. And then third thing is GitOps because GitOps has become the standard deployment route to any enterprise application. So understanding that is very essential. Those are the top three things.
00:46:38
Speaker
Gotcha. Okay. No, I think that's a good way to summarize it. My only only beef in that statement is you said you need to spend one or two days understanding EPS. I thought we just did in a matter of 10 minutes when you were explaining everything. Come on, Jatney. Everybody knows it now. Yeah.
00:46:57
Speaker
But there shouldn't be complexity of Kubernetes sub-drone play. It took me a few months. back in 2015 to understand ah you know what are the components of control plane and how they pop to each other.
00:47:09
Speaker
today's is still less. Yeah. No, but I... guess is A question for you there, right? Obviously, the complexity in Kubernetes has increased, but weren't we having this discussion a few years back, not not you and I specifically, but the community overall, that hey Kubernetes is supposed to be that that ah abstraction layer, that platform where all the complexities are hidden and all you need to do is focus on, hey, how can you deploy things on top? So it's similar to like, hey, only a few people in the ecosystem or of a couple of people at inside an and enterprise organization might understand the nuts and bolts of Kubernetes. But then for everybody else, it's supposed to be that platform. Similar to how not everybody has to know how an LLM um model is is fine trained and fine-tuned and all of those things. They should know how to use those. Do you do think that statement is still ah valid or...
00:48:05
Speaker
people that are moving from VMware to OpenShift virtualization or keyword generically still need to go to that next level of detail and figure out or learn everything to be successful.
00:48:17
Speaker
So it's again, a very, um very complex, very complicated topic. It's black and white. And I want to quote Kelsey Height over here, right? Kelsey sometime in 2016, 27, when Kubernetes was still fairly new, made the statement saying Kubernetes is the platform of platforms.
00:48:40
Speaker
right And resonates it resonated with me back then. It still resonates with me today because i was actually expecting someone in the ecosystem to build a platform as a service layer on top of Kubernetes, which will hide away the nuts and bolts of Kubernetes. yeah But 50 years later, we are still dealing with pods, deployments, stateful sites, services, your balancers, sign and a service knowledge.
00:49:09
Speaker
the whole gamut of controllers you know that we need to deal with. So the ecosystem ecosystem hasn't been able to build a platform later that abstracted the complexity of Kubernetes and Gingrad Gekush to developers, like Giroki or Clouds only.
00:49:26
Speaker
So that has the mind and unaddressed or unsolvable problem, which is a layer it for developers who do not want to understand one wanting nuts and bolts of Kubernetes is all the box.
00:49:43
Speaker
right We don't have the clay. And QWERT just kind of explodes that complexity further because you have to learn Everything from the hypervisor all the way up to the virtual machine instance and anything in between.
00:49:57
Speaker
So the answer is we don't have a platform layer that magically hides away the complexity of Kubernetes. um The closest that we got to is platform engineering and internal ah development portals. like the intermorse But again, they are more of DevOps layers rather than platform layers targeting developers. They are still targeting DevOps engineers and SRAs than developers.
00:50:28
Speaker
And QWord is not an exception. QWord, unfortunately, builds upon the complexity of Kubernetes and makes it slightly more complex. Gotcha. Okay.
00:50:38
Speaker
Thank you for candor when it comes to the new new this new technology stack. um So then what's a realistic migration path, right? if If I've gone through those trainings, I know what these different constructs are. I know how scheduling works.
00:50:54
Speaker
How do I plan out my migration? Because I'm not just running five or 10 VMs. It's in in the order of thousands of VMs that I'm running inside my organization. How do you recommend customers go about this?
00:51:06
Speaker
Yeah, it's definitely not a whip and replace or there is no magic button that you can it can push to kind of see this happen. So but remember, it's it's a transition and extension and not a replacement.
00:51:20
Speaker
VMware is not going to vanish overnight just because keyword has matured and the ecosystem is rallying around behind it. right so i want So the the fundamental aspect is you have to understand This is going to be ah side-by-side deployment and side-by-side environment for a long time to come.
00:51:42
Speaker
And then define your strategy ah very well. So first of all, you have to handpick those standards that are less critical and have low friction to migrate and are fairly independent and not tightly coupled with remaining VMs.
00:52:00
Speaker
So identify those candidates and start performing the migration ah to QVOT. Those are the low angle fruit and those are the best candidates to target the transition and migration.
00:52:13
Speaker
And then eventually, once you gain your expertise and context level, ah create a transition plan and create a migration path to move your workloads one by one.
00:52:23
Speaker
So look at workload as a as an entity rather than VM as an entity because... butcha now Mapping a workload as is to Qubet is more complete and mature and realistic than migrating one VM at a time.
00:52:38
Speaker
A VM is always a part of a larger workload and is delivering an outcome. So you have to think in terms of workloads and outcomes rather than individual VMs. That makes sense.
00:52:48
Speaker
And i I, you know, i ah do want to mention that, you know, during this during this conversation, we've mentioned that all the click offs is dead and those kind of things. And, you know, you have to learn which you do. and you have to learn to love yaml and and kind of dig into these details. But there are flavors out there that do have you know UIs that do help you manage VMs. and and I think you know listeners will discover that you know as they look beyond just the Qubare project itself and into those different orchestration flavors. That being said, i i don't know what the future looks like, Johnny. um Maybe we will get to a place where people aren't using KubeCuddle as much. and And who knows? Maybe in two years we'll be talking about how KubeCuddle with VMs is dead. I have no idea.
00:53:40
Speaker
um Maybe that would make VMapps really happy. ah I'm not sure. But um i want I want to leave you ah you know with one more question before we start to wrap up here. And I know we had probably a list of like 100 questions that we could have talked about with this topic. um But we do want to keep you know time in mind. So ah my last question for you today is going to be sort of what excites you about the convergence of VMs and Kubernetes and and kind of where it's going?
00:54:10
Speaker
Yeah. So for me, you know every time I see Qvert running inside Kubernetes and when I do a QCard in Get Virtual Machines, I feel I'm two steps closer to realizing the dream of universal control plan. That is the most exciting part. I'm going kick out of...
00:54:27
Speaker
looking at pods and VMs side by side. and And this has been the pipe dream for a decade. So that is finally you know happening and it is it is it is the it's happening in real time as we speak.
00:54:41
Speaker
So ah that is one. And second thing is, what excites PD most is the convergence of VMs and containers will accelerate the way we do ML, AI, and Gen AI stuff.
00:54:54
Speaker
i Sure. when we figure out how to do GPUs the right way with VMs running on top of Kubernetes, which is still a pain point at this sign, ah we are going to have a very um very hybrid or diverse set of solutions that will take advantage of partial workloads running inside VMs, partial workloads running on pods, and running the entire LLM workloads and pipelines end-to-end, spanning both pods and VMs together. And that is going to be a very exciting future.
00:55:32
Speaker
they maybe yeah Maybe even we can develop something that runs that way that can just help manage our VMs for us, right? Absolutely, absolutely, right? So now imagine any a combination of VMs and pods handling your entire lifecycle of data processing, pre-training, post-training, and inference and all of that without without you actually knowing which entity is taking care of this specific stage. right now Data scientists will access Jupyter notebooks without worrying whether it is a VM or a pod, and they do their job.
00:56:08
Speaker
like And inference gets served out of the cluster, sparring both VMs and pods, and the whole ML Ops pipeline will cut across all these entities. That's a very exciting future that I dream of.
Episode Conclusion and Thanks
00:56:20
Speaker
Well, on the next episode of Kubernetes Bytes with Johnny Karam, we'll talk about your AI and ML pipelines with VMs and containers.
00:56:30
Speaker
Well, Johnny, I do want to thank you as we wrap up here for coming on the show yet again to talk about such a fun topic. um On behalf of Kubernetes Bytes and Bhavad and I, thanks again for joining our show.
00:56:42
Speaker
My pleasure. Thanks for having me. Thank you Johnny. right, Bob, and I always love talking with Johnny. He's ah so insightful and very thoughtful about kind of the way he... Honest, yeah. And very honest, yeah, which is good, right? the end of the day, um where we're advocates here of technology, but we also are practitioners at the end of the day. Yeah.
00:57:08
Speaker
um Johnny's up to all sorts of fun stuff. And we we were talking to after the show ah and ah suggested that he'll be on in ah in a few months to talk about Jet AI and where that's come with great DMs. And he's doing all sorts of fun stuff. If you're at KubeCon Amsterdam, go see Johnny's talk.
00:57:29
Speaker
That's all I have to say. I won't i won't give any spoilers out. yeahp Thank you for that. Yeah. but But yeah, Babin, know if you want to add anything.
00:57:40
Speaker
No, I think we're good. Let's wrap this up. Yeah. All right. Well, that brings us to the end of today's episode. I'm Ryan. I'm Babin. Thanks for joining another episode of Kubernetes Bytes.
00:57:53
Speaker
Thank you for listening to the Kubernetes Bytes podcast.