Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
eBPF 101 with Matt Lenhard
500 Plays2 years ago
In this episode of Kubernetes Bytes, Bhavin and Ryan talk to the Co-founder & CTO of ContainIQ Matt Lenhard to dive into eBPF and what role it plays in the Kubernetes observability ecosystem. Learn about what eBPF is, what use cases it is used for and how you can get started learning more.
Show News links
- https://cloud.redhat.com/blog/introducing-red-hat-openshift-4.10
- https://thenewstack.io/nsa-on-how-to-harden-kubernetes/
- https://www.crowdstrike.com/blog/cr8escape-new-vulnerability-discovered-in-cri-o-container-engine-cve-2022-0811/
- https://www.businesswire.com/news/home/20220329005227/en/Ondat-Teams-with-SUSE-to-Protect-Customers%E2%80%99-Sensitive-Data-with-Enhanced-Kubernetes-Security
- https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/colocated-events/#attendees
- https://komodor.com/learn/kubernetes-pvc-guide-basic-tutorial-and-troubleshooting-tips/
- https://www.spectrocloud.com/news/spectro-cloud-closes-40m-series-b-round/
eBPF Resource Links
- https://github.com/Morpheus-compiler/Morpheus#ebpf-plugin
- https://events.linuxfoundation.org/cloud-native-ebpf-day-europe/
- https://github.com/iovisor/kubectl-trace
- https://github.com/iovisor/bcc
Recommended
Transcript
Podcast Introduction
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts.
Focus on Cloud Native News
00:00:14
Speaker
We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:27
Speaker
Good morning, good afternoon, and good evening wherever you are. We're coming to you from Boston, Massachusetts. Today is March 30th, 2022. I hope everyone is doing well and staying safe. Let's dive into it.
Personal Updates: Bobbin & Ryan
00:00:42
Speaker
Bobbin, how have you been? We're almost to April and hopefully some warmer weather. What have you been up to?
00:00:48
Speaker
Yeah, first quarter is coming to an end. I don't know. I need to do some personal evaluations. So again, you know this, we bought a house this past month, and that's what our whole quarter was about. Just going to open houses, going through the closing process, and just wrapping that up. So now I guess we need to plan better. This is me and my wife. We need to plan better on what else we are doing for the next year or something like that, or next quarter at least.
00:01:17
Speaker
And in addition to that, I am looking into either building like a completely new or using one of those raised garden bed DIY kits. I'll see what option I choose. Like I was going to buy the DIY kit, which comes with everything you need, but it is expensive than just buying food and figuring it out on your own. So I guess that's how I'll be spending my weekend trying to just see what I can do.
00:01:43
Speaker
How about you? Yeah, I garden is super nice. We usually do a raised garden on the side of my house. We do like we grew peppers last year, green peppers. They came out really good. Tons of tomatoes, tons of tomatoes. And we don't have a full like nice like up right on the ground garden. I'd love to have that. But my property is just like a hill basically. It would be kind of a pain to do it on the ground, but we make do and it's always a good time.
00:02:12
Speaker
Um, yeah, I just mostly had a pretty relaxing weekend. Um, had some family up. It's always nice to see family. So just a pretty chill weekend. Visited a new, um, a new brewery here that just opened in your, uh, it's called Rushford and Sons. Okay. And Upton, it's a kind of a new one. Um, I want to say within this like past year, but they had cousins, main lobster truck there. So.
00:02:38
Speaker
I got myself a nice Connecticut style lobster roll, which is fantastic. So I had a pretty good weekend, all things considered. I guess that's the spring and summer gold, right? Yeah, exactly. It was still kind of cold, but we made it work for me.
00:02:56
Speaker
I'm hoping for some warm weather around the corner.
Introduction to eBPF
00:03:00
Speaker
Cool, well, we have a really cool topic today for those who are kind of paying attention to the Kubernetes ecosystem. We're gonna talk about EBPF and we have an awesome guest to come on the show who is going to chat with us about what EBPF is and we'll introduce him in just a minute. But before that, we want to dive into this week's
00:03:24
Speaker
news. So Bhavan, why don't you start us off? Sure. So this week, Red Hat came out with the OpenShift 4.10 release. So it had like a bunch of new features, but the tool that I wanted to highlight was OpenShift sandbox containers, which
00:03:41
Speaker
were in tech preview or early access for the past couple of releases are now GA. So they are basically based on cat containers project and provide an additional layer of security or isolation so that defense in depth. So you can run your workloads that have a stringent security requirement or if you want to run some third party or untrusted workloads, you can run them as part of a sandbox environment and reduce the blast radius. So that was pretty interesting.
00:04:08
Speaker
And then the second thing or feature that I wanted to highlight from 4.10 was support for ARM processors. So now customers can use OpenShift on the couple of system they list are AWS EC2 A1 instances.
00:04:23
Speaker
and any real supported bare metal platforms that meet the ARM specification. So it should be good. They open up a few edge use cases or a few additional use cases. People who have been using ARM for their infrastructure and were looking for OpenShift support, now they have it generally available.
00:04:44
Speaker
Next on my list was just the NSA Kubernetes hardening guide report. I know we did discuss about this somewhere last year in the October-November timeframe. That was version one. NSA did update it based on some feedback from the community, added some more best practices. I clicked on the link. I didn't read it completely. It is a 66-page long report.
00:05:08
Speaker
But it does dive into some of the more obvious things, things like you should scan for vulnerabilities often on your containers and your images. You should always use least privilege for all your workloads, use network separation, authentication, authorization, and stuff like that.
00:05:25
Speaker
Again, it's a good read. It helps me reinforce the fact that Kubernetes is being adopted by not just the leading edge companies or companies that are working on the bleeding edge, but also now a wider range of organizations and customers that are adopting it as part of their production environments. So that was good. The third one.
00:05:46
Speaker
I like to talk about acquisitions and funding grounds. We did have a new funding round from a company called SpectroCloud. They raised $40 million in series B to accelerate growth of your community's management platform and product development.
00:06:04
Speaker
SpectroCloud, again, I don't have much experience with them, but they have a full stack solution that can be deployed anywhere. So specifically with this new round of funding, they're looking at edge environments where customers can just buy this whole end-to-end stack. As soon as they power it up, it will connect back to their SaaS management portal, something called Palette.
00:06:26
Speaker
And from the pallet interface, you can manage these different edge locations. So let's see, like this is just series B. So we'll see how they go on and raise future funding ground valuations as we have seen not many startups go public this year based on the current financial conditions, but that should be interesting startup to watch. Cool. Yeah, absolutely. But yeah, I think that's it for the list of things that I wanted to talk about.
00:06:57
Speaker
Cool. Yeah. Well, I only had a few things for this week. Uh, the first one I want to talk about is I'm always a fan of sort of, uh, introductory material, basic tutorials and things like that. There's a great article from this week's Q weekly, which targets, um, sort of what our communities persist in volumes, what PV, what are PVCs and specifically how to troubleshoot the, through these basic, um, kind of scenarios and some basic troubleshooting tips that you can use, which I thought was a,
00:07:26
Speaker
A nice addition to a lot of the introductory level one on one content we see about, you know, we've talked about what persistent value claims are what provisioning and binding and, you know, reclaim policies are and how to use them with certain objects like deployments but
00:07:43
Speaker
This one goes into some detail about sort of common errors that you might see, right? So common issues that you might see mounting volumes or accessing volumes and maybe some of the specific errors such as failed attached volume or failed mount errors that we've on the storage space probably know pretty well from a
00:08:03
Speaker
Many stars burned by those quite often. Yeah, exactly. And it goes through some of the common causes and where you can look to kind of go and diagnose the problem. And, you know, if you're in crash loop back off and things like that. So a really cool article. Anyway, we'll put that in the show notes. The next one was a bit of a shameless plug on co-located events. We mentioned it last time. There is a Kubernetes workshop that will be hosted by
00:08:32
Speaker
yours truly and my colleague, Tim at KubeCon Valencia this year. So that will be Tuesday, May 17th at KubeCon EU, which is the day before the event kicks off. And this is a Kubernetes data workshop. So if you like kind of what we've been talking about on this show, we're really going to tackle a lot of the same problems around introductory
00:08:55
Speaker
introducing sort of how to use, um, you know, stateful applications, um, how you can do that on Kubernetes. We'll have some live labs. We'll have some educational materials, some food. There'll be good fun time and some presentations at the end of the day. So, uh, we will put the link to sign up and, um, learn more about that as well.
00:09:15
Speaker
Having gotten a sneak peek at the agenda, I really feel that this session will be valuable for people that are looking to get started because this does have that hands-on component. So you're not just sitting there for a few hours listening to presentations and looking at demos. This is you actually doing stuff. So make sure if you register for it, get a laptop and you'll have some hands-on time.
00:09:36
Speaker
Yeah, get a laptop or bring a laptop. Hopefully you already have one. You don't have to get one, but you can bring theirs. It is bring your own L, learn your own laptop kind of workshop. So yeah, we'll do everything through web browser, kind of through some labs.
00:09:51
Speaker
The last thing I had on here was news from Onda, who teamed up with SUSE to do some things around security. We hear more and more about security all the time. So this announcement is actually more around how to work with secrets and how to securely work with secrets and how to not just use the defaults.
00:10:09
Speaker
in Kubernetes, but how to do that with SUSE and a project called TruSo, which I think I'm saying that right, but uses the Kubernetes KMS sort of project, which allows Kubernetes to interact with all bunch of key management systems. And so this project is kind of just really around, you know, how to work with secrets, how to do it securely, make sure things are encrypted, things are just not short and plain text. You know, everyone can decode base64 if you didn't know that. That's default.
00:10:38
Speaker
I definitely want to look into these security-based topics. We'll put a link to that announcement as well. And I think that's what we had for news.
00:10:48
Speaker
So let's dive into EBPF 101.
Deep Dive into eBPF with Matt Leonard
00:10:53
Speaker
Our guest today is Matt Leonard, co-founder and CTO of ContainIQ. He was a speaker at 2021 EBPF Summit and is super passionate about DevOps, Kubernetes, and anything involving the Linux kernel, which you'll learn most of what we're going to talk about is
00:11:15
Speaker
about EBPF and its interaction with the kernel. And do you actually remember what it stands for? Cause I don't even think. I think it's extended Berkeley packet filter. You are right. Yeah. Just do really while we're doing that. So in case you're wondering, that's what we're going to talk about today. Uh, really interesting stuff. And without further ado, let's get Matt on the show. Matt, it's great to have you on the show. Welcome to Kubernetes Bites. We're excited to talk to you about EBPF today. But before we dive into that, uh, tell us about yourself and kind of what you do.
00:11:44
Speaker
Yeah, thanks for having me on. I'm super excited for sure. But yeah, I'm Matt. I'm the co-founder and CTO of ContainerIQ, a Kubernetes native monitoring and tracing platform that leverages eBPFs to provide APM-like metrics instantly. We're all just super passionate about DevOps, Kubernetes, and anything involving the Linux kernel.
00:12:10
Speaker
Great. Yeah, I will admittedly tell you that I knew nothing about EBPF until we started to talk about this show. And then I dove into a couple of videos and really learned at a very basic level what it is all about. And I'm only scratching the surface, I'm sure. So why don't you tell us and the listeners, what is EBPF and why do we need it?
00:12:34
Speaker
Yeah. So full disclosure, I only got really into it about a year ago as well. So it's, um, it's definitely like, uh, relatively new. It's been around for awhile, but I feel like the kernel support has just gotten to a good enough point where people are really kind of pushing the bounds, uh, with what's possible. Um, but at a high level, it basically allows you to hook into, uh, either Linux system calls.
00:13:03
Speaker
or adding you probes to pre-built binaries to directly hook into those function calls and kind of like a programming interface on top of that. So you can do some pretty fascinating things. And I think you've kind of seen this explosion of tooling around EDPF. And it's always really cool to see what people are building with it now.
00:13:30
Speaker
Yeah, I know that one of the things that we're doing at this coming KubeCon is a day zero event. And I think I saw a whole community event at ABPF, which maybe we could talk about a little bit later, but I think I might have to show up just because it's starting to learn more about this.
00:13:49
Speaker
So since eBPF is sort of tied to the kernel and the system calls, what's the difference between existing tooling that's out there and something like BPF? Yeah, that's a great question. So say, for example, I think like a good comparison of something like perf or maybe like some of the existing tracing tools.
00:14:15
Speaker
What I've kind of seen in production environments is that eDPF is often more performant. The reason for that being it runs in like a VM inside of the kernel. So there's specific use cases where people are building tools that you generally wouldn't run in production now because of some of the performance gains. I also think the entire programming interface that's built on top of it allows you to do things that weren't really
00:14:45
Speaker
possible before, so you're seeing entire applications built around this new set of tooling. And some of the, I guess, functionality is more fleshed out. I think the possibilities are definitely greater with eBPF. Like I said, there's some super fascinating things being built with it right now.
00:15:07
Speaker
Gotcha. So I think when you introduce yourself, you said ContainIQ is built using eBPF. What are outside the monitoring ecosystem, what other use cases are there, or what are the other vendors doing with eBPF today? Yeah, so I think there's a lot of use cases you see in general observability with other tools like Pixi. But I think there's some even more interesting applications
00:15:36
Speaker
uh, applications and things like security. Uh, so like a great example of that is Falco. And what they do is, um, they're basically listening to all of the system calls that happen, like within your cluster by adding, uh, caperbs to them and then, um, providing you kind of like high level security information based on their security or their system calls. Uh, and a great example of this is like, they're able to detect a crypto miners, which has been like a,
00:16:05
Speaker
I have something surprising I've heard recently is people, even employees running crypto miners within their company's clusters.
00:16:14
Speaker
Yeah, not great. And getting away with it too. And so something like Falco can do is they can kind of listen to these, you know, listen in on these different system calls and find processes that shouldn't be making them or things that look out of the ordinary and giving you the ability to kind of like restrict those things or alert on them or, you know, identify when things are happening that shouldn't
00:16:39
Speaker
I actually remember when I was working for a healthcare company, we used to test out Mesos a lot before we dove into actually running it as our platform. But we used to put it up on the internet a bunch. And if you just left it there for long enough in a single day, you'd get miners just popping up on your Mesos clusters. And it was, you know, it's a common problem, but it's an interesting solution to it with UBDF. That's for sure. Yeah, I think
00:17:07
Speaker
We see a lot of people who are using Kubernetes to spin up environments for their users. So they're spinning up a pod that gives each user maybe a development environment. And so what's the first thing people try to do? They try to spin these up, launch a crypto miner on it, and make some money off the company, I guess. And so yeah, tools like Balco can
00:17:29
Speaker
put into those calls and help you catch those things. Interesting. And so I guess one thing for our audience, right? Like the reason EBBF is able to do all of this is because it runs on the kernel. So it can monitor all of these system calls and processes that any of the container that are running on the host can like submits, right? So it can basically be that one single layer where you can have this intelligence built in and it can monitor traffic and like perform all of these actions for you.
00:17:59
Speaker
Exactly. So since you're like cooking those system calls, um, you can really do anything. You can dump the arguments to it. You can dump the return arguments. You can do a ton of analysis on everything that's happening. Um, I guess that leads me into like somebody we can like downsides as well though, is that you do have to, um, we're on the container. That's, that's, um, using this as root or as a privileged, um, some of the newer kernel versions have capabilities for, um,
00:18:28
Speaker
BPF, but if you're not, I think it's like five, seven and up, but yeah, if you're running anything earlier, you need like sys admin with a privileged container. So you're definitely giving a lot of capabilities to the container that's running these BPF based programs. Gotcha.
00:18:46
Speaker
And so in addition to like the vendors, like we just spoke about Falco and contain IQ, how are like customers adopting eBPF? Do they even care about the kernel level or did they just rely on these vendors to offer a security or an observability solution and then they don't care about the implementation details? I think it sort of depends, but in my experience, most people don't really care that you're using the BPF. They just want, um,
00:19:15
Speaker
like it's like information in either an easier way or with like DACA's case, like a way that maybe it wasn't possible before. So like a great use case, I think in regards to like Kubernetes is like, you can hook all of these like system calls, right? Or like with what we're doing, we're adding a BPF program to the traffic control. And so from this, we can parcel all of your network traffic out, right? People don't really care about that. What's interesting to them is that we can then take that network traffic
00:19:45
Speaker
associate it with like my P address and then share you all the traffic in and out of like a given pod based on like the protocol. Right. And so I guess like what a lot of the use cases you see with PPFR is like, it's a faster time to value in some instances because you don't have to install an APM. You don't have to kind of configure like application layer
00:20:10
Speaker
like tracing or something, it's possible to do it with BPF at the kernel layer and then use Kubernetes to attach the metadata about the application. And I think one of the demos that I saw getting ready for this episode, right, where instead of creating individual sidecar containers for each of your boards, you can have the functionality that you just mentioned as part of the EPF program running on the host. And that way you don't have to modify your applications, but still get all of that functionality that you need.
00:20:40
Speaker
Exactly. So like that, that being like kind of a faster time to value something you've seen with like psyllium, um, where like you don't need to, you know, proxy the requests in every single container or pod with like a sidecar container. You can just install a daemon set on the host. That's looking at, um, every open socket and making kind of like the routing decisions based on that, or just, um, exporting the network traffic somewhere else for no later analysis.
Deploying and Future of eBPF
00:21:08
Speaker
Got it. Got it. Now EBPF is deployed, um, in, uh, the same way other applications are then since it's a demon set. Like, yeah. Um, I mentioned, um, if it's deployed on Kubernetes, I'm assuming it can be deployed as sort of like any other application that would be deployed as a demon set. Is that right? Yeah, I think that's how you typically see it deployed. So generally it's going to be a Damon set. Um, where you'd have to like now, um, the,
00:21:37
Speaker
sys folder, and then you're going to have to run the container as privileged. Depending on if you're using something like PCC tools, which is just a framework for building PPF applications, you'll also have to download and install the kernel headers. Which can be annoying because they're dependent on the specific kernel version you're running. So there's a little bit of figuring out the kernel version and then installing the kernel letters based on that.
00:22:06
Speaker
So it's a little bit of extra work. I think what I've seen is a kubectl trace doesn't, this is great. They have like a minute container that runs and basically sounds like, okay, what's your kernel version? Yeah. Okay. Let's get the kernel letters for that version. Let's set up the mounts. We're going to configure everything for you. So that like, it just kind of works.
00:22:28
Speaker
Oh, nice. Okay. Need to try that out. I didn't even know this such a thing existed. So this is helpful. Yeah. Definitely would check out cube CTL trace. It's, it's a super cool and it's an easy way to kind of get started with, um, kind of like EPF cause you can, it basically leverages BPF trace and lets you run these different, um, kind of like BPF trace functions on, you know, uh, different pods in your cluster.
00:22:57
Speaker
Makes sense. Now you mentioned, you know, EPF is sort of a way for you to take an existing system. And this is, I think I see a lot of value in the sense that if you have an organization that's already running a whole bunch of applications and they want some level of, you know, security, you know, observability, right? Dropping in EPF seems like a really good way to say, hey, every application doesn't need a code change or, you know, every application doesn't need a,
00:23:27
Speaker
another container, that's actually super valuable. I see that as a huge win. But where do you see EBPF sort of taking on the future? Like what's the, I guess, what's it targeting for the future? Where do you think it's going to really hold its own? Obviously security being a huge part of this, I think, but maybe some others that you can talk to.
00:23:52
Speaker
Yeah, so one of the coolest projects I've seen, it's actually built by a friend of mine, Sebastian. It's called the Morpheus compiler. And what it does is it basically analyzes your network traffic in real time, and then provides like compiler optimization to basically improve network throughput. So it's kind of like this continuous compiler that's looking at like real time network traffic and making
00:24:20
Speaker
decisions based on that. So in like the benchmarks they've done, there's like been, you know, some really large improvements on things like latency, like I said, like network throughput. And so I think you're going to see maybe people rewriting some of the existing kind of like general networking tools out there using BPF. And I think there's, you know, there's, there's a lot of really cool things that are going on here for sure.
00:24:51
Speaker
Got it. Now, in the networking space, and since we're talking about Kubernetes, is there a is there sort of a path to, you know, helping the idea of, you know, multi-tenancy understanding, you know, shared platforms or, you know, platform or as a service vendors, can they use it to sort of help identify maybe
00:25:14
Speaker
how much network is one tenant over another kind of using in the sense that I see shops kind of do things a couple ways is like every team or part of an application gets their own Kubernetes cluster and they can just do whatever they want or there's this like big Kubernetes cluster shared with multiple tenants. And I think we see this coming up more and more often. So I'm curious about any intersection here.
00:25:39
Speaker
Yeah, I think, um, I mean, the possibilities are kind of endless cause you can, like I said, you can talk into anything, but it's definitely possible because no Kubernetes provides like the metadata for you to associate these things with like whoever the corresponding team or like, I guess like application responsible for that network traffic is. So yeah, I think kind of leveraging, you know, BPF to intercept the traffic.
00:26:06
Speaker
And then Kubernetes to kind of layer on the metadata on top of that, you can get like pretty granular insights into, you know, who, what traffic belongs where, um, and like who's responsible for it, which applications, um, even if you just have like multiple teams and like, you want to see email or traffic per team, like you could do something like that, uh, pretty easily.
00:26:30
Speaker
Nice. And like, I think in my previous jobs, I've worked with tools that do this at the virtual machine level. So like analyze, because they only have one NIC or in most cases, analyze how the traffic is going between them and understand your like communication inside your own data center. And then like the next step was to enforce certain rules. Do you see this
00:26:52
Speaker
EBBF taking as the next step, like also help with enforcement. Like, okay, if you are, if, if it's helping you through metadata and through packet analysis, figure out what's, what component or what part is talking to a different part, can you enforce certain rules and implement those network policies to, to stop certain traffic in real time? Yeah. So, um, psyllium actually does something pretty similar to this. I'm sure where I think at least where you can look at, um, kind of like the,
00:27:22
Speaker
connections between pods and reject them based on data. I don't know if it's on payload yet, but yeah, that's all definitely possible. And there's even specific BPF functions right now being, I think they're like five, four, I think it's verdict, I believe, BPF verdict, but where you can basically say whether to drop the packet. So you can, you can basically look at the incoming metadata of the request and say like,
00:27:53
Speaker
Okay, good pass or like, no, drop it. So yeah, I think that's definitely, um, something we're going to see more of in the future and kind of probably selling them even built on their building on top of like their current implementation to allow more complex mouse of the like packets on decision-making on top of that. Um, like I said, a lot of this is like only available and sometimes newer corals. So it's, um,
00:28:18
Speaker
Some of it's like just playing a catch-up game and waiting for older environments to catch up with what's possible.
00:28:26
Speaker
Yeah, it's actually talking about psyllium. It's actually reminding me I used to work a lot on sort of software defined networking and open flow networks and the things you could do and sort of inside the switch for programming the network, right, being able to identify packets and do some kind of network function.
Challenges and Learning Resources for eBPF
00:28:43
Speaker
It's reminding me a lot of that, but obviously at sort of the sort of Linux and kernel level now, so it's really
00:28:49
Speaker
Really quite interesting. Now you mentioned before that, you know, having to run EBPF itself on the host as root is sort of a downside. Now, is there any other, you know, got you as one getting introducing yourself to EBPF and what it can do? Yeah, so I think if you're not careful, you can add a lot of overhead. So it's definitely something you should be measuring and keeping track of and
00:29:17
Speaker
just like staying aware. It can be dependent on like very wildly between like different hosts and kernel versions and you know, the amount of like network throughput. So you definitely want to stay aware of the performance. If you're installing like the kernel headers, that's obviously going to take off disk space. So again, something you want to
00:29:44
Speaker
stay on top of, um, if you're using something like BCC tools, which is, like I said, it's just like a framework for building out BPF programs. Um, they, they, you're going to have to add a compiler to your, like the container you're distributing. So larger container size, um, differences between kernels can be annoying if you're relying on, uh, like certain kernel structs. Um,
00:30:11
Speaker
And the verifier likes yelling a lot. So there's a lot of times where like, and we see this all the time, where it'll work on certain kernel versions and others, like the verifier was slightly different in that version. And so we're like slightly tweaking our checks, our if statements to please the verifier. So yeah, they're going to be just a lot of fiddling around to get everything to work perfectly, I guess.
00:30:38
Speaker
God, it makes sense. Yeah, I mean.
00:30:42
Speaker
And one of the things that I first came to mind when you're like, oh, you can run sort of a function based on a Linux kernel call. Well, that program, I imagine is very important, like you said, to keep in mind of how much work it's actually doing, right? Because if it's stuck in an infinite loop or it's not getting the right condition, or if you're trying to do too much within there, I imagine, like you said, it can affect the performance. So I'm wondering if there's any best practices around using it that you can share.
00:31:12
Speaker
Yeah. So I think, um, for us, at least I kind of just test, test, test. I mean, I'm sure a lot of engineers say that, but you probably just have to, you know, throw it in like a somewhat realistic environment, um, push a lot of whatever type of like operation. So maybe you're doing like network analysis, like, you know, pushing a lot of data through it, making sure everything works as you're expecting. Um, if you're using like the,
00:31:42
Speaker
a newer kernel, you can leverage, um, some of the new, uh, BPF based capabilities. That's like cat BPF so that you don't have to run a sys admin. So you can lower your security posture now, even kind of just a little bit. I also say on newer kernels, there's, um, uh, BPF type format and like the kind of like compile once run anywhere or everywhere, which will allow you.
00:32:11
Speaker
Basically you do not have to worry about changes in structs as much. So if you're running a newer kernel, I think like that's definitely something you should be taking advantage of. Um, maybe leveraging lib epf or something like PCC just because of the smaller footprint and container. But I think in general, like a lot of it kind of depends on the kernel version you're running and using as much of the
00:32:39
Speaker
kind of support you can get from that formal version. Gotcha. So like all of this is great and like people are empowered to do whatever they want. But just honest opinion, like should end customers or end users try to mess with EBPF programs or just leave it to the vendors and have them scrape all kind of information that's needed rather than trying to do this all on their own? I don't know. I find it fascinating. So I think everyone should kind of play around with it. It's definitely like a cool learning experience. Yeah.
00:33:09
Speaker
There's a suite of tools out there you can get started with really easy. Like I said, check out kubectl trace and let you run super simple or straightforward PPF trace functions or like BCC tools, which again is just, they have a ton of, you know, Python scripts that you started with where you can do things like trap track, you know, TCP connections or look at a function latency is in your Python program.
00:33:39
Speaker
So I definitely think it's worth people kind of playing around with and kind of seeing what's possible. And for me, it's fun. I kind of enjoy that kind of life. And make sure we include a link to all of these tools that you mentioned in the show notes so people can check them out later.
00:33:57
Speaker
Yeah, I imagine that the vendors that produce products with eBPF, ContainIQ, there's also a slew of other products that might be able to use it. We talk about storage, a lot of storage vendors in the Kubernetes space. I imagine they might even have their own use cases for it, so definitely super interesting technology.
00:34:20
Speaker
I do want to make sure we hit on, like Bhavan said, anything you want to include where people can get started. I know, I don't know if you can share anything about the QCon EU and Valencia Day Zero event, but we will definitely make sure to link to that as well. Yeah, so I'll definitely share, I think some tools that I'll start with that people should check out that I think are really cool and kind of help them get started with, you know,
00:34:48
Speaker
hacking away with BPF and that's kubectl trace, bcc tools, libbpf if you want kind of like a little bit less straightforward but less overhead way to develop BPF programs. Again the Morpheus compiler I think if you're looking at you know what's possible with BPF I think that's one of the coolest projects out there and there's a great research paper as well that goes along it
00:35:18
Speaker
kind of explaining how it all works. That thing is super interesting and people should check out. There's also, yeah, the first day of, or day zero, I guess, of QCon is always like EVPF day. So I'm sure there's gonna be some great talks there. But yeah, there's an EVPF summit as well. People should look into. Not, it was, I think, August of,
00:35:44
Speaker
last year, so probably around the same time. Nice. Yeah. There's, there's two days of day zeros now. I don't know. I, I've been contemplating what to actually call that, whether it's day zero, day 0.5. Um, normally in cubecons, there's one day zero and then the event starts, but yeah. And in the Valencia, there are two day zeros. This I think is 16th and 17th. So, um, we'll definitely check that out. We'll be, um, I'll be there as well as my colleague, Tim. So I'll definitely come by and check it out.
00:36:15
Speaker
Um, I guess with that, you know, we'll make sure and include all those, but, um, you know, Matt, I think I've learned a lot and have a lot more to learn about EBPF. I'm definitely going to go check it out. Um, it's been a pleasure having you on the show and, um, I hope our listeners, uh, get a lot out of this just like we did. Yeah. I had a great time kind of like talking to you guys about this. Um, I still have a lot to learn. Thanks. Right. Yeah.
00:36:43
Speaker
Sounds good. Take care.
00:36:47
Speaker
All right. Well, that was a good conversation. I know that, you know, EBPF is definitely a little outside the norm for what we talk about. And I actually liked that, right? I, you know, getting ready for the show, I know I asked you if I was interpreting EBPF correctly and what it does. And I actually find it super interesting. We don't work with the kernel much in our day jobs. So really interesting to see how EBPF is used. And I know I've seen this term come up.
00:37:14
Speaker
more and more in the Kubernetes community and haven't really dove into it. So this was a really cool conversation to have. And I don't know, what did you, what did you really learn from it? This was one of those episodes where I was trying to make sure that I sound as a person who understands the basics.
00:37:32
Speaker
But yeah, really, we needed an expert and Matt did a great job of talking about eBPF and why it's needed. One of my key takeaways was just since it runs in the kernel, it can deliver that better performance. It can optimize the way your applications are built. And a couple of interesting use cases were around observability and security and how running these eBPF programs on your
00:37:57
Speaker
can help you do packet filtering can help you implement certain security rules can help you block traffic even so vendors like falco or psyllium have already been doing such great work that even if you learn like a few things from this podcast definitely personally i will go and check those out and look at additional demos but that was like one key take ever like this.
00:38:22
Speaker
EBPF, even though it's a buzzword, it's increasing in terms of adoption. It's increasing in terms of number of vendors that are using it to implement or build their products. And it definitely has a good trajectory or adoption trajectory ahead of it.
00:38:38
Speaker
Absolutely. I think the big thing that really caught my attention was not having to really change a lot of what existing applications, application code or sidecars are really doing already today. We've actually dove into a few different projects that
00:38:59
Speaker
means you need to kind of adopt certain SDKs or things in your code base to update or to make things work for tracing and things like that. And there's probably really good use cases why that still is the case, but I really like the idea that eBPF and something built on it can sort of be dropped in place and kind of look at those syscalls happening at the kernel level and without a lot of change at all,
00:39:27
Speaker
to any application already running on the system. That is super valuable. I think when talking about your ROI and your time to value of adopting new technology, being able to not change a whole stack of things to make a new thing work is super, super valuable.
00:39:45
Speaker
No, it helps me tie our conversation that we had with timescale DB and around observability and how, like when you're adopting that solution, you do need to make a few changes, add a sidecar
Closing Thoughts and Next Episode Preview
00:39:55
Speaker
container. Whereas this is a different approach of doing things and how, as you said, you can just drop it in without
00:40:02
Speaker
drastically or even like changing your code by a small percentage, which can be a lot, which can be tedious and time consuming and can just push back those dates for adoption. So this is great. Exactly. And that being said, I'm sure there's use cases for both. You know, given that, you know, this is really at the kernel level. So if it's not talking to the kernel, then you really don't know about it. So having, you know, tracing happening within the application within
00:40:27
Speaker
a certain language is probably still a whole slew of use cases that are super valid there. But yeah, really cool technology. The other thing, because I'm a newbie, the be careful statement around you can really mess things up. It caught my eye and it makes sense to me, right? So if you can kind of write programs that do something when there's something else happening at the kernel level,
00:40:51
Speaker
and you're going to make it wait to do something unique, then you could definitely tie down the performance if that thing is really cumbersome in terms of what it's trying to do. And I could easily see how it could affect the whole performance of the system. So I'm sure that that's something I would do probably when exploring this technology the first time.
00:41:17
Speaker
Um, but I also really like, you know, how, um, how Matt was super open about getting involved and really just kind of, uh, digging in and getting your hands dirty with a UPF as well. So really exciting stuff. We'll include all the links that Matt talked about.
00:41:32
Speaker
in the show notes, as well as some news links as well. And like, this is not like a takeaway from the discussion, but just an interesting point that he made, like people, employees are actually using the Kubernetes clusters and running crypto miners and making money on the site. My mind was blown up like, wow, people are just going crazy now. Yeah. Don't do that.
00:41:58
Speaker
That's our message from Kubernetes Spites is don't do that to your employers. Anyway, anyway. All right. Well, you know the drill. Please review Kubernetes Spites wherever you can review your podcast, share it with your friends, and let us know what you like, what you don't like.
00:42:17
Speaker
Please give us that feedback. It's super important for us. And in a couple of weeks, we are having a data stacks on the show, which is really exciting to talk about Cassandra and something, some stuff they're doing around multi-cloud. Really, really excited about that episode. So stay tuned for that. And that brings us to the end of today's episode. I'm Ryan. I'm Bobbin. And thanks for joining another episode of Kubernetes Flights.
00:42:45
Speaker
Thank you for listening to the Kubernetes Bites Podcast.