Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Breaking Down the Diamond: A Look at MLB's Kubernetes-Powered Analytics
1.6k Plays1 year ago
In this episode, Bhavin and Ryan chat with Kevin Backman from Major League Baseball (MLB) and Mike Wagner of Metify to talk about how MLB made the transition to baremetal Kubernetes with Anthos to support all 31 major league ballparks and their infrastructure for capturing on field statistics.
- 03:35 - Cloud Native news segment
- 12:10 - MLB and Metify Interview
- 49:06 - Takeaways
Are you ready to earn extra income from sharing your expert opinion? Head over to userinterviews.com/hello to sign up and participate today!
MLB Article https://technology.mlblogs.com/anthos-on-bare-metal-how-mlb-turned-ball-parks-into-on-prem-k8-clusters-295031253a33
News
Kubecon Sessions are live - https://youtube.com/playlist?list=PLj6h78yzYM2PyrvCoOii4rAopBswfz1p7
Kubecost Cloud Public Beta - https://www.globenewswire.com/news-release/2023/05/09/2664089/0/en/Kubecost-Announces-Public-Beta-of-Kubecost-Cloud.html
Group Volume Snapshots - Crash consistent https://kubernetes.io/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/
Coinbase uses Kubernetes to run staking nodes - https://www.coinbase.com/blog/operating-staking-nodes-on-kubernetes
Kubernetes 1.27: StatefulSet start ordinal simplifies migration https://www.cncf.io/blog/2023/04/26/introducing-sessionize-a-new-cfp-platform-for-cncf-events
https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-on-demand-webinar-run-wasm-shim-this-way
https://www.techtarget.com/searchitoperations/opinion/Takeaways-and-emerging-trends-from-KubeCon-Europe-2023
https://cloud.google.com/blog/products/containers-kubernetes/backup-for-gke-concepts-part-1
https://www.nutanix.com/blog/nutanix-announces-early-access-of-ndk
Recommended
Transcript
Podcast Introduction
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts.
Cloud Native Focus
00:00:14
Speaker
We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
Greetings and Date Mention
00:00:31
Speaker
Good morning, good afternoon, and good evening wherever you are. We're coming to you from Boston, Massachusetts. Today is May 10th, 2023. I hope everyone is doing well and staying safe.
Vermont Trip Chat
00:00:43
Speaker
Bhavan, let's dive into it. How's it going? How you been? I've been great, dude. The weather is awesome. I'm hoping summer is finally here.
00:00:52
Speaker
So we did a short weekend trip to Burlington, Vermont. And when we booked it, right, like we booked it like two, three weeks back, the weather was going to be in the fifties. We're like, yeah, we can still sit inside breweries and have some good beer. But then when we, when we reached the weather was 70 degrees and it was awesome. Like, whoo, that was a great weekend. I call this season fall summer. Yeah, I'm okay.
00:01:17
Speaker
There's always fall, summer, and then we go back a little bit. And it's super cold. I saw it was supposed to be like 38 degrees tonight. So when you do that cold at night, it just doesn't make sense to me. But yeah, I'm jacked for it. I was wearing shorts yesterday, so I'm living it just as helpful as you are.
00:01:34
Speaker
Burlington's nice. I haven't been back up. I'm excited. I'm going back up to Ron a few times this summer. I know like, but I think I'd been to Burlington once, but we did such a, like a fast trip. Like one day we went there, had lunch, had a beer and came back this time. We're like, nope, we have to stay there for at least one night. But you drove to Burlington for lunch. Did I just hear that right? No. So we were camping in the Adirondacks and you're like, okay, on the way back, we'll actually stop by Burlington. Nope. I didn't make the three enough. I'd ride for lunch. I was like, wow, you must really like,
00:02:03
Speaker
Something good.
00:02:05
Speaker
Although the food that I had there, I made a reservation at this Asian place called single pebble. It was really good, like really good food. Shout out to single pebble. Yeah. I know. Great name. Great name. And the best part, like when you, we did take out, like take some food with us. They give, they give you a single pebble, like as part of the, they're like, Oh no way. There's someone living their name. I love that. That's great.
00:02:34
Speaker
Oh, wow. Like a marble pebble? They went on the show. This is a good costume. Or maybe we give them a good pebble. We save our good pebbles for the real people. Oh, wow. That's fun. You had more fun than I did. We've been kind of hunkered down here. My wife just had surgery and she's fine, but we've been kind of laying low and enjoying some kind of hair.
00:03:00
Speaker
people come to us time. Can't complain too much here. But otherwise been chilling close to home. So, you know, nice. Yeah, absolutely.
KubeCon Sessions and Announcements
00:03:13
Speaker
Alright, so we have a really, really fun episode today. We have some great guests from MLB and Medify on the show before we jump on and introduce them a little bit. Let's go into our news a little bit.
00:03:26
Speaker
Yeah, sure. Talking about KubeCon, we did a whole episode, but as a follow-up or one last thing, the KubeCon sessions are now live on YouTube. If you are like me and you didn't actually make it to Amsterdam, I'm jealous. Still Ryan. I didn't get to watch that many sessions, honestly, so I'll be watching them.
00:03:48
Speaker
All the sessions are live, and we'll include the link to the YouTube playlist in the show notes. So I'm sure I'm going through them one day or one session or a couple of sessions every day while I'm doing some work. But that's live. CubeCost, one of our friends, they were on the podcast to talk about Kubernetes cost management. They have a new public beta for their CubeCost cloud offering, where instead of having to deploy CubeCosts on your Kubernetes cluster, all you need is to deploy an agent. It connects back to your hosted or cloud solution.
00:04:17
Speaker
And then you can look at your entire estate and do cost management across all the different Kubernetes solutions that they support. So you can have EKS, GKE, AKS, instead of deploying kube costs in each, you can basically connect everything through agents through that one single portal.
Kubernetes 1.27 Features
00:04:35
Speaker
Very, very cool. Talking about Chill vibes and 1.27. I still love that name. I think you referred to this, but I just wanted to make sure that we highlighted like group volume snapshots are now alpha with the 1.27 release. So again, volume snapshots have been a thing in the CSI standard for a while, at least two, three releases, if not more. Group volume snapshots adds a capability to take
00:05:02
Speaker
PVCs that may be belonging to the same stateful set with multiple replicas and take a crash-consistent snapshot and protect your PVCs. It's in alpha, so I'm sure you'll have to enable that feature flag to actually experiment with it, but it's live and we'll link to the blog article that Kubernetes bytes alumni, Xing Yang, did on the topic. Yes, absolutely.
00:05:24
Speaker
And then final note, I know we have an end user perspective today on the show, but even when we are talking to MLB, I was like, let me see if we can find any more cool use cases. And one came across Coinbase actually uses Kubernetes for running their staking nodes on the cloud.
00:05:42
Speaker
They have an interesting article on how they actually use it, what are the different Kubernetes perimeters. So they talk about using stateful sets. They talk about using PVCs. They talk about how they're using config maps and secrets for configuration informations. And so it's a great read, a quick read. I think it took me five minutes, if not less. But yeah, it goes into a good amount of detail to share their experiences. And I'm sure like they're using Kubernetes outside these staking nodes as well. But again, that's another cool article that I found this week.
00:06:11
Speaker
Yeah, absolutely. I think we've had a few guests in the crypto space that I've also used Kubernetes. So it seems like a popular tool for the job, I guess. I'll have to go read that one myself.
CFP Acceptance and Opportunities
00:06:23
Speaker
The first bit of news I have here is that if you submitted a CFP to KubeCon EU, you might probably have been one of the ones who did not get accepted because they had 11% acceptance rate.
00:06:37
Speaker
Something like almost 2,000 submissions down to 300 or something like that. Don't quote me on the numbers, it's in the article, but it's very low. So anyway, they're getting a crazy amount of volume of submissions and I think they need to grow with the community and how many are being submitted.
00:06:58
Speaker
New platform, if you've submitted before, look out for something new rolling out and read the article. I don't think it's going to be anything major to the end user, but something to keep in mind that if you log on and say, this looks different, it probably is.
00:07:14
Speaker
I think that's another thing, right? I have submitted CFPs for KubeCon in the past. Never, never do they get accepted. I'm still annoyed by the fact, but again, as you said, 11%, yeah, I don't think my CFP was good enough to be in the top 11%. I would just hope like CNCF makes it like a five-day conference instead of a two-day conference so that more people can share what they're working on again. That's a lot of Kubernetes in one week. I don't think either of us would mind.
00:07:45
Speaker
No, you're right. And I think in lieu of getting more people to speak in the community to come along, I think I'm all for that. In general, it's tough. It's not a gamble. There are ways to definitely make your CFP more interesting or maybe tailored to that year. And maybe we should do a whole episode on that. That might be a cool topic to do, actually.
00:08:07
Speaker
Speaking of CNCF, there's a whole bunch of different online events that are done through sort of the webinar
CNCF Online Events
00:08:15
Speaker
series. One of the ones I want to call out for this week is actually the 11th and 12th, 12 AM Pacific. I think I'm reading that right. But you may have to look into that. It's called Run Wasm Shim This Way. We had done a couple. Yeah.
00:08:35
Speaker
We had done a couple of episodes with sort of the topic of a Wasm Web Assembly with Nigel Bolton and things like that. And this seems like a cool topic. Definitely go check it out. I think the whole Web Assembly sort of theme at KubeCon is definitely something that we still keep seeing. So we're going to try to post those things as they kind of gain traction as well. The other one is an article on a backup for GKE.
00:09:05
Speaker
It's a concepts part series. Pretty
GKE Backup Importance
00:09:07
Speaker
cool. I think the whole idea in the blog is really enlightening if you're new to sort of data protection or backup and restore capabilities and you use JKE, which is also relevant today's episode, actually. You know, definitely go check this out. It kind of goes through the concepts of what's provided what and the particularly the piece I like is it really breaks down what it's
00:09:30
Speaker
backing up what's in the backup versus what's not in the backup. Because I think that is confusing to many people where it's like, well, is my image backed up? Is my service or load balancer back up? What about my actual cluster? I've heard that a few times. Yeah. So this article does a good job at that as well.
Nutanix Data Service for Kubernetes
00:09:49
Speaker
And the last one I have is Nutanix. I've been around for quite some time. They announced something called Nutanix Data Services for Kubernetes.
00:09:58
Speaker
which is all about, as you may have guessed, data services on Kubernetes and their way of providing pre-built and managed and those kind of things on top of a number of different platforms. So very cool stuff. Go check out what they're up to.
00:10:16
Speaker
Yeah, that's interesting. I used to work with Nutanix at my previous job, and they had a similar service for virtual machine-based data service deployment called era. So again, this is something that I didn't catch, so I need to go and check it out what they're doing for Kubernetes. Well, it's early release, so it's like, I don't know how much you'll be able to really poke at it, but maybe you could be part of it. In the next six months, I guess.
00:10:39
Speaker
Exactly. All right, now let's jump in and get our guests
Guest Introduction: Kevin and Mike
00:10:43
Speaker
on the show. We have Kevin Backman, who works at MLB, has been there for about 10 years, a Google fellow and senior systems architect there. He's kind of in charge of what we'll be talking about today. So a lot of fun and interesting things from him. And then also Mike Wagner, CEO of Medify, and he'll kind of talk to us about Mojo and what they've been able to do with their bare metal environment. So
00:11:03
Speaker
Without further ado, here they are. We would like to thank our very first sponsors for the Communities Bytes podcast, User Interviews. You might be thinking, who or what is User Interviews? User Interviews connects researchers with high-quality participants to provide real feedback on real products. Participants can earn some extra bucks for their feedback, which allows them to have an impact on how the products are actually being built.
00:11:30
Speaker
User interviews is free to sign up and most studies are less than an hour and pay over $60. Some studies pay several hundred dollars for a one-on-one interview. And as we all know, who doesn't want some extra beer money for the weekend?
00:11:44
Speaker
You can share your opinions with relevant companies who are building products like Adobe, Spotify, Amazon, and many more. Are you ready to earn some extra income for sharing your expert opinion? Head over to userinterviews.com slash hello to sign up and participate today. That's userinterviews.com slash hello. And tell them the Kubernetes Bites podcast sent you by using the promo code kubernetesbites.
00:12:11
Speaker
Hello, and welcome to Kubernetes Bites. Mike and Kevin, it's so great to have you here. I won't spoil it for our listeners. Please give us a little bit of introduction of who you are and what you do. Yeah, why don't you go first, Mike? Cool. All right, so I'm Mike Wagner. I'm the co-founder and CEO of Metafly. And Kevin. Hi, I'm Kevin Backman. I am a senior systems engineer for the systems infrastructure and engineering team for Major League Baseball.
00:12:41
Speaker
Very cool, very cool. Yeah, we're here to talk about Major's League Baseball and all things technology and Kubernetes. And I think that's a really exciting topic. I myself, I played, you missed it a little bit before the show, Kevin, but we were talking about baseball, but I played Division I in college. So it's near and dear to my heart, you know, cross between technology and baseball. I'm, you know, long way from playing any sort of organized baseball anymore. I'd probably hurt myself, but it's a fun little episode today.
00:13:11
Speaker
Ryan played Division I baseball. I just saw one baseball match in my life. But I do have a Red Sox cap. If we want to support the local teams, I can bring that on. Good stuff.
00:13:27
Speaker
So again, I know Kevin and Mike, thank you so much, right? As Ryan said, for joining this episode, we want to start with like that high level overview. Like what does the technology stack at Major League Baseball looks like? And since you guys are on Kubernetes Pikes, I'm assuming it has something to do with Kubernetes. Can you give us give us and our listeners an overview of what that stack looks like and how you use Kubernetes today?
MLB's Tech Stack in Kubernetes
00:13:48
Speaker
Yeah, sure. I can jump in on that. I'm originally faithful. You're a good person to talk to? Yeah, yeah. Some people say I don't agree with him. But yes, no. So we're pretty much heavily invested in Kubernetes. We've gone all in on Kubernetes. The vast majority of our applications these days are running on some form of Kubernetes.
00:14:15
Speaker
I don't like if you pay attention to the news at all. You've seen MLB and Google have a very close relationship in terms of sponsorship and all of that. And so with that being said, we're heavily invested with GCP.
00:14:38
Speaker
all GCP Kubernetes offerings, which in turn led to us rolling into Anthos or the ballparks, but I won't get ahead of it.
00:14:49
Speaker
Yeah, yeah, no worries there. Now, I know, you know, we're going to dive into a little bit about how you're using Anthos and how you got there. But I do want to start with a little foundation about like, what are the applications doing? I mean, we have to put a little context to the story here. You know, where were these applications already containerized? Where, you know, what are they actually gathering? How are they, you know, connected with the actual sport of baseball?
00:15:15
Speaker
So the apps themselves have been containerized for a while now, but obviously they did not start that way. At one point, we're still using the traditional VMware bare metal stack.
00:15:34
Speaker
About, I'd say maybe five years now. I started moving to Kubernetes. Initially, I think it was a lot of AWS Kubernetes and then we started branching out and exploring other options. But to answer the question in terms of what we run, basically everything you see in terms of stat cost is based on Kubernetes. Like that's all all GCP and
00:16:04
Speaker
Anthos Kubernetes. If I'm going into Fenway Stadium, that's what I see running there at each of these locations. It's just Kubernetes clusters on the edge, if I can say that.
00:16:19
Speaker
Correct. Yes, we have edge deployments for Kubernetes clusters talking with GCP as well. Gotcha. OK, nice. That's exciting. Thank you for making it real in terms of what apps are actually being hosted on those infrastructure stacks. We usually tend to focus a lot on the infrastructure piece, but then making it real actually helps not just us, our listeners as well.
00:16:41
Speaker
Yeah. I mean, if you're watching a game on TV and you look in the corner of the screen and you see the score, that's Kubernetes. Oh, okay. Good to know. Do we have something like the AWS stat that keeps coming up during NFL games? Because I watch a lot of those and it's always annoying. But you haven't noticed it, which should tell you the answer, right?
00:17:08
Speaker
That's cool. I'm curious though, in terms of deployment at the stadium, is the infrastructure that's running this actually physically in stadium property or nearby? Or how does that actually work? It's physically in the stadium. We have little mini data centers in each stadium.
00:17:31
Speaker
So I guess to go with that, what about the Hawkeye stuff? OK, because that's the stuff that is beyond cool, right? Are those also running on a different server stack? Or is that running? How does that work? Because that's the lasers and the high speed cameras that actually do all the cool tracking. So that player tracking stack,
00:17:55
Speaker
Hawkeye does a lot of their own work and we provide the space for them to do it. And then their technology stack that they're running will communicate with our on-prem Kubernetes stack to then move the data to MLB.
00:18:14
Speaker
Makes sense, makes sense. All right, so you mentioned you had been using VMware in the past, and this was obviously a part of a new
MLB's Transition to Kubernetes and Anthos
00:18:23
Speaker
rebuild. When did it complete? Actually, like what year is it? Last year? No, so it actually completed. The initial rollout completed in 2020. Okay, right in the middle of everything. That made it probably nice and easy for you. Sure, let's say it was easy.
00:18:43
Speaker
Yeah, a strong sarcasm there, of course. So, you know, there's a lot of reasons, I think, that, you know, we talk to various guests on why they make large changes to infrastructure, large, you know, moves from cloud to on-prem or whatever it may be. I'm curious, what was sort of the catalyst that drove MLB to kind of do this and kind of work with, you know, Medify and everything like that? So we, like the whole initial
00:19:12
Speaker
you know, the bonus of this was we needed to future-proof our technology stack. Our technology stack was a traditional VM running on bare metal, you know, required a lot of resources. It didn't have
00:19:29
Speaker
the redundancy we needed, because vSphere is stupid expensive. And you're trying to now run 30 data centers with this price. It gets a little eye-watering, the cost.
00:19:46
Speaker
And so like we were looking at something nimble, agile, and you know, uh, lightweight that we could just roll out. Yes. Cheap. And then just, you know, something that we could roll out manage easily that, you know, it doesn't take a lot of engineering hours to maintain as young.
00:20:08
Speaker
A lot of vSphere stuff does. So we were playing around with Rancher in some on-prem clusters, just like proof of concept. And we liked the idea of Kubernetes. At this point, we had just signed a deal with Google. And we're starting to roll out a lot of GCP stuff. We saw the benefits of running everything in GCP Kubernetes. And so then Google came to us and said, hey, by the way,
00:20:36
Speaker
see you guys, you know, you like Kubernetes, what about this? And so we decided, OK, let's give it a shot. And it worked out very well for us so far. Yeah, I'd love to hear your perspective on that, Mike, as well, just sort of like, is that a common thing you see, you know, obviously beyond MLB and kind of what they've done as well? It is, yeah, for sure. In fact, so cost drivers reducing VM presence and sprawl and the V tax and all of that. And then there's so there's a cost savings associated with that. And then, of course, there's
00:21:05
Speaker
there's just the simplicity involved in when you're working with bare metal and you know, you reduce overhead and you're speaking more directly to the application. So you get better performance and especially with the build out of edge and hybrid compute in general, you know, the whole term data center or whatever is kind of morphed into wherever the compute closet may be at this point. And yeah, so those low latency applications and certainly
00:21:34
Speaker
you know, just the desire to simplify infrastructure is spiking, I would say. You know, people kind of got, I was going to call it drunk on virtualization and cloud, and then you start realizing the bill is, you know, it hurts the bottom line. And there's simpler and more elegant ways to do things now, and because open standards have kind of caught up,
00:21:57
Speaker
At a low level with this space, it makes enabling these bare metal clouds and managing your own infrastructure a hell of a lot easier if you have the right tools. So yeah. Gotcha. So Mike, talking about the infrastructure stacks, what does the stack look like? Are they general purpose x86 servers? And then how does Metify help MLB bring everything up? And how does it actually work?
00:22:25
Speaker
Yeah, so this is very much a DIY thing, right? So the infrastructure is there. We're a software publisher. We've got Mojo Platform is the name of our platform that makes running your bare metal really easy.
00:22:40
Speaker
Kevin, they've got a couple different... Since we work with any hardware manufacturer, this is all open standards based. The DMTF is the main open standards organization that created the Redfish specification and that Redfish specification has become ubiquitous.
00:22:59
Speaker
So everyone, including the OCP, Open Compute Platform guys, Open BMC projects, Snea, Swordfish, all of the other open standards are making sure and writing their components into the Redfish specification itself. So with that, we work across any hardware platform. And Kevin has had a really unique
Managing Hardware with Mojo
00:23:23
Speaker
a suite of hardware, essentially, that had to work together that provided some cool challenges for us. But yeah, it was a blast to work with. Kevin, I'll hand it over to you for your hardware. I was going to say, I heard a little bit of Kevin, that's a little bit your fault, but we were ready to work on it.
00:23:42
Speaker
Well, you know, technology in bullpox is hard. Of course, of course. So yes, so, you know, as Mike alluded to, we have, you know, a lot of, at least, you know, in terms of overall data centers, including minor league parks and all of that stuff that we support, we have a vast array of different hardware.
00:24:06
Speaker
We have various Dell's, Super Micro's, all of that sort of stuff running. It's tough trying to get something that can talk to all of them at the same time easily. That's why we're leveraging something like Mojo. In our use case, what Mojo does is that it will bootstrap a OS onto the bare metal and
00:24:34
Speaker
that OS image is a custom image we built that has all the required packages to install our Kubernetes of choice. And then all we do is we will drop either an Ansible or Terraform job and it will just roll out the cluster. Now you say we, is that something you built or does Metafide provide?
00:24:57
Speaker
So it's a little column A, a little column B. So yeah, so like Metafy takes care of the bootstrapping and OS image, and then we have a bunch of automation tools we built around Ansible and Terraform that will then come in and take over and actually deploy the cluster itself.
00:25:15
Speaker
Got it. And using those common tools makes it so you can do it the same or similar, I'm guessing, at every park event. Yes. One of our primary designs was that each major league park is identical.
00:25:30
Speaker
Got it. Hardware, software, it doesn't matter. They are all the exact same hardware. And this happens a lot. You have an issue in one ballpark, and then you go to whoever's support is. And they're like, oh, well, is it happening everywhere? And it's like, no, it's not us. It's you. Because if it was us, it would be happening everywhere.
00:25:54
Speaker
Yeah, that's another big aspect as far as use cases go. So distributed workloads and keeping a chain of custody and source of record of what's in that box. What are the rev levels on the BIOS? What are the rev levels on the firmware? You can picture hundreds of servers, thousands of servers distributed around the country, around the world.
00:26:17
Speaker
And if you do a BIOS update and it bricks a particular software application, you have to be able to do a rollback on that BIOS, rollback on that firmware. It might only be to a few boxes specifically, a few servers specifically, so you have to be able to run a report against that.
00:26:35
Speaker
So all those things are enabled through Mojo and all that low level, chip level information of what are the biases, what are the firmwares, what rep levels are they at, all of that is dashboarded and makes life hopefully a bit easier for our customers.
00:26:51
Speaker
Yeah, while people are scared of the cloud price tag sometimes, they still want the experience, right? So it sounds like that's kind of what you're delivering. Man, that's a perfect marketing line for it. Not being afraid. You're right. No more fear of on-prem hardware. I should say your hardware, because it's never on-prem. It's always somewhere else. Well, it's on someone's prem, I guess.
00:27:16
Speaker
I love that saying about, you know, what is the cloud? It's just someone else's computer. What is the edge? It's just somebody else's computer, you know? I'll talk about the edge. What's the edge? Somebody else's computer, man. It's a little closer.
00:27:34
Speaker
Going back to the orchestration piece, you use Mojo to set up those infrastructure stacks, have the OS loaded, have all the packages. Then how do you take that next step and deploy Anthos? Do you deploy it across multiple stadiums at the same time? How does all of that work from OS to Kubernetes?
00:27:52
Speaker
Oh, so yes, we can deploy multiple parts at a time in parallel with Mojo and our Ansible and Terraformed tools. It has simplified our lives immensely because I've actually been with MLB now for 10 years. OK. And when I first started, it would take you an engineer
00:28:22
Speaker
a good six hours to spin up a single park. Now you've got 30 of those parks. Opening day is two months away, but technically a month away because a month before opening day, we start going into code, you know, freezes and heavy scrutiny. And so you have to work very hard in the off season to get everything done.
00:28:48
Speaker
Fortunately, now with the automation, we can get that done in a week.
Stadium Automation
00:28:54
Speaker
One engineer can do it instead of having an entire team of engineers split the parts up. I guess that's another cool thing worth mentioning about Mojo. We have a service catalog inside of it where you can have any number of
00:29:10
Speaker
run stacks, right? Whatever you may need in terms of runtimes and particular stacks for different clusters and pooling of the resources and all of that cool stuff that from a low-level admin perspective, your groups and developers require.
00:29:29
Speaker
So yeah, that's all within the tool and sort of makes it a one-stop shop to be able to handle all those things from OS firmware all the way up to and through the cluster deployment and then the applications themselves. Got it. All of this seems too easy, guys. I'm sure there might have been challenges, right? We can't just... There's hand waving happening, yeah.
00:29:51
Speaker
So Kevin, as you were going through the process, I think I read the blog that you published on MLB's tech site. You maybe kicked off the project in 2019. The deployment was kind of in 2020. What were some of the challenges, apart from all the people challenges that we were going through during the first year of the pandemic? I asked you. I mean, so the pandemic was a huge issue because a lot of this required hands-on
00:30:21
Speaker
hands on for our ballpark infrastructure teams that actually do the hardware installs. From our perspective, the biggest challenges is that at the time Anthos was still fairly new. We're rolling out this new technology. We've never done this before. We have to build this tooling out to help us. Otherwise, we're manually doing everything. I believe the term we commonly use around the office is hand jam. You're hand jamming everything in there.
00:30:56
Speaker
And we're doing this all remotely. Everyone is sitting on Zoom now, and we're still figuring out how to use, or was it even Zoom at the time? I can't even remember. You nearly took off, but yeah. Yeah, it was Zoom. So we're all sitting on Zoom calls trying to figure out how to work.
00:31:15
Speaker
you know, in a remote distributed way. So I think that was probably our biggest hurdle that we had to overcome. Yeah, someone's still got to show up and hook up networking, right? That was the idea.
00:31:28
Speaker
So once you move that on, then you can move on from that. Speaking of sort of remote work, you know, you chose Anthos, sound like you tried a few others. Is there sort of a management aspect that you take on for these ballparks, both from the aspect of Kubernetes itself? And I sort of get an idea of what it looks like with Metify, but maybe we can answer the Kubernetes part and then maybe the Metify part.
00:31:55
Speaker
The one really nice thing about Anthos is that it gives us a single pane of gloss when managing Kubernetes. We manage our Anthos Kubernetes clusters much like we manage our GCP Kubernetes clusters.
00:32:12
Speaker
We have our CICD pipelines just plugged right into Anthos. It's just another GCP cluster. DevOps would manage any other cluster. We use tools like Argo CD for continuous application deployment.
00:32:40
Speaker
we enable the developer team to actually manage their apps directly. We give them the freedom and the tools with the guideline, like little bumpers here and there to help them out. But for the most part, we just let them do their jobs. That is our primary goal, is to let the devs do their jobs and not be a blocker to them.
00:33:08
Speaker
That almost sounds like why organizations like yours need platform engineering, right? You want to make sure that developers are the most productive they can be, while at the same time providing those guardrails to make sure that they don't paint outside the lines, I guess, and get you in trouble. I break that production. Never happened right now.
00:33:33
Speaker
So talking about developers, right? So you said you use ROCD, you have CI CD pipelines in place. So I'm assuming the dev work happens in the GCP like cloud clusters. And then when it comes time to actually do the rollout or push applications, the production actually lives in these stadiums, right?
Handling Game Data
00:33:53
Speaker
Yes. We have virtual bull parks, which the devs will use to develop, obviously test their apps and everything. And then once everything looks good, they will roll out the to production in the bull parks now. With Argo CD, everything is versioned. If something goes wrong, we can easily roll it back.
00:34:18
Speaker
With all the deployments we have in the bull parks, we can do blue-green deployments. If something goes wrong, the end user doesn't normally see it.
00:34:33
Speaker
Yeah, that makes sense. And I think one of the things I was wondering as we were going through this is, I think there was a huge leg up here with things already being containerized. You could plug in, I think I read somewhere, Argo CD to this deployment. Was it truly plug and play? Or were there any lessons learned in this process of getting the applications onto this new infrastructure?
00:34:59
Speaker
If lessons learned were learned before we implemented Anthos, but through our Rancher, playing around with Rancher, like we did, the bullparks aren't 100% a GCP environment. There is finite resources. You can't just auto scale.
00:35:23
Speaker
to me, because if you auto scale, you suddenly start getting memory alerts and CPU alerts for machines and you're going, oh wait, what's going on here? Right, right. You never know, my team will be ready with another server. I know they don't manage hardware, but they might be ready to go with the motor platform. Yeah. I will say that it does make it easier to just be able to, as long as the server has networking, we can just, you know,
00:35:53
Speaker
click a button and merger will kind of take care of the initial part and then we can take care of the rest.
00:35:58
Speaker
So from the flip side, Mike, is there something that was learned on your side of things supporting this kind of use case? Yeah. I mean, so they just had so many cool peripherals and the apps themselves had demands that really provided all sorts of learnings for us. So on the storage side, they had some cool challenges that we wanted to integrate into Mojo. So we did that for their RAID controllers.
00:36:26
Speaker
So they can switch personalities of their, you know, storage devices, check on the health of them, those types of things.
00:36:37
Speaker
There's hardware acceleration as well. They've got Nvidia GPUs in there. I think the 3080s. They're not 3080s. I think that I'd have to look, but they're a consumer version of enterprise GPUs. So just all of the different elements that
00:36:58
Speaker
MLB brings to bear with these kind of rich stacks that have to handle these edge-based low latency, unbelievably high-performance workloads. And then they push up 7.2 terabytes of data each game for all the ravenous fans who can't get enough of their stats in terms of every possible statistic you can imagine. What's the revolutions per minute on a baseball leaving Kyle Crick's hand at 98 miles an hour? It must have. I need to know this information. It's amazing.
00:37:27
Speaker
And I have to tell him, Kevin, so after Aaron Judge hit his 62nd home run, we were on with Kevin, you know, we're on Slack all the time. And I can't even recall what we were doing, but he's just like, you know, hey, guys, you got to check this out. And he showed us Aaron Judge's home run.
00:37:45
Speaker
in 3D as it was drawn by Statcast, as it was drawn by the lasers and high-speed cameras that track that. Unbelievable. Just the coolest thing to see, you know, the exact trajectory of the ball, the total distance flow, and the speed of the ball coming off of that. I mean, it's just beyond cool. The amount of tech that they have in there,
00:38:05
Speaker
It's just exhaustive and cool. So yeah, all the challenges associated with that gave us ample hardware to learn how to work with at a very low level. So it was just a lot of fun. You mentioned the amount of data sort of after every game.
00:38:21
Speaker
Bob and I both have a background in data storage and things like that. And we often talk about this concept. So since you brought it up, how are you using the different resources in Kubernetes, Kevin, to provide storage? Or are you doing something completely different to provide storage to these applications and things like that?
00:38:43
Speaker
So the applications on-prem are on the edge in our ballparks. By the design, they're very ephemeral. If something goes wrong, it's fine. The pod gets redeployed and just keeps going. The storage all happens on the back end, back at MLB headquarters.
00:39:10
Speaker
We've been playing around with using Ceph or a similar shared storage solution for just our on-prem Kubernetes in case of just a blip in network or something like if.
00:39:27
Speaker
if a backhoe decides that it's going to munch on a fiber cable. The great North American backhoe and its fiber munching. Solutions like that we're looking into. We've played around with it and it's shown some promise, but I think that is something we will be definitely moving forward with in the future.
00:39:52
Speaker
So today is it mostly streaming to object or like what? So the streaming like the actual video feed comes straight from our video trucks in the ballpark. The stat cast, all the data, all the player tracking, all of that goes through our Kubernetes
00:40:11
Speaker
stack into GCP. And that is everything. I mean, we've actually had to dial down the logging on the apps, because our apps were making Stackdriver fall over. Oh, nice. OK. There's just a few things going through there, yeah.
00:40:35
Speaker
Yeah, I mean, and that's just a single ballpark would be making Stackdriver fall over. So we've had to tweak our setup a little bit to help Stackdriver along. Got it. Because it is a fire hose of data. Yeah, I imagine. So that's the applications themselves communicating with sort of their parent application, say, in GCP. OK. Yeah, like for example, we're using AMQ.
00:41:02
Speaker
to push a lot of messages and stuff containing player tracking to our AMQ in GCP. Got it.
00:41:13
Speaker
You know, all this talk about sort of everything that you're gathering and all the stats and making Stack Drivers fall over, things like that, you know, I'm curious sort of what's next,
Future of MLB's Tech Stack
00:41:25
Speaker
right? You've gone to this point. Are there next stages for sort of your relationship with Metify, what you kind of have envisioned for this stack? You know, my mind immediately goes to like watching a game in full VR or something like that, but you know.
00:41:41
Speaker
Yeah, I like that. I mean, not to, not to, you know.
00:41:47
Speaker
let anything out but like that is something we are seriously looking at. I would totally do that. MLB and VR would be a huge fan. I'll watch the space closely then it sounds like. You want me to do a tie up with DoorDash or something like that to deliver those hot dogs as well? Like you can't just... I need the full, your peanut hot dogs.
00:42:13
Speaker
There's a lot of fan interaction we want to leverage in the ballpark, especially the way technology is going. It's going to allow us to make baseball way more interactive, way more fun for your casual fan. It's already fun for hardcore fans.
00:42:36
Speaker
Yeah, so I think, you know, in terms of Kuonettis, I think the future is we want to make it smaller, more nimble.
00:42:45
Speaker
If you bring state on, use that safe route combination that you just mentioned, right? Have some state, have some permanence, and also with all these special events we're doing, like the series in Mexico and the London series and all of that. So we want to be able to just ship our stack.
00:43:11
Speaker
to these parks, set it down, turn it on, and away you go. You have the full suite of MLB tools. Oh, that would be so cool. I remember when I went to Ignite and Microsoft Ignite the conference in 2019, they had a Humvee that they were using with the US Army to have these robust edge deployments of their hardware and running something on top. And then that automatically connects to Azure. I was like, wow, that would be so cool to have in a new MLB stadium.
00:43:41
Speaker
Yes. Yeah. Like we, we, right now we have, um, it's sort of, we, we'd like to call them fly packs that we wanted to build. Yeah. And the fly pack is what it is. It's going to sit on a crate. It's going to get on a plane and it's going to go to London. It's going to go to Mexico. Um, or, you know, like wherever we want to and, uh, host events, like maybe even the world baseball classic. Oh, Debbie, I was going to ask about that. Yeah. That makes sense.
00:44:06
Speaker
And what about you, Mike, you know, in terms of sort of supporting bare metal Kubernetes deployment, since this is kind of the use case, what's next for you? You know, so containerized applications, it's like over 50% of workloads now are on bare metal. So containers, it's the preferred application underlying architecture now for a number of reasons.
00:44:29
Speaker
We see things continuing to trend in that direction, especially as EDGE continues to grow. We do work with the military and that Humvee reference was very interesting because we have some work exactly in that area with another partner of ours. It's based out of Virginia Beach.
00:44:50
Speaker
Yeah. And then with, you know, with MLB, it's just a blast to work with them. And we're now the minor league ballparks are beginning to roll on, which is another interesting challenge because each minor league park is its its own little creature as well. But yeah, I mean, I just I'd love baseball. So this is just it's been a ton of fun working with these guys. And as far as the future goes, you know, all the industries and the growth of the edge and the growth of
00:45:16
Speaker
bare metal and people wanting to kind of take back and cloud this cloud repatriation movement that is sort of happening in earnest now all play into our solution and our value proposition really well.
00:45:33
Speaker
Absolutely. I know Bob and I have definitely spoken to a few guests on that topic specifically. It's definitely something that people are thinking about, especially in today's economic climate too.
Robots in Baseball?
00:45:44
Speaker
We're about a half an hour, so I want to get to our little chat GBT question section. You don't have to answer, but if it sparks something in your mind, then just go for it.
00:45:55
Speaker
So we asked chat GPT to come up with a question about technology and baseball and it came up with with the increasing use of technology in Major League Baseball. Do you think it's possible for a robot to become a professional baseball player and break all the records? Oh, wow. Oh, you know, I kind of love the don't answer, but I didn't know if you had any thoughts on that at all.
00:46:17
Speaker
I don't think it will be possible for a robot to become a baseball player. I think it will be possible for a robot to become an umpire. I don't know how I feel about that. Becoming an umpire. We have robot umpires in the minor leagues.
00:46:37
Speaker
That will save a lot of dirt being kicked on the field, a lot of infections. All the hats being thrown out. Yeah, injuries. I mean, that's a big thing. There's nostalgia there for me, though. It's like the umpire is part of it. You know, it's a whole debate. That's a whole nother podcast. This is true. I think that a robot playing in Major League Baseball, well, I think as a demo project for Boston Dynamics, the guys...
00:47:05
Speaker
I mean, they love this. I mean, I think we can put this challenge out to them to have a major league pitcher go up against Atlas, right? It's going to take a little time to teach Atlas to swing a bat. But we will absolutely deliver those bare metal servers for free for them if they want to take on that AI challenge and see if they can hit a ball coming from
00:47:30
Speaker
a major league baseball pitcher, that would be beyond cool. Just no curveballs probably. That would be an insane move. Maybe that's the next step, yeah. Yeah. Especially with all the advances in AI and robotics overall, right? You never know. I mean, this would be a really- You never know. Well, I mean, well, chat GBT itself thinks basically no way I could play baseballs was its answer.
00:47:50
Speaker
Um, uh, even though, you know, they're very advanced, but it's, it kind of, it, the answer it gave was kind of interesting. It says that because of the emotional intelligence and strategic thinking and the ability to kind of adapt to the, the ever-changing game situation is sort of where it, um, kind of went to, but then it also says, coming back to what Kevin says about fans, right? Is baseball is a sport with a strong culture and historical significance. Fans would never let a robot play baseball.
00:48:31
Speaker
Maybe that's the next evolution of Major League Baseball, Major League Robotic Group.
00:48:38
Speaker
Anyway, Kevin, Mike, it's been a pleasure having you here on Kubernetes Bites. I think that was very insightful, and it was great to have such a close to home sort of topic and use case. So yeah, it was our pleasure. Cool. All right. Thank you so much for having me. Appreciate it. Have a good one.
00:48:54
Speaker
Oh, man, that was a lot of fun with Kevin and Mike there. I think my love for the game of baseball mixed with technology made this, I think, just super fun. But, you know, let's go into takeaways you want to talk about. Yeah, I think my primary takeaway was we need to go and watch a game together this year in Boston. I know Red Sox games have already started. We are in season. So let's do that. That's primary takeaway. Although I do get like the basic references, like the first base, second base, the home runs and things like that.
00:49:24
Speaker
I was not aware of the intricacies of having an unhittable curveball and things like that. So I can definitely get to learn. Yeah, there's some kind of embedded tribal knowledge there. Yeah, absolutely. But overall, I think I love the fact that Kevin was able to share his transformation journey. Like we have always seen customers aiming to do something like this, right? Move on from a traditional legacy, maybe virtualization based infrastructure stacks over to Kubernetes.
00:49:51
Speaker
And one of the main reasons he listed us was cost, right? Having to run VMware vSphere on top of his bare metal nodes and then run his applications which might have already been containerized and still having to pay for those licenses and not getting all the features and the resource utilization that he needs.
00:50:09
Speaker
I think moving to containers, moving to something like Anthos that can still run on bare metals, provide that hybrid cloud solution for MLB. That was great. I'm happy that this is one of those public references where people have successfully moved from VMware to Kubernetes because
00:50:28
Speaker
Believe me, there are so many customers, so many enterprises that are still using VMware. So episodes like this or success stories like this definitely gives those users out there some confidence that, okay, it is possible and you can trust Kubernetes to be that infrastructure layer for your production environments and it won't go down. It's not just another open source project that you can't trust. There are enterprises or organizations that are running it in production. That was my biggest takeaway.
00:50:55
Speaker
Absolutely. And I think, you know, even at this last KubeCon, cost is still a huge theme, even though Kevin started this in 2019. I think cost as a theme is definitely something people are thinking a lot about right now. So to hear this kind of thing, to go to bare metal and to be able to manage the complexity, right? I was thinking initially the term, well, isn't that more complex where, you know, they kind of found something that worked really good for them and kind of talked about the simplicity of it.
00:51:21
Speaker
along with sort of the gains of performance and things like that. Which leads me to sort of where I was kind of, you know, my takeaway was sort of the future proofing that Kevin and Mike talked about, which is more or less like building an infrastructure stack that could grow with MLB, right?
00:51:39
Speaker
It could really take on not only the 30-some-odd parks in the US that they were aiming to and be able to deliver those quickly and roll out updates, but also take on things like the minor league stadiums.
00:51:54
Speaker
to be able to future-proof it in a way that they can go ahead and take that and ship it somewhere else and run it the same way, whether that's my leagues or in the special city games in London and things like that. So that's really cool to me. I think building that stack in a way that can grow with you is one of the reasons I've always looked at Kubernetes as a really interesting technology in general.
00:52:19
Speaker
Yeah, I think that's the takeaways from us. Hopefully you got as a listener some of that as well as maybe pick something else out of it like, you know, kind of robot actually hit a curve ball or not and we'll see.
00:52:37
Speaker
doctors about that, I guess. We also need to remind our listeners, right? Like, thank you for supporting the podcast. But like, we still need to grow our audience like it's already up there, but we need to take it even higher. So if you can just share, like I know Memorial Day weekend is coming along, I don't know before with your workmates or with somebody you know, who's still interested in this Kubernetes technology, share this podcast, right? Like just everybody, if they share it with one more people that would really help us out. So I just wanted to do a shameless plug before we wrap up this MLB episode.
00:53:06
Speaker
Yes, absolutely. If you share it with one other person, that goes a long way and really does. So we appreciate that. And again, for your listenership, always means the most to us. And with that, that brings us to the end of today's episode. I'm Ryan. I'm Bob. And thanks for joining another episode of Kubernetes Bites. Thank you for listening to the Kubernetes Bites podcast.