Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
InfluxDB on Kubernetes
341 Plays2 years ago
In this episode, Ryan and Bhavin interview Rick Spencer, VP of Products at InfluxData, previously the VP of Platforms for InfluxData. The discussion focuses on InfluxData, creator of InfluxDB, and how they help customers looking for a Time-Series database solution. Rick talks about the interesting IoT and Edge computing use cases, and how getting that real-time sensor information can be transformational for customers. In the second half of the discussion, we focus on how InfluxDB cloud runs across three major cloud providers, running on top of Kubernetes itself. Rick focuses on their Kubernetes adoption journey and talks about their architecture today. We also talk about how users can leverage InfluxDB for monitoring their large-scale Kubernetes deployments too!
Show Links:
- Datastax raises a private equity round of $115M valuring the company at $1.6B to focus more on Astra DB and Astra streaming - https://techcrunch.com/2022/06/15/datastax-proves-its-still-possible-to-raise-nine-figures-at-higher-valuation-in-2022/
- Platform9 raises $26M bringing the total money raised across all the rounds to $100M - https://techcrunch.com/2022/06/14/platform9-raises-26m-to-help-manage-distributed-cloud-clusters/
- Finout raised $14M Series A funding to help organizations break down the cost of each Kubernetes namespace, each folder in the S3 bucket, each Snowflake Query, etc. - https://www.geektime.com/finout-wants-to-reduce-your-cloud-costs-with-a-mega-bill-and-14m-in-a-round-funding/
- Mercedez Benz runs 900 K8s clusters - https://www.infoworld.com/article/3664052/why-mercedes-benz-runs-on-900-kubernetes-clusters.html
- Weave Policy Library for HIPAA compliance - https://www.weave.works/blog/weave-policy-library-introducing-hipaa-policies
- Kanister Webinar - https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-on-demand-webinar-kanister-application-level-data-protection-on-kubernetes
- Cloud-native storage - Kubernetes Podcast - https://kubernetespodcast.com/episode/182-cloud-native-storage/
- 2021 K8s Annual Report - https://www.cncf.io/reports/kubernetes-annual-report-2021/
Enhancements#3337: KEP-3333 Retroactive default StorageClass assignment
Recommended
Transcript
Introduction to Kubernetes Bites
00:00:03
Speaker
You are listening to Kubernetes Bites, a podcast bringing you the latest from the world of cloud native data management. My name is Ryan Walner and I'm joined by Bob and Shaw coming to you from Boston, Massachusetts.
Meet the Hosts
00:00:14
Speaker
We'll be sharing our thoughts on recent cloud native news and talking to industry experts about their experiences and challenges managing the wealth of data in today's cloud native ecosystem.
00:00:31
Speaker
Good morning, good afternoon, and good evening wherever you are. We're coming to you from Boston, Massachusetts.
Today's Date
00:00:37
Speaker
Today is June 22, 2022. I hope everyone is doing well and staying safe. So let's dive into it. Bobbin, sorry I had to miss the last episode. We crushed it. Thanks for the shout out at the beginning of it. I did listen to it.
00:00:59
Speaker
I don't know. I've been busy. This was the first weekend where we had to move our lawn and that's the life that I've done that. So it wasn't like, again, it was the first time. So it was painful because we had weeds all over the lawn and the edges had grown all the way up.
00:01:19
Speaker
So it took more time. I'm still waiting for the phase where people say that mowing is kind of therapeutic. It didn't feel like that this time around. So maybe in a few months I'll get into the point. Only to some people does it feel that way, Baba. You might never get there. Who knows? Yeah.
00:01:41
Speaker
What have you been up to? Yeah, you know, we were down in New York. We had a family member pass away. That's why I wasn't here. So, you know, unfortunately, but at the same time, you do get to see a lot of family during those times that we're very thankful for. So that's where I was last few weeks. This week, I'm actually traveling. So I'm not even I said coming from to you from Boston, but I'm in New York.
00:02:03
Speaker
And I'm headed down to West Virginia to do a bit of hiking with a few of my friends. So really excited about the week forward. Maybe hit a couple of barbecue spots. We'll see what else we can do down there. But that's what I'm up to. Oh, nice. That sounds awesome. Yeah, no, I'm going to a national park, Bob. And this is, I feel like I'm falling in your footsteps here.
00:02:27
Speaker
I'll look for your reviews on how the new national park is. Yeah, we're just doing a bit that's sort of the northern tip of the Blue Ridge Parkway, which is just only a very small part of it. But I've wanted to do the whole thing. I have to go back, I guess.
Webinar Announcement: Canister and Data Protection
00:02:42
Speaker
Great, so before we get into today's topic on InfluxDB, really exciting stuff and excited to have Rick on, let's talk about the news a little bit. So I have a few things here. The first one I wanted to bring up was there's a really good webinar coming up, cncf.io, or sorry, community.cncf.io on Canister, which is part of Castin's portfolio of products. Canister is actually the piece that
00:03:10
Speaker
You can dig into the specifics around how to do application level data protection. I know we've talked about this many times on this show before, but Canister is actually an interesting way to dive into it. If you're new to it, understanding what does application level really mean, definitely go check that out. Really good webinar. If I remember correctly, Canister is an open source project, right? Even though- It is, I believe. Yeah, unless that's changed. I believe it is. There's also a really
00:03:39
Speaker
great podcast episode on the Google Kubernetes podcast, which is all about cloud native storage.
Cloud Native Storage Discussion
00:03:48
Speaker
One of the previous employees from previously storage OS is on that show and talks all about cloud native storage. So anytime we hear other podcasts talking about cloud native storage, we always want to recommend it. Plus that podcast is great.
00:04:05
Speaker
Definitely go check that out. We'll put that in the show link. But if you're looking for it, it is Kubernetes Podcast.com. Pretty straightforward. You'll find it there. I still envy that whenever you look up Kubernetes Podcasts, they rank higher than us. We'll get there. They have the SEO down pat there.
00:04:24
Speaker
I listened to that episode. It was good. The guy from ONDAT, I did refer to a couple of white papers that the open source community has published around disaster recovery for community storage or for community stateful applications. That's a good resource. Really, really interesting podcast for sure.
KEPP 3333: Default Storage Class Solutions
00:04:44
Speaker
The other bit was, I forgot if you mentioned this last week, but I did want to throw, but did you mention the annual report last week? I highlighted different things. I'm looking at the notes right now and you have something that I didn't even pay attention to. I won't repeat the annual report, but lots of good information in the annual report. One of which I went down a rabbit hole and I found out more up to date from this week is
00:05:09
Speaker
There's a Kepp that's in progress, and it's the enhancements, 3,337, but it's Kepp 3333. That's easy to remember. But it's called Retroactive Default Storage Class Assignment. I wanted to spend just two seconds on it because I thought, as a storage and Kubernetes focused podcast,
00:05:30
Speaker
It was actually an interesting problem that they're solving with this cap, which is we all understand how storage classes, hopefully at this point, work in Kubernetes, and you have to choose one in order to provision storage, whether that's statically or dynamically.
00:05:46
Speaker
and you can set default versions of that. So if you're just provisioning a volume and you don't specify, you're going to get a default storage class provision for you. Now, there's a little hole as they describe it in this workflow, which is if you don't set a default storage class and you try to provision storage,
00:06:03
Speaker
that piece of storage will never be provisioned. It'll just sit there pending until you go ahead and set a default storage class and kick the tires on it. That's the key here with this KEPP and this change is that even if you set a default storage class,
00:06:20
Speaker
after your piece of storage is pending, you have to sort of delete it, the object, and let it get recreated so that it picks up that default storage class that you've set since not having one. This fixes that in the sense that if you have any that don't have an assignment and would have used a default storage class when you set it, it'll automatically kick those tires for you, setting off the whole dynamic provisioning of
00:06:48
Speaker
of storage from that default storage class. So kind of an interesting problem that you solved. I don't know if you've run into it before. I know I actually, I haven't run into this before. I think most of the time I'm working with storage providers. I'm very clear on which storage class I'm choosing from, but I know a large use case is choosing that default one. So we'll have a link to that kept in there.
00:07:09
Speaker
And this is a perfect scenario where we didn't even know this was an issue until we have a solution for it. So I think that's the perfect way to find out about some tweaks that are missing from the open source community. Absolutely.
DataStacks Funding News
00:07:23
Speaker
Okay, I think for me, right, I did have a few funding rounds to highlight and then a couple of news articles as well. So, to start with the funding rounds, data stacks, you might know them as the Cassandra Company, the Astra DB. I think we had Patrick on one of the podcasts to talk about Kate Sandra.
00:07:40
Speaker
DataStacks recently raised a private equity round of $115 million, which valued the company at $1.6 billion. And I think from a couple of articles that I read, they are looking to focus more on that AstraDB and Astra streaming parts of things, which are basically providing Cassandra as a service, but still
00:08:00
Speaker
using communities under the covers to host that managed service. So we see this as a growing trend as people are building more and more modern applications and consuming these modern distributed databases.
00:08:14
Speaker
The second funding round was from Platform 9. They raised $26 million. They didn't share details around what round was it or what the new valuation is. But just looking through crunch space, I found that this $26 million funding round brings their total money raised to $100 million.
00:08:33
Speaker
And I think they want to spend this money focusing on go-to-market and some R&D to seek out those larger scale enterprise deployments. I think they are targeting more larger enterprises now with this new funding. Makes sense. Yeah, we'll have links for both of those. Then I think the third funding round, which is more from a startup, not from well-known names like Datastax or Platform9, an Israeli startup called FinOut,
00:08:58
Speaker
There is like a 14 million dollars series a funding round again, no idea on what their valuation is. But again, another startup that we see in the ecosystem that's helping customers do cost management for your cloud deployments are for their communities deployment, right?
Mercedes Benz's Kubernetes Deployment
00:09:15
Speaker
So looking at the website, it looks like if you have been out running,
00:09:20
Speaker
you can point them to your communities cluster and you can get the cost of running individual parts or running an application inside a specific namespace. They can also help you break down your cost into how much does each folder in an S3 bucket or each snowflake query cost you in your overall AWS bill for example.
00:09:39
Speaker
All of this, I think, goes more towards the trend that people are running these and now they want to make sure that they have optimized their infrastructure. So FinOut is another startup that's trying to go after that space.
00:09:51
Speaker
A couple of additional articles. I know we covered a couple of use cases last time, companies like Coinbase and Airbnb that are using Kubernetes now. Mercedes Benz, they had an article out where I think end of 2021, they shared some numbers. They are running 900 Kubernetes clusters across four different data centers on-prem. So that's a huge number.
00:10:17
Speaker
The thing that caught my eye was they are using cluster API and running these clusters on OpenStack. So, probably have to, unless they wanted just to give an arm and a leg for the 900 clusters.
00:10:34
Speaker
That's true. I think right now, just reading through the article, they do have five different platform teams with a couple focused on just providing that Kubernetes as a service platform for their engineers. Then there are the three remaining platform teams focused on things like database as a service or providing that logging and monitoring as a service and security and
00:10:56
Speaker
including runtime registry and image scanning. So they do have a huge organization to support those 900 clusters. Interesting thing was they are looking at AKS and EKS or even running Kubernetes using the same orchestration on EC2 instances. So they are looking to offload some of these clusters or maybe the new ones to the cloud so that they don't have to scale the platform team from where they already are.
00:11:24
Speaker
It's an interesting article talking about an actual use case and how they are using it. So we link that in the show notes as well.
Vee HIPAA Policy Integration
00:11:31
Speaker
And then the last thing that I wanted to highlight was, I think VWorks had like a GitOps con last week. And one of the interesting announcements that came out of it was,
00:11:41
Speaker
Through Veev policy, which is what customers can use to set predefined policies and it will test against your clusters, they can now also help you set HIPAA policies. So healthcare customers that are dealing with patient information, which is supposed to be confidential in PII, they can now have these HIPAA policies integrated into Veev policy and have these predefined rules. So they can cover not just a single cluster, but using this policy as a service offering, they can cover
00:12:10
Speaker
multiple clusters and multiple phases. So if they can monitor a cluster that's already running, an application that's already running, so run time, running it in audit mode, deploy time, so they can have these policy integrations into the admission controller. So if things don't look right, or if things aren't complying, they can basically
00:12:28
Speaker
deny that application from being deployed. And then during development as well, so whenever they are committing something to the repo, they can check against these policies as well. So helping with the shift left methodology. So I think you
00:12:43
Speaker
Using this, it just felt interesting that now these compliance standards are getting integrated into your CI CD or GitOps pipelines where you can just make sure, you can just rest assured that these tools will take care of making sure you're compliant. And at the end of the day, you can just generate and download some reports to validate that you were compliant against these standards. So something that's interesting, again, we'll have it in the show notes. We want to learn more about it.
00:13:08
Speaker
Right. Absolutely. Lots of good news going on. It seems like the end of June, things are really picking up. Maybe. Maybe that's just me.
Introduction to InfluxDB
00:13:17
Speaker
Cool. So, you know, let's dive into our topic. We have a really great guest on today. Today's topic is going to be all about InfluxDB, sort of time series database. We're going to be talking about what is time series.
00:13:30
Speaker
how InfluxDB runs on Kubernetes. And our guest is the VP of products, Rick Spencer from Influx Data. So he's at a previous role, the VP of Influx Data Platform, where he focused on sort of cloud native delivery, CICD, high availability, multi-cloud, multi-region deployments, and all things scale. So I'm sure we'll have lots of interesting questions for Rick. And without further ado, let's get him on the show.
00:14:00
Speaker
Welcome to the show, Rick. It's great to have you on Kubernetes Bites. Let's just jump into a little bit about yourself and what you do. Sure. So first of all, thanks so much for having me. I'm really looking forward to this discussion. I always love to talk about Kubernetes. Geek out on it. Absolutely. One way or another. So I'm currently the VP of product at Influx Data.
00:14:30
Speaker
But these are fresh wounds. I was most recently the VP of engineering for the platform team. And during the two and a half, almost three years that have been in ad influx, we've gone from running one production cluster to running probably over 15. And, you know, we have a.
00:14:54
Speaker
huge Kubernetes bill. So we really run Kubernetes at scale in a couple of ways. We run really big clusters. And we also run on all three clouds, the big three clouds, I should say. And we run in multiple regions of those clouds. And we do CI, CD to them, like multiple times a day on a typical workday. So yeah, it's where we're.
00:15:18
Speaker
So that's what you're up to now. Yeah, absolutely. Well, you mentioned InfluxDB and probably the good question to answer now is what is InfluxDB in your own words, I guess. Right. So we are a time series platform. Of course, we have a time series database at the core. What is time series? Like the most simple answer I can give you is that's like your time stamped data.
00:15:46
Speaker
And we are able to ingest, process, and support queries of that data at substantial scale. So for instance, the data could be metrics from your Kubernetes cluster or from the nodes that those clusters are running on. A lot of our customers are in the IoT space. They're collecting sensor data. We support nanosecond resolution in that data.
00:16:16
Speaker
And I can go on about the different ways that you can get InfluxDB. But let's just say the probably most interesting part of the Kubernetes discussion is that we have something called InfluxDB Cloud. It's a fully managed service that we manage on your behalf. And that runs on Kubernetes. So it's a very stateful database run on Kubernetes. All three of the top clouds in multiple regions.
00:16:46
Speaker
I mean, that's perfect because that was one of my questions when you answered the first question, right? Like that you run on these different cloud platforms. I was like, is it possible that you're running the InflexDB cloud on Kubernetes? Thank you for asking that. That was the fact that you run them on all major clouds. Is that just for customer choice or is that your choice for flexibility? Curious around that.
InfluxDB's Multi-Cloud Strategy
00:17:11
Speaker
Yeah, so there's a saying called data has gravity.
00:17:15
Speaker
So that's one of the main elements of it, of the reason that we do that is if you're collecting terabytes of data, you probably don't want to egress that out of the cloud provider, transferred across the world to some other cloud provider. Like if let's say you have a SAS product, that's like your HR system.
00:17:42
Speaker
and you're exchanging kilobytes of data and well, if they're using maybe megabytes of HTML or whatever, but you don't really care if like that is in Europe somewhere and you're in California and it's just like such a comparatively small amount of data. You don't care so much about the latency. If you're writing and reading
00:18:09
Speaker
data at the scale that our customers do, they want it in a region that's close to where that data is being produced so that they can keep the latency low and in some cases just reduce their overall cost. So that's why we provide so many options for where to access the SaaS service.
00:18:30
Speaker
Okay. Gotcha. So I think next question that I had was like talking about all the different use cases. I know you mentioned IOT as being one of our IOT or edge computing as one of those most popular use cases that InfluxDB has. So how does that architecture look like? How does it work with a SAS service and then data being generated at those remote locations? Sure. Like, let me, let me break that down for you like real quick. So, um,
00:19:00
Speaker
let's say you're an IoT vendor and that you're writing a custom IoT application. I'll pick something that one of our customers does, like you're monitoring some specific kind of industrial equipment, right? Let's say a big machine, like a fan or something like that. And that's what you specialize in. So every factory
00:19:27
Speaker
that you work in or every plant or whatever, let's say, has 10 of these fans. And those fans are millions of dollars. They're covered with sensors. You're generating metrics, multiple metrics per millisecond. That's going down to the nanosecond granularity. And then you probably have some kind of embedded
00:19:56
Speaker
uh, device on that, which, you know, does not running Linux. It's just like embedded is doing a local control loop. But one thing that you might have on that is this agent that we have called telegraph, which then is able to take all that data and send it to another influx DB location. So telegraph is an agent that you can install, you can build.
00:20:22
Speaker
Put it wherever you want. And it's just great at slurping up data, writing it to an influx DB instance. Let's say within that factory, you have a gateway. Like maybe you bought it from Dell or HP or something like that. And you just have this like huge honking computer that's hanging on a rail in the, you know, boiler room or whatever. And you are running Linux on that, right?
00:20:46
Speaker
So then what you can have on there is InfluxDB, OSS, it's totally free, it's our OSS version. And you're just crushing that with all the data from the fans, right? And then there, if you're, let's say you're on the plant floor and you're like, oh, I just got an alert. How did you get that alert? Because what happens is that local InfluxDB instance can be reading all that data, super high resolution,
00:21:13
Speaker
We have a programming language called Flux that runs within that instance, right? In FluxDB itself. And it can detect the combination of this sensor and this sensor is bad. We should page or whatever, however you do the alerts.
00:21:35
Speaker
we can support arbitrarily complex or simple alerts and transformations on that device. We can
InfluxDB in IoT and Edge Computing
00:21:42
Speaker
also do other computations, but we can also serve visualizations and et cetera. So you get the alert, you log into that gateway, you look at InfluxDB, you look at your graphs and you're like, okay, now I know what's going on. Nothing to worry about here. Okay. And then what you can be doing is like, that's a lot of data that you're collecting right there.
00:22:05
Speaker
If you care about all the data, that's fine. You can just replicate that. More typical is that you'll use this.
00:22:12
Speaker
another flux function which will transform that data and usually quote down sample it by either like aggregating rows like just give me the maximum reading for this sensor every minute or every second instead of every single you know instead of a hundred per millisecond or whatever right so you can collapse that down you can aggregate it you can also enhance or simplify that data
00:22:35
Speaker
Then you write it to a special bucket, which is replicated. It's called edge data replication, which is a feature that that OSS instance of InfluxDB will automatically replicate that data via a durable queue up to our multi-tenant SAS. Now, if you are looking at
00:22:56
Speaker
a hundred plants, each that have many of these big pieces of equipment, now you can get an aggregated view across all the plants. That plant got an alert, et cetera. So we have two kinds of users, basically.
00:23:12
Speaker
One kind of user is actually building a custom user experience on top of it. And so we have all the APIs and developer tools and et cetera. So their customers don't know that they're using InfluxDB under the hood, right? They just think they're using whatever the service provider is. We have another kind of customer that is doing operational monitoring at scale. And they're just required to have a custom solution just because of the scale or whatever other requirements.
00:23:43
Speaker
that we're working at. In that cloud SaaS version, it has all those features and more. So you get Flux, you get the visualization libraries, you get all the developer tools, Telegraph can talk directly to your cloud account that if you want. On that SaaS version,
00:24:04
Speaker
Sometimes you don't have a gateway and you're just writing directly from your devices or from your servers or whatever. You're just writing directly to the cloud. That's fine too. We have a free tier. So if you just want to come in and kick the tires, that SaaS version is also what we call pay as you go. So it's consumption based. So if you're just starting a company or just starting a project or whatever, you just pay for what you're consuming. So you don't have to commit to like,
00:24:33
Speaker
a huge license and you can just use like your department credit card or whatever and spend five bucks a month while you're prototyping and then you know if your workload grows to you know ten thousand bucks a month or whatever that just means your company's successful and you know so that that's um
00:24:52
Speaker
That's an overview from a product perspective. That's very insightful. I think, you know, I had several questions come up in my head during that explanation. I thought there was a lot of interesting pieces of that. One of which that I want to key in on is, you know, you talked about sort of the high resolution
00:25:11
Speaker
and nanosecond data. What does InfluxDB do differently to handle that type of resolution from these different sensors or different time series data sets that say a typical relational or no SQL database can't do? Why go down the route of looking specifically at InfluxDB? Why do you need InfluxDB? Yes. Okay. Why don't you just use MySQL or whatever?
00:25:41
Speaker
What people find is that at scale, if you start to try to query across time, you must have a purpose-built time series database. So for example, we have a function called aggregate window and flux, which says it takes two parameters, a function on which to aggregate, for example, mean, min, max, medium,
00:26:11
Speaker
You can actually put whatever you want. You know, you can write your own custom function to aggregate. And so it takes a function and it takes a time range, like every hour, every 10 minutes, every second, et cetera.
00:26:27
Speaker
What happens is we have a lot of customers that come over to us from document databases, relational databases, where everything worked, that trivial demo scale. But once they hit real scale, they just can't, those queries just start to fall over. They just can't handle it. They also can't handle the ingest. Right. So, sorry, are you interested in, I can peel the onion a little bit. Yeah. Ingest is definitely a big part of it that, uh, yeah, please go down that rabbit hole. Okay. So as you are writing data to influx DB, um,
00:26:57
Speaker
it's got a few parts of the data model. So it's got like a measurement, which is basically like just to group all these together is my measurement, right? Then you have fields, things like which are the numerical things that you're sampling that change a lot, right? Like so
00:27:19
Speaker
CPU load, unless it's packed at 100%, it's going to be different every time you sample at a temperature, vibration speed, all the different things that you might be sampling. Those are called fields. And then there's tags, which is basically metadata. And then, of course, a timestamp.
00:27:41
Speaker
As you stream that data and we're very highly optimized to take that data in like big batches, like we're not a data lake, right? We don't like, you don't like.
00:27:57
Speaker
You don't have to save up a whole bunch of data and then give it to us to process. And we notify you an hour later that we're done processing it. In real time, we can ingest a huge amount of data. It's usually readable in a couple hundred milliseconds. Got it. And then we take that data and we break it up into
00:28:19
Speaker
just conceptually, I won't go into the zeros and ones, but conceptually, we write it into different tables for you automatically based on the tag set. So what is a tag set might be like, plant ID, device ID, customer ID, any kind of metadata that you're going to want to go back and look at later, like, like, if you want to say, you know, show me all of the results by
00:28:47
Speaker
factory or show me all the results for this customer. Those are different things that you're going to query on. As we write, we create those indexes and those separate tables to make those come back very fast. Of course, we order everything by time as it gets written as well. One thing that's important to note about that is we're actually schema on write, which is a sharp knife. It's very handy because if you
00:29:15
Speaker
decide that you want to add metadata later, or change metadata, or add fields or whatever, you can just do it from your client that that right, you know, in point of fact, you can actually tell a bucket like, I don't want to be schema on right, we call that, you know, an explicit, an explicit schema bucket, in case you have that, you know, because as I say, it's a sharp knife, you like add,
00:29:42
Speaker
too many tag values or something you can explode your cardinality and run into trouble. So when you said when you're talking about querying this information right, can we use like SQL queries or is it through Flux? Primarily the language is Flux and Flux is a query language but really it's a data transformation language and another thing to note about Flux is
00:30:07
Speaker
It's a full-on programming language. So we have things like HTTP requests, right? So imagine that you're doing a query and you want to know what is the current temperature outside the factory, right? You can, in your flux, make an HTTP request to some endpoint that has that, you know, weather service temperature.
00:30:34
Speaker
and then use that in your calculations. Let's say you have a database that has customer metadata. You can do SQL.from, query that external database, then query your time series data, join that information together. And by the way, you can also call a remote endpoint, write to another database. So you can write programming tasks that
00:31:04
Speaker
out if you're not if you're not using influx DB, you have to go to your off team and say hey, I need to stay in the VM so I can run this job. And they say sure, give me a couple weeks and then you have to monitor that thing. You have to you know, make sure it's not failing and etc, etc. So you can take with flux, you can take like
00:31:23
Speaker
all of that complexity of writing that application and just hand it off to InfluxDB. And we will run those jobs for you. And you can write alerts if they start failing for whatever reason, et cetera. You can build all that operational intelligence into your InfluxDB. Yeah, OK. That definitely sounds like a very powerful tool. And I imagine it
00:31:45
Speaker
It bodes well for those custom solutions. And it sounds like you work with a lot with various customers. They want to do something very specific. So having this programming language to enable that makes a lot of sense. I'd like to add a couple of things to that. Yeah, please. So we're on 2.3 of the open source product. And we have an enterprise product.
00:32:13
Speaker
We have, uh, you know, the one point X open source product. Um, we support another query language called influx QL, which is sort of a twist on SQL, but that has time series capabilities put into it. And that's really a query language. So if, um, you know, if you're used to SQL or et cetera, that's accessible through the API, that's not accessible.
00:32:38
Speaker
through our UI currently. We also are building a new storage engine called IOCs, which is totally open source. You can go try it, run it, and everything. That uses a query engine that
00:32:54
Speaker
is also open source that we're one of the top contributors to maintainers. That's called Data Fusion. And that is a pure SQL query implementation. So if you invite me back to this podcast next year, we'll be able to talk about our full-on SQL support and things that we'll build on top of that also. Yeah. We'd love to have a follow-up for sure. Well, I want to switch gears a little bit and kind of, you
InfluxDB's Infrastructure on Kubernetes
00:33:23
Speaker
know,
00:33:23
Speaker
direct us back towards Kubernetes a little bit. And it's sort of a twofold question, which is, when you started running Influx on Kubernetes for your own managed infrastructure on the different cloud providers, which you mentioned earlier, what fundamentally changed for the company as, was there any strategic sort of path you took that
00:33:49
Speaker
enabled you to do different or more interesting things because you were using Kubernetes. And the second part of that question would be, do you direct people mostly to use that managed service versus having them run InfluxDB on their own Kubernetes infrastructure? Yeah, those are two really different questions. Yes, two very different questions. Start with the first one. Start with the first one.
00:34:18
Speaker
So first of all, we were very early to Kubernetes. So unfortunately, in many ways, we're trailblazers. You take the brunt of it. We're thankful of it. But it did allow us to, we call it the cloud abstraction layer. So it did allow us to be able to support
00:34:48
Speaker
multiple cloud providers, right? And to be able to take this thing and run it, yeah, in different places. We used a system for that, which was a combination of JSON it and cube config, or cube CFG, which I
00:35:13
Speaker
hadn't worked with the authors in, you know, previous life of those projects, or at least the maintainers. And so that allowed us to templatize the deployments at different levels. So we could have like a date based deployment, then have like, on top of that, like a JSONic configuration that said like, modify the base deployment, one for each of the
00:35:42
Speaker
cloud providers that we support. And then even within the region, then we could add more configuration on top of that, like this region do this, this region do that, et cetera. And so that cascading config allowed us to separate the service development and the configuration, which, you know,
00:36:09
Speaker
Really, especially in the early days was like really useful just because we just weren't at the scale that we are now and so like how to deal with like noisy neighbor problems and that kind of thing was a bit was a bit harder just because you know, we just simply didn't have a scale to Avoid all that and it allowed us to also like experiment, you know to like they let's add a let's add a service just here and just kick the tires on that and
00:36:39
Speaker
You know, like, so we've done some experiments where like, you know, what if we have this crazy service and it's just a matter of adding that service configuration to the right file, which by the way, I don't know if you saw our talk at, in Valencia at KubeCon, but that is also a very sharp knife. But yeah, so it just gave us that like ability to really, um,
00:37:08
Speaker
to move fast and try things. And when you paired that with our CI CD system, which is GitOps based, then that allows us to just have a system where somebody can change code or change configuration by just pushing code or configuration changes into the Git repositories. And then it can be as targeted or as broad
00:37:38
Speaker
as you want to target that. Okay. Before you answer the second part of Ryan's question, I have questions specific to the cloud deployments. So are you consuming like managed services from each of these vendors for Kubernetes? Or do you run these on just cloud VM based instances and have a different orchestration like a terraformer? So to begin with, we like
00:38:04
Speaker
got our own VMs and installed our own, et cetera. But now we run on the cloud providers, default Kubernetes, like EKS, AKS, which is problematic because they do not support the same things. They are not on the same versions.
00:38:24
Speaker
Google comes along and says, oh, by the way, we're going to upgrade all of your nodes with the new Kubernetes version. And you find that out as we discovered. If you happen to be doing a deployment when that happens, your code isn't necessarily well-behaved in those circumstances. So that won us. There's a lot of win there, because now we're not managing Kubernetes ourselves. But that actually injected a bunch of complexity.
00:38:53
Speaker
We also consume other services like information that's not time series data. We keep in Postgres. That we don't manage our own Postgres. We use the cloud providers compatible Postgres, for example. And there's some places we want to do that more of.
00:39:18
Speaker
But just because when we started, there weren't necessarily ways to do all these things. We built a lot of our own things that are a bit harder to migrate away from, but we plan to. So yeah, we're very supportive of using the underlying cloud providers, implementations of common services.
00:39:42
Speaker
Like, um, I, like, if I were doing it, if I were to, to, to make a recommendation to somebody, I would just like, don't, don't spin up your own Kafka. Don't spin up your own, your own Postgres. Going further, I would say like, use noble nine for your SLOs, use edge data for your log management, like.
00:40:03
Speaker
anything that you can offload onto another provider, do it. So then you can just stay focused on like your code, your value add.
00:40:15
Speaker
Yeah, that's that's really important, right? Because I think Jeff Bezos said this right, that only focus on things that actually makes your beer taste better. And he was referring to like the breweries in in Europe in like the 1800s, where they had to run their own power grids, and which didn't make the beer taste better. And then eventually there were breweries who set themselves apart and stopped managing the power infrastructure and just focused on making the beer taste better. So I think that that analogy works here perfectly.
00:40:44
Speaker
I like that. Yeah, that that's, that's good. And I think that there's, you know, one thing is like we talk about the cloud service providers services, which are our which is good. But I think what you what I really recommend people to do is to really recommend to the degree possible, like just opt into the community of third party vendors who are not aligned with the cloud service provider who are not aligned with
00:41:12
Speaker
like a big vendor, like, um, there's like, like, uh, replicated if you need like a solution for on-prem or whatever, like replicated has no, um, motivation to like, get you to buy other services or lock you in or make it harder to like use other, other infrastructure, et cetera. Like I already mentioned noble nine already.
00:41:42
Speaker
mentioned edge Delta, you know, if you want to get to a marketplace, like look at a, you know, company like tackle IO, who will help you get to all the marketplaces and, you know, like they create an abstraction that keeps you from getting locked in. So I think there's this community of
00:42:01
Speaker
companies that are kind of in a second wave of vendors, like people who have managed Kubernetes and built things on top of Kubernetes and then are solving problems for Kubernetes users like us, but not from the perspective of like, Hey, we're like a vendor trying to lock you in or direct you down our
00:42:23
Speaker
whatever our upsell is or whatever our lock-in is, et cetera. So that's like, I'd really strongly recommend that people outsource as much of that as possible and try to opt into that community of providers if possible. And some of these names are new to me, so I'll definitely look those up because, as you said, as much as we can offload, that's better.
00:42:44
Speaker
Going back to Ryan's second question, if customers want to not consume the managed service but run Influx DB enterprise, I think, in their own data center environments, do we still need Kubernetes or that's just running on Linux virtual machines?
On-Premise Deployment of InfluxDB
00:43:01
Speaker
So that case that you're self-managing, that's really your option. So we provide different ways for you to install and manage that software. One way is we give you a Helm chart and you can install the Helm chart in your Kubernetes environment. But we give you other orchestration formats that predate Kubernetes. We have Terraform templates. Honestly, a lot of people just go and
00:43:30
Speaker
that you get the binaries and apply the licenses and run the scripts to configure it. At that point, you'd speak your own poison, whatever they are comfortable with. I mean, that's the whole point, right? If you can't use a multi-tenant SaaS and you have to manage on-prem, then you're already locked into whatever.
00:43:54
Speaker
requirements led you down that path anyway. Gotcha, makes sense. So one more question, while I was getting ready for the episode, I saw that InfluxDB can also help me with Kubernetes monitoring. If I want to use it to monitor what's happening in my cluster, we have seen customer deployments where there are customers that are running more than 100 nodes per Kubernetes cluster. And how can InfluxDB help them with getting that information in real time?
00:44:22
Speaker
So there's a couple of approaches. And the first is just we had just had like Prometheus scrapers just built in. And so if you use the Prometheus library to write out those metrics, you can just scrape those into the InfluxDB. You can just scrape it into your cloud account, set up all your monitoring, everything that you want to that way. A new approach, which I think
00:44:52
Speaker
people are going to start adopting, but I don't know if anyone has yet. I'd be really interested to see is akin to that IOT edge scenario I was talking about where, you know, if you have multiple Kubernetes clusters, you can run a OSS instance inside that cluster, process all the data that's been written. Oh, we have a tool called Telegraph that I think I alluded to before that agent.
00:45:24
Speaker
Tip, sorry, tip, one of the main things that people do is actually they, there's a Helm chart, they install telegraph in their cluster, and then that can then grab all those Prometheus metrics as they're being generated and send them to wherever they want. So one of the things that I think people should look at is to send that data to a local OSS instance running inside the cluster, and then do just what I said about IoT, like all the high fidelity data,
00:45:54
Speaker
throw it away with the retention policy if it's more than an hour old or whatever. But down sample it, process it, replicate it back to your primary cloud account so you can look over it. With those flux tasks, you can do interesting things. For instance,
00:46:13
Speaker
You can like have like the high fidelity data, say just in the cluster with like, say a one hour retention policy. And then if you detect an anomaly, say, okay, actually sync all that high fidelity data back to the cloud.
00:46:31
Speaker
And so that way, if it all goes down and reboots and everything, you'll have the high fidelity data. And if you need to do an RCA later or whatever, you can go back and look at the details. OK. That's a really cool approach, right? We don't have to store everything. But then if there is an event that needs to be looked at, there is an option to ship all that high fidelity or higher resolution data back to that instance. Yeah. So that's for metrics. If you want to do the same thing with logs, take a look at Edge Delta.
00:46:59
Speaker
They're a really cool company that like that does something similar to what I just described. I think this has been a lot of great information. Thank you for diving into like how Influx helps customers and then how you like use Kubernetes to run your managed services as well. I think I have one last question like
00:47:19
Speaker
How can people get started with InfluxDB, the OSS version? I know you already mentioned that, but are there any easy ways to learn more about InfluxDB and then get started with it in their environments? Sure. We have a lot of great documentation. Probably the easiest way to get started is just to go to
00:47:44
Speaker
uh, in flux DB cloud on just go to our website, influx data.com. There'll be a link there. Just click the button, create a cloud account totally free. And then we have some tools there that'll help you set up your developer tools. Like we can walk you through to set up your Python or your JavaScript or your going environment. And you can just do all that for free. And like within 20 minutes, you can be reading and writing data and just
00:48:12
Speaker
like experiencing time series and what it has to offer. So yeah, borrow the free tier cloud account and then take it from there. Yeah, that's easy enough. Awesome. Is there anything else that you wanted to highlight before we call this off? Oh, geez, I don't think so. I'm sure I'll think of it 10 minutes from now.
00:48:35
Speaker
But hey, I really appreciate you having me on the podcast and I really enjoyed the questions. Yeah. Thank you so much for being a guest on the part, right? And we'll definitely reach back out next year when, when, when you have that update up and running. Yeah. Thank you so much. Looking forward to meeting you in person, maybe at KubeCon North America. Sounds good. Yeah. Thank you so much. Bye. Bye. Bye.
00:48:59
Speaker
All right. Well, it was great having Rick on the show. I don't know about you, Bhavan, but I haven't done a ton with time series data. I know it's, you know, a lot of the data out there from, you know, different sources is really time-based if you really think about it. So really interesting solution to dive into with Rick here. You know, one thing I wanted to note was sort of the choice of their managed service and how it sort of evolved over time.
00:49:25
Speaker
because they were early to this whole Kubernetes ecosystem. You mentioned at first, they were deploying their own virtual machines and deploying Kubernetes and having to manage all that. Now, as this ecosystem has matured, as we've seen over the years, they're going with the cloud provider versions of
00:49:49
Speaker
of Kubernetes and one might think that makes their lives a lot easier, but to his point of view there that it comes with its own challenges. You have to be aware of when those versions of Kubernetes go out of use and they get automatically updated or if you're doing certain things while those updates are happening or being aware of what they support or what they don't support.
00:50:19
Speaker
just a really interesting real-world use case of the pros and cons of using your own pre-built infrastructure versus something managed. To his point of view, it really sounds like their focus as a company really is to
00:50:36
Speaker
focus on their own innovations. We've heard this in the past where managed services really do allow you to refocus your efforts where you want to. Offload the operations and management of something else. He mentioned a bunch of them, not just Kubernetes, but specific databases or data services that they're using. I thought that was an interesting point.
00:51:01
Speaker
I know, and I think for me, right? One take away was just the way we did this episode. I think it was interesting to see how InfluxDB and time series database can help customers with different use cases, especially the IoT use case, right?
00:51:16
Speaker
all the high resolution or high fidelity data is available on site, but then you aggregate that or you uplevel that and then send only aggregated information to the SaaS platform. So that was an interesting new approach of thinking about use cases. And then I did like the fact that, okay, they're running on Kubernetes in the public cloud. Yes, but then if customers still want to own everything, even on the control plane,
00:51:37
Speaker
They have solutions, they, Eric mentioned, they have like Helm charts, they have just simple Linux binaries that you can download. They have agents that you can run on your Kubernetes cluster itself. So different ways you can get started with InfluxDB.
00:51:53
Speaker
And I think since they have been using Kubernetes for a while, they have figured out what are the best delivery mechanisms for these on-prem components as well. So I think it was a good balance between how they are consuming it, what were the challenges, how they evolved their SaaS based offering, and then what their on-prem solution looks like as well. Yeah, I couldn't agree more.
00:52:13
Speaker
So I think that brings us to the end of this episode on InfluxDB. And I really enjoyed tackling this sort of specific topic. And we'll look forward to next week. I'm Ryan. I'm Bobbin. And thanks for joining another episode of Kubernetes Spites. Thank you for listening to the Kubernetes Spites podcast.