Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
PostgreSQL in the Cloud
959 Plays1 year ago
What's going on with Postgres? Joining us for his perspective is Raouf Chebri, a Developer Advocate for PostgreSQL and the cloud service Neon. We catch up on what's new and important in recent versions of Postgres, what Neon have been doing to make Postgres work well in the Cloud, and what Raouf's life is like as a professional singer of Postgres's virtues.
Neon: https://neon.tech/
Neon’s architecture: https://neon.tech/docs/introduction/architecture-overview
That CEO quote: https://twitter.com/nikitabase/status/1563913187862335489
Raouf on Twitter: https://twitter.com/raoufdevrel
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Recommended
Transcript
The Future of Software: Evolution Over Innovation
00:00:00
Speaker
Over here at Developer Voices, we're very interested in the future of software. Where are we trying to get to as an industry? What do we think tomorrow will look like? What do we think it should look like? But that doesn't just mean looking at the new and shiny stuff, at least it shouldn't. It also means taking a look at tools that have been with us for a while and asking how they're evolving, how they are moving us forwards.
Postgres: A Reliable and Evolving Database
00:00:26
Speaker
Which leads me to wonder about one of my favourite and staple databases, Postgres.
00:00:33
Speaker
It's been an exceptionally high-quality free relational database for decades now. And it could rest on its laurels. It's definitely earned it, but it hasn't been. So what's new? Joining me to answer that question is Ralph Shebery. He works at Neon, which is a Postgres as a service platform. And we catch up a bit on the state of Postgres and what's new. And then we have a look at how Neon are trying to move the state of the art forward.
00:01:00
Speaker
They've been diving into the guts of Postgres, adding a bit of custom Rust code, doing some architectural tricks that will be familiar if you know how Kafka works.
Neon's Git-Like Enhancements to Postgres
00:01:12
Speaker
And they've ended up with a version of Postgres that starts to look a little bit like Git, a promise of a database that you can easily and cheaply feature branch.
00:01:23
Speaker
Things like that. How can you have multiple versions of the same source database doing different tasks? It's interesting. It really caught my attention when I tried it out last month. So I thought we'd bring Ralph in for the what, why and how of what they've been doing. So I'm your host, Chris Jenkins. This is Developer Voices. And today's voice is Ralph Shepery.
00:01:57
Speaker
My guest today is Ralph Shebry. Ralph, how you doing?
00:02:01
Speaker
Yeah, pretty good. Thanks. Thanks for having me, Chris. Absolute pleasure. You're coming to us live from Switzerland, I believe. Yes, from beautiful Geneva. Oh, very nice. I've only been there a couple of times, but it is beautiful. Yes. So I've brought you in because you are one of my on my grapevine, one of my resident Postgres experts. And you work for neon, they've been doing some really interesting stuff with Postgres. We're going to get into that.
00:02:28
Speaker
But I have a confession. Having loved Postgres in the past, I haven't had much excuse to use it for a couple of years. So catch me up. What have I missed? What's new? Well, Postgres just has been this open source database, relational database for decades. And now it's in version 15.
Why Postgres Thrives in Development
00:02:52
Speaker
And I think
00:02:54
Speaker
One of the reasons that you should pay attention to Postgres and the evolution of Postgres is, well, first of all, it's really the only true open source database. It's not owned by anybody. It's really owned by the community and very active community of contributors who are always striving to making it better, improve performance, adding more features. And I think
00:03:20
Speaker
What's interesting about Postgres is the growth of the community. So according to the Stack Overflow survey, Postgres is the most wanted database among professional developers. And I think also according to DB Engine,
00:03:37
Speaker
among the top five databases. Postgres is the only one that is growing. If you take MySQL, SQL Server, Oracle, or even MongoDB, they're all declining, but Postgres is growing. I think it's really because of its ecosystem.
Neon's Unique Position in the Postgres Market
00:04:00
Speaker
What I mean by ecosystem, there are two things I mean by that. First of all is the ecosystem of extension. So Postgres is very versatile. It's not just a relational database, but it has all these extensions that give it superpowers. For example, we have PostJIS that
00:04:18
Speaker
allows you to do a geolocation. And you have a timescale that helps you to do time series. Or now the hot topic is AI. We have PG vector, which allows you to do a similarity analysis among vectors, which is something that you really need if you're using large language models like GPG. So I think it's just that there are so many things that you can do with Postgres, apart from just storing data.
00:04:47
Speaker
But also, there are so many vendors who are offering, who have their own Postgres offering. So you have AWS, you have Heroku, and you have countless others. You have back-end services, you have distributed SQL. So it's not just this...
00:05:08
Speaker
database that is a standalone and works on better metal, but you can have it as managed service, you can have it as distributed service, and now we have it as a service database with your own.
00:05:21
Speaker
Yeah, so I was going to ask you this later, but I'm going to pull this question sooner in my mind. Because that's because you're a developer advocate for neon Postgres service, right? Surely that must make life hard because there are a lot of places you can get Postgres. How on earth do you distinguish yourself among the many, many Postgres services? It's interesting. I think I think
00:05:48
Speaker
that we positioned ourselves quite early as the database for developers. And I think the neon architecture also helps a lot just to differentiate ourselves from all the other renders. And I think what helps us to be competitive in that space is that you can get a connection string very quickly, and the onboarding process is quite easy. So I think that
00:06:18
Speaker
I think for developers who are just looking for a database, we're just looking for a connection string, you can get started with Neon in a few clicks. So that definitely is something that helps. But what makes us different? I think it's our ecosystem of partners. And we partnered recently with Vercel. We partnered also with Hasura, with Replit. And obviously, that gives us access to that developer community.
00:06:47
Speaker
And it just makes the conversation easier. So we're not necessarily talking to DBAs who want to manage their database themselves. We want to be the database for developers. And I think that's what makes us stand out from the rest.
00:07:04
Speaker
Okay, that's, I mean, that feels like a bit of a marketing headline, the database for developers. Yeah, you're you're I'm going to challenge you on that. So I do think you have great developer experience. And I'm going to talk about that. But like, as a developer advocate, I've done that job, you're going to a conference, Postgres isn't the new and hip thing.
00:07:27
Speaker
There are lots of people talking about Postgres. What do you, Ralph Shebery, get up on stage and talk about to excite people about Postgres?
Innovations in Postgres: Branching and Workflow Improvements
00:07:40
Speaker
That's an interesting question. So what excites people about Postgres? Well, in my case, I think
00:07:49
Speaker
I think is the new thing, the new things that you can do with those graphs. And I definitely forgot to mention branching in that story. Yeah, we're gonna get into that too. I know we need to, we've got a lot of topics to hit. Don't hit them all at once. Yeah, so part of it is, I think, because
00:08:09
Speaker
Well, Postgres has a great documentation. People can go and have a look at it. So they don't necessarily want to know about Postgres when they continue on. So they want to know about all the other stuff that we do. So how does the serverless architecture work? How does branching work? And branching is one of the things that we offer. So I think that what makes them excited is just how can they be more productive using Postgres? And I think that the whole developer workflow story is something that is quite new in the database world.
00:08:39
Speaker
are accustomed to it with things like it where you can create a branch, you can merge it, and you can collaborate with dozens of other developers, but we've never really had that in the database world. Every time we wanted to collaborate using databases, it's just a painful process, to be honest. So, I think what excites people the most when I talk about Neon, yes, developer workflows is the fact that
00:09:06
Speaker
You can create a branch in a matter of seconds, and you can prototype using a database that is hosted in the cloud. You can get the connection string very quickly. So yeah, I would say that's what I talk about mostly. And I try to listen to, I try to listen to the demo. I mean, you've done that for four years, as you said. So you listen to developers, and every time they come up with a set of problems, and you try to be helpful, et cetera,
00:09:35
Speaker
Yeah, and this is how we find things to talk about. Okay, fair enough. Yeah. So we need to get into this whole thing of branching, but we need to come and we need to get the long way because this is the thing that really caught my attention about neon specifically. You have this thing and it's kind of a hip phrase that lots of databases are saying to separate storage from compute.
00:10:01
Speaker
I've heard that in a few different places, including my Apache Kafka days, which are still close to my heart. What does that mean in terms of Postgres?
Efficiency Through Separation of Storage and Compute
00:10:10
Speaker
What are neon doing? Why do we care about separating storage and compute? Well, we care about it because if you want to offer a truly serverless service,
00:10:26
Speaker
then you want to give developers the ability to spin up instances on demand. But the problem with that, as you may know, is
00:10:35
Speaker
When you're dealing with databases, you're dealing with stateful environments. And those are quite hard to manage because, well, every time you want to kill an instance, you need to make sure that that data doesn't disappear. You need to move it around. And what happens if you have an application that needs that data? So it turns out that managing a stateful instance is actually quite a hard problem. And this is why separating storage and compute comes into play and makes things a little bit easier.
00:11:05
Speaker
So in our case, when we talk about separating storage and compute is we've taken Postgres and we added a patch that is about 1,000 lines of code. And all that patch does is give Postgres the ability to talk to our storage engine that we built from scratch and that we built with Rust. So there are really two components to that.
00:11:32
Speaker
And that separation gives a lot of benefits. And one of them is that since we don't have to deal with all the access to the file system, et cetera, then it allows Neon to spin up a Postgres database within seconds as opposed to minutes. So you go to Neon, you create a project, and you have a connection string within three seconds.
00:11:57
Speaker
And how does that actually work? How does writing a new file system driver mean you can spin up instances quickly?
00:12:05
Speaker
Oh, so well, you don't have to deal with all those accesses to disk. So from my understanding, Postgres, that helps boot up the Postgres instance quicker, as opposed to if you had to boot the entire Postgres engine.
00:12:37
Speaker
Sorry, what was I? I was talking about one of the benefits of separating storage and compute. One of it is that you spin up Postgres database really quickly because you don't have to deal with the data yet.
00:12:54
Speaker
And the second advantage is that you can allocate resources independently. So you have that Postgres instance. If you need more RAM, more CPU, you just give it more RAM and CPU. If you want to take it down, you just do it.
00:13:11
Speaker
And this is what allows for things like auto scaling. And we have also scaling down to zero. So if your database is inactive for five minutes, then you just shut it down.
Database Branching and Cost Efficiency in Neon
00:13:22
Speaker
So that's regarding the compute part.
00:13:27
Speaker
The storage part is something that we built from scratch that is multi-layered and it's a distributed system on itself. So we have a durability layer that we call safekeepers. And basically that layer makes sure that your data is always present. So if you insert a row, that is always going to be there.
00:13:51
Speaker
and you have what we call page servers that knows how to reconstruct your data from any given point in time. Because in neon, we don't save just the data, we save the history of the database. Okay. Yeah. And that actually what allows for things like branching is because the page server just knows how to reconstruct the state of your database from any given point in time.
00:14:21
Speaker
So you you're doing like a git like thing where Postgres tries to save some data and your file system layer intercepts that and says, I'm not going to do a destructive update. I'm going to keep it as a history of changes. Yes. Postgres does that through write ahead logs. And this is exactly what we say. So we save write ahead log records. And yeah,
00:14:48
Speaker
And those are immutable files. And actually, actually, the third part that I haven't spoken about is since those are immutable files, then we can offload everything to cloud storage. And this is what we do. So we offload cold data to S3. And that saves up some money also to our end user. Okay, this actually is reminding me a little bit like Kafka, where you make the right ahead log, basically,
00:15:17
Speaker
more important in the architecture than it's previously been. It's not just for creating a backup copy or recovery. It's actually an active part of the idea of how this is going to work. Right. Yes. So this feeds into this thing you can offer of branching whole databases to share with other developers. Tell me a bit about that. Yeah. Well, since
00:15:44
Speaker
Since you can do branching, we really think, our philosophy is that whenever you need to get branch, you'll probably need also a database branch. I can believe that actually, yeah. Yeah, because if you think of it from a developer's perspective, you have production environments, staging, and you have your development environment, and you have databases in all those environments.
00:16:12
Speaker
You don't work by yourself. So you work in teams. And the more people you have, the higher the chance of breaking something, and especially at database level. So if you are making changes, better make them in branches that are isolated, that are independent. All branches inherit data and schema from their parent branch.
00:16:36
Speaker
So one of the advantages is that you can create features that are actually using some production data and i think in some cases that are features where you need a proper testing against production data in order to make sure that it works properly. And then and then nowadays what is interesting is that the staging environment is kinda dead too.
00:17:00
Speaker
People are using more preview environments, so you have these school companies like Vercel who offers developers the ability just to create preview environments for every time they create a PR.
00:17:15
Speaker
If you submit your code in GitHub, you create a pull request, then you have an automatic environment that comes out of the box with it. Your stakeholders or your team members can just click a link and they have this production-like environment. They can have a look at it and see everything that happens. But that's great for frontend. But what happens with the database? Well, we
00:17:42
Speaker
We created an integration with Vercel that does that automatically. So with a few clicks, you can do that. And we also have our API and GitHub actions that also allow you to do that. So it makes total sense to have a database every time you're creating a branch in Git. And this is how we go about it.
00:18:05
Speaker
Okay, so I can actually see that being really useful, because I've definitely used it in front end, and it's incredibly useful. So, so you're saying you've got cheap branching, so I can do a cheap branch of the live data, you've got low compute costs if it's not being used. So I so it doesn't cost me anything to have an extra database for every pull request until they used. Um,
00:18:34
Speaker
So every time you create a branch, well, in the storage engine, you're just creating a pointer. So you have some metadata there. And you're creating also another compute instance to have access to it. You can also create a branch without having to compute, but you would use that as some sort of backup if you need it. But yeah.
00:19:03
Speaker
So you can literally have tens of branches even more. You can have branch for every one of your team and all you're gonna be charged for is really the delta between the branches and not necessarily like the entire, it's not because I have one gig of storage in my main database that I'm gonna have, but then I'm gonna replicate those every time I'm gonna create a new branch.
00:19:31
Speaker
Right. Could you say use this mechanism for like, as a fallback? Like when you do an upgrade in production, could you say I'm going to tag the database at this point in time, and oh dear, the upgrade went badly, I can roll back.
Time Travel and Data Recovery with Neon
00:19:47
Speaker
Well, you can do even better. You have the ability of doing time travel anytime because I said that the storage engine stores all the wall records and knows how to reconstruct data from any given point in time. So let's say one of your engineers somehow decided to manually drop or delete some data and they deleted the wrong data at the database. That's happened, yes.
00:20:13
Speaker
Well, then you can just, uh, yeah, you can just easily roll back, create a branch from, uh, from a past point, um, and, and just say, Hey, I need to have this state of the database from T minus one, not, not T zero. And it does it in, in a matter of seconds. And.
00:20:33
Speaker
and you get back all the lost data. And then you just swap compute instances and that's it. So the backup system as it is now is pretty good. We're still adding more to it because we're getting feedback from a hardcore Postgres users. But yeah, but the story is quite simple for now. What kind of feedback are you getting? Is it like really deep technical or usability or what? Yeah, I think
00:21:03
Speaker
So I think there are things that we're currently working on. So our developer workflow story needs a merge. And it's something that we're working on where we're going to postgres. We're adding things like shorting, replicas. All those things are coming very soon. So yeah, we get those feedback all the time. And we're working on them every day to make sure that we're
00:21:33
Speaker
we get in part with everything that Postgres offers. How much maintenance is that? You say it's like a thousand lines of code at the core of this.
Staying Updated with the Latest Postgres Versions
00:21:46
Speaker
Does that mean that your version of Postgres is very close to the latest? No, it is the latest. We do that for every version of Postgres. We started with version 14 and then
00:22:03
Speaker
When version 15 came out, we were among the first ones to offer it to everybody because all we need to do is just to write that code in version 15.
00:22:13
Speaker
And now you can do that. And it's going to be the same thing with 16. So every time a new version of Postgres is going to come out, we're going to be behind by a week or so, maybe less. Maybe we're going to do it at the same time because we have Postgres hackers within the team that are very, very close to the Postgres community and that contribute all the time to Postgres.
00:22:36
Speaker
Okay, yeah, we can do that. And they have a pretty good track record for doing like release candidates and stuff. So presumably you've got some warning. Yeah, yes. Okay, actually sounds really handy. So let's talk about
00:22:52
Speaker
Speaking of handy, if I'm going to use it, I have kicked the tires on neon and I'm going to sound like this is a sponsored podcast now, but hand on heart.
Lessons from Neon's User Onboarding
00:23:00
Speaker
I haven't been paid to say this. You have a really, really good onboarding experience for like a lot of software as a service companies could go to neon and learn a lot to steal for their own service. Well, I'm glad you like it. I really did. Yeah. It took me about three minutes from hearing about you.
00:23:21
Speaker
to having a database table with data in that I was selecting it out of. Amazing. But I've worked with software service companies where that is the opposite of the experience, right? What's your secret sauce? What's your tip for getting good developer experience? Yeah,
00:23:44
Speaker
I'm not sure if we have a secret side. I think we try to do what everybody does, which is listen to our users, listen to developers, and we want to get better. I think that's what we try to do. If you've looked at the neon console in the past few months, I've
00:24:05
Speaker
I've been around for less than nine months and I can tell you that a lot of changes happen already. Sometimes I have a look at the DUI and I don't recognize it. Things are added all the time, things are changed because we listen to our community. Sometimes we get constructive feedback, sometimes we get bad feedback and we try, or negative feedback rather, and we work with both.
00:24:27
Speaker
And I think we also are in a unique position where we have partners that come from the front-end world, partners like Vercel, Replit. They already offer a really, really good developer experience and we get inspired by our partners to deliver as good of a UI as possible.
00:24:53
Speaker
I'd say maybe like the third thing would be that we have a passionate team and culturally within the company, most of the conversations start with, does this make it easier for developers or does it make it harder? So if it's harder for developers, then maybe you want to remove those frictions. But if it's easier, then maybe it's the way to go. So yeah, I think it's just to listen to our developers and ask ourselves the right questions and
00:25:22
Speaker
and try to mimic the best. So I'm gonna have to push you on this because it seems like every company thinks they're doing that. But I think you're unusually successful at it. What was that? What's the management like? What's the structure for getting changes in? I think maybe it starts also from the top. Like, as I said, culturally, it's like that. And it's important if
00:25:45
Speaker
if you have founders and if you have leadership team that think that this is something crucial. Because, well, we're having this architecture, but we need to find a way to make it easy for developers also to use it. So I think that, yeah, maybe it also comes from the top all the way to the rest of the team.
00:26:09
Speaker
Yeah, okay. Yeah, I could see if if it's an actual priority for the management that really changes things rather than being a lip service priority, which does happen. Does it does it also relate back to this thing you were saying where like, there are a lot of Postgres service providers, so you must know that you have to differentiate yourself.
Neon's Focus on Developer Experience
00:26:28
Speaker
And that onboarding experience when you're in a difficult to differentiate market, that first half hour can make all that more difference.
00:26:39
Speaker
Yeah, there's actually, I just remembered that there's a really, really nice quote by our CEO, and it says on his Twitter. Yeah, I don't remember the quote itself, but basically, when you have a 10x architectural advantage, and here he's talking about being serverless and all the things that you spoke about.
00:27:01
Speaker
then the next thing you yeah, you focus on is developer experience. And that's how you create the separation between yourself and the other companies. I don't quite remember the full quote, but it's an interesting one that you can find on Twitter and Nikita Base. I'll hunt it down and stick it in the show notes so people can have the exact quote. But that I guess that does prove that right at the top you're thinking about developer experience. Absolutely. Which is unusual in a lot of software to service businesses.
00:27:32
Speaker
Okay, so give me a bit of your background. What brings you into a Postgres story?
Ralph Shebery's Journey into the Postgres Community
00:27:43
Speaker
Yeah, I honestly, I didn't necessarily think of myself. I didn't think that I would ever be working
00:27:53
Speaker
with the post-risk community, it's quite new to me. It's something that I'm learning. I worked as a developer advocate and an evangelist a long, long time across the notes. For bigger companies, I worked in NoSQL. But it's just, I think, it was just this
00:28:17
Speaker
It was natural for me to work with Postgres because the community is asking for it and people want it and people want to learn more about it. So I felt like it was an exciting opportunity to be working with the Postgres community and it is such a great time to be within the Postgres ecosystem. It is. Yeah, from my experience, it's a nice community and it's a great general purpose database.
00:28:46
Speaker
But if you've worked in the NoSQL world, right, then you've seen a few different databases. What do you think Postgres is places in the world where everybody's trying to vie for the database that's customized to a specific task? Yeah, I think it's important as a developer advocate also to understand that.
00:29:11
Speaker
there's no such thing as the best database, right? There's no such thing as the best tool. There's only the best tool for your specific use case and for whatever you're doing. So I think that for some use cases, then NoSQL is great. For other use cases, maybe you would go for SQL. We know that for most cases, SQL is probably the way to go. So I think that it's important to understand
00:29:41
Speaker
what your goals are, what is it that you're trying to optimize for, and what's the best tool to achieve that. When I was working in NoSQL before, we had a very performant database. And we've always compared to other NoSQL databases, and we would know that there are use cases where maybe our database would be the best, and maybe there are other use cases where it would be just better to use MongoDB, whatever.
00:30:12
Speaker
So yeah, so I think it's important in the database world to understand why you're doing things. Yeah, that's true. That's true. As ever, as ever in programming, you've got to understand what you're all good at. And you've got to I think you've got a few different things in your toolbox these days. Like, I would say everyone should know one relational database if they can, if they're at that stage in their career, but not just a relational database.
00:30:40
Speaker
You know, the more different ways you've got of solving a problem, the better off you are.
00:30:46
Speaker
And I think you come from that Kafka world where I've seen a lot of architectures where you put Kafka in the center of everything, and then you have all these streams of data, and then you have Postgres, you have Mongo, you have facility B, you have all sort of different databases that are just communicating with each other. And I think this is what smart teams do, is that they understand the pros and cons.
00:31:12
Speaker
of every architecture of every tool and yeah you need to you need to know a relational database but not just a relational database like you need to understand the whole ecosystem as a whole and how things interact with each other yeah well i can think of no better way to get started with a relational database than to start with Postgres and i'll give you that plug you've got a great way of starting with it
00:31:36
Speaker
So, yeah, I think maybe it's time to let people go and kick the tires on your service if they're so inclined. Yeah. Go for it. Neon.Tech. Go have a look. And most importantly, if you have any feedback, please reach out. I'm on Twitter. We are on Twitter. We're pretty active. I'm Rov Devrel.
00:31:59
Speaker
There's the Neon account, and we would love to hear from you. We're still young, we're learning, and we want to offer a great service that can help developers out there. This sounds like it was a decent chance your feedback will actually be acted on. Yeah, that's all we hope for. Ralph, thanks very much for talking to us. Thanks, Chris. Thanks for having me again.
00:32:28
Speaker
Thanks, Ralph. And I'll say it one more time. This hasn't been sponsored by Neon. I was just really impressed by their service. So if you need a Postgres instance in the cloud, give them a try. Perhaps more generally, if you're working on a service aimed at developers and you want to see an example of really good onboarding, I would check them out for ideas to steal.
00:32:48
Speaker
Now, we may not be sponsored by Neon or by anyone else at the moment, but one day we will be. I don't want you thinking I've suddenly sold out, so this space is intentionally left blank for some rampant capitalism.
00:33:02
Speaker
More important than that is the currency of feedback. So if you've enjoyed this episode, please leave a like or a rating or share it with a friend. Speaking of friends, if there's someone out there you think I should be talking to, you'll find my contact details in the show notes. And with all that said, until next time, I've been your host, Chris Jenkins. This has been Developer Voices with Ralph Shepery. Thanks for listening.