DuckDB And The Burden of Success (with Hannes Mühleisen)

Transcript

Speaker: Since we first had DuckDB on the show a couple of years ago now, I've become a real fan of it. It's a really fast local analytics database. Think SQLite, but for analysis rather than transactions.

Speaker: And it's fast in two senses. There's the CPU sense. You can give it a billion line CSV file or a billion record JSON file, and it can process through it in seconds.

Speaker: But it's also fast in the user-friendly sense. You can query those files as though they were already tables in the database. It just works. You don't lose half an hour to figuring out the right create table schema before you can do the job you actually wanted to do.

Speaker: Their goal was fast and user-friendly, and full disclosure, I'm a fan. I think they succeeded very, very well. But success comes with its own problems, doesn't it?

Speaker: If you build something and nobody likes it and it fails, it's sort of freeing. You're free to go and do something else. But if you build something people like, before you know it, you've got feature requests and pull requests coming in.

Speaker: You've got users and maybe customers to attend to. Maybe you end up with staff to manage. Maybe, and in my opinion, this is the worst case scenario of success, you get pulled away from building good tech and you have to spend all your time managing an enterprise sales and marketing team or appeasing shareholders.

Speaker: And the money of that is nice, but it hardly sounds fulfilling. That set of problems, the problems of success, have been visiting DuckDB since we last talked to them. So i thought we'd bring back the co-creator of DuckDB, Hannes Mohlheisen, to discuss them.

Speaker: What problems has success brought him? How do you accept a slew of new features that the community wants merged without becoming responsible for maintaining them forevermore?

Speaker: His answer was they had to build an extension mechanism. But that, in turn, has forced them to do a complete rewrite of their SQL parser. And writing an SQL parser, a fully featured one, is parsing on hard mode.

Speaker: We talk about how Hannes swore he would never do a client-server version, but then he had to walk that back because you can't claim to be a user-friendly database if all your users are demanding client-server and you ignore them.

Speaker: And we also talk about how DuckDB needed to support Iceberg to be a competitive analytics database, but how they found enough problems with it that they felt compelled to create a competing version of the same idea.

Speaker: So in this episode, if you want to learn about peg parsers, Duck Lake, and why all database wire formats are wrong, we get into the technical details. But around that, we also talk a a lot about the reality of building a successful project, including some big news, how they're avoiding the trap of becoming a company ruled by their enterprise sales team by being acquired by AWS.

Speaker: Will that work? The answer to that is a mixture of hopes, fears, contractual terms and friendly agreements. But we'll get there. Let's dive in. I'm your host, Chris Jenkins. This is Developer Voices.

Speaker: And today's voice is Hannes Mullisen.

Speaker: I'm joined today by a returning guest, Hannes Mullisen. Hannes, how are you? I'm great. How are you, Chris? Great be back. I'm very well. It's great to have you back from ah the duck-filled heart of Amsterdam. yes Yes. Yes, we have many ducks. ah and We have actually adopted ah ducks in the zoo here as ah as a company because we have such a deep love for ducks. There's a plaque somewhere and everything.

Speaker: You know you're doing well as a startup when you can afford that kind of sponsorship deal. It's not that expensive.

Speaker: There is something of a theme running through some of our guests, having or owning ducks and cats. Is there? Ducks and cats. Yeah. Yeah, we've got a few other duck-owning guests in the history. Interesting. Let's not get too deep into that because we can go mallard crazy. But ah it has been. i was checking. It has been about two and a half years since you joined me.

Speaker: Took me back to school on the internals of how a database works. Yeah, it's ah it's been a while. Yeah, we need to... So for those that haven't watched the previous episode, let's let's just start with a recap of what's DuckDB and why does it exist? Because aren't there a lot of databases, my friend?

Speaker: There are many databases, but obviously they're all wrong. um but So, i in DuckDB, yes. So DuckDB, we like to describe it as a universal data wrangling tool.

Speaker: okay We have thought about this quite a lot, what it is, what you think it is. But this idea off of giving people confidence to work with data, that's really our guiding principle. um And it's also why DuckDB was built from scratch, because we really didn't want to do that.

Speaker: um But we we realized that there was really not a lot of database developers that had sort of put user or developer experience really first. And so that was really our principle that we that we started DuckDB with, was that this needs to be trivial to set up.

Speaker: This needs to be like friendly in these error messages, and it's you know and and it's to seek the dialect and all these things, and like principle of least surprise, and it needs to you know not have the most conservative defaults, but the most reasonable defaults and all these things, right?

Speaker: And when we sort of looked at these requirements, we realized that we really couldn't just take something else and sort of repackage it. It was it was really the... it It went very deep.

Speaker: um And that's why we... About eight years ago, um we we ah started on this lovely journey. Mark Rasfeldt, the coach co-creator of DuckDub and I, we started on this journey of building a database from scratch, which is a bit of a daft idea in retrospect, but...

Speaker: We have a lot of such lunatics come through these doors. that Yeah, that that that that does. Yes, yes. I mean, I've seen some of your episodes, of course, so um I'm aware. But but it is it is um it is definitely something where people told me it's going to take you 10 years. And I was like, nah. And now I'm like, yeah, it's URI. Yeah. I thought this was to be done in two years, but they they were they were correct.

Speaker: Funnily enough, I've made that mistake as well in projects, and I've realized only bad projects have a fixed deadline. The good ones keep expanding, right? It's also the case ah for system software like databases or operating systems or things of that like sort of statute ah is that um they are never really done. right Stuff changes.

Speaker: like There's no hardware. The operating system changes. The compiler changes. glibc changes. there is there is um There are a lot of reasons. I mean, deep down, i want to kind of i really love the thing that tech is doing, the typesetting thing, that where where it's considered done. And the version number approximates pi better and better over the years, which is kind of crazy. um But it's not it's not semantic versioning. It should be burned at the stake. but um but ah

Speaker: But the idea that software can be done, I really deeply like that. But with databases, not true. I've got to push back against that. i think I mean, like a bad project, no one really uses it. It achieves all its goals and no one needs more from it. It's a good project that people keep saying, what if we did this? I need that. You need to be supporting this platform, right? The good projects never end.

Speaker: That's fair. we did We did do a bunch of stuff in DuckDB that we don't have to do everything, right? So it's not a monolith. It's like ah you they have all these plugins, you can extensions, we call them, that you can install and build and whatever. So that there is a chance that we will get to some sort of stable state with the core itself. But I'm i'm ah also very a realistic person, so I think yeah you'll people will be doing this for a while.

Speaker: yeah Fine. I'm fine with that. ah So I want to pick you up on the thing you just said about usability, right? Because I think it's a very difficult bridge to construct. And I do think you've constructed it. i want to know how.

Speaker: Plenty of people would say, let's make a database that's user friendly. and they would succeed, but it would be technically scraps. I'm not going to single out any particular databases, but there were some, particularly in the NoSQL movement, that were incredibly easy to use, didn't work very well, didn't actually save your data to the disk and things like that. And then you've got academics like you saying, oh, we should do a database properly, and they don't succeed in in usability at all.

Speaker: Yeah, I think i think there's that's an excellent point and I agree with you. I think one one of the tragedies we observed when when before starting DuckDB is what that the state of research and sort of the state of practice was about 20, 30 years apart from each other. Yeah, um life.

Speaker: And indeed, we saw exactly what you described. um So I think the reason why we managed this is because I am actually a bit of an in-and-outsider. I'm a bit of a sort of a weird, weird, what do you call this? Ugly duckling? Like weird, weird, weird duck? don't know. There's some expression in English. A strange duck.

Speaker: yeah Thank you. A strange duck. ah In Dutch, it's the frame they ain't. But um yeah, they have it too. but um the um So I didn't come from the database orthodoxy. i've worked I did my PhD on something in distributed systems.

Speaker: I worked on like ant colony optimization. It was story for another time maybe. um Yeah. ah and then Now I know why those ants are so much more efficient these days. ans are Ants are amazing. um But ah then this like weird thing in my life happened where I moved to Amsterdam and I needed a job. And I was like, all right, i am in a gigantic overestimation of my own skills, I was like, okay, I have used the database before.

Speaker: i can probably work at this database research lab here that has apparently a ah but job opening. Well... yeah um to ah so So they ended up hiring me, which is still I still am amazed they hired me. I don't know why. um And I actually led that lab later and I wouldn't have hired me. So yeah you know it's one of these car crushes of history.

Speaker: it's one of these like car car crashes of history um But the result was that I wasn't i wasn't educated like everybody else because the a PhD in databases is like very structured thing.

Speaker: You go to this seminar, you go to those conferences to talk to those people. These are these are the senior people in the in the community. And there's a sort of orthodoxy that's instilled in everybody. And user-friendliness is not one of them.

Speaker: I can imagine it's not It's not in there at all. And and so and so I think that was why i was... Because I was like i wasn't really outsider. mean, it's still a computer scientist. I'm still a nerd. I still like stuff, right? And I still appreciate the orthodoxy of databases. But it is not absolute to me. i feel like i feel like like this idea that you make something that needs to be sort of... has some have have Okay, maybe maybe that's maybe I should say, if you think about research, you want to have impact. And I realized very early on realized very early on that in order to have impact, you need to appeal to as many people as possible.

Speaker: ah And how do you do that? Well, by being user-friendly, right? um and and and And that that is like really a a great sort of driver because it has benefits in every direction. Like your papers get accepted because people know you.

Speaker: yeah you get grants accepted because because you know you can show that but your software gets downloaded a gazillion times each each day, then that's a pretty good impact statement, right? um Your papers get read, hey.

Speaker: and So so there's like there's really a beneficial sort of, it's like ah a vertical, a virtuous circle um when you go for maximum impact. And I mean, I had i had i got sort of got grumpy department heads telling me what my H factor was down the toilet, you know, but I'm like... H factor? Yeah.

Speaker: Yeah, this is the thing. I've heard this term, H-factor. This is something academics do. It's terrible. So, okay, minor rant. If you are a manager of multiple people and you are too lazy to actually, you know, make a call on who is good and who is bad, then you come up with some metric, right? Right. And one of those is, the i think it's called officially H-index. I don't even know what it's properly called because I don't care about it. But it's like ah a way of looking at your publications as a researcher and then how many times they are cited.

Speaker: And then there is some sort of, some math going on to average all this into a single number and that's your worth as a researcher. Right.

Speaker: Right. Yes. a bit questionable maybe. How can we turn people into numbers that we can sort? Yes. and and then And then your department head, you know who to give the promotion to, right? It's great.

Speaker: Yeah. And more importantly, your boss can't fire you for making the wrong call because you backed up with facts. yeah fact I mean, it's not at all so step susceptible. this this This metric is not at all hackable, right? like we Nobody would ever do anything like that, right? Like this is absolutely...

Speaker: So career-wise, you've just destroyed your H-factor and gone into making user-friendly databases. Correct. i it was like It was a gamble. I was absolutely a gamble. If this hadn't worked, I would have had to find somewhere else to somebody else to work and research.

Speaker: Amazingly, it worked. And yeah, OK. But that's a different story. It was a gamble, for sure. But I only took the gamble when they couldn't fire me anymore when I got tenure at the Institute, which was kind of the moment that I could. Oh, yes. I've heard of tenure. It sounds like a lovely thing.

Speaker: it it's it's It's a very hard thing to get, let me tell you that. it's yeah It's rough. i' I'm trying to get tenure as a podcast, but apparently they haven't taken my application seriously yet.

Speaker: No, no no help the government should maybe get in on this. anyways okay so so Anyway, you've you've you've started to break out of that management loop. um you so We talked a lot about how this was built. but I think one more thing as a recap. So DuckDB is a sort of local, just point it at a file analytics database.

Speaker: What makes that the choice you went after? Because there are lots of kinds of databases and you just said you had a background in distributed systems. You could have done like peer to peer network database. Why go for something that's kind of small?

Speaker: um That's a great question. And I think that was also because there was actually nothing in that space. One answer is there was lot of choice. Back in 2016, 17, there was a lot of choice on distributed

Speaker: sixteen seventeen there was a lot of choice on distributed stuff ah yeah Why make another one, right? um And remember what I said about this needs to be friendly and easy to use. like Distributed systems are the exact opposite of user of being easy to use.

Speaker: like it's an I mean, I have set up these things in my per myself, and this is an absolute nightmare, right? like you're you're You're installing software on 32 computers at the same time, and then one of them says, oh, I don't like your permissions, and then everything crashes down, and you have to sort grab logs on 32 machines, and it's it's an absolute nightmare. um So the single node actually, i think, followed also from from this simplicity sort of a requirement.

Speaker: But it also followed from this insight that people went basically straight from Pandas to Spark. Because the most competent single node sort of data analysis thing there was was Pandas at the time.

Speaker: o And I mean, Pandas is obviously amazing and you know Wes is a friend and we talk, ah but I think he was also would also admit that it's not it's not ah particularly efficient.

Speaker: ae um and yeah And I wouldn't say unless you're very comfortable with it already, it's not that accessible. No, the the the API, that's another discussion. Yeah. um I mean, SQL is also can also be clunky. I mean, we can there's there's there's ah there's pros and cons here. I think the lack of an optimizer is the bigger problem.

Speaker: I think that the problem in Pandas is that you are the optimizer. Oh, yes. Yeah, yeah. Great great thing. We just have another job. But um but I think there's nothing... like If Spandar stopped working for you, that that that was it. You had to go to Spark with all the problems that come with distributed systems and cost, you know.

Speaker: um So we also realized that there was a lot of space between, you know, where Pandas ends and where the capabilities of your machine end. There's a huge gap there. And that's where DuckDB is, of course.

Speaker: um So DuckDB can really we make the most out of the hardware that you have. And hardware has become quite amazing. It's an absolute tragedy. We use ah like a MacBook you know with an M4 or whatever to run Chrome.

Speaker: this thing can do you know This thing can do serious data crunching if you let it, if it has the right software. And so that's also something. so ah Yeah, I mean, we talked a lot about this in the previous podcast, but I still didn't quite believe it until I tried it. But you can crunch, the you can aggregate like two billion rows in a second yeah these days with hardware and the algorithms we talked about last time, right? Yeah. that's And yet it still seems like a struggle to get a spreadsheet up some days. We're doing it wrong. we have ah We have one of our most popular extensions, DuckTV, is the Excel reader, which I think is hilarious.

Speaker: so So you can point actually point DuckTV at your spreadsheet. It's it's its like... Oh, yeah. and as like As a table? So you can say select star from foo.xls?

Speaker: yeah Absolutely. Yeah. Oh, God. I didn't know you could do that. Okay. I'm adding that to my... Because I use that sort of thing. one of the things I like about DuckDB is you can query a CSV or a JSON file directly yeah as though were a table. Didn't know you did Excel as well.

Speaker: I you still think it's a community extension. like i think I'm not sure who built it, but it exists and it's very popular. So ah maybe I should point people at the previous episode if they want to get all the details of how we get to there.

Speaker: But we left the last episode with DuckDB as a local, very fast, very easy to use, um single file database.

Speaker: To me, it was SQL lite, but for analytics. That's absolutely the tagline that we went with. ah And I think it's still a good tagline. i think I think that we may want to talk about the path or the the journey so far since then. But I think it's it's now it's changed a bit like more into an ecosystem, I would argue.

Speaker: like it but It used to be the SQLite analytics, and now people call it the duck stack. The duck stack. The duck stack. Yeah, yeah, yeah. And that is kind of that kind of is that that kind of that is that is true, right? we We did a lot of outside like the single node ah database. Of course, that's still a very core thing we do, and we still push it like crazy. But it is but we have done more things. We have sort of put tentacles out this is from this This is exactly where I want to go, right? because um one of So one thing you decided to go from from that was like a client-server model with Quack? That was actually quite recent. Yeah, yeah, yeah. That was something we... we Yeah, Quack, yeah. I mean, obviously has to be called Quack because how do two ducks talk to each other?

Speaker: Makes sense. They quack, right? Yeah. yeah um We do have a rule in the company, no stupid names. And then I said, OK, we should call it something more boring. And then our some of our dev rel people were like, no, absolutely not. This needs to be called quack. I was like, OK. So yeah, so this is super interesting because ah we actually wrote a research paper way back when, Mark and I, about how terrible database protocols are with pre-DuckDB.

Speaker: OK. Are they? It's actually one of... They are. Oh, you have no idea. No, I want some idea. Tell me what's wrong first. All right, all right, all right, all right. OK, so let me go back to our seminal 2016 paper.

Speaker: um So basically, what we um we had one of the things we were investigating is why people hated databases. um And one of the things that was brought up, it's extremely slow to get data in and out. And we thought, hmm, that's interesting. Why?

Speaker: And so we we started looking at the actual ah client server protocol that all the databases use. Because if you have a client server setup, then get data getting in and out has to go through set protocol, right?

Speaker: And so we did it we wrote a research paper. at It was published at VLDB, very prestigious conference in databases. um And the paper investigated the actual sort of protocols that basically all the major databases were using. So for example, we actually did Wireshark.

Speaker: That was fun. So Wireshark is this sort of low-level protocol analyzer that you can use if you want to sort of reverse engineer your egg timer or something like that. but um But so that we could use, we could use that to reverse engineer database protocol. So reverse engineered like the Oracle protocol, because there's no specification available. So we just reverse engineered, described in the paper. And we we did look at like the Postgres protocol and the MySQL protocol and the, I think MongoDB protocol and ah Oracle and a few others.

Speaker: And what we could show is that they're horrible in the sense that that it is um you are better off just piping the CSV file over the socket than to use any of these bespoke protocols by like a factor of 10.

Speaker: um Oh. Yeah. Yeah. How are they so bad? What are they doing wrong? Well, there is... So Postgres, i mean, maybe let me rant a bit about the Postgres protocol because it's actually quite popular and it's something that irks me a little bit because lots of databases also use the Postgres protocol, even though if they're not Postgres because you can there's there's very nice properties you can use their clients and you don't have to write your own clients. It's very annoying to write clients. Trust me, I've done it. um ah But the Postgres protocol in particular does something like where every row you send back from a query result is its own protocol message.

Speaker: So it has like a header, a length, a type. um And on every column, ah on every row, for every field in the thing, it has like ah has like bytes in there say which data type it is, whether it's null or not.

Speaker: And then there's the actual byte data, which is encoded in sort of a ah binary encoding. This is interesting because it's a relational database where, in theory, all the columns have all the row all fields in a column have the same type. right But because so if it send them if you send back a million rows, every single row tells you what the schema of that row is.

Speaker: Correct. which you could have just done once at the start, and you could have skipped because you already know the schema of that table anyway. Yeah, and i I would say you should have done once. um the reason that The reason they don't do that is because of Postgres' obscure roots as an object-oriented database where every field could be a different object, and that's why they do that.

Speaker: um It's, of course, completely insane from a today's from today's perspective, right? But that's one of the things. So they basically they double, they they already, for an integer, four-byte integer, they send another four bytes of crap.

Speaker: pointlessly. um Then, yeah, as I mentioned, they have ah they have a header for everything, so and everything is a separate message. And and and so the result of that is that's just horrible, the the protocol interaction. So um this is also... and And then also the implementations are not great, right? Like the There's like the the client will process the row at a time because it's a message and it's like just switch has a big switch somewhere where the message time is like. And so so if you shove like ah it's fine if you shove 10 rows over this protocol, it's fine if you shove a couple of hundred rows over this protocol. But once you're looking at analytics and we specifically looked at this from the perspective analytics, um like once you want to transfer a million rows, you are absolutely screwed with these protocols.

Speaker: Right. so So that in itself seems... No, I can imagine that's very stable. Let's call it stable. yeah So that in itself seems pretty easy to fix if you want to add a new protocol to DuckDB.

Speaker: Right. And that was actually the cool thing about Quack because we could sort of design a protocol in 2026. What's more surprising is I don't really see it as a client-server model, DuckDB. That is an an excellent point because you can do much more than that. ah But but the the i mean obviously, when we build something, we think about like what what is the immediate thing we want to solve and what other thing you know what other things do we want to enable down the road. um So Quark, maybe I want to let me mention briefly the motivation there. um

Speaker: So we had always said we are a single node in process database. We don't do client server because it's because we have shown ah conclusively how client server is, let's say, questionable. um But people people what happened then was that lots of people actually bolted on their own client server protocols on top of DuckDB.

Speaker: And there's at least five or six projects out on GitHub where somebody, and there's bunch of companies that have done that. So everybody basically invented their own little RPC mechanism to talk to DuckDB.

Speaker: And then other people bolted something called Arrow Flight SQL to DuckDB, which is like an attempt of the Arrow project to make a database protocol, which is, in my opinion, flawed, but an attempt was made. um And ah so we we sort of watched this and then we were like, you know, at this point, we prop it if all these people are doing this, there's probably demand for it um yeah and yeah Yeah. And at some point, at some point i'm i don't I don't need to be right.

Speaker: I just want to solve people's problems again. right like that's That's the orthodoxy versus the pragmatism sort of divide that we talked about earlier. like At some point, I was like, yeah, i mean I have been out there and said, we won't do client-server. And oh my God, how can I live with myself if I now go back and say, by the way, Here's Client Server for DuckTV. But yeah as i start as I kept thinking about it, I was like, you know what fine. i don't i don't its Consistency is overrated. I want to solve people's problems. If that means I have to sort of eat my words and and and you know and and be wrong for once, that's totally fine.

Speaker: ah Yeah, yeah it's it's our job to solve people's actual problems, not the problems we think they ought to have. Correct. And I think that's i think that this this is a very nice way of putting it.

Speaker: If you tell people what problems they ought to have or how they ought to solve their problems, that's precisely the kind of mindset that that I saw in the database orthodoxy when I started working in in this field.

Speaker: like where they were basically always yelling at the users of like, of there you're holding it wrong, right? which yeah which is Which is a way, which is maybe a way of doing it. It's not the way of doing it, I would say. it's ah yeah And that's that's something where we have a very different approach. Like if we see 15 times the same issue report, we say, hang on.

Speaker: We need a structural solution here. We can't just keep explaining them how they're holding it wrong. We need to some ah somehow fix this structurally. We have to i don have a warning message that will tell. That was actually that was a good example. Recently, there was somebody, Rusty, one of the community but of the valued community members in DuckDB. He said, guys, if like for sometimes DuckDB doesn't have a defined result set order. For example, if you read from a source that doesn't define a result set order.

Speaker: um If you then put an offset without an order by... Okay, so SQL has this offset thing where you can say, give me this row give me everything from row 10,000 to row 20,000 or something. into the own If you have a non-deterministic input or input order and you have no sorting criteria on the under on the query and you specify an offset, the results are completely ah non-deterministic, which is...

Speaker: by definition. Yeah, it is. It is. This is very true. And I was was about to write back like, you're holding it wrong. And then I was like, and then I thought about it for a couple of hours and days. I don't know. And I went back, no, we need to fix this. And now we like we had a i pushed a pull request a couple of weeks ago where, you know, if this in this situation, it will show you a warning, say, hey,

Speaker: This is non-deterministic. You can make it deterministic by adding an order by, but just be warned. yeah We've told you, right? At the very least, yeah. And and and and that's that's the kind of thing that's that that that's the same thinking like with this quark thing. i'm not It's not about being right.

Speaker: ah it's about It's about solving people's problems With data. Yeah. But doesn't it open up a whole host of problems that you hadn't originally designed for? Because like oh yeah by the time you've got client server, now you need to worry about concurrent writers, which is where the whole problem just becomes databases explode at that point.

Speaker: um Actually, we had support for concurrent writers for a long time. um it We just didn't have support for concurrent processes writing to the same database file. different it's These are two different things. DuckDB has support concurrent writers from the different threads in the same process for a long time.

Speaker: um So that wasn't something we had to add. We had also i mean we had greatly improved this recently. but Now we have, like, you can commit while checkpointing, things like that. um The concurrent writer thing wasn't so much the issue with Quok. The problem, like quiet client server is quite ugly in ah in in in like in like the the failure modes, let's say.

Speaker: ah So we suddenly have to deal with the fact like, hey, you know the connection goes away. or the But it goes away in various ways. It can go away like that you pulled the cable, the other side stops responding, you're running on all these timeouts. You have to sort of, you have to clean up resources, but you don't really know when because the other guy might still be coming back or not. We don't know. um There is like exhaustion of of of like and like file descriptors when there's too many connections. Like you have all sorts of like interesting sort of systems problems that we haven't had before. And we spend quite a lot of time sort of trying to work around them.

Speaker: But like the client biggest problem with client server is the sort of the cyber surface. and Right. Well, DuckDB so far has no external sort of accessible port or anything like that. So like if if you couldn't like the the the Russian bots, bots couldn't sort of attack it from the network.

Speaker: ah Right. Yeah. or other countries, there's other countries that do malicious hacking. But ah but ah the um the um we didn't have that. So you know the attack surface for DuckDB was much smaller.

Speaker: Now we have a client server protocol, which means that you know this port is now hanging out in the open internet possibly. Now we have to be actually we have to be very, very, very careful to what you accept and how you accept it and that that you can't do like a a denial of service, you know what happens about authenticated attacks, unauthenticated attacks. So that was a whole new is a whole new sort of space for us ah that we that we hadn't really dealt with before. I mean, obviously, we are not you know idiots. We have we know sort of the fundamentals there of how how you could attack this, right? Like the classical buffer overruns, these kind of things. yeah

Speaker: ah sending malicious packets, these kind of things. But it is it is it is and it was a new surface for us, for sure. The other thing we had never had to deal with was authentication and authorization, two of the massive sort of endless black pits of databases. Yeah.

Speaker: Like authentication is like, okay, you know, is this user authenticated? That's kind of easy, but then the authorization... It's like, can this user read this table? It's like, oh my God. You've got to stick in a whole role-based access control stack, which enterprises love, but there's no fun to implement.

Speaker: And it's a huge, huge expected expected sort of feature set, right? So so that was something. like Postgres has a lot there, to their credit. um ah like As you said, Enterprise loves this kind of stuff.

Speaker: Enterprise, i' maybe another minor rant, Enterprise loves this kind of stuff. And then they complain that their queries are slow.

Speaker: like i can what I see this a lot where where ah where um you have like you have a trivial query and they the you know it take it would take a couple of milliseconds to compute, but to compute the writes, whether this user can do this takes seconds. So that's that's you know like this is also something that I tell people like, guys, you know but I know that your you know your compliance department has ideas here, but there's actual cost associated here.

Speaker: Have you run into that problem then? that Because like DuckDB is sort of built for analytics performance. Are you saying the performance has dropped just for the sake of security, or have you had to solve we had The way we solved it for now is we um we added callbacks that you can define.

Speaker: You can say, OK, so for authentication, you can give a callback, which is a SQL level function. that i I will pass the credentials the user has sent over the protocol.

Speaker: And your function can say yes or no. And if that if it says yes, it will it will let them log in. um So this is something that you can write an extension or you can even write a SQL level function that that basically decides whether somebody should be allowed to log in. That's part one. That's authentication. but Then for the authorization, we have had something similar now where there's a function that gets called before the query gets executed.

Speaker: with the query, with the user that's authenticated, and then your function can think about whether it wants to allow this user to do this action. So if we kind of abstracted ourselves out of there and said, we cannot solve all of these cases, we'll give you the tools to build your own thing.

Speaker: But obviously, if that thing takes a while, your query runtime will have that as a constant factor. There's nothing we can do about that, right? It's up to you. So it is it is um it is a very interesting problem. There are some there are some approaches down in in research that or in people doing open source that are thinking about this stuff that that are quite interesting that push some of this down into queries again. and you know, because some of these things become joins and predicate evaluations and stuff like that. But there is obviously a cost to it. I just find it very funny when the when the and the enterprise people complain about this sort of thing, because it's like, yeah, this is a you problem. This is not a... Anyways. Yeah. anyway yeah yeah

Speaker: I'm sometimes surprised that the whole thing isn't just another standard set of SQL queries. Because a lot of times, authentic authorization is recursive SQL, right?

Speaker: Yeah, that is a great way of doing it. um I think that's not typically how it goes. Typically, this is a completely orthogonal thing in ah in an engine, and I don't think it's a great idea. but that's And I think we are we are working on some some ah there's some ideas internally on how to um make this more a SQL-y thing.

Speaker: um but But yeah, whether this will... i don't I'm not sure yet where this is going. It's just say it's also not something that I am particularly excited about, honestly. i don't To be too quite honest with you, like every company is... if Any enterprise is enter enterprise-y in its different way, and and they have you know they have the money to solve this, so they they should just leave me alive. So you're just sort of giving them the access points to plug it? give them the way of doing it, but...

Speaker: yeah This leads to something we didn't talk about last time, but I wanted to know how it works under the hood, which is DuckDB's whole extension mechanism. Right. Yeah, this is very cool. um So, it's fun background story there. We built this extension mechanism because we didn't want to merge a particular pull request.

Speaker: so So, somebody who we knew sent a pull request, and we were like, ah, we hate this. And this was about... graph stuff in databases. And one of i have I have not so so positive feelings about graph query processing either. I think it's a bit of a distraction.

Speaker: um But we can talk about that. as it ah But anyway, somebody sent this pull request and we like, nah, we hate this. um So let's give you ah hooks in DuckDB so that you can do that yourself without bothering us anymore.

Speaker: Another solution to the you shouldn't want this problem problem. i kind of Kind of, yeah. And also to the feature creep as well, right? Like something like, for example, the Excel reader I mentioned earlier, right? Like not everybody uses this.

Speaker: So why should we bake it? There's other databases I won't name names ah that are just bolting every single feature that they can dream of to the single binary that they ship. And as a result, the binary is like a couple of gigabytes.

Speaker: Great job. um but But so we don't we don't want to do that. So we want to stay small. So we have these extensions. Well, the hooks are pretty straightforward. just like You can imagine every sort of component of DuckDB having like hooks in it that you can hope that you can sort of connect to from an extension. So you can for example, you can say, i've the simplest extension is I'm going to add a scalar function.

Speaker: but I have a function that computes, I don't know, the Ackermann function or something like that. right ah ah That does not ship with DuckTub. We don't have the Ackermann function, OK? As far as I know. um um But let's say you want to do that. Well, then you you make an extension. And that extension will say, I have a scalar function. And it's called the Ackermann function. And this is the implementation.

Speaker: um And then that you load that introductory B, and then it will be added to the internal list of function of the list of functions, and you can call it from SQL. That's functions. That's scalar functions. right get Before you go on, give me more details on that. So what do I do? Do I write a C function and give you the right binary in a header file or what?

Speaker: Well, you you typically you write against a ah a header, yes. There is ah there's a header, DuckTB extensions.h. um And um that will have sort of the the they'll have functions that you need to interact with DuckTB, for example.

Speaker: There will be a function in that in that API that says, I want to register a scalar function. Right? Right. And so you you call that function. And what does it take? Well, it takes a name. It takes like some definition of the parameters that that function takes. Are they strings? Are they integers?

Speaker: ah What type does this return? you know we are type-save-system, so we have to say to type which type. um And you will also give it a callback that has the actual implementation of the function. And so the callback then is ah has to conform to a ah function signature for scalar functions.

Speaker: um And that will then, you know, is a defined sort of thing where here's the parameter that gives you the input, like ah the vectors that data detectives. It's a vectorized system, so you would get vectors as input and you produce a vector as output as a scalar function.

Speaker: um And then there are some helper functions in the API that help you to extract, let's say, integers from this. input. um and i Say, okay, yeah loop over this input, produce give me the numbers.

Speaker: Then you do your own thing to compute Ackermann. And then you there's another set of functions that allow you to put that number back in the output vector. And that's kind of it. There's also other stuff that you know you want to raise an error or you want to and now you want to do other things. So so that's kind of that's kind of very straightforward. is it seef It's a C function.

Speaker: And we are currently moving. ah we had We had two APIs. We had like internal C++, plus plus and we had a public-facing C API. But we are currently moving to move everything to this public-facing C API because it gives us cross-version stability. It's very nice.

Speaker: um And so basically, you you program in this API. And this API is quite expensive. You can create functions. like You can create types. You can create other kinds of functions like ah you know ah aggregate functions or window functions or table producing functions. You can add settings.

Speaker: You can add optimizers, actually. You can add... You can add whole optimizers. You can you can add whole optimizers. You can add logical and physical operators. So you can do things like...

Speaker: im um And here's my new fancy join type, and this should trigger in this and this circumstance. You can ship your operator for it. You can ship your optimizer, detect that circumstance, invokes the ah adds the operator.

Speaker: There's lots of lots of things you can do there. um And actually as actually, it's interesting that you bring this up because in our next release in 2.0, which is coming in in in October, we think, we are finally adding the last bit of the puzzle, which is the pluggable parser.

Speaker: because we, wo yeah yeah, that's another paper we wrote two years ago, I think. Um, cause, because we had, we had everything in DactiB was already pluggable except for the parser because parsers are ancient technology from the sixties. Usually. Uh, uh, uh, Lalo one, you know, Knut himself has wrote, written about this. So you can kind of get an idea how old it is.

Speaker: Um, And so we had the last the last puzzle piece for extensions was, can you actually change the query language? And I'm i'm happy to report that with the next in the next version of DuckDB, you can actually change the parser. We've completely ripped out the Postgres-based parser we had before and and rewritten it into something else.

Speaker: Okay, going to have to give me some details on that. Because ah parsing SQL, it's like the final boss of parsing, I reckon. Ah, what do you mean? syntax it's It's a language that's obsessed with syntax. and Maybe Ruby is the final boss. No, I think you're absolutely right. SQL is theqquel is the final boss.

Speaker: because it's a language that is obsessed with syntax. It's like if you compare, like, say, SQL to Python. In Python, everything is a function call or an assignment or something.

Speaker: There's like 14 things something can be, and that's it. um And then everything the user does is just uses these abstractions to do something. um So there's nothing. like But in SQL, on the other hand, everything is syntax. like everything There's like hundreds of keywords.

Speaker: right like There's like having and you know using. and and you know like there's there's all these There's all these Englishy things in in SQL.

Speaker: So parsing SQL is actually hard. um And you get the thing where like sometimes an identifier can be just an identifier or a keyword depending on where it appears and what kind of statement.

Speaker: Yeah, and then unquoted identifiers cannot be reserved keywords is also really fun. So some keywords are reserved and some keywords are unreserved. The reserved ones you can never use as an identifier and the unreserved ones you can use in some cases. And all of this comes from a um from the apps. This is funny because, I mean, yeah, this is also something I worked on myself.

Speaker: um ah So i'm happy I'm very happy to talk about it. Actually, but now I have actually handed over to Daniel, one of our guys who's who's working on this now full time. But I wrote the first version. So, hey. okay um We need to get Danny and see he thinks about the first version. Yeah, no, no. He's um he's he's cool. um but ah But, you know, you would never get to his version without the first version. there's theres It has its place in history. um So...

Speaker: So the the way everybody does parsing is with yak and bison. I don't know if you've ever worked with yak or bison. It's been a while, but yeah. yeah Yeah, so Yuck is actually the name of the Yet Another Compiler.

Speaker: Yeah, Yet Another Compiler compiler. Thank you. um ah But it's also very yucky, if you if you want to make that joke. I think it's a good one. um Because it is really like the mummy of computing staring you in the face. Like this is stuff that's, I think there's the first paper about Yuck-like parsing is is from Knuth.

Speaker: And it is it is from the 60s, like 1963 or something something ridiculous like that. um Yeah, when you write when you write in Lex and Yak and Bison, you do feel like you've been time-walked back to the early days of C. Yeah, exactly. And then you have this... the problem is not the There's two problems. like One is the abstraction and one is the implementation. Implementation is this weird mixture of ah extended backers in our form.

Speaker: to define a grammar and there's like inline C templates that get expanded from the YAC compiler into an actual C file and and a big lookup table. And the so this one of the problems is the implementation.

Speaker: which is just by now we're using Bison, which is pretty terrible. If anybody like anybody wants to feel ah like everybody has ever seen a shift-reduce conflict, then I'm with you. Yeah.

Speaker: i I now know what this i know without what this is. It's funny because I had to learn all of that for writing the paper. It's like, what what is this actually? um But but ah ah the shift-reduce conflict, but the other problem is the abstraction. So LALA1 is a single look-ahead parser.

Speaker: You can only look at the next token to determine what this token means. Which for SQL is terrible because SQL has a lot of things where it's like i say, you know select X from y order by Z. You will notice that order and by are two tokens.

Speaker: Yeah. yeah So the Postgres parser, for example, has all these wonderful hacks where in the Lexer, they will glue these tokens like order by together again to one token so that you can detect them. um However, this will mean that you cannot have ah you also have to strip all the comments already in the Lexer because otherwise you you could have a comment with your order and buy and you would not be able to detect that it's order buy. Which again means you cannot have things like um annotations in in the Postgres parser. Because you might want to have like comments that give you annotations for you, that give hints to the parser or something. It's not possible because they need to strip the comments in order to detect the the two token tokens that they cannot detect later in the parsing step. It's a complete mess. Yeah, that's nasty.

Speaker: It is nasty. And this is a 20,000 line grammar with C interspersed a nightmare. And yeah, so we have looked at this a lot because we used to use the Postgres parser in DuckDB. And i mean to its credit, it worked.

Speaker: And it's a very complicated language to parse, right? There is a lot of surface area. It is. It is. It is a lot of... come And it worked. And it was only painful when we tried to change things. And as you may know, we did a lot of friendly SQL things where we'd say, hey, maybe we can do group by all or we can do... Select star is optional. You can just do from table. Like these kind of things, right? and that's why So we actually but actually i ended up hacking that parser a lot and therefore had to suffer a lot.

Speaker: And then the biggest problem was it wasn't extensible. Like this was a monolith that was baked together at comp compile time and our extensions couldn't do anything about it. ah yeah which is annoying because you want really, in order to deflect more of these requests of people wanting stuff from you, who need to be able they need to be able to also change the syntax because there are some people that want different query syntax. tears like the There's like some project, I forgot the name now, but there's like some some some project of Prequel, for example, is one of them where people have like proposed different kinds of SQL syntax, right?

Speaker: It's not possible in ah in the Postgres parser. um So we ended up rewriting the parser from from scratch to be accepting that memory is slightly bigger than it was in 63 now.

Speaker: um Yeah. Right. Yeah. ah and And that you know whether you take whether that parsing takes a microsecond or a millisecond really doesn't matter. right like there is there is there's We have some more sort of space in this whole thing now.

Speaker: Yeah, and if you have to hold lots of potential parse trees before you choose to commit to one of them, that uses up memory, but not much. Not a lot, no. yeah And so we've written this to something called a pack parser, which is a much more recent abstraction, which is also something Python has, by the way, switched to. They did this too. okay it's trauma They switched from a a couple of years ago, or maybe not that long ago, I think two or three years ago, like Guido himself...

Speaker: ah switch the Python from a Yuck parser to a Peg parser. um So we basically, and Peg is like a recursive descent kind of thing. It's like a greedy, yeah it's a different abstraction, but it works better. And the nice thing is you can modify that runtime. So now you can load an extension.

Speaker: into DuckDB and that can say, hey, here's my new parsing rule and here's my the transformer that tells you what i want to what kind of logical tree I want to make out of this parsed set of symbols.

Speaker: um So that's very elegant. I'm curious, what do actually Is there some peg library I choose what do you actually use you is there like some peg library i choose for c There is. That's what we use for the prototypes. There is ah the Peglib that's actually very nice. um Okay.

Speaker: Peglib, good name. I think it's called Peglib. um ah But we ended up writing our own um in the... end It's not complicated. Really, it is not. like the the The grammar has like five different constructs and it's you can you can make ah you can really make it trivial. I wrote a trivial...

Speaker: interpreter for it. It's not crazy. i We just ended up writing our own pack interpreter. ah that's yeah it's not It's not so wild. It's a recursive thing. So we evaluate like, oh, I see an alternative symbol here. So now I have to to send these two trees at the same time. I see a multiple times kind of statements. So, okay, so I can invoke this multiple times. There's some sort of optimizations because you don't want to have like a, like you want to this very easy to hit like recursion depth limits there. so you have to be a bit careful with, um with how you manage this kind of state.

Speaker: We're probably still going to rewrite this into an iterative algorithm just to manage. Because one of the wonderful things about C is that, or C-like language, is that you can run out of of stack space. ah but ah But yeah, and I still don't know why. But you can. and and um And so we are we if that we might we've rolled our own. But it is it is really just a normal peg. ah ah yeah grammar that and we've written a sql We've actually done this ourselves. We've written a a SQL grammar for a pack parser that we shipped with DuckToby that actually all the people already have started using, which is kind of cool. Oh, it's open source. so Of course. Everything is open source. Yeah. yeah yes

Speaker: um So we have this Peck grammar now and we have this parser for it. And I've actually heard, and I'm not allowed to say who, but I have so heard that another major major database ah is interested in adopting this parser.

Speaker: So it's it's also something that might have impact beyond DuckDB, which is ah interesting which is ah which is very is very cool to to hear that. like It's like, what, you want to you want to rip out your parser? Really?

Speaker: Okay. That's a show confidence. it's It's, yeah. now it's it's i mean And again, Something I've learned really amazing. Similar to like nobody had ever written a paper, a scientific paper about protocols, like the Klein protocols you just talked about.

Speaker: Nobody had ever written a scientific paper about parsing. And you think there are hundreds and hundreds of joint papers, right? Oh, you mean specifically parsing SQL? For There was never it was never a paper in databases that talks about parsing SQL queries. And I am i am i am amazed by that, right? That's astonishing.

Speaker: ah We wrote it, obviously. um But ah ah but it' it's ah it was amazing to see that. like This is my favorite thing in the world when you... When you, as a researcher, you walk around on this ancient field you know that ancient battlefield of databases, where you think that every square meter has been bombed to bits. But there's these whole areas that are completely ignored. right like This is this is ah something that I find amazing.

Speaker: And think the reason is because people consider it trivial. Well, it's not. no No, that seems like an excuse for no one wants to open that can of worms.

Speaker: Or they they never think about it, right? like I mean, as a database person, I can tell you, if you think about queries as trees, like that syntax thing, that's more like, ah yeah, that's how the how the monkeys talk to us. but But I'm not saying I do this. I'm just saying that's the the sentiment I have observed. Yeah, yeah. yeah okay is there Getting back to user space then, is there if you don't anything to kind of mitigate the danger that someone can create an extension that completely banjaxes the SQL?

Speaker: like no They can break the parsing? They can. You might want to actually restrict the query language that you allow. That's actually a cool feature.

Speaker: Because you could say yeah you could even say, hey, I only want you to be able to do filters on this table and maybe projections, but nothing more, because I want this to be allowed. I want this to always force a streaming execution. You can't do aggregations, by the way.

Speaker: You can ship a parser rule that just disables everything else, and then you cannot create those plans anymore. I think it's a feature. ah Yeah, OK. But i mean, people generally are responsible which extensions they they install and if they want to break their SQL syntax, you know by all means. Yeah. And if you want to load an extension, that comes with a certain acceptance that the author of the extension is in the right. Yeah. Well, that's ah yeah.

Speaker: yeah It's like a Python package. I mean, fair in fairness, a Python package cannot maybe break the parser, although I would probably manage to build one if I wanted to. um yeah it's ah you know It's all the same address space. You can do whatever you want. Yeah, I'm just wondering now how I would do that and I'm sure it wouldn't be hard.

Speaker: No, it's like probably have to find some symbol that they don't export, but it's there. it's this like If you're listening at home, don't do this. It would be fun to you know to mess with your co-workers that don't understand computers fully. Yeah, it's an April Fool's thing.

Speaker: Yeah, yeah. okay It's like there's this old C joke. It's like, define true false, you know? It's like, hide that somewhere in your include files and then, you know, watch the world burn. And that just as you leave the company. Yeah. Yeah.

Speaker: yeah We do not condone this behavior. We don't condone, no, no. Okay, so you've got um you've got quite an extension mechanism built up and extending the parser.

Speaker: That leads me to one specific extension I wanted to talk about. car Because you're in the analytics space, it's something of the hotness in the analytics world at the moment, which is iceberg.

Speaker: o I know since we last talked, you got Iceberg read and write support, but what I'm not sure of is how DuckDB plays in a world where it gives up the analytics to a larger kind larger distributed system.

Speaker: Right. um yeah Yeah, Iceberg is is interesting. um It's something that I think was also one of these things where we looked at this when it first bubbled up and thought, what is this?

Speaker: um And then the people kept asking for it. And then we had to, again, you know go back on our convictions and said, fine. um But we're going to complain the whole way.

Speaker: um So yeah, for I mean, Iceberg is this idea. is this i mean it is It is a great idea to to to have to fix this problem of naked Parquet files on S3. um Being the way how you store data, like this whole data lake idea, right? Mm-hmm.

Speaker: And so this is this is one ingredient. And the other ingredient you need and is change. like People always treat change as an afterthought, right? It's like, oh, the data has to change. Oh.

Speaker: And then... and yeah how Having spent a lot of time in the kind of real-time change tracking world, yeah, it is often an afterthought at best. Correct.

Speaker: Yeah, exactly. Exactly. I mean, that's... And also the academics are guilty of that because we we do static benchmarks a lot. It's like, ah, it works fine. Moving on. But like ah but the ah but the changes, changes changes now everything flows. We know this. ah The Greeks knew this. um So ah so so it's ah so it's um this idea of Iceberg is to some somehow have a sanity in changing data that sits on a data lake. right You have a bunch of parquet files on data lake.

Speaker: You want to be able to add ad file add rows to this. You want to able to be able to delete things. You want to be able to update things. want to be able to change a schema with some level of sanity. And just to fill in that gap, parquet files being let's store everything in a column-oriented fashion for analytics performance.

Speaker: Right. Yeah. And then the the iceberg thing on top is the catalog that says where all the data is and manages adding. not and Yeah, no, not really. There's two layers there, actually. there's The first is the change management lever. That's another set of metadata files that sits on top of these parquet files.

Speaker: um The worst format in the world, it uses Avro. um I don't know if you know remember Avro, but it's from the Hadoop. Yeah, you do? Yeah, we was well might come back to that. We might have to have that debate, but carry on. I have some randest there. So first, on on Iceberg, it's two things. Iceberg is first, this metadata level of files, metadata stuff.

Speaker: that tells you about which versions exist of that table, ah which files are part of it, which schema exists, that kind of stuff. And then on top of that, you have the Iceberg catalog stuff.

Speaker: And those are not files, that is a service, right? And that's like a REST API kind of thing that indeed does tell you which tables exist and where they live and um gives you credentials to maybe read from S3, things like that, right? Like there's there's two layers there. And actually the file metadata stuff was first and later on they tacked on the... um the the REST API.

Speaker: It's also called the Iceberg REST API. That came later. um yeah So that's kind of Iceberg. And yeah, it's quite popular. People like it. I think the reason people like it is they don't want to be locked in. But in principle, the idea that people want to their data and make sense of it and it not being locked up behind. sometimes be able to query it from lots of different places without having to move all the data from vendor A to vendor B.

Speaker: And without having to pay, like if you do a full table scanning like Snowflake, it will be expensive, right? Like if you if you do this if you read the same files from S3, it will be much cheaper because in the end, Snowflake is just ah ah a margin on top of Amazon cost, right?

Speaker: but um I feel I have to declare I have worked for Snowflake in the past. So i I'm staying out of that. i like it I like these guys. they They have a very competent system. Don't get me wrong. It's just i think the pricing model...

Speaker: of companies like Snowflake and Oracle has kind of created the lake house formats in the first place. That's just my my theory there. Because they were at quite expensive.

Speaker: But anyway. yeah Yeah, their whole business model, we can certainly say the business model is value add on top of the data. And then you can debate on how much value and how much add.

Speaker: Yes, exactly. Exactly. Yeah, so that's that. ah But we we looked at Iceberg. Yeah, sorry. So you wanted to get into that world of Iceberg is becoming the lingua franca of analytics data or? Again, we were kind of pushed into it by people.

Speaker: i think it was like the most requested feature if for like two years straight. And then um we also worked together with bunch of companies that ended up sponsoring a lot of the Iceberg work. um And thiss like this is also how our company, Doug Labs, was kind of operating or is operating, is that if you if you care strongly about a feature and you're a company that has lots of money, then you can pay us and we will prioritize that feature. So that's how it happened with Iceberg. Right. um um So we got kind of begun we worked on it. And we also built it. I mean, again, i' wrote wrote the very first prototype of the Iceberg integration for DuckDB myself, which is how I learned a whole lot about Iceberg. like it's a yeah it's this That's the way to do it. That's the way to do it. ah Do you know what? if If DuckDB can read natively from an Excel file, then reading from Iceberg isn't really something you should exclude. right

Speaker: That's true. And again, yeah, absolutely. Now we we should be able to read from Iceberg. And I think think we have actually, i think we're getting close to be the most feature complete open source implementation at this point. Really? It's quite bold claim.

Speaker: I know. 2.0 will have it. I think blog post is in the in the works. We are, think we're getting there where we actually, the we have the the sort of the most check marks in the big, matrix of of iceberg features. And because it has been so important and we have actually have a team at this point working on this here at our in our offices. So so we have we have really spent a lot of time. We spent like three years on this at this point.

Speaker: of Wow. is it one of those thing So you've done it as an extension rather than in the core DuckDB. Is it one of those things where you found limitations in the extension mechanism? I think the extension mechanism is far fine for Iceberg.

Speaker: what we What we saw was that um there were some sort of other infrastructure items that were relevant to get good Iceberg performance that are like not an extension related thing, but more like a general thing. For example,

Speaker: um we needed to, for 2.0, we're also adding asynchronous I.O. um Because these Iceberg files are typically stored on object storage like S3, and to get good performance from object storage, you kind of have to have asynchronous I.O., which basically me just means that you're you're not blocking your database engine on on reading from from network, from HTTP, ah but you're have a separate sort of worker pool of of threads that keep the pipes full of data for the other thing to process. So that's something that we've um something that we added. that's like it's more like That has has nothing really to do with the extension mechanism. It's more

Speaker: it's more the um it's more the yeah you know the general engine behaving differently because this use case in Iceberg is often like this remodeling. That's in the extension or that's gone into the core?

Speaker: That is in the Parquet Reader. The Iceberg extension uses the Parquet Reader to read Parquet files. um And the Parquet Reader now has this async stuff, so it will just be faster and that benefits everybody that reads Parquet, including the Iceberg extension. One thing that we had to figure out for Iceberg is how how extensions have dependencies with each other, which is absolute nightmare. Because Iceberg obviously uses Parquet, the Parquet extension. We're not going to duplicate the Parquet reader for Iceberg.

Speaker: Fun fact, by the way, Spark, in its infinite wisdom, ships two Parquet readers, one to read Parquet and one to read Parquet pals it files as part of Iceberg. Why? Ah, no idea. um Is that Conway's law, two teams not talking to each other within Spark? My prediction is that the people that built Iceberg, the tabular people that got acquired by Databricks, they um they needed to move and the Spark people weren't moving.

Speaker: And so they just said, fine, we'll just ship our own completely independent from scratch reader for a parquet. as part Which now is the funny thing that if you read a parquet file from from Spark, you can have different results whether you read it as a straight parquet file or you read it from Iceberg.

Speaker: No. um It's 2026, and computers are still broken. Oh, yeah. Right. But um but we didn't do that. So we we added dependencies between extensions. So the Iceberg extension uses the BK extension. We also have an extension that talks to that reads Avro files, um because obviously Iceberg uses Avro files for the metadata. It also reads also uses JSON.

Speaker: Because it wasn't it wasn't enough to have Avro files as a metadata format. It also added JSON as a metadata format. So then the Iceberg extension also depends on the JSON reader. And we had to figure out how to do extension dependencies, let's say, to get this Iceberg stuff to work well.

Speaker: But it works. It's fine. ah it' Okay, then I'm going to pick you up on your Avro thing because you said the problem with Postgres and lots of database protocols is like they're putting the type in every row, right? this is one of the things that Avro completely fixes. It like puts all the metadata first, then just streams out compact binary data. I would have thought you'd love Avro.

Speaker: No, I don't. Regrettably not. um I also built an Avro reader from scratch, by the way, to understand Avro. And I've also built a Thrift reader from scratch to understand Thrift.

Speaker: it's ah How do you still look so young? You should look old and haggard by this day. I shaved today. ah I'll do it.

Speaker: Now, I love serialization. It's one of my one of the joys in my life is to figure out how to take multi-dimensional data structures and put them into a one-dimensional storage medium and back. It's great. its somehow Somehow, I love that.

Speaker: ah but then but then it's i just ah and And so that's, i mean I guess, why I spend so much time on this stuff. But Avro was built for Hadoop, right? It comes from that world. it was It was originally built as an RPC format. And later on, somebody invented a file format around it. And indeed, you have the you have the metadata that contains a schema as JSON for some reason.

Speaker: um like One of the things that bothers me, if somebody invents a file format that then depends on another file format to describe something that could have i been done in the file format. But OK. um So ah so they invent they they use JSON to describe the schema. Fine.

Speaker: um But the the problem with Avro is that um There is like every value still has a type prefix because the um because um every field, like there there's like there's like things like unions and structs and they have to say which field is actually there and which is not. So there's still a ah byte that will will will tell you which which which which field is actually there or not. That's fine.

Speaker: My problem with, a structural problem with Avro is that you cannot read it without a schema. um Whereas a Thrift file, you can read it out the schema.

Speaker: they they speak They were such byte bit pinchers that they removed the structural information that would be required to decode it without knowing the schema. And the Thrift guys have that.

Speaker: And Protobuf, by the way, as well. And so that means that an Avro file is completely useless without the schema. and you cannot build a generic Avro reader, which would be really nice. but Especially if you... if Right. So that is my gripe with Abro, let's say. There's other sort of problems like then they they they chunk a bunch of rows together in one compression block in the file, which means that de facto to read a single row, you have to decompress that whole block, which kind of defeats the purpose of a row-based format, at which point you might as well use Parquet.

Speaker: It's a, it's a, there's some, like the idea, the the idea is right, but the, in the, in the execution, let's say there were some, some, some, some problems. But yeah, it is, it's what Iceberg uses, which is why we have a reader for it.

Speaker: um This almost makes me wonder if you've come up with ah the duck format for solving these. but You clearly have strong opinions on the way that data should be serialized.

Speaker: Why is there not.duck? We have a DuckTB storage format. DuckTB has its own storage format. But do you have like a wire protocol format? Ah, yes. Well, Quack, yeah, sure. Okay.

Speaker: Quack has that. Yeah, yeah it it uses our internal serialization for sure. um Yeah, but but Iceberg, I mean, because we looked at Iceberg and we saw some issues and um we ah think the biggest problem with Iceberg that I see is that they so tried so hard to not have a database, right? So all this all this change tracking metadata is in these Abro files and the JSON file.

Speaker: And they tried so hard to not have a database because databases don't scale. Their words, not mine. um and And then they slapped this Iceberg Rats catalog on top of it, which has a Postgres in it.

Speaker: Yeah, yeah. Which is which is the was the point where we said, So this thing exists now in your sort of diagram of components. Why don't you just forget about this whole metadata tracking file nonsense and just use the database for it? And that is exactly what Duck Lake is, right?

Speaker: like it's that Yeah, yeah. yeah Yeah, but you have the database already. You just added it to the to your to your Visio diagram. Yeah, except that you've got a database, now you should pull all that metadata and actually use a proper database. they never understand and yeah And that's the funny thing. so we did this with Duck Lake. We said that the the quote was, we have a database and we're not afraid of using it. It's by the way, a Shrek quote in my mind. I have a dragon and not afraid of using it. Oh, okay. um Yeah, it's great film. But but um the... So that's kind of the idea of Duck Lake. And the funny thing is that the that it seems to be the case that the iceberg world has grudg grudgingly accepted that this is a good idea. And I see sort of movement from them to go more into that area with the yeah iceberg REST catalog query planning API that's also trying to pull a lot of this metadata out of these Avro files into the centralized thing to be able to do query planning without having to read

Speaker: 1700 Avro files, right? like that's yeah that's ah That of an issue. Presumably, you get exactly the same optimizer benefits because you're doing exactly the kind of similar kind of planning for where to look for the data.

Speaker: Right. So DuckLake's query planning is a single query and in in the metadata catalog, and that's really elegant. and Yeah. So Duck Lake is an implementation of Iceberg, a replacement for Iceberg, a proof concept? It's a replacement.

Speaker: It's a replacement. now it's not a No, it's not a proof of concept. we have we've released 1.0 recently of Duck Lake. um It's used. It's actually the Duck Lake extension is downloaded as much as the Iceberg extension for DuckDB. So this is actually used out there. We we know it's used out there.

Speaker: and We have ah customers working with Duck Lake. So it's it's actually very cool to see that. um We use the same Parquet files underneath. So you can actually take an Iceberg table and fairly cheaply but with a metadata-only operation import it into Duck Lake and vice versa.

Speaker: So we use the same format for um data files. They're just Parquet with some of little asterisks on them. um And then we also use a compatible format for the deletions. So the way Iceberg stores deletes is have additional files that have tell you which rows are deleted from the um data files. We are compatible there because we figured, you know, we don't have to break compatibility where we don't need to do it. Like we are, you know, we're pragmatic people. We don't break things just because we want to break things. We we really we do usually do things because we think they're better that way. And Duck Lake is definitely one of these things where we looked at this iceberg spec and we thought this cannot be the state of data engineering in the year of our Lord, you know, 2024 whatever it was.

Speaker: Right. ah so What do you think is going to happen then? Because if you get the sense that they're already starting to pull metadata into the database they have now accepted they've got, are you all do you think you're just ahead in the race to a similar position in which iceberg metadata is a database?

Speaker: You know, it's very fascinating because I think there's multiple pages. I think if they end up doing what we proposed, I consider this a massive win. you know It's like we made a format that... we that you know It could also be that I could also see that DuckLake gets way more adoption, right? It's also possible. Depends a bit. you know If a large player would say, hey, we'd be betting on DuckLake,

Speaker: That's it. All right. that That would also be very, very good. um There's multiple ways for this to, um to to like, it's already having impact. So it's we've already crossed the first bar of it of it being ignored by the world. That hasn't happened. It is absolutely part of that world.

Speaker: And now the question is just, what is the endgame from here? Is it going to go into obscurity? are the other guys adopting it? And any of those sort of paths, I consider a win, right? Like if if if our tiny team from like of like database crazy people from Amsterdam can make that kind of thing happen, ah you know, sho shove shove Silicon Valley around and I consider that a massive win. This is the slug.

Speaker: yeah Yeah, we've got to implement influence those people across in California. Sometimes. Right? Sometimes. Yeah, yeah absolutely. yeah Okay, so but that kind of puts you in a new space, doesn't it? Because again, i think of DuckDB as this small thing in my command line toolkit, but now you're you're getting out into the enterprise world.

Speaker: That's true. um Yeah, absolutely. I think the Lakehouse stuff definitely is is something that you know we we see be people we see people storing their stuff in in Iceberg. We see people using DuckDB to interact with with Iceberg stuff a lot. like That's sweet something we see a lot. It's one of our most downloaded extensions, as I said.

Speaker: um We also see ah like lots of pushes of... There's also services out there like S3 tables, which is like a managed iceberg from from from AWS that we interact a lot with from DuckDB. So there's there's a lot there's a lot of this happening. And yes, you're absolutely right. And this is also something we've seen is that the the kind of people that use DuckDB has shifted over the last two years from, here's you know here's Chris.

Speaker: typing on his on his on his laptop. If you're listening to this on Spotify, you've just mimed typing with two fingers and I'm personally offended. i am not. i am not i am not. I'm so sorry. i didn't mean that. This was more like my I'm hacking. Please, please, please.

Speaker: Fine. And i have a duck on my keyboard. For those of Spotify who cannot see this, here it is. ah Nice duck. Anyways, but um but the um what we've seen is from it went from sort of somebody running DuckDB on their laptop. They still exist and we were grateful.

Speaker: But to also enterprise adoption, right? Like where where where people are betting huge sort of data pipelines on or DuckDB, for example. um I can't talk about all of them, obviously, but there are some that have publicly spoken about this. Recently at DuckCon, we had a talk from Spotify haha ah that described how they have this whole AI agent infrastructure for you to ask questions about your music history.

Speaker: And that just runs DuckDB under the hood with ah with an agent on top that basically turns the user's questions into SQL queries on that database. that they they they instantiate for every user once you start sort of asking questions. So that's just running DuckDB.

Speaker: right so Wait, they create a separate database just for the user each time? Yeah, it's it's de ah it's it' there's a talk at DuckCon, as far as I know, that's how they it's what they what I remember from it, that they would just instantiate the duck DuckDB instance. Once the user starts asking questions about the data, they they they shove all the listening history into that instance, and then the agent goes to town on it, essentially. Right, which you can only do reasonably with a database that's backed by a single core file. So you pick SQLite or DuckDB, right? Yeah.

Speaker: Right, but it's analytics, so you probably want to use DuckDB because SQLite isn't... I'll give you that. Yeah. So that was something that was presented at DuckCon a couple of weeks ago in Amsterdam.

Speaker: ah So we also saw then, for example, we also saw somebody talking about how they changed their their ah big pharma data analysis workflows from Spark to DuckDB and, you know,

Speaker: saved hundreds of millions of, I don't know the number, I i don't remember. But like you see a lot more enterprise adoption. like I didn't have wouldn't have expected that somebody like Spotify or a Big Pharma would suddenly bet on DuckDB. And I think that has been the story for the last maybe two years, it's a good way of thinking about it, of of just this massive adoption. And I think it's also something that pushed, let's let's say, our download numbers. Right now for DuckDB, we're beyond like a million a day, which is which is totally wild. That's crazy.

Speaker: um How has that changed life for you, for the company? Because here you are kind of rebel academic building an open source database, right? And now suddenly you're thrust into the enterprise world. Are you going to be going all corporate and ringing the bell at NASDAQ and all this stuff?

Speaker: It's a great question. um So I think i haven't I haven't been a rebellious academic, maybe in my mind, but like e de facto. I mean, our company has like 30 plus people right now. It is a fairly, i mean, it's not huge, of course, but it's a fairly big operation to run on a daily basis. um Also revenue wise, right? Like the amount of money you have to shovel into that fire every month is is considerable. Yeah, yeah. 30 staff is eye-watering numbers.

Speaker: I mean, for a for a small self-funded sort of company from Amsterdam, it is it is quite a lot. yeah I mean, love everybody. They are great people. There's not many teams out there that can that can do data engines on sort of this level. That's that's also something I had to learn. I was like, well't everybody can doesn't every company have people like that? It turns out to be not the case. No.

Speaker: now um So we have... we have um But yeah, it's definitely been a bit of a it's been a bit of a a change. um Dealing with corporate purchasing departments is also really not fun.

Speaker: This is like... hey what Getting them to sign off in invoices and stuff like that. Yeah, or pay their invoices. But it's a basic thing you would imagine, right? But um no, but I think i think that's that's all fine. And we're really really grateful with our team here that's working on that. But it's it's I've noticed that you know interacting with with these massive enterprise customers is certainly something that like a hack team of hackers is maybe struggling with a little bit. Yeah.

Speaker: Yeah, that that's often the case for like, when you're when you're a coder who just wants to build something and people will find useful. Yeah. Firstly, the first hurdle you normally fall at is um getting people to use it because most of us hardcore programmers aren't marketers, right? Yeah.

Speaker: But then if you get that hurdle, you've got the reality that you gradually get pushed out of writing version two of a peg parser into worrying about corporate relations. Yep.

Speaker: Well, that's actually true. it's funny though It's funny, though, the marketing aspect. We had a post blog post yesterday about DuckTab 2.0 coming out in in fall, and it was on the top of Hacker News for like all afternoon, which is something that that is that still that still feels to me like we've got de marketing ah we've got the marketing down still. But um but no, if absolutely. The... um And i think the I think going forward, it's also interesting like what kind of company we want to become. right like in the in the It's always been clear that we you know that we we want to continue doing this. We want to keep pushing database engines. um

Speaker: But it has also become clear that we would have to probably become a different company if we wanted to really push DuckDB to the next two or three orders of magnitudes of of of adoption. right like that That we would have to have I don't know, sales, marketing, corporate stuff, account managers, these kind of things, right? Like that's that's definitely something that that we have been sort of thinking about for the last couple of maybe one year. It's like, hey, how are we actually going to scale up? haha ah How are we going to scale at this company's operation? Just also just to to to to continue our mission as so as as making data better for everyone.

Speaker: Yeah, you can't just build databases. You end up having to build the soft systems that a business is made of. Right. And I think we have been doing extremely well ah in in in many ways. We have never had a a month where we you know haven't been profitable in the last five years, which is something I'm very happy about. Not many companies can say that, even the huge ones, right? Exactly. um But but it's also it's also clear that you know we we have to do something different.

Speaker: So what are you going to do Are you going to spin up your sales team and become that kind of guy? Or... No, I know. and it's it's It's hard for me to talk about in a way because it's been it's been such a such a sort of process.

Speaker: um but yeah i mean But we actually, um by the time you hear this, ah we we are we we are joining Amazon actually as a company, as a new subsidiary. So Duck Labs, the company, ah will be will become part of Amazon Web Services, which is part of Amazon, one of the biggest companies in the world. Yeah, I've heard of them. as ah Yeah, they're big. They come to your door on a regular basis. No threat. ah but it's Soon drones will be dropping ducks out of the sky.

Speaker: No, but it's it's very fascinating as ah as ah and as ah as a direction. right like i mean I would have never thought we would ever get to this point that we would be acquired, which is which is like ah something that I heard other people talking about. and and we were like but yeah just so this But it it is actually something that I think is we have thought about this a lot, as you can imagine. right You can only do this once.

Speaker: You can only do this once. and you I'm i'm not probably not going to start another a company. like I mean, i was I was a sort of i sort of a reluctant founder in the first place. um um yeah but yeah Tell me about that from your point of view. because You must have been thinking, well, we could start up the whole sales and marketing engine and try and go down that route to the path at NASDAQ. Ring the bell. Or did you get courted by Amazon? And you're thinking, well, maybe this is a road to go down that's less business work for me. So it's actually fascinating. Yeah, less things I'm not excited about, let's just say. Like, I want to work on technology. I want to push technology.

Speaker: And I think this is also the case for Mark and everybody in our team. We want to push technology. I'm not that interested, let's say, in in building a sales team. Let me just be absolutely absolutely frank. And I think I could have found somebody to do this.

Speaker: Yes. um But um it it would it would it would become a different company. I'm fairly sure about that. Yeah. And Amazon, we've actually been working with Amazon for for two years already. They've been one of our one our biggest customers, actually. um You know, for example, working on Iceberg stuff, they're one of the people that have been funding work on Iceberg inductively for the last years. So so it it when they then fought when they when they basically showed up one day and said, hey...

Speaker: Can we maybe talk about this? ah it was it wasn't it wasn't coming It wasn't coming from like ah it wasn't like ah you know a bolt from the blue. Is that something you can say? i don't know. yeah um It wasn't a bolt from the blue. it was it was It was like, hey, we actually know these people. We have a working relationship with them um and we like them.

Speaker: And they have ah you know have a really good vision about where things should go. um But it was still a bit of an interesting discussion because we, you know, we'd be like, okay, but what what we will we do as part? You know, they have they have thousands of thousands of of employees over there. Yeah. at at What will we do with our 30 sort of people team?

Speaker: But I was really happy about the the the vision that... that um that they presented to us. And so one of the part of this vision of that vision is that DuckDB and DuckLake and Quark and all this stuff is currently not part of the company. Many people don't know this, but DuckDB is actually is actually part is's actually organized or governed, if there's good word, by the DuckDB Foundation, which is a non-profit foundation here in Amsterdam that is is like a lot like, let's say, the Apache Software Foundation or something like that. Right? Right, yeah.

Speaker: So all the projects are under that foundation. and the amazon and And so we are not selling that. We are selling the company Duck Labs, formerly called DuckTV Labs, which is kind of the commercial sort of side of things where we did all these projects with with commercial partners.

Speaker: We are not... The foundation is untouched. We leave that alone. That stays where it's at. Right. We're selling the company. So it's very similar to what happened at Iceberg, for example, where Tabular, the company, got sold to Databricks.

Speaker: But the Iceberg project was and is an Apache project, which... wasn't at all part of that acquisition. This is the model that we're looking at. And it's not only that we're going to... It's also not only that we're going to let the project sit in that foundation and sort of, you know, never touch it again. But I'm really excited that the AWS guys are also really committed to us keeping, pushing DuckDB and DuckLake and all that stuff in the foundation for years to come. So in in fact, we're going to have more people working on that stuff than before, right? So so that's really fascinating.

Speaker: Yeah, is this a thing where you're going to have to think about making sure you open up the foundation to non-DuckDB and therefore non-Amazon members?

Speaker: Well, Amazon isn't isn't part of the foundation. They are donors to the foundation. We highly appreciate that. But I think the fund opening up the foundation to to to get input from all DuckDB users more than maybe we have been doing this in the past is definitely part of this. We want i mean we care about the ecosystem, of course, that we've built. And we want to but make sure everybody that uses DuckDB continues to thrive.

Speaker: um and and And I think that's that's that's kind of also is really unique. I think that AWS is really on board with that, right? Yeah, yeah.

Speaker: do you Are you not worried? i would be worried that what they say today will change three years from now as their management structure turns over and suddenly you find yourself in a different set of promises.

Speaker: I think that's an excellent ah point. um We have spent a lot of time thinking about this in the various legal documents that have been flying around. yeah um But we do have a long-term commitment all the way up their sort of structure that this is what they are planning to do.

Speaker: for the you know for the time going forward, independent of individual people being in individual positions. So we have their commitment on that, that they will they will let us continue to do that. They want us to continue being effective and you know push things in DuckDB. And I think that it that is... um i am I'm confident that we have the the commitments that we we need ah to to keep to keep doing that without um it being dependent on, the say, one person sort of changing jobs, maybe. like That's definitely a case we have been thinking about.

Speaker: Yeah, you don't know if you're in a situation where your one champion AWS retires or... No, no, no, no, exactly. but That is something we have actually spent quite a lot of time thinking about, talking about with them.

Speaker: We have gotten commitments from them that we but make us confident that this is really their long-term strategy. um And and and that's that's really exciting. I think that's that's pretty unique.

Speaker: and And again, you know, like DuckDB... the foundation doesn't doesn't change. right like it might even We might even end up opening up more to more people to more people having input on the foundation roadmap, for example. But it's not but like the projects are not moving. the The license doesn't change. If anything, we plan to to increase the speed of of with which we're doing things in DuckDB, like the you know the the the amount of people being able to contribute to to Core.

Speaker: um So that's that's ah that's exciting. Yeah, this deal must come with certain material benefits for you and Mark, right? But it must also come with a lot more resources for the company, I assume? Yeah, no, absolutely. Absolutely. like Yes, of course. I mean, there's there's there's that they're paying to acquire a company. i mean we' not We're not be not going to tell and and people how much it is because it doesn't it's not important.

Speaker: um I hope you've been suitably rewarded. Jeff Bezos can afford to put a few pints in your back pocket. i think I think my beer budget is safe, let's just say, for the foreseeable future. ah It's all that matters in the end.

Speaker: um But I think the um the resources, ah for sure, there's there's some things I'm excited about. is the It's, of course, ah resources in terms of people.

Speaker: but like we we have We had to be quite conservative people in in adding people to the team just because we we are self on the company and we we need to whenever we signed a work contract with somebody, we need to make sure that we're able to pay that for the for long term.

Speaker: So we have been quite conservative in hiring. im looking We are actually don't have to do that anymore at the same degree as part of AWS. um We also get, of course, a lot of infrastructure, right? Like as part of AWS, there's a lot of computers we can use. there well You've done it all for the free AWS credits.

Speaker: Absolutely. now yeah But i think I think what's also super interesting and something we've always been locked out of was the ability to see real-world workloads. That's really a problem. And maybe i should talk about this a little bit.

Speaker: So you worked at Snowflake. Snowflake sees every single query that people run, right? I mean, they yeah but might pretend they don't, but they do. ah the and they I'm not sure how much I can say legally, but that is a valuable source of intel.

Speaker: Anyone, any reasonable person would assume that. Let's just say. um and you know i don't have a site never to done in india with with i think I don't have any insight there. but it's I don't think they look at the data, but they can look at the access patterns, right?

Speaker: Right. and And the kind of queries people run. and it's something' So DuckDB, because we are European and we we respect people's privacy, privacy um we don't have any telemetry in DuckDB. We don't send out the queries that you run to our central sort of command. like We don't. We don't have telemetry.

Speaker: We don't know what what on earth you're doing with DuckDB. And that's actually been a limiting factor for us because one of the huge advantages of the Snowflakes and the Databricks and the the Amazons of this world is that they see the workloads and they can to steer development and sort of optimization towards where it really matters.

Speaker: Yeah, the real world edge cases. Yeah. We have always been locked out of that because sometimes people open bug reports and say, hey, this query doesn't work. um And then we fix it. But we have no idea if that's just some dude in their garage you know in in Uzbekistan or the biggest company in the world that has this problem. Because for us, it's just a GitHub account with a bug report. right Yeah, yeah. So so um for us, we've always been locked out of this of this optimization path of um of real-world workloads. And that's one thing that I think will be extremely valuable for the for the for the project, for DuckDB, to basic say, OK, you know what? We see a lot of queries that, I don't know, group on 15 string columns at the same time.

Speaker: Not sure. Maybe that is a thing. Who knows? But we can then say, all right, and now we're going to actually optimize you know everything about these things because we see them being such a relevant part of real-world workloads. And I think that's one part where it's really It's really going to come together in terms of um you know ah us being part of Amazon, really improving DuckDB itself.

Speaker: right yeah that's That's going to be very exciting. Yeah, there's an old thing about open source ah being good because all all bugs are shallow and you have 10,000 eyes on it. But that's only true if you have the full feedback loop available, right? Right. And we don't. And we we don't have the feedback loop, right? I mean, we we also have this this this well-known issue that like one in 10 people that experience a problem will report a bug at best, right? Yeah.

Speaker: So we also have this thing that there's just this this undercurrent of stuff that maybe doesn't go so well. We don't know. But we never hear about it. And and i mean we have... We were fairly happy with our sort of issue tracker sort of load at the moment, given that the insane amount of users. um But but like we are really, really interested in in and sort of closing that feedback loop to say.

Speaker: Yeah. I mean, also, there's also exciting stuff that, I mean, obviously, Avalyous is planning with DuckDB that I can't yet talk about. But, you know, down the road, there will be there will be um will be announcements, of course. And and that that's that's also something that I'm i'm looking forward to is ah what you know what can be done as part of ah such a huge organization.

Speaker: And again, i think the thing i really I'm really i'm really i'm amazed that, and it's also a bigger part of you know why we were considering this in the first place, is because they said, we want you guys to be this entity that has just encomput like that just contains database query processing expertise,

Speaker: As an independent entity, we're going to be Duck Labs, we're going to remain Duck Labs, it's just going to be an Amazon company or something like that. right like that's that's ah that's ah That's, I think, the part where we're not going to be absorbed or anything like that into 15 different product teams.

Speaker: um but we're going to got of stay together as ah as a team to to push query processing. Hey, that's something I care about. And I suppose you can't 100% guarantee that will happen, but if you went down the I'll start a sales pipeline and grow that way path, you couldn't 100% guarantee that would work either.

Speaker: Fun story. i So I talked to to to ah some founders of some big database companies that I cannot mention um that are they're running da they're running SQL as a service kind of stuff.

Speaker: yeah And they, very in the beginning, because we were thinking about whether we should make a, but open i have ah have a service like a SaaS for DuckDB. And they told me like, look, couple of years down the right line, the sales people will run engineering because they will dictate what you have to build for them to close more deals because that's what the whole company is sort of and their share and the stock price is predicated on. And I heard that and I thought, that's interesting and it's not necessarily a world I want to be in.

Speaker: um Yeah. Right? yeah it's not Yeah, it's not an expansion that comes for free. It comes with a set of shifting priorities. Right. And I think AWS is kind of big enough for that to be maybe a little bit more disconnected. Like be where we where can we can we really to push on technological excellence and know that that will spill over much like they already do in other areas, like with the CPUs is something I find quite impressive where they build their own CPUs, right? For the data centers, the Graviton thing.

Speaker: And and and they they they push everybody. Like, this is something that people benefit. And it's more efficient. it It's nice to the planet, all of that. ah So this is I think it's really it's really going to be fascinating. it's a you know i Obviously, I don't know a lot yet. is By the time we're recording this, we are we're in the final stages of negotiations.

Speaker: um And ah yeah, it's but it's it it's it's going to happen. Okay, one more question about that before we get to another technical thing I want to talk about.

Speaker: how does that i just want to know, because you don't often hear this story, how does that affect you? Because you you you know you are the co-founder of a successful company that's been profitable every month for five years. You're a big fish in your own pond.

Speaker: Right. And i can see the pros and cons, but how does it feel for you? Oh, you know, it's super fascinating because one one idea I really i had to come to terms with is having a boss again. Yeah, yeah.

Speaker: I know that feeling and it doesn't go easy. Thankfully, thankfully ah the the person who is is's going to be in in charge, I've been able to work with him for the last couple of years, as a as I mentioned.

Speaker: So I think I have a pretty good idea what what he's like. Yeah. So that's that's reassuring, but it is it's definitely going to be an adaption for me. i ga mean Obviously, in our company, it' like the only restrictions we had were what you cannot legally do, right? I mean, that's there's a very sort of wide, especially in Holland, you can do a lot of things. It's like there was, there was like there was I mean, the the Dutch are just historically business friendly in that sense, right? So there's not a lot of restrictions on companies.

Speaker: yeah um So we had to follow the laws, but that was it. um and And I think that that that, I mean, it's not that we did anything crazy, but it was very sort of nice, you know, like,

Speaker: can we Can we buy this ridiculous thing for our team's entertainment? Absolutely. right nobody yeah Nobody can tell me that that's not OK.

Speaker: um So ah so that that's going to like yeah that's that's maybe that's maybe a change. um Another thing I've noticed, this is going to have to performance reviews. It's been a while since somebody did a performance review on me. Oh, yes.

Speaker: You may find, as some people in your position have found, that you've at some point in the past five years become unemployable. it's it i Yeah, it's possible. um ah But um i'm I'm okay. I have i have made my peace with that idea. It's okay. maybe maybe i don't have to Maybe I don't have to look at the performance review. How about that? yeah It gets written, but I never see it. That's kind of my lot my my thinking right now. Ignore the performance review, ignore the sales department, and just carry on writing serialization formats.

Speaker: Pretty much, yeah. um I think i think that's that's, of course, I think I'm going to be fine. ah Don't worry about that. But it is definitely going to be different um different in terms of you know the the parameters. Obviously, I mean, it's a big company. They have they have rules, and I understand that. it's got But that that's we'll see how that goes.

Speaker: Yeah, I suppose there's also the advantage that they're a big company. they They won't be paying that much attention to you in the global scheme of things, I guess. That's also true. Yeah, yeah, yeah.

Speaker: Interesting point. We'll get you back in 18 months and see how it's actually gone in reality. That could be. Yeah, happy to. So there's one more technical question I wanted to ask because sure in a way this is surprising you haven't got it already.

Speaker: um I saw coming in the pipeline um triggers. Is that forthcoming DuckDB feature? Triggers are coming in 2.0. Yep. We're going to have ah tons of tons of... I mean, we are we have this blog post out yesterday that...

Speaker: that has like a preview for 2.0. um And yeah, it has ah it made it listicle, you know, 10 things coming to DuckTV 2.0, number 8 will shock you. It's kind of ah yeah so the harking back to... Obligatory.

Speaker: It's gonna... Yeah. the The question then is not why triggers, but why now? ah Because we have the year of the server in DuckTub. This is the whole... this is the whole sort of We had the year of the lake house last year, and now we have the year of the server.

Speaker: And um triggers are one of these things that you kind of need when you're running a server because you're you know you you want things to happen based on your data changing, and we are we that's, I think, why we are doing it now. um And ah there's also other things done in the pipeline that's still coming past 2.0, like ah we're going to do a um a PL SQL kind of thing where you can write actual programs like stored procedures in DuckDB.

Speaker: And it's going to be fast this time around. So we have this we have ah Dennis here, who is a who is awesome, who' who who did his PhD on compiling PL SQL to recursive subqueries. And we're going to we're going to implement that. He now works for us. And we're going to implement that in DuckDB in the future so we can have stored procedures Written in pure SQL, but fast. um So that's going to be that's going to be exciting. It's coming to DuckTV.

Speaker: Written in pure SQL, so it's not like a programming language wrapped around SQL. There's not PL Duck coming. Because SQL is like it's like you have like if, else, loop kind of constructs as part of SQL.

Speaker: but Right. So it's it's to that kind of old spec rather than just SQL or you're inventing your own language. I think it's, I mean, we're probably going to make it nicer because it's quite clunky. You're right.

Speaker: um we have We have some ideas on how to make the syntax better. ah But the idea is that this is a SQL level thing that allows you to express arbitrary computation in a SQL-E sort of thing.

Speaker: um And that's getting evaluated by DuckDB in an efficient way. Okay. Okay. And is that going to, because you compile DuckDB to WASM, is that also going to be available in the browser?

Speaker: Yeah, for sure. and Everything. This is all this is all we all coming to the browser for sure. can ask you a million more questions. but We can do a follow-up. i' Yeah, there's i mean there's there's lots lots of exciting things to that's coming for sure. and We are hard at work getting 2.0 out of the door. that's that's ah That's always a bit of a crush, crunch.

Speaker: um Yes. ah That's the team is working hard on that. I'm very grateful. I shall leave you to that. At the end of the year of the server, when we found out how the AWS acquisition has actually played out, you must come back and tell us.

Speaker: I will be happy to. Excellent. Hannes, thank you very much. Thank you so much. Thanks so much, Chris. Absolute pleasure talking to you. Always. Cheers. Thank you, Hannes, and best of luck. I hope the deal works out really well. I'm told it's being inked pretty much as we speak.

Speaker: So may you live in interesting times. However it plays out, I think you'll end up writing more code than if you've gone down the enterprise sales management route. So I think the world's a better place.

Speaker: As always, you'll find links to everything we discussed in the show notes, including a link to the previous episode where we really got into the weeds of how you build an analytics database from scratch.

Speaker: Personally, my homework is to go and learn more about PEG parsers because I'm a fan of parser combinators and I don't know much about PEG parsers. I'll go and make that my holiday reading. Before we all go away and learn stuff, please do take a moment to like, rate and share this episode.

Speaker: And make sure you're subscribed because we're back soon with, I think, a compiler expert. Stay tuned for that. But until then, I've been your host, Chris Jenkins. This has been Developer Voices with Hannes Mullerheisen.

Speaker: Thanks for listening.