Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Building the SpacetimeDB Database, Game-First (with Tyler Cloutier)
3.1k Plays5 days ago
Eighteen months ago, Tyler Cloutier appeared on the show with what sounded like an ambitious (some might say crazy) plan: build a new distributed database from scratch, then use it to power a massively multiplayer online game. That's two of the hardest problems in software, tackled simultaneously. But sometimes the best infrastructure comes from solving your own impossible problems.
The game, Bitcraft, has now launched on Steam. SpacetimeDB has hit version 1.0. And Tyler returns to share what actually happened when theory met production reality. We cover the launch day performance disasters (including a cascading failure caused by logging while holding a lock), why single-threaded execution running entirely from L1 cache can outperform sophisticated multi-threaded approaches by two orders of magnitude, and how the database's reducer model - borrowed from functional programming - enables zero-downtime code deployments. We also get into how SpacetimeDB is expanding beyond games with TypeScript support and React hooks that make building real-time multiplayer web apps surprisingly simple.
If you're building anything where multiple users need to see the same data update in real time - which, as Tyler points out, describes most successful applications from Figma to Facebook - SpacetimeDB's approach of treating every app as a multiplayer game might be worth understanding.
--
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@DeveloperVoices/join
SpacetimeDB: https://spacetimedb.com/
SpacetimeDB on GitHub: https://github.com/clockworklabs/SpacetimeDB
Our previous episode with Tyler: https://youtu.be/roEsJcQYjd8
Clockwork Labs: https://clockworklabs.io/
Bitcraft Online: https://bitcraftonline.com/
Bitcraft on Steam: https://store.steampowered.com/app/3454650/BitCraft_Online
WebAssembly: https://webassembly.org/
Flecs (ECS for C/C++): https://www.flecs.dev/flecs/
TigerBeetle: https://tigerbeetle.com/
CockroachDB: https://www.cockroachlabs.com/
Google Cloud Spanner: https://cloud.google.com/spanner
Erlang: https://www.erlang.org/
Apache Kafka: https://kafka.apache.org/
Tyler Cloutier on X: https://x.com/TylerFCloutier
Tyler Cloutier on LinkedIn: https://www.linkedin.com/in/tylercloutier/
--
Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
0:00 Intro
2:01 The Architecture of SpacetimeDB
5:01 Client-Side Prediction in Multiplayer Games
11:00 Reducers and Event Streaming
15:00 Launching Bitcraft on Steam
19:00 Debugging Launch Performance Problems
26:56 Hot-Swapping Server Code Without Downtime
30:01 In-Memory Tables and Query Optimization
42:00 Is SpacetimeDB Only For Games?
51:00 Performance Benchmarking For Web Workloads
55:00 Why Single-Threaded Beats Multi-Threaded
1:00:01 Multi-Version Concurrency Control Trade-offs
1:05:01 Sharding Data Across Multiple Nodes
1:10:56 Inter-Module Communication and Actor Models
1:17:00 Replication and the Write-Ahead Log
1:24:00 Supported Client Languages
1:29:00 Getting Started With SpacetimeDB
1:39:02 Outro
Recommended
Transcript
Introduction to Relational Databases and Gaming
00:00:00
Speaker
Is there something that links relational databases, functional programming techniques, and entity component systems from the game world all together in one place?
00:00:12
Speaker
Because on the surface, it seems like they're all quite disparate ideas. Well, about 18 months ago, we had a guest in, Tyler Cloutier, and he was either very good at seeing the connections between these sorts of ideas, or he was a little bit crazy because he decided to create an MMO, massively multiplayer online game, and he decided that the best way to create one of those was to build a new kind of distributed database from scratch.
00:00:43
Speaker
That sounds like two of the hardest projects you can take on. And he was doing them both in parallel with a fairly small team and a fairly small development timeline. It sounded crazy, but the more he explained it, the more I thought he was onto something.
00:01:00
Speaker
At least in theory, that didn't guarantee it would work in practice. Well, roll forward 18 months, the game has launched with a few lessons learned, and the database itself is becoming both the heart of the game and the heart of the company.
00:01:16
Speaker
So what ties those ideas together? How does it work? And what were the lessons learned? What was the gap between theory and practice? But more importantly, by digging into it, how can we begin to see more systems as kind of massively multiplayer online games?
00:01:36
Speaker
maybe serious games, but still massively multiplayer. Somewhere between game architecture and database architecture, is there a more universal architecture we could learn from?
00:01:49
Speaker
At the very least, SpacetimeDB and Tyler's perspective on software are a very interesting point in the design space.
Tyler Cloutier's Journey with SpaceTimeDB
00:01:57
Speaker
So let's go and explore it.
00:01:59
Speaker
I'm your host, Chris Jenkins. This is Developer Voices. And today's voice is Tyler Cloutier.
00:02:17
Speaker
Joining me is a returning guest, the brain behind Space Time DB. It's Tyler. How are you doing, sir? I'm doing very well. Thank you so much for having me. I'm glad you're back. I really am, because um the last time we had you on, you were looking at both a database launch and a game launch.
00:02:36
Speaker
And I was genuinely wondering how well that was going to go. Yeah, me as well. I you know i appreciate being back on. um it's been It's been a wild ride. I expect it will continue to be a wild ride. um But yeah, it's always going to be exciting when you decide to write your own database just to be able to write your own game.
00:02:59
Speaker
And ah that has not disappointed. Yeah, well, now I've played the game. I know you're not completely mad for doing it that way. And you must feel reassured that you're not completely crazy either having launched right? I do. I think, though, it's at some point in the development of it, it became so retrospectively obvious that it was a good choice that even before we launched the game, um we knew we were on to something. um What I think is different is that once you actually get it out there, you really know for sure that the performance is not a pipe dream, but it's actually something that...
00:03:33
Speaker
can't support a real time our game. yeah Yeah, because we talked a lot about the potential performance traps in there. But before we get into it, I think it was a long time ago, and I'm sure not everyone has listened to that episode.
00:03:45
Speaker
Give me a recap on the basic architecture of space time DB. What is it that you're trying to do but other people don't do with databases? Yeah, that's a great question. um So let me give you an idea overall of what SpacetimeDB is. So SpacetimeDB probably should be called SpacetimeOS. That was something I sort of left out of towards the end of the ah the last one. But essentially what it is is it's a database, a distributed database.
00:04:12
Speaker
You take your program, you compile it to WebAssembly, you upload it to run inside of the database, and then clients connect directly to the database. They call functions on the database and then subscribe to data, which gets updated in clients in real time. And so the reason I say it it should be called an operating system is because it's essentially a system that ah schedules your program, runs your program, ah and then persists data to hardware and sort of manages all of your hardware together um in a way that you don't have to care about, um which allows your program to run and then clients to connect to it, essentially.
00:04:53
Speaker
And when you say you are uploading your program to the database, that's you as, like for instance, the game developer. But there's also a certain amount of computation happening on the client side, right?
00:05:05
Speaker
That's correct. So the um the game developer is going to upload the the main portion of their program to the server. The client... um is essentially a read-only replica of the database.
00:05:18
Speaker
But the difference is it's not a full replica. It's a partial replica. So you are sort of going to say, hey, I want i have X number of tables. I have X number of rows in those tables.
00:05:29
Speaker
um I am going to select some of the rows in those tables from some of the tables. And I would like them to be replicated to my local client as quickly as you can um and then kept up to date over time.
00:05:43
Speaker
And so that is that's what the the client is doing. And then obviously the client does an additional amount of things on top of that ah in games, for example, um anything that doesn't need to be synchronized across clients. none of the multiplayer stuff. So if you have like particle effects and things like that, that are just purely for visualization, that's all done by the client animations at times.
00:06:04
Speaker
A lot of it is triggered by things that are on the server. So if there's an explosion that everybody needs to be able to see at the same time, that gets sent down as a piece of data, and then the client renders that. So the client is doing the rendering. The server is doing the sort of source of truth of the data in your application.
00:06:20
Speaker
So the server projects out a subset of the data to each client, and then the client projects out a visualization of that subset to the user. That's exactly right. It's it's very functional functional programming in spirit. and Remind me about writes, because if I'm running a game client, ah I want to write my new position, say, back to the server, but I also want to immediately visualize that change. And you've got collision detection on that, so there is logic over whether the write can happen at all or not.
00:06:52
Speaker
That's right. So um ah in in our game, in BitCraft specifically, there's not a lot of collision detection, but there is some. So... um A lot of games have more. If you're doing a first-person shooter, that's that's very you have to be very careful about timing and latency matters a lot. Different games have different sort of latency requirements.
00:07:10
Speaker
um But yeah, so generally how this would work is through something called client-side prediction. So that is um my client is going to... It has a replica of the world, so it it knows where things are as of a certain point in time in the past um on the server.
00:07:26
Speaker
And it will look at the world and it will say, OK, I know what the server will do because I know what the code and logic of the server is. So I'm just going to do it locally, assuming that the server is going to say that's OK. Then i'm going to also send it to the server at the same time.
00:07:42
Speaker
So locally, I'm rendering it as though it has actually happened in the server. It hasn't. I send it to the server. If the server eventually comes back and says, like, yep, that happened, no problem, then the world has proceeded as I have predicted. And it looks the same, and I just kind of throw away in a way what the server does.
00:08:00
Speaker
um Now, if I send it to the server and eventually it comes back and the server said like, whoa, you can't do that, I will have to go back in time on the client and undo what I had done and sort of replay forward from there.
00:08:14
Speaker
And so that's what client-side prediction is for games. And is that done, is is the reason that works is it's probably done on a matter of sub-second intervals. So at the worst case, I can just sort of jump the client's play as I mean it depends on you know what your latency is and and how much collision is going on and so on it can be a more or less good experience ah but yeah generally speaking this is you know it from you to the server something like a hundred milliseconds um in a relatively bad case ah so ah yeah you you're doing this in real time you can see like this will be um scenarios where people kind of pop back around
00:08:52
Speaker
in different ah regions. If you see that like in a game where they like rubber band and stuff like that, that is often a cause of rubber banding. Yeah, i've seen I've seen it a lot more with other players not being where we thought they were going to be than with my player, right?
00:09:07
Speaker
Well, that's right. Well, yes, true. I mean, very it's it's not so frequent that you will um invalidate what you did, but it can happen. I mean, um you can imagine a scenario where you.
00:09:21
Speaker
Your client believes that you killed somebody. This is actually a a long topic of discussion, uh, to get into, but, um, there's, there's some legendary, uh, talks that people have done at GDC, for example, about like how I remember this one specifically, uh, from overwatch, uh, about how they do the client side prediction on their end. And it it gets very complex because, um, you also, uh, want to do different techniques. Like for example, lag compensation, which is, um, you might actually, okay, let me, let me give you the example of where, uh, this can
Game Dynamics and Technical Challenges
00:09:54
Speaker
go wrong. So if I,
00:09:55
Speaker
shoot you and I, and my bullet goes out and I think it hit you. But actually, although on my client, you were going in front of the bullet because you kept going, you had actually turned around and I didn't know that yet.
00:10:07
Speaker
Right. So you, um, I dodged, but you didn't know. Correct. I dodged, but I didn't know. My client will... What will happen in certain games is it will animate the ah bullet collision. So you'll see like a blood spatter or something. But actually you won't die because they don't actually compute the die until the server actually says for sure you're dead. That's to avoid people like dying and then coming back to life again. Right. So there's sort of advanced techniques that you would want to make sure... ah you get right. And there's some very legendary talks about it. They do very fancy things like, for example, um they'll use UDP and they will put in the last 10 frames or seconds or whatever it is of operations so that if you lose a message, the next message will have the message that you had intended to send, ah but that got lost. So you use a redundancy. You've got like a rolling window of what's happening.
00:11:01
Speaker
That's right. Exactly. And so it's it's a way of trading off um bandwidth for latency. So you you have to use more bandwidth, but you get lower latency okay because you don't have to retransmit. Okay. Okay.
00:11:13
Speaker
we're We're going a bit too far into game development theory, and I kind of want to stay on databases. But so but you have hinted at this. is like From the client side, this is like an event streaming model, i get a subset of all the events in the system.
00:11:28
Speaker
As a client side program trying to figure out what's going on, is this also like a functional... um It's like a fold, isn't it? we're We're reducing over a stream of events to find out the state of the world at the moment.
00:11:41
Speaker
ah that's Yeah, that's exactly correct. And more than that, we actually have named the functions that you call on the server reducers for exactly that reason. um So let me talk a little bit about that. So the the security model of one of the first questions I get when somebody says, when I tell people that your clients are going to connect directly to your database is, was that secure? Right. And reasonable question, I think, for the old permissions model and architecture of databases is, But the thing that SpaceMDB does is because you're uploading your your actual server program, you take all the same logic you would normally write in a regular server and you put it in your database.
00:12:19
Speaker
So it's not as if you're not doing all of that checking. You are doing it. It's Turing complete. You can do whatever calculations you want to. We're just taking it a lot like um sort of from a performance perspective, putting it in the same process as your database data.
00:12:34
Speaker
And that is important for speed and performance. ah You would not be able to achieve the types of things that that you're doing. Those functions that take those inputs and modify the database state to produce a new database state, we refer to as reducers because they're taking all actions and reducing them into the current database state and producing a new database state.
00:12:53
Speaker
Right, yeah. And is it functionally pure? I mean, if you lost the database state, could you replay the event log? Yes, it is. And so that's one of the the most important components about um reducers is that they are replayable. So for example, what we do currently do in space time to be is we actually store the set of changes that you've made to the database state on disk. But alternatively, you could actually just store the code as of as of that point in time and the input to the function. And you can just replay them and then actually get the same exact state that you would have gotten back in time. And that means managing things like access to random numbers access to timestamps, um anything that could potentially change that you would have to have as input into your reducer ah to produce things. But once you do that, you can start to do very, very cool stuff, which we haven't ah released to the public. In some cases, we haven't fully developed yet.
00:13:47
Speaker
ah But for example, a point-in-time debugger. So you could set your database step state back to a particular point time. start up the reducer as it was, as it had run at exactly that point in time with production data, if you wanted, and then step through the code as it would have happened at the time that it did happen, which is pretty fun. That's something I've used in um Elm front-end language, oh yeah that time-traveling debugger thing. And it's really good when it works well, right? Yeah, it's really fancy. Now imagine you can do it for your entire...
00:14:20
Speaker
MMORPG, right? So somebody reports a bug, you're like, that shouldn't be possible. Let's go back to the point in time when it happened. Let's see what actually happened on the server. And have you are you is this something you're actually using in production? I mean, have you solved bugs with that?
00:14:35
Speaker
So we we have not built specifically this point-in-time debugger. We absolutely have gone back in time to the ah commit log and so and ah looked at the changes that a particular transaction made.
00:14:49
Speaker
ah But we haven't we haven't built a thing that actually steps through your code yet. That that would be super fun. Oh, right. We haven't done that yet. But you've got all the data. So you haven't got the nice UI on top of the basic principle.
00:15:00
Speaker
Correct. That's right. Okay. So we should talk about... God, there's so much to talk about in this, but I think we have to now get into... You were just about to launch last time we talked.
00:15:12
Speaker
You've got this database. You've built a game on top of it, an MMO. um And then you were just about to launch. and i will Tell me about how the launch went, because I know you had some performance problems.
00:15:26
Speaker
So did it go well overall? What did you learn you should have done differently before you released? Yeah. Yeah, so for people out there, the game is BitCraft Online. You can download it on Steam. It's in early access. um And it's an MMORPG. Just to give a brief idea of what the the game is overall, ah you start in the wilderness in a shared world with all the other players, and you rebuild civilization from the ground up. So you can edit the terrain, you can cut down trees, you can build buildings, you can ah level your character up. it's It's similar in concept to RuneScape plus Minecraft, if that makes any sense. yeah
00:15:58
Speaker
um And ah in ah June of this year, we went from our alpha phase, which was sort of a single region ah and a small number of players into Steam Early Access where anybody who bought the game could get access and play the game.
00:16:16
Speaker
Um, and shortly before that we released space time DB into 1.0, because if we were going to use it in production, we felt that it was ready, uh, to be used in production. Um, and we've been working for a while on, on getting into that state.
00:16:29
Speaker
Um, and then, uh, launching the game into early access was, was very interesting. Um, prior to that, we did a, we did a preview on seam, uh, which is like, um,
00:16:41
Speaker
you could get access to a play test of game right before early access went out. That, uh, was interesting because, ah we discovered during that, either it was during that or actually in early access.
00:16:55
Speaker
I think it must've been in, in that preview. Uh, we actually discovered that our hardware was bad, which I have as a software engineer have been trained to never blame the hardware. yeah Almost never is.
00:17:08
Speaker
um but in this case, ah We saw two of the regions. So had nine regions in the game. And each region is is managed by space time to be database. And that handles all the players in the region. And as they move between regions, we move the data from one region to the other. okay um just sort of a way of sharding it spatially so that you can get more players into a single world.
00:17:29
Speaker
ah And we saw two of the regions were lagging quite badly. And this was during one of our tests with players. We thought, well, that's just, yeah, I don't we don't i thought that was suspicious.
00:17:41
Speaker
But we thought, something's wrong with our software. We'll look into it more. ah We couldn't figure it out. We said, all right, well well, see what happens again. Well, in the either the preview or the early access, I don't remember which one.
00:17:54
Speaker
ah we saw the same thing again. Just two of the regions, two or three of the regions, really suffering. So eventually, after much debugging, we ran like a basic Intel benchmark. And two of the nodes, or or three, I don't remember, were doing like 25 percent of the performance on c on the cpu benchmark that the other two nodes the other nodes were doing so we swapped those out we like migrated the data over and bam uh it was going great uh so that was that was interesting i've never had that before i mean now i've been in a situation like that once before and you have to check a heck of a lot you you'd end up checking a lot of things before you start suspecting the cpu
00:18:38
Speaker
Absolutely. i mean, yeah, it's one of those things where it's like, you know, is it going to be the CPU that everyone is using that you think you would have heard about this problem? Or is it going to be your crappy software? 99% of the time it's your crappy software. But in this case, it actually, it was not. So that, that was interesting. Um, I, again, I don't remember if that was in the actual alpha that we did the swap or it was the test right before. but that, that was a, that was a shame because it, it sort of, uh,
00:19:08
Speaker
made it seem as though we didn't have our stuff together when we did. But it turns out we didn't fully have our stuff together because ah when we actually then released, we still had performance problems. So this was that we resolved the hardware, we're still having performance problems, ah and we had to figure out what was going on over a period of like 24 hours. So this was the this is like prime time. When you launch a game on Steam, you've got one chance to get it right because people are going to leave reviews immediately. And those reviews are permanent.
00:19:39
Speaker
yeah And if if they can't play the game, they're going to leave a bad one, and that's what you're going to have to do. So this was an absolute race to minimize the number of negative ah reviews. So when we put it out there, I mean, the game worked. It wasn't as if it didn't work, but I think somewhere around like 500 to 700 people per region, we were starting to get like enormous amounts of lag. The worst one was like we had like a minute long...
00:20:06
Speaker
lag spike on particular regions. So some regions ah would just start taking a huge amount of time, like full server freezes. Nothing was happening.
00:20:18
Speaker
Everything was ah dead for like a minute, which is a very, very long time ah in computers. In fact, it's so long of a time that it's like something is catastrophically a problem. ah A lot of times computers take seconds. That's when something is broken. Like things don't take seconds in computer land unless something is broken. If it's minutes, there's there's a disaster yeah yeah happening.
00:20:40
Speaker
And that actually turned out to be the case. So one of the things that we noticed was this would – everything would be fine. And then after a period of time, it would get bad. And then it would get bad. And then it would get really bad. And then we'd get these big freezes.
00:20:54
Speaker
and What we discovered actually was what was happening is we had – an AI agent that was listening to the game state of the world. ah And um periodically it would crash because there was a bug in the external client.
00:21:09
Speaker
And it would disconnect. it It would disconnect a lot of different agents all the same time. And um we were logging something um an enormous number of times ah when you disconnected. That is to say, all of the messages that we had not yet delivered, we logged once per message.
00:21:31
Speaker
Uh, right. Right. So, so if I had been, if I was slow, if I was a slow client and I had accumulated a lot of messages, uh, I would be disconnected and then we would log something about that message every time. Uh, and moreover, um that log still happened, I believe while we held a read lock.
00:21:49
Speaker
So, um it would take more time, which by the way, would cause other clients to build up a queue of messages. And then when they disconnected because they were so far behind,
00:22:01
Speaker
they would print a lot of logs. And then the printing of logs would cause more people to disconnect, and there was this sort of cascade. And so you'd see, yeah like, you would get worse, and then it would get really worse. And then everybody would freeze and disconnect. And then it would it would start be, like, kind of OK again. And then it would it would sort of get bad.
00:22:15
Speaker
um And so that was that was very interesting. And so we should have been logging asynchronously yeah in this case, but we were we were not. Actually, we probably shouldn't have been logging like what was essentially a trace message in debug.
00:22:26
Speaker
um Yeah, but that time that's exactly when you like when you launch. You want that kind of level of debugging. but You do. And I think these are the types of things about performance that people...
00:22:37
Speaker
um don't really think about like you think about oh memory allocations and caching and all that stuff yeah even if you get all of that done like a log in the wrong place uh at the wrong level while holding a lock uh can just s sink you yeah yeah so that was one thing that went on there were a couple others i don't know if you're interested in in the further uh details of of what went uh what went on there I mean, like I can see how, like, things like, there's always fundamental performance, like, um ah is this order, is this constant time or is this order n squared, right?
00:23:11
Speaker
Right. And that you can't fix unless you go right back to the drawing board. And then then there's that incidental, annoying stuff, like logging messages need to be asynchronous or, and that you fix, I'm assuming, fairly quickly, and then you you suck your wounds and learn,
Database Optimization and Performance
00:23:24
Speaker
right?
00:23:24
Speaker
Yeah, we had that fixed within four hours. There they were basically within, sorry, 24 hours. we There was four problems, essentially. ah The first one was um
00:23:36
Speaker
There was a workload that was different in production that then was either in our test cases or in our alphas. And that had to do with the the multitude of regions. So in the the large number of regions that we had, um when you opened a UI element, which, by the way, our bots didn't do. So we had like load testing bots. They would walk around. They would pick things up. They would mine things. They never opened any UI. wow.
00:24:02
Speaker
And when you open the UI, we would do these one-off queries to get like, you know, various data from the thing. So this is not a subscription to the data. It's just like, hey, I don't want this data long-term. I just want it for a while you have your UI open. So please get me the data and we'll do that. So we we did that. And in specifically the large full world,
00:24:22
Speaker
ah We had map locations. So when you opened up the map or you opened up certain like trade windows and things, we would go get all of the data. And there was a set of queries that were not optimized.
00:24:35
Speaker
ah We just like the optimizer just didn't optimize those particular queries. um And so they would scan a lot of rows. ah And we did not notice this in our bot testing on the full world because they didn't open the UI. And we didn't test it in the alpha or the preview because we didn't have as many regions. And we didn't have we sort of had one-ninth of the total amount of um actual UI, the things that we need to render on screen. okay So that was something where when we went live, immediately we started to see this. and we started to see, especially in the tutorial where people are opening up all the windows,
00:25:10
Speaker
ah having a lot of performance problems there. Right. Yeah. So that was that was interesting. We added the optimization for that kind of on the fly and then just published the new database and that started working fine. And then there was the the the disconnection and the freeze issue that I talked about. And that led to a thing that we called region eight disease, which is where region eight would start getting bad and then it would get really bad and then it would be very, very bad.
00:25:34
Speaker
And then the final thing is um that we had detected some of these freezes before launching, like the night before or like two nights before. um And the thing is with Steam is you've set your date. So there is no changing the date because you have to get approved. Oh, yeah, yeah.
00:25:51
Speaker
So you've just got to be ready for when it's when it's happening. And so we had to be ready. We saw these freezes and we rolled back to an earlier change because we thought we had they they had been introduced, but they actually hadn't been introduced. They were always there.
00:26:04
Speaker
um It was just under certain conditions, the mob monitor would crash and cause the cascading logs problem. um And so we didn't ship with a set of optimizations on queries that um we should have shipped with. And so that final one, we applied that and you could see our graphs go from way up here, just wooho and then they would be super low and very quiet. And so after about, I think it was between 24 and 48 hours, we had resolved all of those and it was relatively smooth sailing.
00:26:34
Speaker
Okay. After that one. That raises a whole bunch of questions, but but I'm going to ask first for the big picture. Since that launch and you've got actual traffic coming in how has the database fundamentally performed?
00:26:49
Speaker
um Well, since the launch, it's been fine. Certainly for the game, it's only gotten better, right? So as as an MMO goes on, typically it comes down, you release an update, it it pumps back up, and then it comes down again. um So we've had ah a steady ah but um slowly declining player base since then, and performance has been great.
00:27:07
Speaker
um We haven't really had any problems with that. What it has led to is us looking into and using more so some of the other features of SpaceMDB. So, for example, one of the very cool things that we can do with SpacetimeDB, because we have the whole server logic, is we can actually swap the server logic out at runtime while all the clients are connected and not disconnect any players. So if we have to do maintenance on the server or change the logic of the of the server, we can just do it and no downtime is required.
00:27:39
Speaker
How's that working? you've got You're capturing an event log. You've got the state of the world. What do you do? You introduce new code and hand the state of the world off to a new process.
00:27:51
Speaker
Yeah, so what we do is you upload the new program to the server. um We let all the existing transactions finish. um We swap in your new logic and we start the new next transaction and off it goes.
00:28:05
Speaker
So the key ability for us to do this is that we have distinct periods of time when... ah in between when you're running code, right? we can we can it's it's logic It's not as if you're just running code all the time, you're doing transactions. And so we can say in between transactions, we're going to swap out the code. And then as long as the clients are still compatible, so all the API is the same or you haven't deleted things that you like functions that they're calling,
00:28:30
Speaker
they will be none the wiser. ah So you can change things like the physics of the world if you wanted to at runtime in between transactions without disconnecting clients with no downtime.
00:28:40
Speaker
Okay, the thing I'm not quite understanding there is this, maybe i'm Maybe I'm coming with assumptions from how reducers normally work in a functional programming language, but do you have to serialize the state at the end of the transaction, save that so the new code can pick it up and know where were?
00:28:59
Speaker
um we ah Yes and no. So the, so the first part is true. We do serialize that we put it on disk so that we can recover to the state that we need to, as of that particular transaction. Um, but we do not use that state, uh, that those sort of diff states to have the new state be applied to the database. So, um, what am I mean by that?
00:29:24
Speaker
Uh, in it In a transaction, you're basically inserting rows and you're deleting rows. We record all the rows you delete and insert, and we write them to disk. We also just apply those changes directly to the in-memory representation of your tables immediately. So I shouldn't say immediately. It's staged, and then when you commit, it's applied.
00:29:43
Speaker
And that is important so that you can have ah atomicity. So you can roll it back if if you don't actually apply it. Yeah. um And so that's how that works. ah but But fundamentally, you are correct. ah The reducer produces a set of changes. Those changes are applied to the current state, and they're also written as changes to disk. Okay, so there is an in-memory database which the new code will just connect to?
00:30:07
Speaker
Correct. Okay, so it's the in-memory database that is that state that gets passed between iterations. Okay. Yep. Your tables are represented as in-memory tables and you access them. it's It's a key component of the performance. So we can do things like physics calculations. And like I have a demo that that I can run where you have a bunch of balls all bouncing around and colliding into each other. That's all happening within a transaction, each frame.
00:30:31
Speaker
running within a transaction, doing all the collision logic, and then returning all the changes of positions of all of the ah circles. And then we write all the changes to disk, and we go to the next transaction. OK. Then that begins to explain something I didn't understand in something you said earlier. You had some queries. One of the reasons you had performance problems is you had some queries that weren't optimized.
00:30:54
Speaker
But, I mean, what's that look like if all the data is basically in memory? Because you're not indexing? Yeah, so, I mean, unoptimized queries can be this. um So, in the game, there are location states. And the location state is just...
00:31:10
Speaker
Let me take one step back. So how do we represent data in the game? um We basically use ah the entity component system model, which is very similar for Bevy. But it's ECS is sort of a subset of the relational model. So the relational model is ECS plus more.
00:31:25
Speaker
um And okay what I mean by that is I can build an ECS inside of Postgres. And that is as simple as saying I have one table, which is the entity ID table. And i will have one column and it will just be all entity IDs. And I will have other um tables which have a foreign key reference to the entity ID table.
00:31:48
Speaker
um So, for example, I might have health being a ah table. And I might have like combat stats being a table, something like that. yeah And then for any entity that I want to give the property of health to, I insert a row into the health table that has a foreign key relationship to the entity. And now that entity has health. So I can look up all entities which have health and combat stats and food or whatever it is that i want to apply to this this thing. And I can look that up. So that's how we represent ah ah data in the game.
00:32:22
Speaker
One of the things that we have, one of the components that we have is a location state. So those are Things in the game that have a place in the world that is fixed.
00:32:33
Speaker
So a tree, a building, um things of that nature. yeah There's about 20 million of them or something that are about in each region, if I'm not mistaken. Okay. And so ah what you can do that's wrong, if you misoptimize your queries, you can say, okay, I would like to get... um all Let me see if I can come up with ah an example. I want to get all resources um nearby this player or something something like that.
00:33:03
Speaker
And so what you can ah do is, and maybe there's only 10 of those, or let's say 100 of those resources in the world. ah What you're going to want to get is also the location of those resources. So what you do is you say, okay, I would like to loop over all the resources of this type.
00:33:18
Speaker
yeah And for each one of those resources, I'm going to scan the 20 million rows of the location state to find out where that resource is. And I'm going to look it up by its entity ID.
00:33:29
Speaker
Right. And so that's a lot. Yeah. It's 20 million times 100. And so if you're not if you're not doing it, the way you'd want to do that is you'd want to have used the index that we already have on location state to say, look up the location state by the entity ID rather than scanning
Innovations in SpaceTimeDB's Architecture
00:33:44
Speaker
the full table. Right. Or actually one, I said that backwards. In some cases, it is even worse. some cases, you will scan the entire location. You'll you'll go location by location to see if they have a look if they have a resource in the resource table. So you'll look for every re location.
00:34:02
Speaker
Try to find if it has a resource in the resource table. And that's looking through 20 million things when you really only had to look through 100. Right. Yeah. Yeah. so there's So there's many various kinds. You have to use the indexes that you have. Nice. And then you have to also um do, I forget what the particular name of the optimization is, but you have to decide the join order so that you are ordering your joins to do the smaller things first. yeah You don't want to do, you want to eliminate or you want to p apply your where clauses in a sense.
00:34:33
Speaker
ah before you just go to town. You are being the human equivalent of a query optimizer in a database that's had 30 or 40 years of engineering to it. Exactly. And that's the beauty also, I suppose, of query engines is you do not tell it to do it a certain way.
00:34:49
Speaker
um You simply tell it what you would like to the results to be, and then it figures out how to do that. The thing is, um it's only as smart as it is, so you need to always kind of improve it. And then ah it leads to the classic database problem of like, oh, I didn't have an index on that column, and now it's slow.
00:35:09
Speaker
It's as if you wrote a a Instead of a hash table, you just had a big array. Yeah. Yeah. There's there's no getting around those fundamental data type issues, right? Yeah. There is not. I think one of the one of the really cool things about this architecture for how you represent in-memory data is that um changing...
00:35:28
Speaker
the the physical representation of the data is just configuration. So you add an index here, you remove an index there, you um can change the optimizer to do all these cool things, but the logical model doesn't change while you're doing all of that stuff. So you are accessing the data in the same way that you were. you don't have to rewrite your program to change the performance characteristics of your program.
00:35:50
Speaker
Yeah, yeah. um And decoupling those things was the original... um motivation for the relational model in 1970. Yeah, yeah, yeah. yeah the ah The way the data is stored shouldn't be... The way the data is used shouldn't determine the way the data is stored because you need to be more flexible than that, right?
00:36:13
Speaker
How it's stored and and how it's accessed. I'm sure you've you've had ah the classic problem, which is you have a list of things and they're in a hash table, right? You just you have a hash table. Yeah. And it's maybe it's keyed on like the username or something like that.
00:36:26
Speaker
And you're like, gee, I really would also like to key it on their first name or something. And now you have two hash tables and you have to keep them in sync. And so you have that sort of classic problem where it's like, okay, now every time I do an operation here, have to make sure try I do the operation there. And that is what a database can do for you. And then you can add or remove different hash tables as you want. And your code doesn't have to be rewritten every time.
00:36:48
Speaker
But did you end up doing, um like implementing all those algorithms to basically build an in-memory database from scratch, like the internal in-memory database? Is that built from scratch or are you leaning on, I don't know what, they I'm sure there are off the shelf in-memory databases that you could have picked.
00:37:08
Speaker
It's a good question. um Not so much, we don't use anything internally, it's it's it's hand-rolled. Okay. um If I knew, i don't know that that's a great way to go. i think it, um, I think it has upsides and downsides. So the upsides are you fully understand it. Like I, there's a, you don't want to fall prey to the, it wasn't developed here, uh, disease, yeah but at the same time, there is a very strong benefit to having a full understanding of the system and the ability to change it for the particular workload that you have.
00:37:43
Speaker
ah because that can make an enormous performance difference, as we've found. And so it was too much of a risk for us to say, like, all right, we're going to go tinkering with somebody's thing. And then, ooh, actually, we didn't understand that this was the assumption that that person made, and it doesn't quite work for us. um So we just built it ourselves. It's not really that complicated in a way. It's just sort of a way of organizing your program memory.
00:38:11
Speaker
um If that makes sense. and And what I will say is like from the module as of today, ah the way you access the ah in-memory data is through what we call physical queries. And those physical queries are not through the query engine. So from your module, like the internals of your database, your program, your server.
00:38:32
Speaker
You do physical, like what we call data store operations. So the data store is the thing that is the in-memory data structure that holds your your data. And you can do things like, I would like to use this specific index to look something up, or I would like to scan the table, more or less.
00:38:49
Speaker
Um, That's to be distinguished from the clients subscribing who say, I would like to run this query, and then we optimize the query for them. But we give the power for the um the server writer to do whatever order of specific physical operations they want.
00:39:08
Speaker
ah Very shortly, we already have in some to some degree, we also want to provide that query engine to the the server writer so that they don't have to care about the particular operations that are done physically. but They can kind of rely on the query engine to do that. Or they can, of of course, just drop it down to the physical layer and say, like, no, I want to i want to do these particular operations.
00:39:30
Speaker
Right. So does that mean that as a writer of a program for Space Time DB, I am... What's my language there? Is it an API? Is it two APIs? One that's low level for memory structures and another one that looks like SQL? Yeah.
00:39:45
Speaker
Yes, that's that's essentially correct. So we have a query builder, a typed query builder, um that you can construct queries, and then you can send them to the engine, and then the engine will return the results, and they will be typed as well, because the query builder is typed. So you can say...
00:40:01
Speaker
um you know select but it looks like sql It looks like an ORM in the in the SQL Builder version of this. So you basically say, you know select from this table, where this, join this other table. And then we know the type of it because that we've built that up in the query. And then we can say, hey, here is your this row and your that row. And then you have them in your in your module. Alternatively, if you want to use the physical API, you can say, i would like to iterate this table.
00:40:28
Speaker
And so it will just... It'll give you all the rows in that table just as you asked for. um You can also say, I would like to, so you can do like, um the way you access all of the data in the database is through a reducer context, which gives you sort of access to the the database. So you hit CTX, which is the context,.db.
00:40:47
Speaker
um Then you would say.your table name, so like.player. then you would say if you had an index on a particular column, let's say it was the ID, you would say.player.id.find, and then you would look you put the value in. Okay.
00:41:02
Speaker
Yeah. um If it was a range query, you'd say.filter, and you'd provide the range, and you would get the values within that range. Okay. Yeah, that makes sense. it Also makes sense that maybe you wouldn't go to writing actual SQL because you want to keep that type information within the compiler, right? Well, Exactly. So that's actually the tricky part. The clients currently ah write SQL, like string, and that is fine. It's not really very complex SQL queries, but we are also going to be adding the type of query builder to the client because why not? We have it. We're going have the building for the server. So why not also have it for the client? And that way people can just get the, um that they can construct their queries of the right type. Right now we do return queries, um,
00:41:46
Speaker
typed. ah But that's sort of relying on you putting the right table name into the query. Right. Yeah. yeah Okay. ah and So from there, i kind of want... Because we've got this dichotomy or hopefully symbiosis you building a game and a database, right?
00:42:09
Speaker
But if you're really a database company, while you're being a game company, Something about the way you said ECS made me think, well, is this database just useful for game-like things?
00:42:23
Speaker
And is that fine? Or do you think the model applies more broadly? do you have ambitions to be a generic database? This is a very interesting question. um So ah what I would say is actually...
00:42:37
Speaker
We are fully a relational database. There is actually no feature in Space NDD that is specific to games at all. um It happens to be that the ECS model is essentially the relational model.
00:42:49
Speaker
There are other reasons you would use the ECS model in games, which is the physical representation of the data and the way that games typically query those things can be very well optimized for the current hardware and caching and stuff like that.
00:43:03
Speaker
But the actual model is, in some sense, the relational model. They are one and the same. And and more ECS systems have recognized this and started building more and more powerful query engines. And the one one i would call out is is Flex, F-L-E-C-S. F-L-E-C-S. It's written in C, I believe, C and maybe C++. plus plus isnt ah And they are adding additional... um query capabilities to that.
00:43:29
Speaker
But to your point about is this specific to games, um it certainly is not. And in fact, we're making a big push into web. So we've just added TypeScript support on both the server and the client. So you can now write in SpaceMDB, as of this moment, um TypeScript server modules. And we think this is actually extremely powerful for web in a way that um I think really has not been explored.
00:43:56
Speaker
ah And one example of how I would say that's true is ah it it fits really into the the modern web frameworks extremely well.
00:44:11
Speaker
So something like React, actually the the name reducer was originally inspired by React Redux, yeah which has the concept of reducers. In fact, React has a use reducer hook that is conceptually very similar to SpaceMDB.
00:44:27
Speaker
um And what you can do in in our React library that connects to it is you can just say in your component, as though you were saying useEffect, you can just say, useState, you would say useTable.
00:44:42
Speaker
And what we will do in the background is, using just our regular sort of TypeScript SDK, synchronize the state of your table with its with the particular where clause that you want,
00:44:53
Speaker
into your React state. And then any time those results change, we re-render your React ah client a And so having a real-time, and I mean real, true, honest-to-goodness real-time React app is as simple as saying use table and use reducer. And use reducer is a function that you can call that will cause tables to change. And that's literally it.
00:45:17
Speaker
And presumably from what you've said so far, when the server says that the table has changed, you are just shipping the individual rows over the network. So it's bandwidth. That's correct. Yeah.
00:45:28
Speaker
Right. And then when I, as a user of that web page, do something to a particular table row, you're going to treat it just like the collision detection thing you were talking about. you're going to optimistically say the table has been updated, send it to the server. And if the server rejects it, it will reject my local change.
00:45:47
Speaker
So that is correct. we don't I just want to be clear. We do not currently do that in the TypeScript SDK, but there's nothing preventing us from doing that. In fact, if we were to take the database engine and run it in the client, you could do all of this stuff like incredibly real time where it looks as if you're interacting um instantaneously with the server so the server is locally and we're actually making changes and only if there's a disagreement do we roll anything back or say that like hey the you can't do this transaction because x y or z happened so you tried to um transfer out of this account into another account but you had already done it in another tab you would want to do that because you don't want to say like yeah it's great it works great uh unless that actually is the case right I think because I know we touched about on this the last time we spoke, but I know people are going to wonder. So I think I have to get you to recap the security model for this, because if I'm running essentially a subset of the real database on my machine, it sounds like a security hole on the surface.
00:46:47
Speaker
um So I guess there are a couple of things to think about. So ah assume we're not doing client side prediction just in the in the first case. We can always add that in. um If we're not doing server side prediction, what we have is almost the exact same model that we have today. So you have um your server, you have functions, you're calling on your server. They're just RPC calls. There's nothing really fundamentally different about that.
00:47:09
Speaker
um Those RPC calls, do um authentication, ah they do authorization. um So we we're OIDC compliant, so that's OpenID Connect. So the thing that you see, like signing with Google or whatever, um that's all something we support. ah So your users authenticate with SpacetimeDB. We then send your module that you wrote like, hey, this is the ID card, more or less, of this person who is connecting. You do whatever internal logic you would like to do to decide whether or not they can make a change to the server or not.
00:47:44
Speaker
Um, and then, um, if you say no, by the way, you just roll everything back and it was as if they had never been there. So that's, that's even more secure than it currently is. Cause what, what can happen in, in regular servers that aren't transactional is you can throw an ever in between transactions and half the thing has happened or something like malicious people can kind of mess with your database state if they know how things work internally. Um,
00:48:07
Speaker
It's very hard to guard against that because you need to make sure that your error boundaries are right and you've got your transactions all set up and there's performance considerations because now if you're doing very long running transactions, you're locking rows and all that stuff. But otherwise, it's it's quite the same.
00:48:19
Speaker
And then on the read side of that, um you can have private tables. Those are just not visible to anybody outside of the database. You can have public tables. Those are visible to anybody. And then what we have now, which I only talked about theoretically last time, was the concept of views.
00:48:35
Speaker
right So a view is a table that you construct with a ah procedural function ah inside of the module, and you return rows from that. And then it looks like a regular table, but it's actually constructed um with your procedural code. So you can also do whatever arbitrary logic you want to do there. So, for example, I can say...
00:48:59
Speaker
ah This player, let's say I had an inventory table where private or a DM, let's say we're doing a chat chat app and we make a DM ah table.
00:49:10
Speaker
I can see in that table only things that were addressed specifically to me. So there's a two column. And what I do is I just return. i look up by the person who's calling the view.
00:49:24
Speaker
the All of the rows in that table, the sort of the underlying private table. that are addressed to my tape user and I return those. And then Those ah rows are sent to me and only to me because anybody else would not have that same situation.
00:49:41
Speaker
And the better part about it is they're also updated in real time. So anytime anybody sends me a DM, it's automatically sent down to my client. And that's how we apply that. Right, yes. And it's automatically parameterizable, right? Because I could do this in a regular relational database, but I'm not going to create 20,000 views for my 20,000 users. Correct. And I need something like parameter level security or and know
Transaction Throughput and System Efficiency
00:50:06
Speaker
Exactly. And we we actually went through an experimentation phase with rel of security because i can there's a performance considerations with views that I could talk about. um
00:50:14
Speaker
But yes, you can you can take a view. So if you use a regular function, you can pass in parameters. um Your identity is a parameter. I think we don't expose like the timestamp because we you could do that. But that's like, well, now we have a lot of views. We have a view for every second.
00:50:29
Speaker
um ah we So we return these things and we materialize it. So if anybody else wanted to query the same thing. So in in this case, ah nobody else would be querying the same thing unless I had multiple devices all with the same identity.
00:50:42
Speaker
um But ah in a lot of cases, for example, in games or in something else, you might have something like a leaderboard and you want to compute that on the fly. yeah And everybody's going to be sharing the same view. So we will materialize that. And then if somebody else shows up, we just return the results. So we don't actually call your view again.
00:50:59
Speaker
your Your view function, yeah your view constructor function or whatever you want to call that. And that really helps with with performance as well. um But yes, fundamentally, they are parameterized. Okay. This makes me want to ask, because if this were any other database project, the next thing I would ask, which is slightly different because you've already but written a game on it, I would say, like, what about performance and what about network problems, right? If you've got 20,000 users collecting to their own sub view and occasionally messages get lost, what's going on there?
00:51:30
Speaker
Yeah, so this is the only benefit of having developed a full MMORPG just to prove out your database, which I would not necessarily recommend. But it really helps us on performance because we know without a shadow of a doubt that we can do these things. And so 2.0 of Space 9DB is all about performance.
00:51:50
Speaker
We're coming to web. ah And um we've been doing some some profiling, some performance ah benchmarking against some existing databases. ah and It's really fast. So we're very pleased about that. um i'm I'm not going to share the numbers right now because we're going to release them when we've got them solidly.
00:52:12
Speaker
ah But we can do um multiples times more transactions per second for like a web workload. So imagine you have a bank or something and you want to transfer money between different accounts. Yeah.
00:52:26
Speaker
One of the problems that you have ah in something like a normal relational database that you host separate from your server is that ah if you want to do a transfer, you have to get the balance of somebody in your server.
00:52:41
Speaker
look at that balance, get the balance of another person, look at that balance, make sure that they have enough money to to do the necessary transfer, and then write both of those rows back into the database.
00:52:53
Speaker
If you're not doing that in some kind of stored procedure or so or something like where you're preparing the full query and sending it over in one go, you're holding those locks on those two rows for...
00:53:06
Speaker
milliseconds like hundreds of hundreds in a worst case but like tens of milliseconds in sort of a normal case or even let's say at least one millisecond which means you either open yourself up to deadlocks or um you've got like a huge serialization problem on the entire transaction system you've got a huge serialization problem and your um transactions per second just goes into the floor. Consider, for example, even sort of ah a normal case for something like where tracking balances.
00:53:35
Speaker
um You might want to have an account which which represents All of the the maybe you're making an AI startup ah and you have a point system for like how how much LLM juice you've used. Right. So everybody's got a balance. And then then you want to track like the total amount of balance used. And so you ah did every time that you do an action, you deduct from their um LLM balance and you add it to like total amount used because you're doing double entry bookkeeping.
00:54:05
Speaker
Well, every single user that's doing any action is taking a lock on that some summary account, the sort of the total used account. So now you've got this serialization problem where actually you can only do one transaction at a time, and each transaction is taking at least one millisecond. So you're capped out at 1,000 per second. That's just...
00:54:24
Speaker
maximum you can possibly theoretically do, yeah yeah assuming that's happening. um And so that that turns out to be very, very slow. ah In space time DB, because you're in the same process as your data and it's in memory and it's just right across that WASM boundary, um which is important for security purposes, but it's just right there, you're talking like tens of nanoseconds to be able to access it.
00:54:49
Speaker
So that's that's a well, I suppose from milliseconds to, net like, let's call let's call it one microsecond from, it's a thousand X performance improvement, right? on the On the ability of you to get that thing and hold that lock for a very small amount of And you have some numbers to back that up because games do these kinds of transactions pretty regularly, right?
00:55:11
Speaker
Yes, well, it's not so much that... We're doing a performance benchmarking now, but it's not so much that. It's that um it's the only way... We kind of have backed into it It's the only way that you could get this to work for a game because you really can't spend...
00:55:25
Speaker
very much time on this. I'll give you one example of something that like people might not think about for performance profiling or or have maybe a different viewpoint on that only through bashing our faces against it have we fully appreciated the importance of. um Very commonly, for example, in ah let's say the web world, what you would do is you have um an asynchronous scheduler. So a in JavaScript land, it's single threaded, you do sort of one thing at a time, that's actually pretty good, ah and it goes in an event loop, right? And so you get a request in that triggers a thing, you do a function, you do a callback, you return, and so on.
00:56:05
Speaker
um You might think like, okay, this is really great. We're going to multi-thread the crap out of this. So we're going to take that thing and we're going to do a bunch of ah transactions um all simultaneously and we're going to return there.
00:56:19
Speaker
um The problem is if you're modifying the same data, those are being scheduled to different threads because you're actually are sharing a thread pool and those threads are being scheduled to different cores.
00:56:31
Speaker
And now um when you go to access your data, it's actually been modified on another thread So it needs to be flushed out of the cache on that thread and loaded into the cache in the thread that you now have. And so you're sort of always operating only in L3 cache, which is shared across all cores. yeah So L1 cache and L2 cache, they basically might as well not exist.
00:56:54
Speaker
Everything is in L2 cache and you're constantly sending data across cores from one to the other. yeah um Not to mention things like false sharing, which is... you appear to not be accessing the same data, but actually they're preload, they're prefetching the data from one thing to the other. So if I modify, so like, let's say I grab a ah collection of memory cause I want to load it into memory. Um,
00:57:18
Speaker
And another thread is modifying something that I am not modifying, but I have prefetched. Well, once they modify it, it needs to be flushed out of my cache. And now because I need to get the fresh data if I'm going to read it. yeah And so there's all of these sort of hardware things. So what we have actually found to be the most performant by a mile is single threaded.
00:57:41
Speaker
put it all on one core, one thread per core. just you this is your This is your computer. You get the whole thing and then just using L1 cache very efficiently. And that gives you way, way more performance.
00:57:53
Speaker
Then do you have a mechanism for set? Because it seems like you're wasting cores there and that's becoming very important. you Have you got some mechanism to say, well, these tables only live on this core?
00:58:05
Speaker
Yeah. Those tables only live on that. Can you shard tables up? It's a good question. I guess what I would say is the naive parallelization thing that you could do where you just say like, OK, throw a job over to this thread, do that job and move it back and forth. You're constantly swapping between chords.
00:58:20
Speaker
We were in a situation where we were seeing, um, bit sort of small transactions. They were taking like a millisecond. So we could only do like 16 in the timeframe that you could, on the client, render a full frame with millions of triangles. So like something was not right. um And it turns out that was L3 cache.
00:58:42
Speaker
This simple version of it is extremely fast. Could you do something more intelligent and say, There is no data sharing here, um so I'm going to sort of load up these sets of tables. You certainly could. We do not do that yet. I think what we have learned is that you really want to suck all of the performance out of a single core and have a full mastery of how the hardware works before you even begin to think about multithreading. And that's in a language like Rust where we don't really have um a bunch of... ah
00:59:19
Speaker
you know, thread synchrony bugs, which you would normally have if you didn't have RUF kind of protecting you from accessing things from different threads. So it's a very, I guess it's more advanced. People think of multithreading as advanced. It's way more advanced than you would think. You can certainly do it. Will you get better performance?
00:59:35
Speaker
Probably not. the The same thing that we've experimented with on the same vein is is multiversion concurrency control. So um are you familiar, I suppose, with multiversion? maybe I can give kind of a brief. yeah give me I know it from Postgres, but give me your summary.
00:59:49
Speaker
Sure. So the summary of multiversion concurrency control is I'm going to do a lot of transactions um all at the same time, concurrency. And in order to achieve that, the transactions are going to look at multiple versions of the data.
01:00:02
Speaker
ah And the way that that actually works, for example, in Postgres, as you're talking about, is um when I make a change to a row, actually, I will create a new row that which is only visible to my newly running transaction.
01:00:17
Speaker
um And old transactions that are still running, instead of having to lock that row for that person to write on or something like that, I will just continue to look at the old transaction that I was, the old row that I was looking at, and my transaction runs, and I can sort of keep them in separate worlds. This is this, you get this version of the data, you get that version of the data. Actually, Git is an example of multi-version concurrency control. I have my local repo.
01:00:41
Speaker
You have your local repo. We have to do merge conflicts when we merge. But I am operating at the same time as you're operating. sort of like slow, human-scale MVCC. Yeah, yeah. I'd not thought of it that way, but it is. Yeah, okay.
01:00:55
Speaker
It really is. And actually our original MVCC model was inspired literally exactly by how Git works, where you have sort of like a a main branch and you're appending ah transactions to it more or less. And as long as they don't overlap. So if you didn't touch any of the files that I looked at, yeah then we can't possibly have. ah But Git has the luxury on conflicts of like, OK, that's user problem. They can sort it out manually. You don't have that luxury.
01:01:22
Speaker
Well, we can we can like we sort of have the ability to look at what your eyeballs looked at. So if we if the analogy would be if you recorded every line of code that you saw when you downloaded it, if you didn't look at anything that I changed.
01:01:39
Speaker
then you couldn't have been influenced by my change in any way. And as long as you didn't write to anything i didn't look at, we can sort of just merge these, no problem. okay But if if we detect a problem, what we're gonna do is we're gonna basically say, pick one of them, and then have the other person redo their work on top of it, sort of like a rebase.
01:01:59
Speaker
um and And so now you can look at the things that I have done, because I have done them first. We've decided the order. ah It's a little hand-wavy because in code, like there's sort of long-distance dependencies, but for a database, it actually does work.
01:02:16
Speaker
um But it turns out, by the way, if you do all this, and I only know this because we have done all, if you're not very careful about it, the overhead of the bookkeeping of multiple versions can quickly destroy your cache performance to the point where It's just not worth it. It's just like actually just better to put those suckers in cash and just run over them one at a time.
01:02:40
Speaker
The old fashioned way. Yeah. Yeah. It's funny. This is this is constantly surprises me. And I remember talking um about DuckDB and it surprised me. But the if you if you line things up for like the very core of the CPU and the way it's architected, you get such a performance boost that.
01:02:58
Speaker
that a dumb approach that way just blows a sophisticated approach that doesn't think about the hardware out of the water. Yes. And so I think that the point there is not like, you could you get more with, with parallelization?
01:03:14
Speaker
Probably, but it's not even, it's not the first order solution or even it's like, you really have to be,
01:03:22
Speaker
Very careful. But if you do, I bet you could you could get more performance. ah Not in all cases, either. But yeah, it it is it is such that like cash is king, as they say. And um ah yeah, it it just it can be 100 times faster. If you're going to L1 cash instead of L3 cash, I don't know the particular numbers, but I'd be really interested in looking at latency numbers um to to give you a number at least 10 times faster. I think we went from...
01:03:51
Speaker
Yeah, something like ah a millisecond per transaction down to like 10 microseconds. Yeah. so it's up of whole So that means I have to be running on 100 cores anyway to just get that same speed up. Yeah, yeah.
01:04:03
Speaker
And then you've got contention problems. So what if we just did them really fast? There's another database, actually, that has a very similar um conception. And I give them all the credit because, ah you know, I think we...
01:04:16
Speaker
we believed that the numbers we were seeing come back and we knew this was a good direction sort of because of them. Uh, and that is Tiger Beetle. So they have always had this, uh, philosophy as far as I can tell from the conception of the database, single threaded, they do like a million transactions per second. Um, which means by the way, you only have one microsecond to get them done. yeah Um, and, uh, they just say, you know, dead simple, right in Zig, know exactly what you're going to do No dependencies. Uh, we're going to,
01:04:46
Speaker
We're going to make sure it's um it's fast. And I think there's something to be said for that. OK. Tell me about. And I guess the the the last thing I'll say there is there's also like you get the benefit of, hey, there's no contention. You just you don't get a performance degradation if you're accessing the same rows. In fact, you get a performance boost often because it's all in cash.
01:05:07
Speaker
Yeah. So you're good to go. Okay, but what about in the large? Because you've um you've said that you've had to shard your game into multiple regions.
Scalability and Intermodule Communication
01:05:18
Speaker
There is there there is a size at which you say, even under these wonderful performance conditions, it's so large, we need to split this onto different nodes.
01:05:26
Speaker
That's great. does DuckDB support that sharding natively for you, or is it something you had to write on top? Space Time DB? is but so What did I just say?
01:05:38
Speaker
I think DuckDB. Sorry. ah but that's I've got that loaded into L1 cache. I apologize. Yes, fair enough. yeah not least similar We all relearn the same lessons, I guess. um So ah does it do it for you? um Right now, in in a sense, it does. and In a sense, it does not. Right.
01:05:59
Speaker
But it's a very interesting issue. So like let's let's talk about for one brief moment, I'm going to take an aside, if that's all right, ah to talk about like what would be an alternative architecture. So for Space Time DB, we load your database in memory. It fits on one machine.
01:06:13
Speaker
We put it in single-threaded, and you can just do the transaction super fast. All is well. What happens when your data is very large? So you just have a lot of data. So forget even about throughput of data. Just like quantity-wise, it can't fit in memory. um One thing that you can do is you can say, okay, well, I'll start caching it in memory only and I'll write to disk or i will write to object storage and I'll pull it in. That's going to hurt your your performance, obviously, because you need to go get it. but It's like sort of an L10 cache. It's very far away from the CPU, if you will, but the same idea otherwise.
01:06:50
Speaker
um The alternative is to say, um actually, what I'm going to do is... shard my data across a bunch of different machines and then sort of preemptively building that L10 cache.
01:07:07
Speaker
And then when I go to access that data, I will maybe pull it all into one machine, I'll run my query on it, and I will return the results or something like that. So that's something like... um In the case of like NoSQL, something like Cassandra. In the case of, like I would guess what it calls NewSQL, something like Spanner or CockroachDB, which are very they can handle quite a lot of load. um But you're always paying the cost of that L10 cache. it's sort of like um In order to do anything, I need to touch the network, right? So I cannot do a full transaction localized to one machine because by its nature, the data is sharded across. If I have one table, it's sharded across all the different things.
01:07:50
Speaker
Very important if you have a huge amount of data. But for almost any other workload, it's much slower ah because... You don't have the data. So you have to go get the data and you have to do the thing or you have to do the you have to ship your code over to the data and do it over there or whatever it is. It's one of those two.
01:08:09
Speaker
um And so ah for Space Time DB, it is always the case that we we want you to be able to do certain transactions locally.
01:08:20
Speaker
We intend to probably relax that requirement. So you can imagine a new table in Space 9DB, which is just called, which has a property called like distributed or sharded or something like that, where the latency of access goes way up because it's actually spread out over multiple machines or it's,
01:08:37
Speaker
partially cached in S3 or something like that. And you don't actually know if you look up a row or you do a scan, is that going to look it up in memory? Is that going to go to disk? Is that going to go to the network? Is it going to go into another region? ah Who knows?
01:08:50
Speaker
um So you certainly can do that same model. ah But for games, it's just completely untenable. You could never access that in sort of any of your hot loops. So what we do instead for SpacetimeDB is is um we give you, the programmer, the ability to shard the data yourself so that you know within a particular module, you will always have the data that you need in memory.
01:09:18
Speaker
um So what we do for you is, and this isn't released publicly, it's only something we use internally, although you can actually, with a new feature, you can hack and hack it into your own stuff.
01:09:29
Speaker
um ah We do intermodule communication. So what that means is, You can send a message from one database to another database, um and we will ensure that the message gets delivered with exactly once semantics is the idea.
01:09:45
Speaker
and so I've got to pick you up on that. You can't say exactly once semantics is the idea. Do you have exactly once or not? So in the internal, this is why we haven't released it publicly in the internal version. It's, you know, uh, it's, it works, but it's not quite, okay we want to release it publicly when we can actually do that.
01:10:02
Speaker
Um, and we can in, so exactly once semantics is very difficult to achieve, it requires that you remember all the messages you received because you need to know, have I gotten this message before or should I give it to them? So if you can verify that you have or have not gotten a message, um,
01:10:19
Speaker
ah then you can know whether or not you want to like surface that to the application. Yeah. You need item. Fortunately, we're storing the full history of all the transactions. So we do have that information. And practically speaking, you you have a cutoff on that or something. So it's like, you know, it hasn't gotten to me for a day, it's not quite exactly once. It's like exactly once as long as it's not delayed by more than a day. Okay.
01:10:41
Speaker
um In which case you probably would get um potentially more than one function call. And you can have that be configurable. You can say like, hey, look, i i I either want to hear about it more than once or I never want to hear about it, that kind of thing.
01:10:56
Speaker
um Okay, so so so you can send messages. ASC, do you have a follow-up question to that? Yeah, um because you've hinted at it, and we have talked about this offline very briefly, but the idea of databases communicating with other but databases. i know you have plans for the game where people can start to program their own space-time modules and upload them into the existing game.
01:11:22
Speaker
ah Correct. and And so they would be able to, I'll give you an example of how that's possible. So we have this system of being able to communicate with with databases, which is not unlike the internet. You can sort of send a message to another machine. The only difference is each actor in this model, this actor model, if you will, ah is persistent.
01:11:42
Speaker
And so it has all the, we handle the persistence for you and all that stuff. And you can call functions on other databases, more or less. um And so what we could do, for example, in our game or in your application or whatever, is um have a developer register a module with my module, the BitCraft module, and I could delegate out to it various questions. So, for example, if I had an alliance system, I could say...
01:12:14
Speaker
um An alliance owner can create a piece of software that will vet new people who apply to join. So a player in my game goes up to the alliance board or building or whatever it is and says, like, hey, I would like to join this alliance.
01:12:32
Speaker
Sends a message to the BitCraft module. The BitCraft module says, oh, this alliance looks up its registry, has a module that ah determines whether or not this person should be able to join.
01:12:44
Speaker
I send a message to that ah module. That module looks at all of the data that it wants to about the game that it needs to. So like, is this player above a certain level? do they Are they friends with those people? Are they in that alliance do that we don't want to associate with? Whatever it is they do. And then they send a message to ah the Bicraft module and we allow them into the the thing or we deny them as per whatever the alliance said. And in this way, you can sort of have applications that are not only um sort of sharded or or made of many actors, but made of many actors that are controlled by different real world people.
01:13:22
Speaker
And they can do whatever whatever you program them to do realistically. And so that is why we we do really think of the database as sort of an actor model system.
Resilience and Licensing in SpaceTimeDB
01:13:33
Speaker
ah And
01:13:37
Speaker
I think the thing it's, it's really a distributed operating system and that it's sort of similar in some sense to like beam and Erlang and so forth. Yeah. Yeah. Um, but it also builds in the concept of, well, I don't need to trust.
01:13:52
Speaker
it It builds in identities and permissions, uh, which is really important if you don't trust everybody that's sending you a message. Yes, yeah. But it also builds in the idea of a persistent log of messages, which is something I've sometimes wondered if Erlang should have.
01:14:09
Speaker
Yes, it does. And and I think that's that's important. It's not important for just ah persistence, but also exactly one semantics and and that kind of thing. And I think that's the thing that really makes them... um I think across it, crosses there's like a threshold where if you can get across it...
01:14:28
Speaker
ops and DevOps and needing to be woken up in the middle of the night goes away. And and I think that is, um you know, if you look at the cloud right now and you have Kubernetes set up and maybe you get fewer alerts, maybe you get more alerts, I don't know, but one of those two, you cross a threshold um in principle if you can get the system to manage enough stuff where
01:14:55
Speaker
you are not having to manage your infrastructure quite so much. the the The operating system manages your infrastructure for you. um
01:15:07
Speaker
And you write your program. Now, that obviously does not solve you from bugs that you write, right? So if like I... write a module ah that gets into some kind of problem with the data, it's just, it's like, that's just what it is. I'm still going to get woken up for that one. If we have a gold dupe, for example, in Bitcraft, because somebody had integer overflow problems, which we have had, um then ah like, that's just what it is. ah But I am hopeful that we can build a system where ah you don't have to worry about like,
01:15:42
Speaker
It's abstracted over the machines, right? So for example, you don't to worry about machines failing and that kind of thing, ah which, you know, Kubernetes tries to do, and I think does do for the stateless version of it. But if you have persistent state, you know, then I have a stateful set here that that's not replicated, I have to do all this. So then you combine it with database. So now you have like databases plus Kubernetes. If you could just get it all together and you have replication, then in principle, you have sort of an unkillable operating system that's not localized to a single machine.
01:16:15
Speaker
it's It's a system that is distributed. a system a distributed system. and And that's resilient. yeah ah And so the dream is that your your operations problems, and your infrastructure problems can go away. And I would argue that there's one domain for which this is definitely true.
01:16:34
Speaker
um Something like Ethereum smart contracts achieves that thing, right? If I were to write one, it's very expensive. It's very slow. There's a lot of other problems. But um one thing you can say is that nobody wakes me up in the middle of the night to make sure that my Ethereum contract is running.
01:16:53
Speaker
And I think it's a very important property. So there there is an existence proof that this is possible. um We just have not achieved it for the cloud. Yeah. OK. OK. I always tense up when people mention some blockchain related solutions, but I will concede but that that feature. This particular one. Yeah. I would change the way it's implemented. But that feature is. Yeah. just I would change it, too. And I think we have changed it. But but that one feature is true. Just yeah kids it's very hard to bring down. Yeah. I have to concede that.
01:17:22
Speaker
um So does this mean you've thought about replication? Oh, yes. ah We support replication in the SpaceTimeDB cloud version of SpaceTimeDB. So SpaceTimeDB comes in two flavors, SpaceTimeDB standalone, which is available on our GitHub. And then we have the sort of more proprietary SpaceTimeDB cloud, which is a clusterized version of SpaceTimeDB.
01:17:42
Speaker
And it schedules all of your databases to various nodes and then does replication between those nodes. it's the replication basically because it's a write-ahead log-based system. You can just copy it Correct. Okay.
01:17:53
Speaker
Yeah, it just uses that. That makes me think, um and feel free to duck this question if it's like going way too far out there, but I've long been a fan of um Apache Kafka as a log-based system.
01:18:08
Speaker
One real strength of that is that it's basically a firehose for data. It can take and a massive quantity of input data and deal with it sort of asynchronously. And I don't think you're trying to replicate that.
01:18:21
Speaker
But one thing that Kafka's been trying to replicate, and I don't think has entirely successfully done it, is build an operate um build a usable operating database on top of that log-based system.
01:18:36
Speaker
Yes. So there, I think it's ah it is, you say it's not really the same idea. I think it is a similar idea. They're coming at it from like, ah we're going to start from just the logging component of it.
01:18:48
Speaker
um and build the database on top, it sounds like, and i I'm not super familiar with their um effort to do that. I certainly have used Kafka in the past. In fact, the original, when we we prototyped SpaceTimeDB in TypeScript, and the the backing store for the write-ahead log was actually Kafka. Oh, really? Way back in the day. Yeah, it was actually Kafka. So um I'm not familiar with their efforts to do this exactly, but it is the same idea. where're We're coming at it from like, a hey, we need the data,
01:19:18
Speaker
um I guess I would call it like the data
01:19:23
Speaker
reduction or like sort of like the state of the thing to be materialized. And we needed in memory and we need to use that. And that has sort of semantic meaning that like, this is the current value of this thing. And the log is sort of ah a thing that we have and is, is there. And, but it's a little bit more secondary to the product as it is right now.
01:19:42
Speaker
um, But there's a lot of cool things we have planned for that. There's no reason you couldn't listen to that log as a stream, that you couldn't publish it to Kafka, that you couldn't um run real-time analytics on it, run some kind of streaming algorithms to say, like, oh, as I see transactions come through here, I'm going to count the number of ah different events that are happening, and and I'm going to do some various logic based on that.
01:20:05
Speaker
um we're not We're not focusing on that so much right now. Mm-hmm. Although we have some fun had some fun ideas there. um But it is, I think, a similar ah similar dream okay to what what you're describing. Yeah, there's this been this long push to, I think was Martin Kleppman who said, turn the database inside out.
01:20:24
Speaker
Start with an event log and build up an entire tower of database on top of it. Absolutely. and And I think that is, i mean, every database of database fundamentally is just a write-ahead log. Like, it's a write-ahead log and then it's really squint.
01:20:39
Speaker
And then they build, you know, various things on top of it. So... um I don't know of any database that doesn't use that. That's a gross simplification. I think a database obviously is also organizing the data in memory. Not in memory, but just organizing the representation of the data. but A very important property of ah databases, not all databases, but most most modern databases, is the separation of the physical representation and the logical representation. So that... ah
01:21:13
Speaker
you don't have pointers in your database data. That's crazy. um Then your database would be like stuck on one machine because that's like where it is physically located on that machine or you'd have to have the same kind of memory model, whatever that is.
01:21:26
Speaker
um That was the original problem that Edgar F. Cobb is trying to solve with the relational model. The thing that I find to be interesting is we did all of that for disks, right? Sort of separating the physical representation from the logic representation.
01:21:42
Speaker
For some reason, we didn't extend that over into memory for reasons that I can't really understand. um i guess in-memory databases do that, but like I argue that even program memory should probably do that and that you should have sort of like a a separation between my program that operates on my data model and the physical representation of my data.
01:22:08
Speaker
And then I could sort of crack open the hood and change the physical representation of the data. But that doesn't actually require any of my other code to be rewritten. And we have that. Because like if i if I use java.util.map, I have no idea how that's laid out in memory. I only care about the logical representation.
01:22:28
Speaker
You have that? Yeah, I mean, I would say you have that to a degree, but you can also, in Java, you have pointers in your data, and you don't know it. So, for example, Java is a particularly weird one, because like if I have a class, and I have a member of my class, and it points to another class, how do I serialize that object?
01:22:47
Speaker
Well... there's not anything I can do to serialize that pointer. I have to like walk the other thing. And so I have to build a program that actually like walks through nodes or or represents classes as a graph. And then I have to kind of build out that graph and provide some kind of representation for that.
01:23:05
Speaker
And that is fundamentally weird. it's mixing um It's mixing your program data with how it's physically represented. So even that, the hash table, not so much. In principle, you just have keys and values.
01:23:19
Speaker
But any objects, like objects are weird. Objects are a mixing of your ah sort of logical and physical data, I would argue. Yeah, okay. Yeah, I can see that. there's there's a there's ah like Especially if you're coming from like an ECS way of looking at things.
01:23:38
Speaker
I would certainly agree with that. ah If you're coming from like a C thing, I think it's even more manifest because you've had yeah your data that you actually are storing is pointers. Like in Java, they kind of put it in the background and sort of, oh, well, some pointers here. um But in C, like you have an array and it's an array of pointers. So if you want to serialize that, well, it doesn't make any sense to – if the data moves, the data changes.
01:24:03
Speaker
And I think that's the fundamental difference between them. Like if you should be able to move your data around and not have any of the data change. Yeah. That's like a property. Yeah. Portable data, I guess you could call that or something. Okay. um and And that I think is the, the key component. And that's not true of, for example, Java objects. Like if were to move one thing from another place, the actual data in my original object has to change because its pointer has to change. Yes. Okay. Yeah.
01:24:30
Speaker
Yeah. Now i see what you're saying. Um, Okay, I want to pull sort of up into a bit more developer space on this. Sure. Let's talk about client languages, because that gives a hint of where you're going. you You started off in Rust, partly for memory and partly because it compiles to WASM and that's a useful property.
01:24:53
Speaker
Yes, technically we started in TypeScript just to, make well, way, way, way. Oh, okay. When we had Kafka as the yeah and the engine. um And TypeScript is actually very, very fast. The problem that we ran into there was it wasn't consistently fast. So occasionally it would just take a millisecond to do something. And it was unclear why, presumably because it wasn't JIT compiled or something like that um in the engine.
01:25:19
Speaker
And so we switched to Rust ah for that reason. OK. And now you're coming back to TypeScript and C-Sharp? Coming and all the way back. C-sharp. So we support actually um four different languages.
01:25:32
Speaker
ah Rust, C-sharp, C++, sharp c plus plus and now Unreal as well on the client. ah And TypeScript and JavaScript, technically. Because it's just sort of... so We technically only support JavaScript, but you do can do write TypeScript for it. And we have a typing for all your stuff. What makes those the languages you've targeted?
01:25:54
Speaker
So ah Rust, because we we wrote the database in Rust, and um we wrote BitCraft in Rust. So we originally wrote BitCraft in TypeScript, where we had the thing which called janktimeDB, which was the sort of the TypeScript version of that.
01:26:07
Speaker
um We rewrote that in Rust because we needed the performance. We needed to be consistent. um The client was always in C Sharp for that. If we had a C Sharp client and we were doing Unity, we needed a C Sharp backend.
01:26:22
Speaker
So now we have a Rust backend and we have a C-sharp backend. We wanted to support Unreal because we had a lot of people requesting to be able to make Unreal. So if we're going to support Unreal frontend, we support C++ plus plus backend.
01:26:34
Speaker
And then TypeScript is the most popular, or TypeScript JavaScript is the most popular language in the world. And it's really important for web. ah we We have people deploying games to the web. We've had people deploying apps to the web. So we've always had the TypeScript SDK. And it sort of logically makes sense to do TypeScript on the server as well. OK.
01:26:53
Speaker
I can see the games angle. tell Tell me about how people are using it or how you hope they'll use it for something like the web. I mean, what's the least gamish application of this database?
01:27:05
Speaker
The least game. So ah we have, well, I would say the least gaming one is actually not necessarily a web one even. So we've had, uh, ah you know we there some of we have some enterprise clients, and I wouldn't want to give up exactly who they are whatever, but we have financial clients that are doing order book matching, that kind of thing. So um I have a list of people who want to sell things and a list of people who want to buy things, and I want to find out who's going to buy and sell them in the right order, and I want to match those things and and sort of have transactions fulfilled.
01:27:39
Speaker
ah that is obviously not very games. um We've had that for a long time. They're doing it in Rust, so it's actually not TypeScript. You certainly could do it in TypeScript. um We've also had people build, like, for example, um there's ah one ah startup that's doing... um something like Figma, but for stream overlays. So the idea here is if I'm a Twitch streamer, I can have my mods moderating my stream overlay while I'm on stream and I don't have to actually do that. So they they will drag in like memes and things and counters and stuff like that.
01:28:16
Speaker
And they will be represented ah on the screen and they can move around in real time and all that stuff. um It's like a canvas that they can paint on and that, That is updating in real time.
01:28:27
Speaker
that's So that's one example something. It's a live and collaborative broadcast application. Correct. Yeah, exactly. And they I believe they have plans to also allow ah Twitch chat to interact. I think they do already because they have like polls and things, but um maybe even to manipulate the canvas. So you can imagine Twitch chat drawing on the canvas that is shown ah overlaid on the stream.
01:28:55
Speaker
Okay. Yeah. and And inevitably, given the permissions model you've got, some Twitch streamers get to pay to get more access to the canvas.
01:29:06
Speaker
ah Yes. well that that I assume... you know We're not involved in those decisions, but that this seems like a reasonable thing for them to do. Yeah, yeah I can totally believe that. Okay. So...
01:29:18
Speaker
I think it's time to move on to how one might write something with this. I get the basic model. What does it look like if I say, I'd like to play around with this and build some collaborative web thing?
01:29:31
Speaker
Yes. So the goal, what we're trying to do ah is have this be as simple as possible. A lot of what we've done in the past is make it very, very performant, um make it very well designed.
01:29:42
Speaker
correct, um and then production ready and able to run our game. We've launched our game, and I think the goal now is to make it as easy to use as possible. And we're doing a couple of things to to achieve that. One is we're adding a bunch of templates. So the idea is...
01:29:57
Speaker
ah You should be able to come to our web page, look at a template, say, like I would like that template, run spacetime init, the template name, and bam, you have a running application locally.
01:30:09
Speaker
ah We'd like to do that for web apps. So like a very basic one that you might want to build is like a real-time chat. Maybe you want to build something like Discord. yeah um That's either the perfect thing to build on SpacetimeDB in some sense. It's real time. It's not so easy to build if you're going to do with regular server. It's all persistent.
01:30:24
Speaker
um You have complex permissions that you need to keep track of that are editable and all of that stuff. um So we want to build some some example applications that people can just kind of pull off the shelf and really get an idea of like, what can I build with this thing?
01:30:36
Speaker
um We want to become more accessible. So we want to ah do things like support TypeScript, um support all of your favorite web frameworks, so Svelte, React, Vue, whatever else is is out there that's popular so that you can just kind of one-click drop-in use table and be ready to go.
01:30:58
Speaker
um ah In Unity, right we have an SDK that you can just drop in and get started there. more tutorials there. We're really looking to improve our documentation. And then one thing we'd also like to dramatically improve is our pricing models. So we launched our cloud platform um in March.
01:31:19
Speaker
We actually haven't charged anybody any energy yet. So people have bought energy. They're sort of stockpiling it. A lot of people, some people stockpile a lot. Um, but, uh, we haven't charged any, we wanted to gather statistics on how is this being used before we actually, um, begin charging people as part of that. We've learned like, Hey, we should just really simplify the model. We're, we're getting very granular here. It's kind of hard to understand.
01:31:46
Speaker
There's not really a point to that. Like what people want to know is how much is this going to cost me fundamentally? So, uh, we're doing a couple of things. We're adding a free tier. ah So people can just deploy to their cloud version of SpaceMDB for free and run basic apps, no problem.
SpaceTimeDB's Future and Community Engagement
01:32:04
Speaker
We're also going to be doing a pro tier. So that's going to be $25 a month. And then you get a bunch of energy, sort of ah credit every month. So you can just keep spending up to that limit. And then we're going to do a team tier. And the team tier will be like a named organization on GitHub. So you can have your whole startup join and um you can...
01:32:24
Speaker
ah have complex permissioning to allow these people to publish or these people to do other things like manage users and things like that. um And then we're also adding a first, we have already added actually a first party auth solution called Spacetime Auth. And that's kind of like Firebase Auth or Supabase Auth or things that people are probably familiar with.
01:32:45
Speaker
Just lets you get started right away. You get auth out of the box. It's OIDC compliant. So you can add sign in with Google, sign in with Facebook, sign in with whatever. ah Twitch. um I don't think we have Steam yet, but we would like to add Steam. Makes sense. And that kind of thing.
01:32:59
Speaker
Okay. What's the licensing for the free version if I want to run it locally? And how how useful is it? so i have to Is that just for development? Yeah, that's a good question. So it's not just for development. So there are there's a free main cloud version. That is basically, we're going to run it for you. You don't have to worry about anything. It's going to run on Space 9DB Cloud. ah You can deploy your app. There is always the Space 9DB standalone.
01:33:25
Speaker
variant, which is the one that you can get off of GitHub and you can you can run it. We we use the um business source license, um which the way that that works is ah you can use it for development for anything.
01:33:39
Speaker
You can use it ah in production for anything except a set of things that the developer provides, and I'll talk about those. And then after four years, it goes open source.
01:33:51
Speaker
Right. So that's the that's the agreement um in in summary. The things that we prevent, sort of we say like there's a limit on, is you can only run one node in production, and um you can't resell it as a service.
01:34:04
Speaker
Yeah. And so let me explain why. um We would like to be able to provide the the cloud version, to sort of the highly scalable cloud version. And we would really not like Amazon or someone like Amazon to have Amazon SpacetimeDB.
01:34:21
Speaker
And then sort of we've done all the development for it, but they have all the clients. And so they'll they'll bring them there. So that's that's the reason for that license. yeah ah But ultimately, it becomes open source. Okay. I don't know there are lots of people out there that dislike those sorts of licenses. I think they're entirely fair myself.
01:34:36
Speaker
It's understandable. I mean, I think the thing is to um what you don't want to do, what we said we didn't want to do from the beginning is, oh, hey, we're open source and then realize this business model doesn't work and then kind of bait and switch people and be like, oh, we're changing the license.
01:34:49
Speaker
which I think a lot of databases have stumbled down that path. And so we had the benefit of kind of looking at that in hindsight and saying, well, let's start a little bit more restrictive. And then if it's like, you know what we think we can manage this business model, we we make it less restrictive. Nobody's upset when you make it open source. People are upset when you migrate it into closed source. Yeah, yeah. It's when you reduce what they can do for and no net. And and right, so it's the user,
01:35:15
Speaker
Correct. It's a change of the the deal. The deal is changing. yeah Always for the worse, always in favor of the companies when that happens. It can be. One thing I should mention is that ah we have announced that BitCraft is going to be fully open source. So the game. Oh, interesting. Fully 100% open source. We're choosing a license. The thing we're currently dealing with there is um we, in certain cases, we've used Unity Assets Store assets. So we have to actually remove them before we can make it open source. Otherwise, we're open sourcing somebody else's thing, which we can. Right. Well, that leads me to perhaps a closing question. What's the future of ah BitCraft? Because you spent all this effort building up a platform for a game. Is the game still very much at the front of your mind?
01:35:59
Speaker
So the game yeah is doing quite well ah for MMO. um We ah have been selling it on Steam. We are evaluating um when it makes sense to go into sort of full release. And the game was always meant to be free to play. So we have in-app purchases. Basically, can there's an empire system where you can have these skins and things, and you can put out banners in your world. And we're trying to make sure that that monetization model works. um So the game is is plugging along just fine. um ah It is hard to imagine, though, that Space Time DB...
01:36:34
Speaker
ah is not sort of the magnum opus of the company. It, you know There are a lot of companies for whom this has been true. um So like Discord started as a game company. I think Slack started as a game company.
01:36:48
Speaker
um Unity started as a game company originally. yeah It just turns out that it's so difficult to build a game that you end up having to build infrastructure that yeah tends to be useful for other people. I was reading about, who is it, Gabe someone, the um guy behind Valve who ended up... Gabe Newell. Gabe Newell.
01:37:06
Speaker
and he's recently taken delivery of a super yacht. He used to be a game company. think he took delivery of the yacht company, if I'm not mistaken. I think he bought the whole thing. That's because he migrated from games to being a game delivery infrastructure company, right?
01:37:21
Speaker
Yes, yes. And I think, i mean, that's true. There's no doubt that it's a good model. I mean, Lord knows we've paid a good amount of money to Steam at this point. We've definitely helped, we've contributed to said yacht. That's definitely true.
01:37:34
Speaker
Um, but I think that, um, the game is fun. We, we, we love our players. The players, uh, love the game. We want to make it more fun, but I think that
01:37:49
Speaker
there's just such an opportunity to not have to slog through server software development, um, that, that, That's our dream. I mean, I really, really, really would like to improve the general
01:38:09
Speaker
user experience of creating multiplayer, real-time applications. And so that's that's what we've got our eyes sit on right now. Yeah.
01:38:20
Speaker
I've built some web applications in the past, which You wouldn't describe them that way, but de facto they are multiplayer real-time experiences. And it's not been fun. It's certainly not as been as fun and easy as you are describing.
01:38:36
Speaker
so I think on that note, people should go and try and... I think so as well. And and I guess what I would say is if you look at some of the most most successful web apps, right, very few of them are are single player. You've got, I mean, look at Figma, right? Look at Facebook, look at Twitter. All of them are these multiplayer experiences, if you want to think of it that way. And they're real time.
01:38:57
Speaker
um MMO, I mean, ah Facebook is an MMO. It's a weird one and it's a text-based one um for the most part, ah but it is fundamentally massively multiplayer. And I think that is where ah a lot of the value that people get out of applications is, is interacting with other people. And it's been so hard. It's it's very hard for people to get started.
01:39:18
Speaker
We have definitely had people write to us and say, like, you know, we tried to build this thing for years. ah We... always hit a wall at some point and thank you so much for building space time db because now i can finally get past that and that is the most rewarding part of this whole thing yeah i can see that but also you've got the game and also plans on the yacht you're going to invite me on the yacht when you get it right on the yeah yeah i'm i'm not sure about that i'll be too busy building databases so we'll we'll see fair enough one for the retirement years Yes, I suppose so. On that note, Tyler, thank you very much for joining me. I look forward to bringing you back to Spacetime TV version 3 in a couple of years, right?
01:39:58
Speaker
Sounds great. oh Look forward to it. Thank you, Tyler. You know, one thing we didn't get into in that was how this all feeds into agentic programming. Because I think that really is changing the way we program. It's also changing how easy it is to adopt new technology, new techniques, like brand new databases.
01:40:18
Speaker
For now, all i'm going to say on that topic is they've been putting a lot of effort into their documentation for humans and for agents. So if you want to give Space Time DB a try and see what it's like, now might be the best time. Link in the show notes as always.
01:40:34
Speaker
While you're heading to the show notes, please do take a moment to like, rate or share this episode. It helps spread the word, especially as we're back for a new year. And we are planning more episodes in 2026. So please stay tuned and make sure you're subscribed. But for now, I've been your host, Chris Jenkins. This has been Developer Voices with Tyler Cloutier.
01:40:54
Speaker
Thanks for listening.