Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
What can game programming teach us about databases? (with Tyler Cloutier)
1.9k Plays10 months ago
The world of game programming might seem a million miles away from 'regular' programming. But they still have to deal with the same kinds of data, scale and concurrency problems that we’re all familiar with in the software world. And that makes the gaming world an interesting place for new ideas - under the hood they’re solving those same problems we face, but often with some novel ideas about the solutions. So this week we’re off to the massive open world that is game development, to see what we can learn that might make lives easier in the non-gaming space. Joining us for that is Tyler Cloutier, the founder of Clockwork Labs. They’re building SpaceTimeDB, a curiously-distributed database built to be the underlying platform for their new MMORPG, BitCraft. Digging down into the architecture of SpaceTimeDB, we pick Tyler’s brain for nuggets of information on event sourcing, request/response vs. subscriptions, transactions, security and much more. All in an effort to make our programmers and data scientists’ lives easier.
--
SpaceTimeDB: https://spacetimedb.com/
BitCraft: https://bitcraftonline.com/
“4X games” defined: https://en.wikipedia.org/wiki/4X
Plan 9 O.S.: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
Tyler on LinkedIn: https://www.linkedin.com/in/tylercloutier/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Kris on Mastodon: https://mastodon.social/@krisajenkins
--
#programming #podcast #softwaredevelopment #software #gamedev #gamedevelopment
Recommended
Transcript
Challenges in Game Programming
00:00:00
Speaker
One of the areas of computing that I'm really curious about, and I think the software world has a lot to learn from, is game programming. I mean, probably not on a surface level. I've never worked for an e-commerce company that needed collision detection. But get below that surface, and a lot of modern games, especially once they go multiplayer, they're dealing with things like global networking issues, multi-user concurrency, competitive concurrency,
00:00:27
Speaker
massive data volumes under brutal latency requirements. They have a lot of the programming issues that we're familiar with, but under much harsher conditions.
Introducing Tyler Cloutier
00:00:38
Speaker
And being a somewhat separate world, they tend to approach the solution from a novel angle. So this week, we're going to go digging for system design ideas in the gaming world. And my guest for this is Tyler Cloutier. He's got a background in distributed systems and data science for the gaming industry.
Data Handling in Gaming
00:00:56
Speaker
And he's currently building Bitcraft, which is a massive multiplayer open world game, and to support it, a really interesting flavor of database called SpacetimeDB, from which we're going to mind some ideas about concurrency, transactions, data security, query management, lots more. A lot of juicy ideas solved from an angle that I'd never considered.
00:01:21
Speaker
So let's get going. I'm your host, Chris Jenkins. This is Developer Voices, and today's voice is Tyler Cloutier. My guest this week is Tyler Cloutier. How are you doing, Tyler? I'm doing well. How are you? I'm very well.
00:01:48
Speaker
You're going to take me to a new world for me because I have a long history in programming, but one thing I've never done is computer game programming. Along with that, the one thing I've always wondered is they must have a lot of the same problems that the rest of us industrial programmers have. Dealing with graphics and story and stuff, but then there are data problems everywhere.
00:02:15
Speaker
That's right. Actually, I would say what I have experienced is that they have the normal problems that everybody else has, but times 100. Because not only do they have to build the thing, but it has to interact with all of the data in the program besides just sort of individual parts. And then it has to go really, really fast. Yeah. Yeah. All our problems except at 60 times a second. That's right.
00:02:41
Speaker
So your background is originally in game design or data science for games, or what's your origin story?
Tyler's Professional Journey
00:02:49
Speaker
My original background is actually in chemical and biomolecular engineering, which is completely unrelated to games. And then I did my master's in computer science, focusing on distributed systems and machine learning. OK. After that, I did some time at Bloomberg and then Apple and then a company called Machine Zone, which is a game development company.
00:03:09
Speaker
Right. And what did you do for them? So there I worked in their data science and engineering department as a data science engineer. And there we were building two things. One, pipelines for data, so making sure that we got the real live data as quickly as possible into a form that we could then feed into our models.
Free-to-Play Business Model
00:03:30
Speaker
So then the second part of what I did is also build those models, which predicted various things about how players are likely to behave.
00:03:39
Speaker
How likely is this player to turn? How likely is this player to spend money? Will they give us a good review? That kind of thing. This is a pay to play game. This is a free to play game, actually, that is quite expensive, ultimately. Because it's the business model where most players don't pay anything, but some pay for
00:03:58
Speaker
cosmetics, upgrades, that kind of thing. And they spend a lot. No, no, it's much worse than that. So it's a mobile game. And the whole idea of the game is it's supposed to simulate what it feels like to be a king. And so what that ends up being is that you have this little city.
Game Mechanics and Monetization
00:04:15
Speaker
It's called a 4X game, which is for the four different types of play that you're going to do.
00:04:24
Speaker
You build the city up, you upgrade your buildings, and then you can start sending marches out to attack other players. And ultimately, you want to capture what's called a wonder, which will make you the king of the kingdom.
00:04:40
Speaker
Capturing that wonder is quite an expensive endeavor. So the way it works is that they sell you speed ups. So upgrades take time. You can speed things up by paying for it. And some people pay quite a lot. There were individuals who spent upwards of several million dollars in that game. You heard that correct. If I did not see it myself, I would not have believed it.
00:05:09
Speaker
Gee, I can't. I struggle to compute why someone would do that and how they can be rich enough that that's their disposable income. There were Saudi princes. There were people of that kind. I mean, it was a global game. And so it attracted a lot of people who were interested in simulating what it felt like to be king. There was one person who was rumored to have hired at least one person or a team of people to actually
00:05:36
Speaker
purchase and open the packs because it's actually mechanically a lot of work to open $1 million worth of $100 packs. That's 10,000 packs. So there's that. There was another person who used to fly his entire alliance out to Las Vegas to be closer to the servers so that they could do the Super Wonder event more effectively.
00:06:01
Speaker
I mean, A, that's really weird, but B, is it that much weirder than traders putting their computers right by their main exchange? I don't know. I don't, I suppose not. I mean, these people really cared about the game. I asked players, you know, why are you so interested in this game? And I remember one told me,
00:06:25
Speaker
I'm a security guard at a place or something to that effect. And I just sat there all day and I downloaded this just to pass the time. But over time, I found that I had like real actual friends in this game. And when I would log on, they'd say, oh my God, this person's here. And he felt like somebody, whereas in real life, he did not really feel like someone. And that was important to him. OK, I can see that.
00:06:49
Speaker
But OK, so curious game mechanics, but that's not why I want to talk to you.
Historical Data Challenges
00:06:55
Speaker
The thing that's the reason I get into that is because you're there clearly in a background where there's serious amounts of data coming in live and serious money to be made in understanding the flow of that data. Certainly. Yep. Tell me about that and how it led into what you did next.
00:07:18
Speaker
Sure, so now this is an interesting story because it sort of obliquely leads into what we actually ended up building because certainly my time at MachineZone inspired it. But it's not the way I would explain exactly what we're doing, but let me tell you, I suppose, the origin story. So while I was at MachineZone, always we wanted historical data. So we wanted to know not only this is the current power of this individual or this is the current set of items that they have,
00:07:47
Speaker
But what is the full history of what they have? So we can predict, for example, hey, look, this person was attacked. They got zeroed out. And now they left the game. And they're not likely to come back. We always wanted to know that data. And machine zone didn't have that data because the traditional infrastructure of companies is to have their game data or their really website data in normal
00:08:13
Speaker
relational databases. In this case, I believe it was MySQL. And the problem is when you update someone's power in those databases, the old power goes away. So you need to have some kind of a way to actually get that historical data.
Lambda Architecture for Data Pipelines
00:08:29
Speaker
And what they started to do was they were snapshotting their databases every 12 hours.
00:08:34
Speaker
And we would then get that snapshot data and we'd try and piece together a historical data, but that was very sad for two reasons. The first reason is that the data itself was awful because a lot can happen in 12 hours that could cause you to leave the game, right? So you don't really have that information. And I should also say there was another stream of data, which was just event data, but it was very loosey-goosey event data that was sort of whatever people had slapped together.
00:09:00
Speaker
So you tried to build up a picture of what had happened historically from these two sources. And the other reason the snapshotting data was bad was that it was enormous. Because if you think about it, 99% of the data in a database does not change in 12 hours. If 99% of your players have turned, you're just copying this old data every 12 hours. And so eventually they had to purge the old data so they couldn't keep it forever.
00:09:27
Speaker
And they spent millions of dollars trying to clean up this data and get it into a form. We built a system which was based on the Lambda architecture. And if you're not familiar with how the Lambda architecture works, you essentially set up a streaming part of your data pipeline, and you set up a batch part of your data pipeline, and you try to weave those two together.
00:09:51
Speaker
all of your big, well-formed data in Hive, which is a right, append-only database made by Facebook for large data, and then you would have something like Flink or Apache Spark taking your real-time data and trying to make decisions based on that and bringing it in with your batch process data as well. There's a huge amount of work, and I would say,
00:10:18
Speaker
95% of the data science was actually just getting the data into the right form in the right place at the right time.
Building Bitcraft for Data History
00:10:26
Speaker
That's a very familiar statement that spans way past gaming. Absolutely. So when we began to build our own game, I decided I'm not going to have it. We are going to have
00:10:41
Speaker
Uh, the full history of the data. So I want to be able to go back to any point in time and actually see what the game state was. But more than that, I want to actually be able to replay it at that time so that you could hop into the game at that time and actually see it being replayed on that level of granularity, on that level of granularity. Right. So you're not just storing events, but like player thumbstick movements and stuff. Correct. Um, and actually I saw, uh,
00:11:11
Speaker
On an earlier podcast that you had, I think it was maybe two weeks ago, you were talking about event streaming. And the guest there said at the end, this doesn't always work for everything. It doesn't work, for example, for games. And I thought, aha, how wrong you are. In fact, this is exactly what we're doing. So event sourcing is essentially what Space Time TV does.
00:11:34
Speaker
Okay, that's colossal amounts of data, very widely distributed user base, high response times required because you've got to deal with things 60 frames a second ideally. That's a big challenge. How do you start to break that down into something? What's your approach?
00:11:58
Speaker
I think the best place to start is to first understand what the game is that we're trying to build. And then from that, you can see why Space Time DB is a necessary requirement. So we have two products. We have a game called Bitcraft Online, which is a massively multiplayer online role-playing game.
00:12:16
Speaker
You can sort of think of it as like a combination between RuneScape, if you're familiar with that, and Minecraft. So there's this very long-term skilling and progression in the game, but at the same time you can actually change and edit the world.
00:12:32
Speaker
and build your own things within the world. So that's the game that we set out to build. And in order to do that, notably the first thing you think is, well, we need to put everybody in a single world logically because you can't have people occupying the same space in the way that you could in a normal MMORPG because they actually are editing the world.
00:12:56
Speaker
So if I was a normal MMORPG, I could put many, many instances all in the same city, right? It doesn't matter. In our game, it certainly does because you actually need to do that. So now you have a very interesting distributed systems challenge on your hands. Yeah, you've got a large global mutatable state. Correct. And it has to be persistent. So if your servers crash, people want to have their buildings that they spent their time building and stuff like that.
00:13:20
Speaker
Yeah. So that's the first thing you have to understand about how we came at this problem. And so in order to do that, we realized, OK, well, we need a system which is built around the persistence. Because if we're going to be making these permanent changes to the world and it is everywhere, either we're going to be
00:13:46
Speaker
doing the normal architecture of games, which I'll take a brief aside to explain. So normal architecture is you'd have a game server. You'd have your databases. The game server itself, unlike a web server, would have quite a bit of state. And you, at periodic times or when people did important events, would write to the
SpacetimeDB Architecture
00:14:06
Speaker
database with a transaction. And then you'd get that back. But you, as the developer, are doing a lot of work
00:14:16
Speaker
maintain, I suppose, synchronicity between your database and your game server state. Because for example, if I kill an enemy, there's a lot of things you can do that make things pretty crazy. But if I kill an enemy, that enemy is probably not
00:14:35
Speaker
Their position is probably not stored in the database, but that there was an enemy might be. Or that you picked up items probably is, because if Susie enters your inventory, you don't want to lose that. Or you could do it periodically, and then they could do replays and stuff. What you end up getting to is a place where you're spending a lot of your time as the game developer not thinking about the actual gameplay programming, and rather thinking about the distributed system environment in which you're actually building this game.
00:15:05
Speaker
Yeah, I can believe that. So what we decided to do is say, we are going to make that all, I suppose, opaque to the game developer. And we're going to put them in a context where they're operating inside a transaction already. That transaction is going to be manipulating in-memory data. And then we are going to do all of the necessary things to persist that data to disk so that the gameplay programmer does not have to think about that at all.
00:15:35
Speaker
What's that look like for the programmer? They're treating their local client-side instance as though it's a relational database. Is this what you're saying? That is one consequence. But the main point of what I'm saying is that on the server, all things inside of BigCraft happen.
00:15:59
Speaker
within a database. So let me explain just a little bit about how SpacetimeDB works and what it is actually. So we built this game and we wanted to do this in this way. So we built a system called SpacetimeDB, which is fundamentally a database. So it's very focused on persistence. The way it works is you take Bitcraft, you compile it to a WebAssembly module,
00:16:23
Speaker
You upload that into the database, and then clients connect directly to the database. And now I hear a lot of people in the audience screaming, oh, you can't do that. You couldn't do that. And it maybe was a bad idea for databases like Postgres and so forth. But we have built a permissions model around Space Time TV that allows you to do that safely. And so the way that works is,
00:16:51
Speaker
You, as a client, call what we call reducers on the database. They're very similar to stored procedures. And then those stored procedures, which are written in whatever language you want, that compiles to WebAssembly, will access things from the database and write them back to the database. So just as an example, let's say we had player move. And notably, everything in Bitcraft, including all the player movement, all the chat, all the trees, all the ground, everything is stored within the database.
00:17:18
Speaker
So if we want to move a player, what we do is we call a reducer called movePlayer on the server. That updates some rows in the database and then commits those rows. And that's it.
00:17:28
Speaker
Then other clients will subscribe to the database state. So they'll say, I want to select star from player position. Where they are near me, basically, would be what that word clause would say. And then all connected clients that have subscribed to that, when that player moves, will hear about those rows in their updates and then automatically update it in the database. I'm sorry, on their local client. So hang on. Where is the database?
00:17:58
Speaker
The database is on a server, stored, in this case, I believe in New York. Okay, so how on earth does that possibly work when I'm moving and expecting things to update 60 times a second?
00:18:10
Speaker
So okay, this is the first thing that's interesting about games. You would not have a tick rate on the server that's 60 times per second unless you were making a game like Counter-Strike. So typically, I'll give you sort of a range. Minecraft updates 20 times per second, so every 50 milliseconds. RuneScape updates, I believe, four times a second, so every quarter of a second.
00:18:34
Speaker
So there's a variety of different levels that you can do. Generally
Enhancing Gameplay with Client-Side Prediction
00:18:39
Speaker
speaking, the larger your game is, obviously, the fewer updates per second you want to be doing, because there's so many entities to move within a time frame. Yeah, but I mean, I'm in London. I'm not sure I can guarantee full frame as a second to New York and back.
00:18:55
Speaker
I see. I understand your question now. With any game like this, you do not wait for the round trip.
00:19:06
Speaker
How long would the round trip be from New York to London? I actually happen to know it's about 80 milliseconds or 70 milliseconds, something there. So it's actually not crazy. You can play it without what's called client-side prediction. But the typical way that you would actually do this is you'd run client-side prediction. What does that mean? That means that I as a client have some subset of the server state. And when a player, my player, decides to do something,
00:19:32
Speaker
I can predict what the server is going to do to my local state, assuming that it will work. So based on the state of the world as I know it, if I try to move, I should be able to update my local state.
00:19:45
Speaker
Assuming the server will agree with me and so I will do that and then I will immediately see the results of that on my own local client But I will send something to the server now if the server agrees we basically come back we reconcile No problem if the server disagrees Let's say somebody exploded a bomb that your client hadn't heard about yet, right? Yeah, right on top of you And the thing you try to do is invalidated by that. Maybe you're dead now. Yeah, what will happen is
00:20:14
Speaker
You will have sent a request to the server that says I want to move the server inner Jackson says actually you died What will happen is your client will say oh I understand it will roll back to the point at which you were going to move and
00:20:31
Speaker
And it will then play forward the updates as they actually happen from the server and then try to replay your move. So it would go originally, before you heard about from the server, it would go, you're standing here, you move, you actually move. Then what happens is you hear about the bomb, you roll back to the point at which you were about to move.
00:20:50
Speaker
You then blow up because of the bomb, and then at that point you try to move, but you can't move because you're dead. That's how that reconciles. This sounds very like transaction on the client side. Yes, it does. Is there a database on the client side?
00:21:07
Speaker
Well, I believe basically everything is a database. I have more I could talk about that. But essentially, yes, although typically people don't think of it that way. So typically the way that people think about it, first of all, in games is with a tick. So on the client, you would have
00:21:25
Speaker
a frame essentially that happens, they would call it like a server frame if it's not the actual render frame. So the render frame always happens at 60 frames per second or sometimes now like 120 or 144 or whatever your monitor actually has. A server frame typically doesn't go beyond 60 frames per second.
00:21:48
Speaker
It assumes there's a loop, and basically we're going to update all the state once a frame. SpacetimeDB actually doesn't make that assumption. You can do that with SpacetimeDB, but it's not a requirement. So I'll just say that. And there's latency versus throughput trade-offs with that. That's essentially what that will end up with. Because if you have something taking 60 frames a second, the minimum latency that you can have is 1 60th of 1 second.
00:22:13
Speaker
because you could have the wrong time. You could have tried to do something just as the frame was starting and now you have to wait the full frame time before you actually, that effect is applied. So now you ask, is the client a database? What is happening on the client is there's really two ways of doing it and so I want to be careful.
00:22:38
Speaker
In the one way of doing it, the client has a deterministic simulation of the game world. So that means that all of the inputs that are going to manipulate the game state are being sent to the server and replicated out to the clients. Those clients then receive that input. And then they run the game forward a little bit, like one frame.
00:23:02
Speaker
and they'll find out what actually has changed in the game state and they move their state forward. That requires having total knowledge of all of the state because if you don't have total knowledge, you're non-deterministic because you no longer know what you don't know. You don't know that that bomb might come in from outside and actually
00:23:24
Speaker
If you want total knowledge, you have to have the entire world so you can see events that might be coming over the horizon. This type of server synchronization really only happens with match-based games. Games with small states, like League of Legends, that kind of thing. Or RTSs, which is another match-based game.
00:23:48
Speaker
where the inputs are quite simple, but the outputs are quite complex. So you might click here to move a group of guys, but 500 guys might move. And so that's actually a lot of data to say where all the positions of them are. But you don't have to do that. You just have to replicate. I clicked here, or this player clicked here. And thus, when I play my deterministic simulation forward, all the guys will move within my simulation. I don't have to communicate that data over the network. OK. Yeah, I can see that. So that's one way of doing it.
00:24:15
Speaker
For MMOs, part of the reason they're so difficult is you can't do that. Because I cannot possibly have the total state of the world on my client. It's too big, fundamentally, by design.
MMO State Management Challenges
00:24:27
Speaker
I can't put the whole world of Warcraft and all its players on every single machine. Correct. Alas, you cannot.
00:24:34
Speaker
Instead, what they do is, typically, the way this would work is you have your game server. That game server knows what a game client is. And when they connect, they know where the player is. And they have a bunch of special logic to say, OK, I know what this player is. I know what they need. I'm going to send down that data to the client. And then I'm going to send down that data to the client
00:24:57
Speaker
let's say once a frame. So every frame I will compute, okay, what has changed on the server and I'm gonna send a bunch of messages down saying these are the new positions of all the players. That's the typical way of doing it. Now, notably, this means that you have baked in what your client wants to know about into your server code.
00:25:22
Speaker
because you as the server need to know what they need to know, because you're going to do this streaming update to them. So it's more complex than, for example, let's say a web site with a GraphQL query. Because with a GraphQL query, you can say, oh, I'm this client of client, and I want to know all about this data. And I'm this kind of client, and I want to know about this data. But because games are streaming, and they need to go fast, and they have this tick-based thing built into it, historically, people have built them so that,
00:25:50
Speaker
You write all the code for synchronizing the clients, and you build in some concepts. You probably build in the concept of positions and of players, and that players want to know about things that are around them and all of that good stuff. So if you were to then go build an AI that doesn't care about where certain players are, maybe it's trying to regrow the trees or something, and it wants to listen to the data, no can do. You've already built in the particular query that wants to be done on that game server state.
00:26:17
Speaker
Right, so we're inverting the control, so the server
00:26:24
Speaker
The server knows what kind of things you would want and pushes those to you. Correct. So that really bakes in the server then has to have very fixed ideas about what kind of people connect and what they might want to do. Absolutely. Correct.
Optimizing Streaming Updates
00:26:39
Speaker
What space identity does is the opposite. We actually treat it from a formal database perspective and say, actually, clients are just going to write queries, which are going to be executed with a query engine.
00:26:50
Speaker
in a subscription-based streaming way. So first, we connect and we send a query and we say, I want to listen to all of these players in this region or whatever we are interested in. And then that data will be incrementally streamed down. So as the data changes in the database, we compute that query for all the subscribed things and we send it down to their clients. Then you can think of their clients as having a replica
00:27:17
Speaker
of the server database. Now that replica is a subset of the data and it is only prefix consistent, so it's not strongly consistent. What do you mean by prefix consistent? You have the database state as it existed at some point in time in the past. So you have all of the updates
00:27:38
Speaker
in a prefix of the message log, or of the write ahead log, up to a certain point. That's what I mean. Now it's a subset. So it's not, I have the whole database. I have some set of the data as it was in time. So it's not eventually consistent. I don't see any weird things about it. I will see the database state as it was maybe like five milliseconds ago, or if I'm far away 100 milliseconds ago. And so what that allows us to do is then you can query your local database
00:28:09
Speaker
as though it were the actual database. And so you can get this information out from your local database much more conveniently and faster than you might otherwise do in a normal game server. You'll forgive me being really boring here, but I'm translating this into a non-gaming world. And I can imagine I, as a client of a bank or trading platform,
00:28:37
Speaker
I might want to have all the data relating to my accounts and maybe some of my counter parties, but not the entire bank's data. And then I want to be able to optimistically make transactions on that data. They get sent back to the central server and I get told if that works, but I can progress as though it did. Yes. And that would be exactly the same architecture we're talking about in the gaming world.
Unifying Backend Systems with SpaceTimeDB
00:29:03
Speaker
100%. Yes.
00:29:06
Speaker
What we are, in a sense, trying to do is unify across both of those things. A lot of people, and why is that important? Because many people have tried to make a game server backend kind of thing, like a game engine, but for the server. So there's Unity. You've got Unreal. Wouldn't it be nice if somebody made that for the backend? The problem that people have is that,
00:29:35
Speaker
When you think about what a game is, on the client, it's the same across all games. If I'm playing chess or solitaire, what I want to do on the client is very similar to what I want to do in World of Warcraft. Let's say I'm making a 3D solitaire. I am rendering objects on a screen, and I have all of that stuff. It's all kind of the same across both games. I want to render a 3D world. I want a loop that applies some logic to the state of that world.
00:30:05
Speaker
And then I want to turn it into 3D objects, and I want to project them onto a screen, and I want to do lighting effects, and I want to do sound effects, and I want to do all of that. Every game from the client perspective is not identical, but they rhyme. They have a lot in common that an engine could do that we don't have to write over and over and over again, essentially.
00:30:24
Speaker
On the server, if you think about what chess versus World of Warcraft is, those architectures share nothing. They might as well. One is like a web app kind of, like a chess move. I could build that with a web server and Node.js and all that. And the other one is a very complex
00:30:44
Speaker
multi-user fast-changing state thing, which synchronizes data persistently to the database and updates positions and all of that. So what we're trying to do in some sense with Space Time DB is close over all of those things. And you really have to go all the way back to the database for it to be general enough to actually apply to both of those scenarios. Yeah. OK, I can see that.
00:31:09
Speaker
Sounds like a colossal amount of work to do well, though.
Simplifying MMORPG Development
00:31:13
Speaker
It does. Nobody knows this better than I do. Let me put it to you this way, though, with respect to that. When we decided we were going to make Bitcraft, we were committed to making such a system. The fact that it's available as its own standalone thing is not really that much more work. Every MMO that you have ever
00:31:37
Speaker
scene has an architecture which is at least as complicated as Space Time TV. And I actually know that some of them, I can't necessarily name them, operate in the same sort of stored procedure way because it's the sort of convergent evolution that they arrived at. But they just didn't formally call it a database. So we, in some sense, an easier problem because if you treat it as a database formally,
00:32:08
Speaker
You get to use all of the research and learnings that 50 years of database research has brought about. You do not have to reinvent the wheel is what I'm saying on a lot of these things. And so we were always destined to create a system that's like this.
00:32:30
Speaker
As soon as we decided we wanted to actually create this kind of thing. Ours is arguably just not sort of shoestring and duct tape, not to disparage anyone else. It's very hard to build an MMORPG, but that's kind of how I would think about it. Let's say rather than that, it's not an afterthought. Not an afterthought, yes, correct.
00:32:53
Speaker
Okay, so what we've got here is a system where I as the game programmer, someone moves the joystick up, I update the Y position of their player's row in a database. Correct.
00:33:12
Speaker
Magically, that's going to be synchronized to the server without me worrying about it. And roll back if it turned out it didn't work. And then I just have a rendering function that's also looking at my local database and the screen.
Balancing Latency and Throughput
00:33:27
Speaker
That is correct. Yes, that's essentially correct. Okay.
00:33:34
Speaker
Let's start with the first objection. That's going to be too slow, even if you don't have to do the server round trip every time. So let me ask a follow-up question to your question. What specifically would be too slow? Because what I want to ascertain is exactly what you're talking about. There is a perception that databases are slow, and perhaps that's what you're driving at.
00:34:04
Speaker
Okay. It's got to be transactional. It's probably iterate. Once you get into things like all my bullets flying across the screen and hitting people, it's updating a considerable amount of data. And if collision detection, it's got to happen a lot of times a second. I won't give you the number. You can give it to me, but that feels like that feels like that's going to grind on a transactional database.
00:34:33
Speaker
Okay, so this is a great question and I understand where you're coming from. And we had to be a little bit crazy to think that this was a thing that should be done originally. But for several reasons which I'm about to outline, I think you will come to agree that actually that's completely possible and plausible to do within a database context.
00:34:55
Speaker
So the first thing I will draw your attention to is that we're not, by no means, the first to do things like this in history. There is a database called Times 10, which was developed in the 90s. That was bought out by Oracle, wasn't it? It was bought out by Oracle, correct. And it actually has a very similar architecture to Space Time TV. So a couple of things. One, it's fully in memory.
00:35:21
Speaker
The whole purpose of that database was that for certain very high throughput, low latency applications, current databases weren't hacking it. Not even that, current server architectures weren't hacking it. So what they decided to do is to have a database, have in-memory state in that database, put the logic of your program physically within the same process as that database, and then
00:35:50
Speaker
have you access the data within the same process. So you're literally reaching into your current program memory. You're treating your program memory as though it were a database. And then what they do is all the updates to that data, they append in an append-only fashion to a write-ahead log.
00:36:10
Speaker
Okay. And this was developed for telecom processing, like routing calls, these kinds of very, very low latency high throughput things. That's the almost identical architecture actually to how Spacetime2B operates.
00:36:31
Speaker
use with WebAssembly and whatever language you would like, and some nice things on top of that, including subscribing to the database, which I don't believe Times 10 actually provides that information. Okay, but that's that right-hand log that we've got into persistence, which you said is important. Isn't that a blocker to the performance?
00:36:54
Speaker
Not typically. So first thing I would say is that appending to a write-ahead log is actually quite performant on modern hardware. So that's actually how Kafka works and it's how it's assumed to work. And Kafka is known as sort of a low latency streaming thing. It's not that low latency because of details. But it's relative to what a lot of people use, very low latency.
00:37:17
Speaker
Yeah. Okay. Yeah. The other thing that's important to know about Kafka and systems like that is that you can trade off throughput for latency. So in the case of Kafka, you can batch more things up, which will cause the latency to increase, but will cause a throughput to go up. You can always say, I care about latency more than I care about throughput. So I will.
00:37:41
Speaker
decrease it down to just one transaction. So that would make sense for I want a really high fast-paced game where I really want the lowest possible latency, or I don't really care if things come in late. So that's one thing. The next thing I would say is that for games, or really any application, choosing the level of durability that you want should be configuration and not code. So what I mean by that is,
00:38:09
Speaker
I ought to be able to decide that I want to listen to data that might not be persisted to disk. Because I don't actually care about that. For a player movement, if my server crashes and they move back 10 feet.
00:38:25
Speaker
don't really care about that. If I'm running a bank transaction and it rolls back the last 10 seconds or whatever of bank transactions, it could be a problem, because they might have already given away the item that the guy bought,
Data Security and Permissions
00:38:39
Speaker
right? OK, yeah, yeah. So when you say that's configuration, are you configuring it on a per object type basis? Could I make some match levels of persistency guarantees?
00:38:53
Speaker
you actually configure it on a subscriber basis. So you would say, hey, I'm going to subscribe to this data. And for this particular subscription, I want to see the data as soon as it updates. I don't care if it's ready.
00:39:07
Speaker
I want to hear about, so there's sort of levels at which you can listen in. So there's a pipeline of data that comes in. My message happens. I update the data. That changes the stuff in memory. I write it to disk. I replicate it to other machines, all of this. And at any point in time, you can decide like, you know what? It's good enough. I want to listen in here.
00:39:25
Speaker
So I want to listen after it's been updated in memory, versus I want to listen after it's been persisted to disk, versus I want to listen only when it's been replicated to five machines. That's sort of a different level of listening, if that makes sense. I'm jumping around trying to get my hands all over. So how does that work programming? Let's say the score, the player's score, is gradually ticking up. It's not the end of the world if it maybe rolls back a little bit.
00:39:53
Speaker
Am I writing some code that just subscribes to the score changing and just renders that corner of the screen?
00:40:02
Speaker
You certainly could do that. Typically, what a client will do is they'll subscribe to all the data that they want right up front. So they'll say, let's say it's a match, chess match. You'll say, I want to subscribe to all the piece positions. Or I want to subscribe to, it depends on how you program that match. But let's say you're going to do it in a certain way. I'm going to listen to all the piece positions. And I want to subscribe to the score. A lot of cases, you can actually compute the score on the client based on the state of the game.
00:40:28
Speaker
But let's say you can't. For some reason, in this game, you would subscribe to the score as well. And then that will be updated in a row. And you'll just say, from score table, subscribe. Select all star from, essentially. And am I joining those data sets? So select all from pieces, union, select all from score. Yes. So in this case, you're going to basically
00:40:52
Speaker
select a subset of each server table. We do not yet support subscription joins. We do, actually. So we support what's called a semi-join. So you may filter out rows from a table based on a join from another table. So for example, I might want to subscribe to all players who are friends with this other person. So I would write a join.
00:41:22
Speaker
And I could, but I would always get the whole player row and I'm not going to get any like player plus others data. If you want to do that, you would subscribe to the other table as well. And then we union all of those together and send them down. Okay. And are we writing this query in SQL? SQL. Currently. We, there's no reason we can't also support other query languages like GraphQL in the future. It's just for right now, for building an M O RPG, we need SQL.
00:41:50
Speaker
OK, so as a game programmer, I'm writing, like you say, very much like stored procedures that have a mixture of SQL and coding. And what's the language? So the language is the module that you're writing is a WebAssembly module. So it's any language that you want that compiles to WebAssembly. Notably, we support Rust and C Sharp in terms of building a library of nice things for you to use in those languages. In principle, anybody else could
00:42:19
Speaker
do whatever language they want that compiles to WebAssembly. But yeah, those are the two that we support right now. OK. I risk framing this all as objections, but I'm trying to think. No, no, please. There is an objectionable idea that happens to work. And so it's quite exciting. OK, so the other thing that people always complain about is still procedures. I mean, a lot of people dislike still procedures, and I think the reason is
00:42:50
Speaker
I think there's two reasons. One is the language can be weird for stored procedures. Personally, I reject that one. If it's valuable enough, you'll learn the language. The real one is management of stored procedures is a misery, generally. Correct. Yes, it is. I would actually go a little bit further, too. The permissions model of stored procedures at times can be arcane as well.
00:43:15
Speaker
I believe it's really, to your point, fundamentally a user experience problem, not a theoretical or technological problem, if that makes sense. The developer experience rather than this simply doesn't work.
Improving Developer Experience
00:43:31
Speaker
Yes. If you actually think about server procedures as they were, it's a nightmare. You have data that's in your database operating that's opaque because somebody updated it, but it's not in version control. You don't have any idea what was running. Did somebody change it? Where is it stored? It's just a nightmare.
00:43:52
Speaker
It's a great point. What I will say to this is, actually, we didn't set out to build the database and store procedures. What actually happened is we built a system that had the right UX for what we wanted our developers to have, and then looked at it and said, like, oh, from this angle, actually, this is just a database with stored procedures. So it was very much a, we backed into it, we didn't arrive there. So that's number one. Developer experience is the most important aspect of space-time DB. And if it is bad, there's no point to doing it. That was why we created it in the first place.
00:44:20
Speaker
The way we solve these problems are, number one, is we put all of the stored procedures as the root of your database. It is all in a single module that's based from a single repo, in this case, that you conversion in version control, and then you can see the versions of it. The thing I would liken it to from a developer experience perspective, and now I'm going to say something that will
00:44:46
Speaker
maybe trigger a lot of people. But it's similar in principle to smart contracts. Nobody thinks the developer experience of those is bad, except the fact that they have to deal with the blockchain. And the programming language can be pretty terrible. And the programming language can be pretty terrible. But fortunately, we've solved both of those problems by removing the blockchain and making it so you can use whatever programming language you like.
00:45:16
Speaker
But it's the same idea, right? You do not need a DevOps team to maintain or an operations team or any of that or AWS credits or any of that to run your smart contract.
SpacetimeDB Modules vs. Smart Contracts
00:45:29
Speaker
What you do is you say, publish.
00:45:31
Speaker
You said it. You forget it. You walk away. You don't have to deal with that ever again. It's running. Someone's running it for you. You really, truly don't care. And that is the promise of the developer experience that I think we can provide with store procedures. And it's very easy, in the case of smart contracts, to keep them in sync. Normally, actually, in a lot of systems, you can't update your smart contracts. So that's one thing. But in our case, you can update a space entity module. And it comes from a database. And you can see the version that was up there. And the version is stored in the log.
00:46:02
Speaker
The fact that you're updating your whole database and you can do migrations within your module, and you're doing the whole module at a time, vastly, vastly improves it. Then you have the language that you want to work within, which is a normal programming language. And then on top of that, we have built a permissions model that allows you to have complex logic, which is easy to understand by the developer, if that makes sense.
00:46:30
Speaker
OK, let's go through the permissions thing. For instance, if I've got access to subscribing to data from the server, I would very much like... I wouldn't. Hypothetical black hat me would very much like to use it to cheat on the game. Absolutely. By subscribing to other people's data. OK, so first thing I'll note, all games of the first type that I described, where they have a deterministic client and they're replicating inputs,
00:47:00
Speaker
must know about all data in those games. So League of Legends, you can cheat. In fact, they're doing like a kernel extension to prevent people from cheating. But that's beside the point.
00:47:14
Speaker
require you to see all data. So if you see fog of war in an RTS, everything under that fog of war is there. You could remove the fog of war on your client and see all of the units. So sad times. So first of all, they're just right out. They don't provide that to you at all. In our case, what you can do is
00:47:32
Speaker
There's two types of permissions. There's write permissions, and then there's read permissions. So if you want to update the database, clients are only allowed to update the database through the module. And so what that will be is, let's say I wanted to move a player, but I try to move a player in a way that's illegal. I'm trying to go into this place where I need to be level 56, and I'm only level 50.
00:47:58
Speaker
What the server will do is it will check the level of the player because you're just writing the logic and you'll just fail the transaction. So you'll just say, no, you can't do that. We roll everything back and we throw it away. So that's the first thing. The way that works is each client has an identifier, which is called the identity. Makes sense. It's kind of like an Ethereum address. If you want to liken it to something in that regard, you can see who that person is.
00:48:21
Speaker
And then you can say, all the procedural checks you want in the whole world, is this player friends with that, or do they know each other, whatever it is, and then fail the transaction if it's not allowed. So that's the write. So write is super simple, very, very easy to do.
00:48:38
Speaker
So from a read perspective, there's a couple layers that you can do. So first of all, what we implement today is private tables. So that's just, hey, this table is only viewable by the owner of the module, so basically the database creator.
00:48:55
Speaker
And we would like to add, so we have not yet added, because it's not yet 1.0, both column permissions and then column read permissions and then row level security. So what that means is you should be able to write a function inside of your module that says,
00:49:17
Speaker
Well, in case of column, you're just going to annotate columns as being private. And that's pretty straightforward. For row level security, that means like, can this person see this row? So if they subscribe to these players, maybe this player is invisible, and I shouldn't be able to see them right now. So you want to be able to write a filter function.
00:49:37
Speaker
on a table, so a filter function that applies to a table, that allows you to do arbitrary procedural logic that basically says whether or not this player should be, or this row in this table should be visible to this subscriber.
00:49:51
Speaker
So if I had a hypothetical card game of some kind, where I have cards that only I can see, cards that all my teammates can see, and cards that the opponent can also see, would I be able to model that? You certainly would. So what you'd do is you'd say, let's say you just have one table called cards, and you'd write a function that says, this is the subscriber.
00:50:15
Speaker
which is like this is the identity of the subscriber. Do you want to show this row or these batch of rows or however we end up ultimately implementing it for performance reasons? And you would look through the row and you would say, ah, who is the owner of this card? I am the owner of this card. I can see it. Oh, I'm not the owner of this card. Is the owner of this card my teammate? OK, I can see it. And so forth and whatever complex logic you want.
00:50:42
Speaker
OK, and I'm writing those functions in the same language. Yes, in Rust or whatever language you use. So the language to define security rules is the general purpose
Future Plans for SpacetimeDB
00:50:52
Speaker
language you use? The general purpose language. And it's a procedural language, not going very fancy. Obviously, you can do what, for example, Superbase does, which is they have you write those row-level security rules in SQL. So we may also support that. I'm not sure right now.
00:51:11
Speaker
Boy, it is a lot easier to write Rust than some arcane SQL query about row level security. I'll tell you that. OK. Right. Where does that leave us? So is my experience programming the server side similar to my experience programming the client side? OK. This is a fantastic question. Let me tell you where we are today.
00:51:37
Speaker
and the vision for where we want to be with Space M2B. So where we are today is,
00:51:45
Speaker
You write your server module. That runs on the server. It's written in, let's say, Rust. You write your client. We have a Rust SDK. And what that does is gives you a bunch of functions that you can use, like a subscribe function where you can pass in all your SQL queries, and then you can get all the data back. The Rust SDK currently stores that data internally. So it has this little mini database, if you will, like a little mini memory. And you can query that data.
00:52:12
Speaker
The querying of that data is relatively rudimentary. It's based on code generation that we do.
00:52:21
Speaker
your module has a bunch of types and a bunch of schemas and all that stuff. You can take a module, and then you can extract the schema from that module, and then you can code generate whatever type of clients you want. So for example, you can call a particular function from the client. So if you have a Rust module and a C-sharp client, as we do, you can get the C-sharp equivalent type to the Rust type on your client, if that makes sense.
00:52:51
Speaker
Which is important because that's another thing with with stored procedures is that like oh the type Like you hopefully the types work because it's like dynamics who knows it's just crazy The way they do things or you have to sort of like apply a type and you have to maintain the types. We are Code generating a lot of it's like protobuf right so you you have your schema we scoop that out of your module and
00:53:16
Speaker
It's like a protobuf representation of your schema. You can then co-generate on whatever client you want, whatever types you need. So TypeScript, we support four right now. TypeScript, Python, Rust, and C-sharp for clients.
00:53:31
Speaker
But when I call those functions, they're still going to the local space-time TB client instance. Okay, so they do, and then they get sent out to the server. We don't automatically do client-side prediction right now. That is something that, for example, in Bitcraft, we have to replicate the
00:53:51
Speaker
the logic of. So if you move a player, you have to move them like yourself. You have to rewrite the logic in C sharp, and then you have to write the logic in Rust. That's typically how a lot of people do these things, and it is a huge pain. They duplicate the logic, and they have to do this thing, and it's a huge pain. Some more clever people, actually I know of some that are developing
00:54:15
Speaker
and RTS use WebAssembly. And they run the server both on the client and the server, and they do that. And so that's ultimately where we'd like to go with this. So ultimately, we want to run space NDB not only in the server, but also in the client and have them synchronize between each other automatically based on your subscriptions. And then you have a fully running module. So the same module that's running the server is running on the client.
00:54:43
Speaker
And when you update, when you do a call, actually we run the actual server logic on your client, update that, and then that does the whole reconciliation. So you automatically get client side prediction for free. That's what we're doing. Really, how far away do you think that is? It's a good question. So in a sense, we're already doing it on the,
00:55:04
Speaker
what we call Space NDB Cloud, which is our cloud offer. So we have, OK, there are two versions of Space NDB. There's Space NDB standalone, which is the open source version that's on GitHub. You can take a look at that, everything. That'll run like a single node clustered as though it were your own personal instance of Space NDB. We also have Cloud, which is a distributed system which will run many machines and coordinate between them all.
00:55:29
Speaker
And the way we replicate from one to the other is sort of the normal way in which you would replicate a client. So they're all just clients of each other is an interesting thing. And that has a lot of implications for strong consistency, but I don't think we have time to get into that. But either way, we're working towards that, I suppose, on the server. And then we will do that as soon as we can on the client. We're also building an MMORPG, so we're a little bit busy.
00:55:56
Speaker
I'm not sure exactly when that'll be, but it is still useful in the way that it is right now.
SpacetimeDB as a Distributed OS
00:56:00
Speaker
That is to say, we're not automatically doing client-side prediction. But we will eventually do that. I envision a world. So here's the secret. Here's the real secret to what Space Time DB is. Actually, it's not really a database at all. What it really is is a distributed operating system in the spirit of Plan 9, which has never sort of taken off. Let me explain.
00:56:25
Speaker
Briefly, so space time to be cloud, as I mentioned, runs over, let's say, 100 computers, right? So you've got this thing that's running. From the outside, it looks like just one instance of space time to be. So you can't really tell that it is made up of 100 computers. And what you're doing is you're taking a program and you're running it on that distributed computer. So it looks again, like a single
00:56:54
Speaker
computer, and you're running a program on it, and we're abstracting away the hardware. Boy, that does sound like an operating system, doesn't it? And so that is really ultimately where we'd like to go, is a place where the cloud is not this collection of hardware and services that you have to piece together in this grotesque fashion. But really, it is just a giant computer.
00:57:22
Speaker
And you're going to take your program, and you're going to run it on that giant computer. And that's it. This is going to be even more blurred when you've got a series of clients connecting into that who are themselves similarly programmed. Correct. And so what you might say is that you're building a gigantic distributed operating system that the whole world runs on top of. Right?
00:57:47
Speaker
You could say that if you were so bold, and I don't know that we are yet, but one day perhaps we will. So the idea would be that you're all operating on the same protocol to speak with each other and that you can't really even tell
00:58:04
Speaker
I mean, there's a lot of details in this one, and to be quite honest, I haven't thought through all of it. But if everybody's speaking the same language, you have all of these modules subscribing to each other, it's just the actor model. You know what it is. It's very similar to Erlang, right? It's got the same kind of spirit. You've got these actors, and they are sending messages to each other, and they're listening to messages that are being sent to them, and they're updating their state, and they're moving on. So it's very much in that spirit.
00:58:31
Speaker
Okay, let me ask you this, and you may not like this question, but I'm going to ask it anyway. If someone thought, this is a great idea, but I'm not waiting for you to do it, I'm going to build this on Erlang myself. What parts would they be missing? What parts would they find hard? Yeah, so the whole database part, right? So don't forget about that. So I've thought about this.
00:58:53
Speaker
In a sense. So let's say you want to build this on Erlang. Cool. What is Erlang missing? Well, it's missing the persistence. I know they have persistent actors. But the performance of that is key. The size of each actor is key. So within a space on DB actor, if you think about them as actors, we also do multi-version concurrency control.
00:59:16
Speaker
so that we can run as many transactions as possible within one machine as one actor as you might possibly be able to do. So you want each actor to be as large as possible before you start going into other actors, because as soon as you go into distributed systems, it's complicated. And you can do a lot with a single machine, it turns out. Although each actor could, in principle, be more than one machine, but I digress. That's another direction to go in down in the future.
00:59:44
Speaker
And then there's the whole relational model. So you need to build on top of Erlang the ability to do queries on the rows and get the actual row data out, all of the type system stuff. You'd want to be able to run in whatever language you...
00:59:57
Speaker
you would like to because maybe your programmers are familiar with C sharp because they're unique developers and all that. Let's say also now what about the subscription so actors in the early models I understand you can send messages to other clients but that's kind of like the old way of doing it with the game servers where I need to know what this other actor wants to know.
01:00:17
Speaker
or build a subscription system where they send me a message, which is their subscription, and then I run the whole query engine, and then I send them what they need to know, which is what we have done. So you have to build that whole query subscription system up from the ground as well. So good luck to you, and I would love to see and use your system if you do that, because we wanted to make a game. We are doing this because we must.
01:00:42
Speaker
OK, then perhaps we should wrap it up with the last two questions. If someone decides they don't want to do that, what state of SpacetimeDB in for me as a user?
Access and Testing of SpacetimeDB
01:00:53
Speaker
Can I go and play with it? Yes. You absolutely can. So you can go to spacetimedb.com. You can play with a demo. It's right there. You can also very quickly go to our Quick Start Guide, install SpacetimeDB, get a local instance of it running, Spacetime standalone. You can upload a module to that. You can connect to that. You can call functions on that.
01:01:11
Speaker
You can also upload to our testnet. So our testnet is a version of Space NDB Cloud, which is relatively nascent. But it's meant for you to play around with what the Cloud version will be. It's completely free. We give you a free amount of energy. And energy is what powers these things. It's not actual energy, to be clear. It's just points. You can give it to AWS credits. We give you AWS credits.
01:01:41
Speaker
and you can go to town on that. And then, notably for the testnet, we reserve the right monthly to wipe the data because we're still updating the ABI and we don't want to be locked in yet. But early this year, so we're trying for let's say April,
01:02:00
Speaker
to move into 1.0 and the main net. So the main net of space NDB will be the version of space NDB where we guarantee that your data is going to be there forever. And it will be persisted and replicated and all that good stuff. And so you can begin building your applications now for a launch post April.
01:02:21
Speaker
OK. Here's another dangerous question, because there's only really one right answer. Is your game, Bitcraft, going to be running on that testnet? On that mainnet? It already is. So we, yes. 100%. And we are working on, so a lot of our focus right now is getting the performance to where it needs to be. So Bitcraft used to run on what we called jank time DB, which was late space interview, but it was the thing that we built first, and it was not its own product.
01:02:48
Speaker
And that worked quite well. But it actually was more like the traditional old servers where the server knew what the client wanted. And it was relatively performant. And we're now getting back to that point that right around now, we had the same performance of JangtimeDB. And now, as we gear up for the alpha, which by the way, sign up for the Big Craft alpha, it's happening early this year as well. I'll put a link in the show notes.
01:03:16
Speaker
Yes. And we are getting to the point where we are at that performance level that we need for that alpha. So that's like, well, I don't want to, I don't know how much I can say without upsetting the Big Craft team. But it's many more users than we had previously concurrently running in the game.
01:03:33
Speaker
Right. Cool. Well, you've got a busy year, busy few years coming up. We surely do. We surely do. Awesome. Well, good luck with space-time TB. I hope it takes off. Good luck with Bitcraft. I hope that takes off. And if they both take off, are you going to invite me to your private yacht for a follow-up? I don't know that I'll have one of those. I'll be too busy on the next part of space-time TB. Good luck. I need satellite from there.
01:03:57
Speaker
You'll be able to afford it if both of those work. I guess so. Tyler, thanks very much for joining us. Thanks for having me. And that's all from Tyler. Thank you very much. You know, in among all the things we discussed in there, I think Tyler must be something like our third guest to reference the Plan 9 operating system. And I don't know much about Plan 9, except it never took off, but it was a huge influence for a lot of people.
01:04:23
Speaker
I think we might have to have some kind of retrospective what we could have learned as an industry episode on Plan 9 one day. So if you're a Plan 9 expert, or if you know one, please get in touch. And the way you get in touch is the same way you send us any feedback. My contact details are in the show notes. If you're on YouTube, there's a comment box just down there. Spotify has a Q&A thing for each episode these days, and so on and so on, check your app.
01:04:52
Speaker
On the subject of feedback and of future episodes, if you enjoyed this episode, please leave a like or a comment. If you think other people should find this podcast, please rate it or share it with a friend. And make sure you're subscribed because we're going to be back next week with more interesting voices from the software development world. Until then, I've been your host, Chris Jenkins. This has been Developer Voices with Tyler Cloutier. Thanks for listening.