Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Can Event-Driven Architecture make Software Design Easier? (with Bobby Calderwood)
2.3k Plays11 months ago
This week’s guest describes Event Sourcing as, “all I’m going to use for the rest of my career.” But what is Event Sourcing? How should we think about it, and how does it encourage us to think about writing software?
In this episode we take a close look at systems designed around the idea of Events, with guest Bobby Calderwood. Bobby’s been designing (and helping others design) event based architectures for many years, and enthusiastically recommends it not only as a system-design technique, but as a way of solving business problems faster and more reliably.
During this discussion we look at the various ways of defining event systems, what tools we need to implement them, and the advantages of thinking about software from an event-based perspective. Along the way we discuss everything from Clojure, Bitemporality & Datomic to Kafka and more traditional databases - all in the service of capturing real-world events and building simple systems around them.
–
EventStoreDB: https://developers.eventstore.com/
The CloudEvents standard: https://cloudevents.io/
Datomic: https://www.datomic.com/
Adam Dymitruk’s Event Modelling Explanation: https://eventmodeling.org/
Bobby’s Event Modelling course: https://developer.confluent.io/courses/event-modeling/intro/
Bobby on Twitter: https://twitter.com/bobbycalderwood
Boddy on LinkedIn: https://www.linkedin.com/in/bobbycalderwood/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
–
#software #softwarepodcast #programming #eventsourcing #eventdrivenarchitecture #kafka
Recommended
Transcript
Object Orientation vs. Procedural Approaches
00:00:00
Speaker
How do you solve problems with computers? I mean, fundamentally, how do you think about taking something that's happening in the real world and making it into something that can run inside a machine?
00:00:12
Speaker
We have a few different answers to this in our industry. One is object orientation. Think about the problem as a series of objects or creatures that have behaviors, and those objects talk to each other and trigger each other's behaviors, causing the system to move forward to the solution.
00:00:32
Speaker
That's one approach. Another popular approach is the procedural one. You think of problems as being like things that need a recipe. You break a complex problem down into simple problems. You pull the recipes off the shelf that solve those simple problems and combine them back until you've got a solution recipe for your big problem.
00:00:53
Speaker
And there are more. Those are two ways of tackling it. There are more ways of thinking about software design. And whichever of those approaches you pick, it's going to fundamentally shape the way you write software, and the tools you choose, and the way you think about solutions themselves. So I always think the more of those perspectives you have loaded into your brain, the more flexible a thinker you can be, and in the end, the better a problem solver you become.
00:01:23
Speaker
To that end, we're gonna try and add a new perspective this
Introduction to Event Sourcing
00:01:26
Speaker
week. We're gonna talk about an old but new way of looking at this called event sourcing, which I would categorize as the two what's. You look at the world and you say, what happened and what does it mean?
00:01:40
Speaker
That's my summary, but it needs some more detail. So joining me this week to provide it is someone I've always had great fun chatting with, Bobby Calderwood. He's been building event-based systems and consulting on event-based design for years now. And he's going to take us through what event sourcing is, why it matters to software developers and to a business, how you do it, how you think about the world as a series of events.
00:02:08
Speaker
and how you do it well, what tools you need, and why you should add event sourcing to the list of ways in your brain you trigger when it's time to solve a problem. We have a lot to cover in this, so let's get cracking. I'm your host, Chris Jenkins. This is Developer Voices, and today's voice is Bobby Calderwood.
00:02:42
Speaker
I'm joined today by Bobby Coldewood. Bobby, how you doing, man? Good, Chris, how are you? I'm very well, very glad to see you. It's been a while. Yeah, likewise, likewise. I very much enjoy coming on podcasts where you host some. Well, thank you very much. Yeah. So I'm going to pick your brains about a big, wide topic, which you claim to be an expert on. I know you are an expert on. We'll see. Yeah, the jury's out. We'll see at the end of the podcast.
00:03:08
Speaker
Yeah, yeah. We'll give you a final score, like a ballroom dancing thing. Yeah, perfect. But the topic is event sourcing, right? Which I got into event sourcing from a weird angle. I got into it from Elm as a front-end development tool. Oh, yeah. So my definition might be different to everyone else's, I sometimes think. So I want your... Sure, sure. So defining event sourcing,
00:03:34
Speaker
Greg Young has actually written on this very recently. Greg Young is the father of event sourcing, coined the term and identified the pattern. The pattern's been there in information systems management for hundreds of years. We've had this idea for a very long time, but he
Event Sourcing: Recording and Reconstructing State
00:03:52
Speaker
named it, claimed it, built up a lot of developer mindshare around the idea. Kudos to Greg Young. Just recently, he's tweeted out a very short text
00:04:03
Speaker
trying to get what is the essence of this thing that we call event sourcing. Because over the years, a lot of stuff has grown on it, like sort of barnacles on a ship hull. But really, at the core of it, he and others in the community, myself included, believe that it's just that system state
00:04:20
Speaker
is a reduced function over the events that have happened in that system. So by writing down the events, we can do a reduce or a left fold and come to the current system state. And that's really it. That's the whole magic of this big thing, event sourcing. It's really just that the system state is a function of the events that have happened in the system. You derive it from things that actually happened.
00:04:46
Speaker
Yeah, yeah, that's right. And that's that accords with our notion of cause and effect and the way that our brains record narratives, like we're very, you know, storytelling is a big part of the human condition and how we think about things. And that's because we think about things logically, we think about, OK, this thing happened and that this other thing happened.
00:05:04
Speaker
those two things together caused this third thing to happen. And now that's why we're in the state we are now. So that's kind of how our brains work. Is that true in the sense of particle physics? I don't know. That's a philosophical debate we don't have to quite go into. But in terms of keeping track of things, in terms of keeping records of what's going on in our systems of automating business processes or whatever, it's a good enough approximation. It's a model that works. It provides something really valuable to our understanding of
00:05:33
Speaker
the causal history of some particular thing that we're interested in studying or interested in keeping track of. Okay, well, let's dive into that. Can you say it provides value? What value does it provide? Why should we care about event sourcing?
Event Sourcing vs. Snapshot Systems
00:05:47
Speaker
Yeah, that's a great question.
00:05:49
Speaker
As we stand here at the dawn of the AI age, where we've got these machines that can think in terms of cause and effect and can take the accumulated experience that they've been shown and then infer or induce new ideas from those past ideas, our information systems more than ever need to remember what happened.
00:06:13
Speaker
we got into this bad habit. So the alternative to an event source system is a snapshot-based system where we've got the current state of the world, which is necessarily a summary of everything that happened up to this point. And all we've got is that summary. But we haven't remembered the story. We haven't remembered how we got into that state. So it's a bit like watching a football game by just taking snapshots of the
00:06:36
Speaker
of the scoreboard, right? And we're looking at the scoreboard. The scoreboard summarizes everything that's happened in the game at some level. And every time the scoreboard changes, we just sort of look at the scoreboard. That would be exceptionally boring. It wouldn't give us very much rich information about what was happening in that sporting event. And it sort of defeats the purpose. The scoreboard's important. It captures some important summaries of what's going on in the world. But I think we'd all much rather watch the game, right? So that's... Yeah, and there are others like...
00:07:05
Speaker
My knowledge of all sports is wafer thin, but I'm going to try and join you on this metaphor. You get in games these days the amount of time a player spent in a certain zone on the pitch or something, and that's something you can only do if you kept the narrative of all the things that happened and reanalyzed it in another way. Absolutely, absolutely. In American football, so I played American football in college,
00:07:33
Speaker
In college, in high school, it's really fun. In college, it's sort of like a job. It's like you sit and you watch hours and hours of game film of your opponents. Wow, really? Yeah, absolutely. In preparation for the game, I would probably watch between 20 and 30 hours of film on this week's opponent. The reason you do that is you see what types of schemes they run, what types
00:07:56
Speaker
which player you're going to be facing, what their techniques are like, and so forth. So you do all of that, because you're deriving something very important, apart from the outcome of the game, right? We these are games that were played in the past, we know who won these games. But that doesn't really matter. For our purposes, we've taken that same information, and we've repurposed it for the purpose of training ourselves to play an opponent, rather than
00:08:19
Speaker
as a fan of watching the game or observing the game or whatever. It's taking the same information at the course of the game and repurposing that information for a new use. Same thing is true in event source information systems. You can start with what you think the information is, the information system is supposed to be, or maybe it's transaction processing role.
00:08:38
Speaker
and you capture all the information that happens in the course of transaction processing. And then all of a sudden it occurs to you or it occurs to some smart product person like, hey, we can use that same information and derive new value from it. Just like your new statistics, right? How much time a person's been in zone or how many shots they took on goal or whatever other statistic we want to summarize apart from the score that's captured on the scoreboard.
Accountability in Finance through Event Records
00:09:00
Speaker
You have all that information so you can come to those new conclusions. You can't derive those new insights from those events, those event data, which you can't. In a snapshot-based system, that information is lost, irrevocably lost. It's gone forever. And you made that choice. You made the choice to lose that information by doing a snapshot-based system instead of an event source system.
00:09:20
Speaker
Yeah, yeah. I often think the kind of poster child for this is like, without wishing to criticize, object orientation is based around the idea that you have a state, stuff happens, you react to it, but you forget it. That's right. That's right. And you destructively mutate that state. And now whatever it was before is gone. And within a single process space, like a single memory space in a program,
00:09:44
Speaker
that comes really gnarly when it comes to sort of concurrent programming and parallelization where it's like two different threads could be looking at that thing and all of a sudden it's something different and now it's like everything all the work that I've done up to this point is meaningless because somebody changed what I was looking at kind of thing so that becomes really difficult within a process space but zooming out from that to sort of the broader enterprise information context
00:10:09
Speaker
when I'm a business and I've forgotten everything that happened right up until the present moment. And all I have is sort of like, here's where we're at now. How do I reason about the future? How do I extrapolate from my past experience and try to predict what's going to happen? So both from an sort of
00:10:26
Speaker
process perspective with object orientation and some of these other practices, as well as from sort of an information management, information systems perspective at the business value level, you're sort of losing stuff all the way down. So it's like this cascade of lost value by not hanging on to these things, hanging on to what's happened in the past.
00:10:44
Speaker
Yeah, I kind of think one of the places where we've always remembered this is probably finance, because they have the regular, the auditors drop in at the end of the year and say, what's your past stream of events? They don't phrase it that way, but that's the question they're asking.
00:11:01
Speaker
Sometimes they phrase it a little more aggressively like, if your current account balances aren't in harmony with what's happened over the course of the year with these transactions, somebody's going to go to prison or get fined a lot of money or whatever. That's exactly right. We have to be able to justify, why is the account balance this? Or why did we assess this much interest on this loan?
00:11:21
Speaker
And you have to be able to go back and justify, well, we assess this much interest because the balance was this at this point in time. And so you have to be able to point to the past to justify the present, which is common in all human endeavor, right? We always sort of have to keep track of those sorts of things, certainly in finance.
00:11:36
Speaker
I mean, double entry bookkeeping has been around since the 1400s. So we've been doing this for a very long time where we say, record the facts. OK, now derive new facts from those original facts. That's something we've done in finance for a very long time. So I think the finance world is sort of in good habits around that. What I learned, though, when I was in the finance world, I was working at Capital One for a lot of years.
00:11:57
Speaker
There's a bit of domain blindness going on there. So within the very specific domain of the financial ledger, kind of the core banking systems, everyone was very disciplined about keeping history and making sure that we could audit everything and all that. And then sort of in every other domain in the company, right? What our customers' addresses were, eh. Just throw away everything and just keep their current address. It's like, well, come on, you can't like, you know. So there is a bit of this domain blindness where it's like when it comes to one particular domain, we see the value of event sourcing.
00:12:25
Speaker
But then we'd go build another information system that's sort of outside of that core financial domain. And it's like, yeah, just credit is good enough. Keep current state, throw everything else out. Why do you think that is? Is it? I mean, let you speculate, actually. I have no idea. Domain blindness is very common. And I think the
00:12:46
Speaker
From the dawn of the computer age, everything was so expensive, everything was so cost prohibitive that we wanted to keep our programs efficient. We wanted to keep the disk space that we required really low, certain information systems, mainframes and other types of systems.
00:13:03
Speaker
embedded those cost-oriented assumptions. And so I think it's just a developer experience thing. I think that's just the habit that we've gotten into as developers is, yeah, let's just mutatively in place, mutate these data structures, and just keep the summary of what's going on. And maybe we'll keep some of that other stuff around, but we'll keep it in a non-core or non-system of record place. We'll stream it into Splunk. Somewhere in a log file somewhere. Yeah, that's right. We'll stream it in Splunk with our application logs or our analytics or whatever.
00:13:32
Speaker
But yeah, it's sort of not in the core domain. And I think the purpose of event sourcing is sort of, you know, flag that idea, say, hey, you know,
00:13:41
Speaker
We've been keeping records for most of human history in this history preserving event sourcing kind of way. We bailed out on that from like 1960 to 1995. And now we're trying to like, or maybe I'm generous, maybe it's like 2015. We sort of bailed on that idea and we're really focused on minimizing disk usage and just keeping what we absolutely needed to keep.
00:14:08
Speaker
And now we're getting back to the point where it's like, hey, disks are free, right? We can just keep all the information around, but that's going to require sort of a paradigm shift of this several generations worth of developers' habits and impulses and inclinations to change, to make event sourcing the default like it probably should be instead of the exception.
00:14:29
Speaker
I always think there's a learning curve to go back and forward to doing it the old and new way, right?
00:14:41
Speaker
Oh no, please. I always think developers will pay that learning curve cost if they see the value. And you could argue that maybe we just need event sourcing in finance because the value of it is so obvious and so high, especially when the auditors come knocking on your door and maybe the IRS and the FBI. But does it have value in other domains? Do you have any examples where it's...
00:15:09
Speaker
Yeah, absolutely. So, you know, anywhere where you can imagine deriving new facts from the facts you've already captured. So, you know, the primary domain, you know, sort of the transaction processing state, sometimes people call that like the aggregated aggregate route, right? This is the thing that you use while doing these sort of transaction processing stuff. If you can think of deriving
00:15:36
Speaker
other conclusions besides that sort of transaction processing state from your set of events, there's definitely latent value there that you're missing. So think about the aggregate route is like the scoreboard of your football game. The events is the stuff that happens in the football game. If there are other things that you think you can derive value from other than just the score of the game,
00:15:58
Speaker
then there absolutely is value in event sourcing, right? So we've already identified some in our sports metaphor, we've got sort of these alternative statistics that are useful and interesting to sports geeks. And then there's also the value of observing that game to prepare to face that opponent, right? So there's already sort of these two different domains outside of just
00:16:19
Speaker
scoreboard and who won the game, where the events of that game are interesting and have value in that sports metaphor. So taking that to other business domains, like, yeah, absolutely. I would struggle to think of one where I couldn't derive new facts from the work that I do or the things that I write down as events in the course of automating a business process.
00:16:42
Speaker
There's sort of two that obviously jump to mind. One is analytics. We've talked about that a little bit. It's kind of like statistics. Hey, there's a statistics that we can derive from what's going on that are interesting apart from the results or the outcome. There's something interesting about the process that maybe we can do process improvement. Maybe we can do better in the customer journey. Hey, there's a hotspot here where something is happening over and over and over again. And maybe it shouldn't be. Maybe we can smooth that part of the process out. So there's a lot of sort of analytics
00:17:11
Speaker
analytics we can do on that kind of stuff.
Event Sourcing for Analytics and Integration
00:17:13
Speaker
You always see that in online sales, where job one is to actually capture the sale and process it. But then you've got any success in that. The marketing department want to know what's stopping us getting more sales. That's right. Reduce friction. Where's the next friction point? Absolutely. It's all of that, really. And we've gotten in this bad habit in software of separating out the analytics from the work that's actually being done in the information system. There are two different systems, and you have to
00:17:39
Speaker
Do them differently in the code you gotta have your analytics layer that's just writing these sort of events out and then you gotta do the actual real work of processing but it's like why would you do that when you could just do all of it. By storing the business events right so that's one area so about that i would like in to sort of.
00:17:56
Speaker
these alternative statistics, right? Shots on from a way of time. Yeah, that's right. The other one is one that I'll more like into sort of preparing to face that opponent by watching game film. And that's training AI agents, right? We were
00:18:15
Speaker
future opponents to that team that we were watching on film, and we were training ourselves by exposing ourselves to how they were going about their process. We could anticipate them, or we could tell what kind of plays they were going to run and what type of situation, that sort of thing. So we were training ourselves on that data. We can do the same thing with automated agents now, where you say, hey, we've got all this event data. It tells a very clear story. It's this sort of mostly structured, say, JSON or whatever kind of data.
00:18:44
Speaker
let's train up an automated agent and see if it can reduce friction or see if it can do some magic where it can build a conversational interface on top of our existing user interface or whatever. The sky's really the limit on training these automated agents on your event data, which you just simply can't do with a succession of snapshots of the scoreboard. You just can't do it.
00:19:06
Speaker
Yeah, or simple stuff not quite as glamorous but very useful of like suddenly all our customers are searching for, I don't know, a blue hat with a feather in it. And if we could react to that quickly before we hear in the news a week later that some star is wearing that particular kind of hat, you can see it.
00:19:28
Speaker
pulling the signal out of the noise. Yeah, absolutely. Yeah, reacting to the leading edge of what's happening. Yeah, yeah, absolutely. So yeah, I think there's a lot of benefits that way. The other benefit that is sometimes hidden, because we get really excited about the analytics, and we get excited about the friction friction reducing, we get excited about the machine learning and AI stuff. The other one is a little more pedantic, but but still very costly for enterprises. And that's just system integration, right? When you have a
00:19:55
Speaker
a system that's recording everything that it's doing, you can create a really loose coupling integration with that system by simply just observing what it's writing down, right? Okay, we're just observing the events that they're writing passively. We're not interacting with that system. We're not issuing commands to change the state, whatever. We're just seeing what it's doing. And then downstream, we can do
00:20:14
Speaker
all sorts of different stuff. We can create new projections of state. We can kick off some process. We can issue a trigger to, hey, this user bailed out on a cart. They put a bunch of stuff in their shopping cart, but then never checked out. Let's kick off an email and send that email to them saying, hey, did you forget something? That sort of thing. So there's all this systems integration stuff where your systems become much more easily extensible and more loosely coupled.
00:20:41
Speaker
the more you employ this pattern throughout your information systems. Because you're making the events available. And that relates to one of my mottos, which is the least coupled two things can be is just with data. That's right. The maximum decoupling point is just information. Yeah, and there's sort of a bit of a decoupling reuse paradigm. I don't know exactly how they're proportional, but everyone talks about reuse. So hey, let's reuse this. Let's reuse this.
00:21:10
Speaker
Reifs is good, but it depends on what you mean by reuse. Are you talking about reusing the same class or the same interface in your code architecture? Or are you talking about reusing the same business occurrence to generate additional business value? I think the latter form of reuse is infinitely more valuable
00:21:30
Speaker
than reusing my interface in my class or in my code architecture. But we focus a ton on reuse at the code level, and we don't focus much on reuse at the data level. Like, hey, let's just see what business value we're generating in the course of these information systems and multiply that and reuse that and keep deriving value upon value there.
00:21:53
Speaker
that's going to produce a lot more for the bottom line than like, oh yeah, I was able to dry up my function and only have two function implementations instead of three. And you're like, okay, that doesn't really matter, right? In the macro scale, that's not saving a ton of money or generating new value, whereas this sort of reuse at the business data level absolutely is.
00:22:12
Speaker
And we should have, I feel like we understood that at one point in our history and we lost it because I'm sure Edgar Codd had figured that out back in the IBM days when he was deriving SQL, right? Yeah. Relational data, make the data accessible without assuming how people will use it. That's exactly right. And that brings us to sort of the last really interesting aspect of this idea of event sourcing as being
00:22:38
Speaker
system state as a function of all the stuff that happens in the system, which is that there's no real query language. There's no imposed usage pattern.
00:22:50
Speaker
By storing the events, you can sort of build your own query language of just the different functions that reduce over those events to come up with new things. And we've already touched on this briefly. You'll almost always have one that's sort of the official store. That's the transaction processing state. That's the one that's making sure everything's consistent and we're doing our jobs right and all that kind of stuff.
00:23:11
Speaker
But then there's an infinite number of other functions that can operate on that same sequence of events and derive new facts and new conclusions or recommend things to do next or whatever. There's sort of a whole bunch of functions in that domain space where you can derive new things. And it almost becomes its own query language. So in event sourcing, there's not really a query language necessarily. It's just a function of the stuff that's happened in the system.
00:23:40
Speaker
Yeah, big old basket full of folds. Yeah, all the folds. All the folds. So we're getting into this. We are beginning to imply certain architectures and certain technology choices. I can feel you almost about to say public immutable log.
00:24:01
Speaker
Things like, if you like this idea and you want to start implementing it, what are the technological choices you make?
Implementing Event Sourcing Technologies
00:24:09
Speaker
That's a great question. Every few months, a blog post comes out saying, like, event sourcing is hard, or you can't do it with your current tech stack, or you can't do it on Kafka, or you can't do it with X or whatever.
00:24:22
Speaker
It's such a simple idea. You can really do event sourcing with a file on disk, right? You just append to the file. And at some level, that's how Kafka and systems like it work. It's just like, okay, it's a pend-only file and we've got a NIO file handle to that thing and we just sort of append new stuff to it and flush it to disk and all that kind of stuff. So you really can do that, right? If you are smart about how you structure your code,
00:24:50
Speaker
The repository where you store your events is maybe the least interesting thing about what you're doing. You can store it in a flat file on disk. You can store it in a relational database. There's a lot of different ways and techniques you can use to store these things. But all these events stores have to have a set of characteristics in common. You must be this tall in order to be an event store idea. And that is you have to be able to
00:25:20
Speaker
It should be immutable. You touched on one of those things, right? Your log of events should be immutable. But there's also this sort of idea of I need to be able to see the events that are in there already before I write the next one. Because I want to make sure that
00:25:38
Speaker
you know, I don't double write an event, for example, you know, it got a guard against certain of these sort of distributed systems problems, right, where we got two writes of the same information in there. And now you've got extra events downstream, you have to sort of filter out or whatever. So there's sort of that idea of, I want to make sure that the world hasn't moved on.
00:25:59
Speaker
from where I thought it was when I issued the command to write these events down. So there's techniques like optimistic concurrency control that allow for that, where you can tell your event store, a good event store will have this API or this sort of characteristic built into it where it's like, hey, write this event down, provided the world hasn't moved on from the last time I read. That gives you this ability to say,
00:26:27
Speaker
read my events as of this point in time. As of event number 55, read those events, build your state, show the user the state based on those events. The user makes a decision, issues a command. In response to that command, you say, OK, now we're going to append some events into the world. But some other user somewhere else may have already seen that state and issued
00:26:48
Speaker
a similar or a, you know, contravening command. And so you want to be able, before you write any events down, you want to be able to look in there and say like, hey, has the world moved on? If not, you know, succeed and write the event. If the world has moved on, fail the right, alert the user, tell them, hey, the world's moved on, you might want to reconsider what you were about to do.
00:27:08
Speaker
I can see how you build a system that way, but I'm going to have to challenge you on this. Absolutely. Isn't the event sourcing way to say, well, user A said do this, user B said do that. We record both of them.
00:27:22
Speaker
And then it's the fold that makes that decision what that means. Maybe, maybe. So by having, yeah, the classic engineering answer is it depends. And it depends on what you want to do. If your domain says that both of those inputs are equally valid and we need to sort it out after the fact, like certain CRDTs like AutoMerge do exactly that.
00:27:46
Speaker
auto merge with all of its fanciness. It's an amazing piece of technology. It's just an event log at the bottom, right? You're just writing these events down and it's clever about how it sequences those events that come in concurrently from different actors. And then just as you said,
00:28:03
Speaker
the fold which actually produces the current value or that converges the current value so everyone sees the same value given the same log, the fold sorts that out. And you can either have multi-valued fields where it could be this or it could be that, or you just say last one in wins or whatever, right? You can sort of tell that fold function how you want to resolve that. So certain types of event source systems do that.
00:28:27
Speaker
But in other types of systems, you want to say, no, there needs to be some level of, I don't want to proceed if things have moved on. If the world's moved on from a certain point, I want you to fail that out. And so in a good event-sourcing event store,
00:28:47
Speaker
It'll just give you the option, not the obligation, but the option to use this sort of optimistic concurrency control setting and say, hey, if this particular event stream has moved on past this offset, fail the right and we'll try again. And that gives you lots of different ways to sort of retry until that thing succeeds. Or it just gives you the option. You can just do fire, forget, just capture all the events and then figure it out in the fold. Or you can say like,
00:29:14
Speaker
No, we need to be more careful than that. We need to be more consistent than that and stop the world a little bit.
00:29:19
Speaker
Can you give me an example of where both are good? Because I'm kind of leaning towards the capture both so that we can one day have an analysis process that says, do you know what? We've got a lot of concurrent users trying to edit the same document in this. And that's the fact that's interesting. Yeah. So you're sort of alluded to the classic use case for keeping both is concurrent document editing. If I'm building a Google Docs-like thing where I'm trying to
00:29:48
Speaker
changes coming in thick and fast, and we need to just capture them all and figure it out later. That's great for systems that are highly concurrent, lots of edits and changes coming in really fast. That idea of perpetual reconciliation and just capture everything and then reconcile it after the fact, that's definitely the right solution there.
00:30:10
Speaker
for a lot of enterprise line of business applications that change more gradually and the consistency is more important than the availability. In editing, availability is paramount. We just want to accept everyone's edits and we don't want anyone sitting there with a spinner on their document because that makes them really frustrated. It's a terrible user experience. That's why you're accepting all those changes and figuring it out for the fact that the right use case in editing case. But for things like
00:30:38
Speaker
a financial ledger or whatever, making sure that the same transaction doesn't come in twice because someone kicked out the power cord. We just have to be a little more consistent than that. We have to make sure that we don't double charge someone's credit card. You know what I mean?
00:30:56
Speaker
things that happen and have an immediate adverse user effects. And then if we sort it out later, we can give their money back. But it's like, hey, sorry, you already drew, you already overdrew my account. And now I can't afford diapers. Like, you know, that's not a great user outcome for that person. So being able, you know, having this ability to sort of introspect on the log and say, so long as the world hasn't moved on from here, I know that what I last showed the user is the current state of the world. And so that they're making a decision with the latest and best information possible.
00:31:26
Speaker
And then they write and then someone else comes in, does the same thing. They see a view of the world. It's consistent at the time that they read it. And at the time that they try to write something else down, they have, you know, the system has at least the option to say, yeah, we only want to commit to this decision if that information that we showed the user was the very latest possible information and nothing's happened in the meantime.
00:31:51
Speaker
Yeah. Yeah. Okay. Um, so does that imply a certain tech stack though? I'm trying to make you commit here. Yeah. No, no, absolutely. And I'm, and I'm trying to, to be the consummate engineer and be like, I don't know. It depends, but it does. It doesn't apply. Give me a menu. Give me a menu.
00:32:09
Speaker
So the original, sort of the original category, or the original system in this category of like event sourcing event stores is Event Store DB. It was started by Greg Young back in kind of 2012, 2013 timeframe. Great products. Now it's a company event store. I'm friends with the management team over there. CEO is a really great guy. So really good event store.
00:32:34
Speaker
It's written in .NET. They've got clients in a lot of different languages. And it was sort of the first one to offer this sort of data model around streams of events.
00:32:44
Speaker
And then this optimistic concurrency idea of saying, OK, for this particular stream, succeed if it hasn't moved on, fail if it has moved on.
Features of Event Store DB
00:32:53
Speaker
And that gives you that kind of optimistic concurrency control guarantee. They also have another really neat aspect of their data model, which is the idea of an entity or a subject in the stream. So if you've got a big
00:33:06
Speaker
information system. So, for example, like Autonimo, which is the little toy information system we built for the Confluent course, it's sort of this
00:33:17
Speaker
autonomous vehicle ride sharing thing where, okay, I'm an owner of this really cool, you know, autonomous vehicle. I park it in my parking lot, you know, to go to work, but I don't want to just sit there baking in the hot sun. I want it to go make me some money. So it drives itself off and goes and picks up, you know, kind of ride sharing like riders who request a ride and then the system schedules it for them. And then they get in the car and it drives into their place and they get up. So it's that kind of a system when you've got, um,
00:33:45
Speaker
a lot of those coming in, there's lots of different rides and there's lots of different vehicles. And if system state is a function of all the events that have happened in that system, to find the state of any particular ride, you'd have to replay all the events for every ride that's ever happened. And that's prohibitive. That's not going to be performant. You're going to have lots of issues there. So the ability to
00:34:11
Speaker
filter the event stream and say, I only care about events about this one particular ride. I'm just tracking the history of this ride because I want to figure out if it's got scheduled yet or if it's dropped off its user yet or whatever. I want to be able to display in that user's app.
00:34:26
Speaker
your driver's going to pick you up in five minutes or whatever. I want to be able to display the current state of the ride somewhere. You don't want to have to reduce over every event that's ever happened for all rides ever. You want to just focus in on that. So that's what we call fine grain streams are good for. You can say, hey, I'm going to write to this named stream, rides. These are all the events about the rides in the system. But I'm going to indicate
00:34:51
Speaker
that this event is for ride 12345, whatever, that will aggregate back up to the rides event. So it'll show up in the overall stream of all events. But it'll also allow me to say, I just want to see the streams about this ride really quickly reduce over the five events that have happened to that ride and figure out where it is in its process rather than the 5 million events for all rides ever. So that notion of fine grain stream is important.
00:35:16
Speaker
How's that implemented? Is that just indexing the events as they come in? Yeah, and really what an event store is doing is just a write-ahead log of events, and then you build this index of streams on top of that. Really, an event store is an event source system in its own right. It's got its own little write-ahead log of the transactions that have come in, and it's got the set of indexes built up around.
00:35:41
Speaker
stream and subject, the event store to build several other projections around event type. So you can actually just see all the events of a particular type in total order and all those things. So you can view it in slice and dice a couple of different ways. So yeah, that's sort of the first
00:35:59
Speaker
system of its class. There's several other in that category, along with Event Store DB. I just spilled the beans. My company, Evidence Systems, has just written one that's very similar. It was inspired by Event Store DB, inspired by a database that I worked with a long time ago called Datomic, a couple of other systems. And it's basically
00:36:24
Speaker
solving the problem that a lot of bloggers have pointed out where it's like, hey, you can't use Kafka for event sourcing, right? So when people say that, what they're saying is, yes, Kafka is a durable, immutable log, but because it doesn't have these characteristics like
00:36:40
Speaker
the ability to look at a stream of events, or the ability to fail a write in that optimistic, concurrency-controlled way if there's already an event on the stream. Because Kafka is so eventually consistent in how it processes those events with the producer and consumer, a lot of people have stumbled. They've tried to use Kafka as their event log, ran into a few places where the eventual consistency was not OK for their use case, and they're like, ah, we can't do it. It's impossible. It's not the right thing. Can you give me an example?
00:37:10
Speaker
Yeah, so one of the classic APIs in an event store is look up an event by its ID. I wrote this event down last Tuesday. I kept its ID around someplace. I would like to see that event, please. Kafka just can't do that. You'd have to spin up a consumer.
00:37:28
Speaker
Consume over the course of all the events that have come in, identify the one with the ID, write it down someplace, and go on. So that's doing a full scan like that every time. It's just not performant. It's not the best developer experience. It takes a lot of distributed systems complexity to do a very simple job. So that's one. And Event Store DB can do that. Our product, Evident DB, can do that. You can just say, hey, show me this event. And it'll pull from the index this particular event. Or show me all the events on this stream.
00:37:57
Speaker
Easy. Show me all the events on the stream for this particular subject. Easy. And you just get back those things. So it's really those indexes on there that set
00:38:07
Speaker
Events Store DB, EventDB, some of the other event stores apart from Kafka. Where EvidenceDB is a little different, we sit on top of Kafka, so we're actually writing down all of our data to Kafka. It's all a big complex Kafka Streams app, and we're writing down our batches of transactions to Kafka as a log, and then we're building those indexes to make it really performant to query.
00:38:31
Speaker
you know, where the database is in terms of that thing. And then we also provide that optimistic concurrency control API that Event Store DB also does, which is, hey, you know, when you go to write, check those indexes, make sure that the world hasn't moved on, and then only succeed if the database is in the state we expect it to be in.
Requirements for Event Sourcing Systems
00:38:50
Speaker
Right. So what we're saying really there is for an event sourcing system you would like, let's not try and say anything too definitive about the entire industry, for an event storing system, you would like some kind of reliable right-to-head log. Yes. Plus some kind of mid-tier state manager that's really building indexes as a kind of
00:39:15
Speaker
What's the word I'm looking for? A state you can access on the way. An intermediary state. An intermediate state that's got the right operational characteristics to be able to query it quickly and cache it and all those sorts of things. And then combining those two things,
00:39:31
Speaker
at write time, I want to be able to consult those indexes before I commit a write to the write ahead log. So it's putting all those things together. It's the write ahead log of transactions. It's the indexes built on top of those transactions. And then it's the ability to fail a write in a perfectly consistent way, a linearizably consistent way, fail a write to that write ahead log if the indexes aren't in the state that the writer expects.
00:39:58
Speaker
Right. Is there a particular set of indexes you always create or is it like custom defined or both? Is there like a default menu of indexes plus you can custom define? Yeah, no, that's an excellent question and it's sort of an interesting
00:40:16
Speaker
It's a little bit of an interesting database theory question, right? Like it's a relational database. Yeah. So a relational database has a write ahead log and it has a set of indexes that it maintains on top of that write ahead log. And then it has a query language for writes and reads that ensures that the log is in the state that is consistent with those indexes that consistent with the writer or the user's expectations, right? So very similar type of system
00:40:43
Speaker
And different indexes build, sorry, different databases build different indexes depending on configuration and user preference and all those things, right? I can tell Postgres to build me an index on this column, on this table, so that it's really fast. I don't have to do a scan of that table to find the thing I'm looking for. I can just say, hey, when it's equal to this, I can pull out that record.
00:41:07
Speaker
it's much more complex in the relational world. And there's tons of math that like, I don't understand that, you know, governs how those things come into being and how you can, um, elevate certain query clauses to make the queries more performant. I mean, there's just tons of really excellent work done in the relational world to make that a performance thing in the event world. It's like a lot simpler, which is why I felt, Oh yeah, I can probably build this, you know, trying to build a relational database. I'm like, forget about it. I'm not enough. I'm not a good enough engineer to like even tackle that. But in the event world,
00:41:37
Speaker
It's rather simple. The schema that we use for our events is an open standard called Cloud Events. Cloud Events is great. CloudEvents.io, it's really great. I think it's a Cloud Native Computing Foundation standard. I can't remember exactly, but it's a standard that's been adopted by a lot of the vendors.
00:41:58
Speaker
And it's a very simple schema, and the way they've managed their standards process is really great. They've got a very well-worded standard, and it's very simple. But the idea is each event has a schema. And so there's certain fields that we can count on being there. Any of the required fields in that cloud event we know have to be in there.
00:42:18
Speaker
So all of our events in EvidentDB that we write down are cloud events. So we can count on the presence of certain fields being there. And then there's another field, which is optional, which is subject, which is exactly that sort of fine-grained stream. It's an entity identifier that says, this event is about this thing. And that allows us to build a subject-based identifier. There's also a bitemporal,
00:42:41
Speaker
extension field on cloud events where you can say, you know, effective date, I think it's what it's called or something like that. But there's some sort of field that allows you to say, I know this event came in this order, but really, we want to think of it as if it had occurred, you know, last Tuesday or this morning at 9am or whatever. Like when you're backdating a deposit.
00:43:01
Speaker
That's right. Backdating and deposits is a great example. A lot of banking use cases rely on this sort of trickeration to say, yeah, these transactions came in in this order, but we're going to fudge it a little bit and shuffle them around so that we can come up with a better customer outcome. They actually do this in favor of the customer more often than in favor of the bank.
00:43:21
Speaker
because they're forced to by the government. If certain transactions would come in in a way that would cause multiple overdrafts, for example, they resequence them so that only one overdraft will happen, or at least they're supposed to. It depends on the bank you go to most of the time. But they're supposed to process those transactions in such a way that it will minimize the number of overdraft fees that the customers hit with.
00:43:51
Speaker
write a check and it overdraws my account and then I make a deposit and then another thing hits and I make another deposit, it will have overdrawn me two or three times in there. It'll say like, no, put all the deposits first and then put all the debits second and then we'll see where we're at and see if they increase. If there's any reality in which this wouldn't have occurred. That's right. Yeah, that's the one you have to pretend actually happened.
00:44:15
Speaker
That's right, exactly. So that bitemporality becomes really useful. So in EvanDB, we build a stream index. So what type of event is it? Or what bounded context is this event coming in? So in our autonomous example, there's sort of a stream that's about vehicles. This is people registering their vehicles for the service, making their vehicles available, requesting the return of their vehicles, that sort of thing.
00:44:41
Speaker
And then there's a stream that's about rides, which is why ride got requested, the ride was scheduled by the system, the rider got picked up, the rider got dropped off, all that kind of thing. So two separate streams of events, and we index on those streams so that you can say like, okay, here's the total order of just the events in this stream. We maintain a total order of all the events per namespace. So we're a multi-tenant, we allow for many database kind of namespaces within our system.
00:45:11
Speaker
Okay. And all of those, all the events within a particular database namespace are totally ordered and we maintain a total order on those things. But then also you can sort of drop down into a particular stream and say, Hey, just show me this particular stream. And then within that stream, we can say, Hey, just show me the events about this particular subject.
00:45:29
Speaker
and then you can totally order them per subject. You can even, if you get clever, you can do that across streams and things too. So yeah, so really stream, subject, and then bi-temporal indexing, those are kind of the big ones. There are a handful of others, so each event has a type.
00:45:45
Speaker
So you can, if you want, build an event index on type. I don't know how useful that is. Event Store DB has that. I don't know what anyone uses it for in Event Store DB, but they have it. And I'm sure they have it for a reason. And I'm sure some customer at some point asked for that. So you can index the events or look them up by type and that sort of thing. So yeah, that's really the main API, though, is really I'm going to transact a batch of events.
00:46:09
Speaker
And I can specify sort of constraints on those events. And if all those constraints hold, the whole batch goes in. If any of those constraints are violated, the whole batch fails. So you get that sort of atomic level right of multiple events at once, which is actually something evident DB has that event store does not have. Event store is one event at a time that you can write in. So we have the ability to sort of atomically transact batches of events in or not, depending on those constraints, those OCC constraints.
00:46:35
Speaker
So that's kind of the right API. And that's it. That's the only right API. And then the read API is, you know, pull me a stream or pull me a stream for a subject, or pull me a stream for a subject in a bi temporal reordered kind of way. So that's, that's how we
00:46:51
Speaker
We provide those APIs to the end user. Very straightforward. It allows for reasoning about the system at a particular point in time. We're able to do a ton of caching in our clients for doing that sort of thing. We have a unique sort of feature in our database that in the client
00:47:10
Speaker
very much like Datomic. So Datomic had this really awesome notion, one of Rich Hickey's best ideas. And that's saying a lot because he had a lot of really great ideas. But one of his best ideas was the idea of the database as a value. So it's sort of the operational part of a database, which is the thing that you write against. That's sort of the connection. That thing's stateful. It's like, you know, got
00:47:31
Speaker
network stuff in it, and you're talking of this thing that's a machine, and it's going through a succession of states over time. So there's the connection, and that's kind of the stateful machine bit. But then you ask the connection for a database value. You say, hey,
00:47:46
Speaker
show me the database as of this point in time or show me the latest that you have access to or whatever, and then you get a database value back. And that database value has true value semantics. So no matter what happens, rights can keep going into that database.
00:48:03
Speaker
as much as they want, they can fast, but that database value maintains its consistent point in time view. So you say like, once I've asked it for this database, every time I ask it for some stream or for some subject or whatever, it's always going to give me back the same answer. So you can use these database values in functions, in functional programming, because it has value semantics rather than reference semantics that are going to point to some changeable thing. For the sake of listeners that aren't quite the closure fans that you and I are,
00:48:32
Speaker
Why don't you define what you mean by true value semantics for a database? Yeah, so value semantics means that it's a value like the number 42 or some immutable map or something where it's stable over time. One of the key constraints in functional programming is
00:48:51
Speaker
If you call a function with two arguments or with an argument multiple times, it should always give you the same answer. So if it's a true function in the mathematical sense, if you pass it arguments that are the same, you should always get the same answer out the back end.
00:49:12
Speaker
There are exceptions, of course, like you have to have random number generators and some other types of things. But really, you should be building most of your programs up out of these pure functions because they're very easy to reason about. You don't have to think about change and when things change and good currency or whatever. If you have pure values and you're just passing those values to functions and always getting the same.
00:49:30
Speaker
Answers back if you just leave it a reason about your system and tested and all that so that's one of the big virtues of functional programming you introduce databases in the mix and everything becomes complicated now you have to have monads all over your program like i don't know everything can change at any point and so i have to just hedge through my whole program if you have a database that provide you nice semantics.
00:49:52
Speaker
you can take that database value and even though under the hood it's composed of references and it's referencing out to these other places, it's stable and so it doesn't change over time. Regardless of what happens to the database, more transactions are hitting it or whatever, that database value is what it is and it'll always return you the same results when you ask it questions. You can pass that database to a function and always get the same answers again. It participates in pure functional programming in a much nicer way than
00:50:21
Speaker
these nasty stateful machines that you have to wrap in all kinds of monads and stuff to make sure that you're sure. So by treating the database as a pure value, you get things like consistent queries across the database for free. That's right. And in this type of database, because it's all made out of events, there's tons of cacheability in there. So when I have a database value,
00:50:47
Speaker
When I first get that from the connection, there's probably a lot of network activity. I'm downloading all the events and make sure. But now they're all cached. And so now as I'm asking it questions, it feels like it's inside of my program. It doesn't feel like I'm talking to something way over there across a wire. It feels like everything's right here in my hands. So now we can think of this thing not as
00:51:09
Speaker
the stateful connection, intermediated by this stateful protocol, but something that's present, something that's in my program in a much deeper sense than if you're dealing with an ORM or if you're dealing with executing SQL queries and getting back to these big data sets. You have to be really conscientious about
00:51:29
Speaker
when you ask for that data set. Because sometimes with SQL stuff and ORMs, if you ask the same question twice, it'll give you two different answers. Because the database will have changed under the hood, and there's mechanisms for letting that ORM thing know in memory. So yeah, there's a lot of complexity built into these sort of interactions around databases. We're trying to simplify that out and say, hey, let's simplify the whole notion of a database down to just the things that happened, and then submit x's on top of those things. And then let's simplify the interaction with the database so that we have value semantics in memory.
00:51:59
Speaker
so we can use these things in functions, so we can ask it questions and always get the same answer, right? So we're really trying to sort of really dramatically simplify it down, push that complexity out to the edges to be handled by the platform or the system itself, and not to burden the developer with all the crazy distributed system stuff.
00:52:16
Speaker
Yeah, there's a lot of clever stuff going on under the hood, but at the programmer level, let's try and make it look just like a series of values. Exactly. Which is nice to program with. Are we implying that in order to do event sourcing or to do it well, you have to get into functional programming?
00:52:34
Speaker
No. And in fact, I, like you, sort of came to event sourcing from a weird angle. Most of the event sourcing thought, you know, from Greg Young through the sort of domain-driven design community taking it over, has come from an object-oriented perspective, right? People talk about aggregates as if they were objects. People talk about, you know, sagas and, you know,
00:52:55
Speaker
projections and commands and events and stuff as if it was sort of this object hierarchy of things. And that's fine, right? If that's your style of programming, I think that works has been a lot of really good thought put into that stuff. But functional programming, again, I'm not a great engineer. So all that stuff is too complex for me, I can't keep it all in my head. So I need something simpler. And for me, functional programming is much simpler. And in fact,
00:53:21
Speaker
But it's weird. Nobody's really talked about functional event sourcing until really just the last couple of years. People have started to apply this functional programming lens to event sourcing and to distributed systems as a category. Applying these really simple ideas from functional programming
00:53:39
Speaker
to this event. Sourcing idea goes back to what we said at the beginning, right? System state is a function of all the events that have happened in the system, right? So that's that sort of left fold idea, or the reduce idea. That's really from my lens. If you ask, or if you read on the internet, if you know, read Martin Fowler's Blickie, or if you read, you know, a lot of the stuff that's been written about event sourcing, it really does have a bit of an object oriented lens on it, in sort of the popular
00:54:04
Speaker
mindshare. But again, so there's really great writers out there right now who are talking about the decider pattern, right? So the decider pattern is this idea that there's really in most information systems, especially sort of line of business information systems, there's really only three functions that you need to write. And that governs all the business logic in your whole system.
00:54:26
Speaker
And it's decide, which is, OK, the user's telling me to change state. But I don't know about that. I don't know if the system is in the right state. I don't know if what the user is asking is reasonable. So let's look at the current state. Let's look at the command that they're sending us. And then we'll decide, OK, if this is a reasonable thing, we'll emit some sequence of events to be appended to our event log. If it's not a reasonable thing, we're going to return error and say, hey, try again, man. What you're asking for is not reasonable. The system's moved on.
00:54:56
Speaker
your expectations have been violated in some way. So we'll give you a chance to reconsider this idea, right? So that's command. That's really our sort of transaction processing moment, where we can tell the user no, and we can say like, you know, we're protecting the consistency of our system here. So and that has a very, it's a function and it has a well known signature and we understand kind of what that is, right? Yeah.
00:55:19
Speaker
evolve is the second function, and that's really just the reducing function that you'd have in a left fold where you say, give me the state, give me the next event, and then I'll return you the successor state. So if you start with some initial state, and then you can just process each event
00:55:35
Speaker
with Evolve in a left fold kind of way, that allows you to compute the state as of any sort of point in time along your event stream. So that's Evolve, and that's really just about projecting these events into and building state the way that we've been describing kind of the whole podcast. And then the last one is called React. It's a less used function, but it's the idea that like,
00:55:56
Speaker
when some condition outside of my world that I'm observing is met, or if some condition inside my world is met, I need to take some sort of action. So this allows for integration between systems. I'm going to observe some upstream
00:56:11
Speaker
or stream of events, and then when some trigger condition is met, I'm going to issue a local command and do some stuff locally. Or I'm going to look at my internal state, and when some condition is met, I'm going to reach out and invoke some foreign command to cause change in some other system. So this is the way that we sort of bridge between these event streams and between these different systems. But anyway, all of that is to say, you have this little thing,
00:56:38
Speaker
that has three functions in it so you can build a functional type out of that thing. People have called it the decider, they've called it some other things too. But that little decider really encapsulates the entire business logic of your whole system. And so that thing
00:56:52
Speaker
You can now do all kinds of stuff with it because it's pure functions. It doesn't know anything about databases. It doesn't know anything about distributed systems. It doesn't know anything about event store DB or evident DB or event logs or Kafka or anything, right? All it knows is just the pure business logic of the domain of the business problem that you're solving.
00:57:10
Speaker
You can compile that down to, say, WebAssembly, and it can run in your client. And you can do all your event sourcing and deciding and all that stuff in the client, writing stuff down to IndexDB or whatever. Or you can compile it and run it in the server as an API. And now you have this really well-designed, stateful API that you can run. Or you can compile it down and run it
00:57:33
Speaker
inside of your database, like evident BP. So we can do a lot of things like that where we can start to build projections. So going back to a question you asked a little while back, what indexes do you build? And can there be custom index as well? We build kind of the baseline set of indexes that I described, you know, subject, stream, bi temporal, and then
00:57:57
Speaker
In the future, this is a future state of the world, users can provide their own code, their own deciders, their own evolved functions, and we can build projections on the fly. As new events come in, we can just run their logic, build up the next state, stick it into cache someplace, and now you've got this perpetual user-defined projection, which is basically the index that the user wants it to be. That can run on the database side instead of in the user code,
00:58:24
Speaker
And so now that's very fast to query and all that kind of thing. So there's a lot we can do with that where it's like this user defined indexing or this user defined materialized view like you might have in a relational database, but it's a materialized view that's just a function. It's a function in your native programming language that we've compiled to WebAssembly and now we can run it anywhere.
00:58:43
Speaker
Right. Yeah. Yeah. It's sufficiently universal. It seems like a new entry into the attempt to boil down what is programming to as simple as...
WebAssembly in Event Sourcing
00:58:53
Speaker
That's exactly right. It falls in the tradition of the JVM and LLVM and all these really very sophisticated attempts to find the sort of universal computing paradigm. WebAssembly is super interesting because it has sort of one
00:59:09
Speaker
in the sense that it's already embedded in your browser and it's already in the platform of the distributing computing world, which is the web. It's in those standards. The JVM never got there, LLVM never got there. It's this place where you can
00:59:25
Speaker
compile any language down to a common bytecode format. These things just recently in the WebAssembly world, they've crossed the threshold into the component model, which allows you to write one of these things in Rust and one of these things in Go and one of these things in C and one of these things in Java, and they can all interoperate with each other
00:59:46
Speaker
through the component model, so through this standard bytecode interchange format. So really interesting world right now in the WebAssembly space. There's lots of really cool stuff. But this idea of taking these functional programming ideas, pulling from a world of event sourcing and domain-driven design, pulling from the world of Kafka and event streaming, pulling from the world of WebAssembly and database design, we can sort of triangulate on this really interesting set of ideas
01:00:14
Speaker
to make it very simple to write event source systems in the language of your choice with a very simple sort of infrastructure set up. Our goal at my company, Evidence Systems, our goal is to make it so that line of business developers who are trying to build these sort of business capabilities, these APIs that encapsulate business capabilities, they should only have to write business logic. They shouldn't have to care about web requests. They shouldn't have to care about databases and Kafka and distributed systems and all that kind of stuff. They should just be working with their
01:00:44
Speaker
business matter, subject matter experts, and their product owners, and they should just be encapsulating everything that they learn in code, pure, stateless, functional code. Once they have that, they should be able to plug that into the platform, and the platform should take care of all the data persistence and all the web request handling, all that kind of nonsense.
01:01:05
Speaker
sort of facilitate teams going much faster this way, right? Let's just focus on that business logic. You don't have to be the full stack developer that has a PhD in 17 different things. You can just write code that encapsulates business logic and you can have these really well-defined APIs with these good characteristics and these really nice operational, non-functional requirements met and all that kind of stuff. Yeah. That'd be a nice future to get to. Yeah.
01:01:29
Speaker
I think that does leave one hole that we need to address. And this could almost be a whole podcast in itself. So let me see if I can get a quick answer from you. And you've hinted at this with like, we're just leaving the programmer to make decisions about the business domain. And you've also hinted at with we've got this ride sharing app, which is just a stream of rides and a stream of cars. There is a decision to be made in every system about how to structure your events.
01:01:59
Speaker
Which events do you capture? How do you encode them? And how do you segregate them into different streams of events? Can you give us some guidance on the design space? Yeah, absolutely. And that's the space that I've really spent the last sort of three years really diving into.
01:02:16
Speaker
And my, my recent training that we were just released with confluence with on confluent developer, it's called practical event modeling. Uh, and addresses exactly sort of the question you just asked, which is, you know, how do I define what an event is? How do I figure out how to structure it? All those sorts of things. The best thing that I found for doing that is a technique called event modeling. And, uh,
01:02:35
Speaker
Adam Dimitric, who's really close with Greg Young, have sort of come up together in this world of event source system design. He invented this technique sort of related a little bit to Alberto Brandolini's event storming, similar sorts of ideas, but simpler and more focused on the actual system design rather than on sort of exploring the domain and problem space.
01:02:54
Speaker
So it's, you know, you can build this event model. It's a very simple sort of graphical modeling technique. You have your subject matter experts in the room. You have your product owners in the room. You have your UI designers and your architects and your data people and your security folks. They're all in the room with you sort of doing this collaborative exercise where you define out this world of events, right? Command, event, read model, which is like a snapshot of state and interface.
01:03:21
Speaker
And you define out all the interactions in the system. And by capturing it graphically, you enumerate all of the different state changes in your system. And so your whole system design is based around the state changes that take place in it. It's actually really interesting. One of the
01:03:42
Speaker
One of the issues that the industry has been wrestling with lately is on the heels of this McKinsey paper, how do you measure developer productivity? How do you measure the contribution of a given developer? And right, McKinsey and their team of consultants, they think they've cracked this code. They know how to measure developer productivity right away. A bunch of the sort of signatories of the Agile Manifesto are like, no, you can't. We've already gone down this road several times. You know how to measure developer activity.
01:04:11
Speaker
not developer productivity, and there's worlds apart. Anything that you measure, developers are going to game in order to make themselves seem more productive and get the raise that they need to feed their family. You can't equate productivity and activity. When you've designed the system in terms of the state changes,
01:04:32
Speaker
on the event model, you can break this model down into what we call slices, which is basically like, okay, here's one state change. User clicks something on a user interface that issues this command that gets handled by our web tier, and then we record this event to our event store. That slice is a state change. It has a concrete, well-defined business value.
01:04:52
Speaker
You should be able to after you've done this for a while, your development team should be able to say like, okay, this is going to take us two dates because we've done a thousand of these state changes in the past and they all take about two dates. So now you can really start to hone in on, here's the cost for this slice.
01:05:06
Speaker
here's the value or the business benefit of this slice, and now you can start to make real business decisions around developer productivity, which features we tackle first, which features should we cut, when should we stop developing, all these really hard engineering management questions that we've tried to answer over the years and we've had lots of different answers for, and Agile tries to handle them some ways, and Waterfall tries to handle them other ways.
01:05:29
Speaker
By having a design and by having that design really define the value of your information system in terms of its state changes and its state views and its interactions with other systems, well now you can really assign pretty concrete value to each one of these slices and you can
01:05:45
Speaker
break these slices down. So the event model, to answer your original question, yes, it contributes immensely to the proper design and discovery of what events happen, how do we group them into streams, and so forth. If you do my training on Confluent Developer, it'll take you through that process and will show you how we designed Autonimo using this process. And it'll show you kind of each step of the way of how we're finding out that. Yeah, it's a working example. So we start with that example in the very first module, and then we take it all the way through.
01:06:14
Speaker
So it helps to arrive at the proper system design. But furthermore, by making that system design the contract that we're all working towards, it actually gives a lot of these management benefits. So product owners can see what they're going to get. Managers are going to see, here's a clear estimation of when we're going to get each of these things. And here's the ones that are done. And here's the ones that we're still working on and whatever. So in terms of IT management and having a good working relationship with the business stakeholders,
01:06:43
Speaker
Event modeling is the best thing that I've found for both designing the system and managing its delivery. Okay. Okay.
Event Modeling and Developer Productivity
01:06:51
Speaker
I feel like I'd need to go through that a few times myself to believe you, but I want to believe you. Absolutely. No, it's a big claim. It's a big claim. Do not take my word for it, please. The training that I do is great. Adam Dimitryk's website, eventmodeling.org is great. You can go on there and he has a lot of resources.
01:07:11
Speaker
in there. But this is something that I'm doing with customers right now and showing them how to both design these systems and manage their delivery this way. And it's given huge benefits. So for me, this is it. This is all I'm going to use for the rest of my career. I found the way to... It's not a silver bullet. There's no silver bullets in engineering. But in terms of business information systems,
01:07:33
Speaker
I think we've cracked a bit of the code of how to quantify and enumerate the business value, how to gather the requirements, how to properly design out the system in terms of its state changes, and how to transcribe that visual diagram into code using those techniques that I described, even with the decider and some of these other things.
01:07:52
Speaker
There's a lot of value there. And look, it doesn't apply to every type of software, right? You can't write a compiler this way. You can't write a video game this way. You can't write a graphics processing engine this way. It's not a software design pattern. It's really a business information systems design pattern, which is a small, small subset of kind of the software world. But for that small subset,
01:08:10
Speaker
Man, event modeling, functional event sourcing, hard to beat those two things in 2023. I think I'm going to leave you on that campaign slogan. That's it. That's it. Functional domain modeling. Love it. Bobby, thanks very much for taking us through it. That was a lot of fun. Chris, thank you so much. Happy to join you. And hey, if we have to do another podcast to really dig into the nitty gritty, I'm happy to do it. Excellent. Thank you very much. We'll see you again soon. Thanks, Chris. Bye.
01:08:38
Speaker
Thank you, Bobby. You know, one thing I think we will have to do a follow up episode on is using event based systems for building user interfaces. I think it's a great way to structure UIs. And as I said, it's how I got started down the event systems rabbit hole, I may have biases here, but I love it. And I'd like to find a way to share that with you.
01:08:59
Speaker
But until such an episode, if you want to find out more about event systems in general, you'll find links to all the things we talked about in the show notes, as usual. If you're wondering where to find the show notes, they're just down there, down in your app, suspiciously close to the like and subscribe buttons and the share and rate buttons, all those lovely little feedback mechanisms in general. So if you've enjoyed this episode, please take a moment to send some feedback my way before you head off.
01:09:29
Speaker
And head off, we both must. So until next week, I've been your host, Chris Jenkins. This has been Developer Voices with Bobby Colderwood. Thanks for listening.