Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
MicroServices For Better And Worse (with Ian Cooper and James Lewis)
0 Playsin 7 hours
What have we learned from more than a decade of deploying microservices? Was it a good idea? Are we any better at figuring out what a microservice is, or where its boundaries lie? Does splitting things up create fragmentation problems? And is it too late to put the genie back in the bottle? This week we’re going to look at all these questions and more as we reflect on the lessons learnt from this big architectural idea.
This interview was recorded live at GOTO Copenhagen, with two microservice experts and thinkers: James Lewis of Thoughtworks and Ian Cooper of JustEat.
–
Residuality Theory: https://leanpub.com/residuality
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Ian on Mastodon: https://mastodon.social/@[email protected]
James on BlueSky: https://bsky.app/profile/boicy.bsky.social
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on BlueSky: https://bsky.app/profile/krisajenkins.bsky.social
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Kris on Twitter: https://twitter.com/krisajenkins
Recommended
Transcript
Organizing Large Software Files
00:00:00
Speaker
One thing you learn pretty early on in your programming career is that you have to be able to split software up into subsystems. I think most people's introduction to that idea is the day you write a file that's several thousand lines long and you completely lose track of where everything is and what's going on. And you just have to admit defeat and start breaking things up. And so from that day, you gradually learn to separate software out and organize it and recognize the boundaries between different tasks. And that skill deepens and keeps getting more important.
Microservices: Hype vs. Reality
00:00:36
Speaker
And at some point in your career, you may find yourself using those fundamental skills in high-stakes mode, where you're breaking things up not on the scale of files, but on the scale of whole companies. How do you break an entire business down into subsystems?
00:00:53
Speaker
For a while it seemed like we had a magic answer to that. Use a microservices architecture and all your scaling problems will be over. At least that's what the hype tried to tell us. Some people found it to be true. Some people found it to be much less of a silver bullet than they were told. But it was very hard to assess the technique in the middle of the hype about the technique.
00:01:16
Speaker
And now that hype has passed on through blockchain to AI or something, we have an opportunity to sit back and ask some tough questions about microservices.
Critical Questions on Microservices
00:01:26
Speaker
Was it a good idea? Was it a good approach? When it failed, have we learned anything about why it failed? How large should a microservice be? And how do you figure out what counts as a service? Because a lot rides on getting the boundaries between services correct.
00:01:45
Speaker
Do you get architects to define those services? Do you get teams to define the services? Or do the services end up defining the teams?
Expert Reflections on Microservices
00:01:54
Speaker
Joining me to try and clarify what we've learned from a decade of deploying microservices are James Lewis of ThoughtWorks. He co-authored the original microservices article that helped thrust the idea into the mainstream.
00:02:10
Speaker
and Ian Cooper, who's currently living with the consequences as a software architect at Just Eat. And they are going to both reflect on the past and bring us some advice for the next 10 years of software architecture.
Live Conference Discussion
00:02:24
Speaker
This episode got recorded live before a studio audience at the recent GoTo conference in Copenhagen. So let's head to Denmark.
Re-evaluating Microservices Impact
00:02:33
Speaker
I'm your host, Chris Jenkins. This is Developer Voices, and today's voices are Ian Cooper and James Lewis.
00:02:52
Speaker
find out what we're going to say. yeah So what we are going to say is microservices. You're both giving talks about microservices. From perhaps an optimistic and a pessimistic perspective, perhaps we'll get into that.
00:03:08
Speaker
But I think one of the nice things about talking about microservices now is we are at least two hype cycles down, right? It's no longer called, it's no longer hit, and we can talk about what it actually is and whether it worked.
Successes and Failures of Microservices
00:03:20
Speaker
So let's just start with the big picture. Ian, was it a good idea?
00:03:26
Speaker
so So yeah, when i when I give talks about um where did it all go wrong, I've got another one about TDD. it's depth It's meant in a very loving fashion. So there's a famous story of George Best, who um was in his hotel room ah with a Miss World lying next to him naked with champagne and cocaine around the room. And a bellboy brings in basically some more champagne and you know some caviar, looks at him and goes, George, George, where did it all go wrong?
00:03:53
Speaker
And so it's funny in that context that I think in some cases I look at where people think a practice has broken down, ah why and and what has what has led to that that viewpoint. So I think of late, there's been a bit of press saying, oh, you know, microservices are a disaster, you know, Amazon can't figure them out. You know what, there's a whole Prime Video episode.
00:04:20
Speaker
um Why are we at this point is the question i kind of wanted to kind of get to and and i think i don't want to necessary jump over jamie so i will surrender the floor and trying to get what i introduced it i think the problem is occasionally like a lot of things in tech.
00:04:38
Speaker
ah People take ah what is relatively straightforward set of ideas, rules, and principles, and things get confused. right Things get confused as we go forward. There are misunderstandings around key terms, so micro-unfortunately is a bit of a thing that caused, in my opinion, some misunderstanding. right um Even though in the original in the original paper it tends to be quite clear about what that actually means. And I think some of these in the DevOps community and the simultaneous arrival of containers and communities also caused other certain misunderstandings. And I think the other thing is when you get into practice of this monolith to microservices journey, there are decisions people quite often make because they're doing it at speed. And when people,
00:05:29
Speaker
Because often the thing that causes people to want to use microservices is that they need to scale their organization rapidly. They need to bring many more developers to bail on the problem space because they have some kind of scale up and now are growing at velocity. And the problem with that is that that's often the worst time to be making decisions about how you partition your software and how that software interoperates because it's all done at pace.
Lessons from First Generation Microservices
00:05:52
Speaker
And I think i think part of the problem is there's a first generation of those decisions which people look back on and regret.
00:05:58
Speaker
But I think that actually they've learned a lot about how they should improve it. For now, they've gone beyond scale up to be much more a kind of stable state. And they can look again at that infrastructure and go, you know what? We learned a lot. Maybe we should redraw the boundaries a different way. um But I don't think that always comes over in the discussion about microservice. I think what you get a lot of is it's all been a terrible mess. And that's what the choices you made have led you there.
00:06:23
Speaker
um But it doesn't mean you need to abandon microservice, it just means, okay, we've learned, we'll arrive apply those lessons to how we move forward with this. But yeah, I think a lot of it is that misunderstandings and often application at pace.
Defining Service Boundaries
00:06:38
Speaker
Yeah, the thing I want to pick up there is boundaries, because it's terribly hard to know where to draw the boundary lines around services or microservices or bits of the monolith that live in different libraries.
00:06:50
Speaker
You're a ThoughtWorks person. You can answer that. Tell me where to draw the boundaries. Right. So what you need to do is you need to get a pen and you write all the things on it and then you draw around it and that's your boundary. It's fine. Easy as that. yeah Easy as that. How much do you charge for that advice? I don't know. You can't afford me though.
00:07:07
Speaker
ah oh um But it's interesting. i mean like I think you said something fascinating there about um Prime Video, for example, if you came across this this this ah this thing where, hey, we're moving away from microservices, we're going to refactor everything into this new into this into this model. They didn't at all. ah it's yeah They actually refactored a bunch of functions into a microservice. So it wasn't it was completely missing like miscommunicated, actually, in a way. um i mean i i actually I agree, actually, with everything you just said. This is going to be a terrible panel, because we're just going to agree the whole time. um i I agree with everything you
Adapting Microservices to Business Needs
00:07:42
Speaker
said. you know it's
00:07:44
Speaker
There is a there is a there are generations there are there's a generation of organizations that adopted it early who are now hitting the boundaries, I think, and finding new problems, actually. um But also, there's, I think, a misunderstanding somewhere along the lines. I did a talk several years ago now, maybe 2017 or 2018, called Building Software That's Never Done.
00:08:03
Speaker
And one of the core principles, one of the ideas behind microservices is on the shift of products versus projects. you The project thinking was about getting finance for a thing that's then finished. and you're done right So you decide the beauty you decide the boundaries, and then we're going to build the thing to the boundaries, and we go, oh, we've got the boundaries. Well, damn, microservices suck. Well, actually, the whole point of it was that you were supposed to be able to morph these things.
00:08:27
Speaker
You're supposed to work in teams that were able to make their own choices about where those boundaries were. And if they were wrong, you should be then going on to change those boundaries without having to get another project-based funding for that, you know because you'll fund it as a product team over over a long period
Product vs. Project Thinking
00:08:45
Speaker
of time. There is another question, I think.
00:08:47
Speaker
related to that, which is, well, I think this is fascinating. Were they a zero interest rate phenomena? And actually, is product thinking and product team thinking a zero interest rate phenomena where we were we were talking about scale ups we were talking about, whereas at the moment, everything everyone's talking about cost efficiency and um economies of scale. I think that's a different question as whether product continuous funding this project, you know, short-term funding. You're not talking about technical debt interest. You're talking about because money was so cheap, you could do all these crazy scale. We might have thousand developers, and we'd have these thousand developers do the thing, and as long as they were at adult we were making money hand over a fist, and then that's fine, right? Whereas actually now, there's
00:09:26
Speaker
What I'm seeing in the market is a retrenchment away from that back towards this idea of what' have much smaller teams. um So there's ah there's an idea continuously, you could grow your way out the problem. yeah but Whatever the problem was, it didn't matter because you had 20% growth and you'd basically have the capacity to solve the problem with just more people. right and now ah can I can remember having conversations with you know, staff engineers in in my team who would say to me, Ian, no one seems to care that this is incredibly inefficient. And I said, you're right. They don't. And that's not going to be an argument that you're going to win with them with because they're just going to go, well, it's all right. We'll grow out of it next year. I think everything's interesting.
Managing Technical Debt with Flexible Design
00:10:00
Speaker
I kind of want to bring my half. I'm going to touch on a little bit when I talk on Thursday is so there this is problem, effectively, which is the continuous thing you mentioned, which is um you but what what we do is software as we build a thing with a structure.
00:10:16
Speaker
And the problem is... we we create that structure based on a moment in time. And that'd be great if the business we worked for never changed. Of course, the the business keeps changing and evolving and having new ideas, et cetera. And so then we defined this complete misalignment then suddenly between what the business does and our structure, which is what we tend to think about as technical debt. And that is just that domain driven design has this hydro context mapping, which is this kind of way of trying to say, what is the actual domain now compared to what I what i
Microservices Adaptability and Replaceability
00:10:44
Speaker
actually build? And oh my gosh, everything seems to be misaligned.
00:10:47
Speaker
So one of the things I think, in case people miss about microservices, is is what did where did this idea of small come from? I don't think small was focused, but replaceability was the focus, was this idea that, well, we need we can need to keep evolving because the business is going to keep changing. And the only way to basically solve that problem is to change our software, right? And changing our software is proven to be hard when we're inside a monolith because we just get entangled into a ball of mud. So is there is there a technique which makes change easier to cope with?
00:11:15
Speaker
So I think I partially agree with you on zero interest rate, but I wonder actually if it's more businesses are changing less yeah that is actually
Adaptability in High-Speed Environments
00:11:25
Speaker
causing the phenomenon. Or rather that's the, the that group of scale ups and hype scales have turned into more mature organizations and are slowing down a bit. That's a very interesting point. I mean, I think looking at So one of the original inspirations going back, about 2005, 2006, 7, I worked in investment banks a lot front in front front office sitting on trading desks with traders building software. I don't know if you've worked in those environments, but they're incredibly fast paced. The idea that you can say, put that feature request in my backlog or get to it in three months.
00:11:57
Speaker
because you know because I've got this big monolith and it's really hard to change and we've got loads of other equipment. That just isn't going to cut it because you need to take advantage of of an opportunity in three days. We need to start trading this new exotic in three days. Can we do it or not? I wish I'd had that experience. but go i right i Okay, fine. So I can name those. I was at Citigroup, I was at ah RBS, and I was at Dresden at Kleinwald as well. in capital markets, equities and interest rate derivatives. I'm not going to name the banks I've worked for, but my experience was everything would take 18 months unless it was a regulator requirement, in which case you just move heaven, earth and hell. right right right Because you had no choice. And that no and that that does happen as well, often with the middle and back office, more middle and back office systems.
00:12:40
Speaker
um But sitting you know sitting with traders, where they where they you know can we do this in three days? I want to be able to start trading on this with this product and on this product on you three days. um You need to be able to say yes or no you know based on on those sorts of timescales. And the way you could do that was by building small bits of software that did individual small bits of things. So you wouldn't take the big, yeah you've you've got ah you've got a you firehose feed of pricing information coming in from Bloomberg and other places, right?
00:13:09
Speaker
um So you wouldn't you wouldn't take the thing that's publishing the prices that you've already got and change that to add the new things that you need and deploy that again. You would build another one, maybe you can copy and paste that thing and just change the internal so it was it was so was the new thing. um And so to doing that, you ended up with this estate of lots of these small small applications doing small things. That was, again, one of the things, you know, and that was actually just absolute pace of change, pace of change, pace of
Optimizing Microservices Dependencies
00:13:37
Speaker
change. Yeah. So one of the things that I did talk about, I'll be very quick, in my talk is back in 1975, you ordered in fact to accept a talent and structured design, there's a little graph, right? And it talks about what we call intra-module, inside the module dependencies. And it says, of course, the smaller you get, the more you reduce the intra-module dependencies. So the lower coupling you have,
00:13:57
Speaker
But the more services you have, the more you you increase the inter-module dependencies. So you get a curve a bit like this, right? So I refer to this as the Goldilocks point. There's a point for your business where effectively you have maximized, effectively you've got as small as you can without increasing the number of services to the point that... But that's different for everybody.
Choosing Microservices Based on Needs
00:14:16
Speaker
you yeah Unfortunately, i there isn't like someone can say to you, that number's always 42. And once you get beyond 42, microservices is a disaster.
00:14:24
Speaker
that you can have The difficulty is you have to find that number. Everyone is obviously going to get that wrong and then have to try and adjust. Now I've got a better clue for my business, what that number is. And of course, as your business changes, that point may change, but you're just, that's what you're trying to do. Hit that point where that balance between getting smaller, but having too many things that you're then replacing it with dependencies between things is right. And that's been true since 1975. But that connects back to the idea that we're now two hype cycles down at least, right?
00:14:53
Speaker
Have we learned anything about finding that number? Have we got any wiser about going into a new project with microservices? so Again, maybe ill i'll i'll I'll answer that question. I'll answer a different one, which is always a good technique if you're a consultant or a politician, actually.
00:15:09
Speaker
um so
00:15:13
Speaker
i think what we've I think we've learned quite a lot about I think yes. but sorry back up My perspective on this is still and has always been that it's a particular architectural style that you should use if you've got a particular set of problems. If you've got the requirements that mean microservices make sense, then use microservices. It's the reason we've never had it on an adopt on the thoughtless technology radar is because you can't say it's a sensible default architectural style because it's got sharp edges. You're going to cut yourself with it unless you do a bunch of stuff. It's a bit like XP. you know If you just take one of the extreme programming practices and say, hey, we're going to be successful because we're doing
00:15:56
Speaker
um on-site customer, right? But you forget it about pairing and you forget about test room development and you forget about user stories and you forget about
Infrastructure for Effective Microservices
00:16:03
Speaker
um you forget about ah but all the other ones, right? um Then you're not going to be successful. I suspect that's, I think that's very much the same with microservices. yeah You can't, you can't expect just because you've got lots of small things to be successful. You need to think about all the rest of it. Are they independently deployable, right? Do you have the infrastructure or do you have the ability to um to manage them in in in production. you know you know Are you running build it run it teams? yeah That was the original idea was that you'd actually be building these things and running them yourselves. So what have what have we learned? I think what we've learned is i what I've learned is people still tend to make
00:16:43
Speaker
decisions based on the hype cycle ah yeah because, hey, this is cool, right? And let's do that. yeah um In the same way, I think, you know, it's like the the default these days is, hey, everyone must use Kubernetes, right? Everyone has to. If you're not using Kubernetes, you're not cool. Well, actually, you can run stuff on bare metal. It's fine, right? so That's actually okay, too. a Do you have the requirements to necessitate or lead you towards microservices? One of the original talks I did, Java the Unix way, was back in 2012,
00:17:09
Speaker
in Qcon San Francisco is before Martin and I wrote the definition up, was about a, I think it was called, how the subtitle was how how we built an event-driven system based on you based on services that have a thousand lines of code each or something like that. um And that was that was because we had a particular set of requirements. and We needed really high throughput, we needed low latency, we needed to be able to burst overnight to to do sort of auto-scaling stuff for our inconvenient consumers in queuing and queuing.
00:17:38
Speaker
We had this these requirements that sort of pointed us down this direction of of building the system in a certain way. And I think that i think that's something that we... maybe need to hold to a bit more than how many is the right number,
Organizational Influence on Microservices
00:17:53
Speaker
right? right yeah That reminds me thinking of banks. I'm going to throw this one at you, Ian. So I think one of the requirements that leads to needing microservices has never been technical. It's always been organizational. You hit the point where your biggest problem is in order to release something, you need two other departments to coordinate with you. And those are the projects that fail or slip behind by six months.
00:18:17
Speaker
Are microservices really a technical organization or are they a social organization turned into a technical problem? Well, I think microservices and organizations go hand in hand, right? Because of sense, once you adopt microservices, the I think you enhance the impact of colonized law a bit, right? Because you are effectively saying, I'm going to create a collection of people, process and technology and use that to deliver a microservice. and um One of the anti-patterns I've seen occasionally is, ah this team seems very busy. These people don't seem busy. we' We'll give them basically they can build a microservice. And actually, you probably shouldn't have broken those two things apart because, for example, one of the forces that I would say means you shouldn't break things apart would be, there needs to be an asset transaction here, not a base level transaction. So as soon as you get that and you split their part, they end up with this very chatty thing that's not very reliable.
00:19:09
Speaker
And you did that because you decided to adopt your organizational solution to the problem, right? So I think you have to, the problem is if you can't orient your organization around the delivery of microservices, you will struggle because the two forces will keep keep continuously bashing against each other.
Socio-Technical Issues Revealed by Microservices
00:19:28
Speaker
yeah um And you know we also get, I also see tough wars about, you know, um Surely we own this responsibility and this microservice, and you can't own that bit because and people are human beings, right? In the end, it's always a socio-technical problem, as we say, right? And it's that autumn demarco that it's always a people problem, right? And I think that
00:19:50
Speaker
it just becomes clearer with microservices because you are orienting around groups of people delivering, given the things, you surface that problem. I suppose it exists in the monolith, but you kind of, you kind of, you bury it under the mound. Whereas with microservices, it just becomes clearer that you need to solve socio-technical problems as well. I just want to get back on the the partition thing and the question of Because it's the most fundamental problem for microservices, right? And only architecture, what what architects do, if you believe in that term, is partition software. So how do we partition
Introducing Residuality Theory
00:20:22
Speaker
software? And and I but don't want to abuse the idea, because I'll give you a short version of it. It'll be bad. The most interesting thing I've seen recently is Barry O'Reilly's Residuality Theory. um So Barry decided that we don't really know what we do as architects.
00:20:37
Speaker
we we Boy, does he tell everyone about it. Yeah. He's the architect who will admit it. He's great. He's very good. and And what he's saying that to is is more, we we have these kind of practices that we use, but none of them really... The the answer is we just employ them, but none of them are successful. What's actually successful, we just have some intuition and experience, essentially, that we're applying to the problem. And we rationalize it by saying, oh, well, yes, I used domain-driven design to figure this out, and we probably didn't. We probably just didn't reapply domain-driven design to justify the decision we'd already made.
00:21:08
Speaker
But he has an interesting take on how do you solve this problem of it's a chaotic world and you're trying to build a structure for it. So his idea effectively is actually some kind of a i got horney's probably going to hate me but an style approach where effectively you create what you call these series of stressors.
00:21:24
Speaker
There are effectively an arbitrary number of things that could happen. ah You take your naive architecture, you subject it to the stresses, you understand what that results in. Generally, the results tend to cluster as things that would happen to the system, and then you say, well, what's the fix? And then you buy those fixes to pay the business priorities. So that's the way I'm trying to simulate what might happen over five years, so that, essentially, rather than having to experience those five years and go, wow, it turns out, you know, 2000 microservices wasn't the best idea, that you could actually simulate that and prove to yourself beforehand, okay, that wouldn't work for us. Probably my simulation suggests we need about 30. And it's a very interesting idea that we could, rather than experience the pain, we actually simulate the pain, and as a result, at the end of that go, okay,
00:22:13
Speaker
If you want to avoid this problem, we should do this right now,
Simulating Future Stresses
00:22:16
Speaker
right? or Or at some point, we're going to need to do this. You could defer that, but you're going to need to do that before you get to this point. And if you not checked it out, it's very interesting. i It's definitely a slight problem. barr Barry wants to basically teach people how to do architecture. And I think at the minute, the main audience for it is probably people who do architecture.
00:22:35
Speaker
so But it needs to get slightly broader. But I think if you are struggling with that problem of how the hell do I at this point make decisions about how to partition my software and anything that's more reliable than that, it is definitely an interesting technique. It definitely is. i mean i've i've been Because yeah and it's it's founded in complexity science and chaos theory. We talk about the the the the the the way that a system could evolve by applying these stresses, you identify the attractors, if you like, in the complex system. So there are different, there tends to be different, even though you're applying different stresses to to what you have, that there are these attractors, these clusters or clusters of future states that are more likely depending
Complexity of System Design
00:23:20
Speaker
on the... on the Give me a strong, great example. um good oh Well, there's there's some this really interesting ones in his talks. There's a great one about some electric ah car chargers. Do you come across that one?
00:23:31
Speaker
Yeah, I have seen electric cars. I just want the end. And what happens if somebody loses their key fault? Right. So they say, OK, well, we'll put in um we we'll just pay for it by effectively having number plate recognition, your drive up and we'll just take off your account.
00:23:46
Speaker
and that's interesting because later they raised the problem of what happens if somebody tries to steal basically our electricity it's okay because that attract to where we recorded your number plate means we have your number plate so we just you know send you a big bill in the post and there was another one right with the meat yeah yeah there there was some there was so something apparently that started happening i think it was in germany but i'm not entirely sure where some folks were getting quite upset with the idea of having to um Yeah, they've derived towards um EVC, yeah, natural vehicles. And um so these protesters were starting to stuff mincemeat into the charges.
00:24:26
Speaker
And so this is, yeah, this is an example of actually like totally unintended thing because when they were trying to work out how to stop this, it turns out they'd already solved the problem by having implemented the feature of number pick recognition and et cetera, et cetera. So by solving a set of other problems, they'd solved for a wiser set of problems. And that's essentially the attraction of the problem. Yeah. So the idea is these attractor states are ones that effectively There are only a certain number
Using Simulation for Architectural Decisions
00:24:53
Speaker
of them. As you keep creating more scenarios, in theory, you don't create more of these tractors. So that gives you a feel for what could I build that would end up being a resilient system that would work for me. So it's definitely the most interesting way I've seen to try to solve this problem in a long time without the current solution, which is try, experience, pain, and adjust, which
00:25:16
Speaker
Which works and all animals basically work by that mechanism but it's not it's not the kindest way for us to figure out what boundaries look like, right? There's a brown tradition of fall around and find out, right? Yeah, there is basically. could try yeah So are you saying this is a technique for figuring out what your microservices boundaries would correctly be ahead of time?
00:25:37
Speaker
I don't think correctly. I mean, it's it's similar to a set of practices that that have already existed, right? So if you think about, right, we've got a current system as a state, it looks like this. We've got these boxes and we've got these these software systems and they communicate in this particular way. What should we do? How should we proceed? How should we migrate away from whether it's a legacy, I don't know, pricing slip, whatever it is, right, some legacy software. I mean, what what would I do in that circumstance? And often I'm asked in to help with these sort of questions is well I would think I've come up with ah the current state and then I would think about a number of future states and I would think about some pros and cons of each of those okay what happens if I so if i split things up into self-contained systems say what happens if i we we go full microservices or what happens if we just replace the model with another model it's because you're going to buy it from a supplier somewhere or whatever it is. um
00:26:25
Speaker
And then you try and understand what what the risks are associated, the constraints, the pros and cons are reaching these things, and then you make a recommendation. That's generally my job. I'm a consultant. um And then I walk away and everyone else has to solve the problem. This is one of the guys who solves the problem, of course, um because he does real work for a living as opposed to just wave his arms around. um But the the the idea of residuality is putting structure, much more structure to that, and crucially mathematical underpinning of it, such that these are, because you have these attractors, the attractors but will mathematically, well there's a mathematical relationship between the attractors and the number of um
00:27:07
Speaker
I guess states that the architects couldn't turn into in the number of ways it can fail in the future. yeah So it's it's a sort of more. each like You can figure out, um you have to get a big spreadsheet, but you can figure out by a big spreadsheet our thing is when stresses happen.
00:27:23
Speaker
are are things impacted together or are things that basically, are ah so those things that are impacted together, you have to ask the question, okay, if a whole load of times these things always seem to be impacted together, a question would be, are they really part of the same thing?
Constant Change in Software
00:27:39
Speaker
Should should I really partition those? so Are they always actually subject to the same forces of change? Which is something we tend to say is,
00:27:45
Speaker
you know, things inside a a given, any kind of module is things that subject to the same forces of change. So it would say to you, these things subject to the same forces of change. And it might also say, this thing is actually subject to quite diverse forces of change. And then maybe you should break it apart. And that would actually make your architecture more resilient.
00:28:01
Speaker
So it does try to figure that out for you as well. So it's like, and I think it's ah it's for you're exactly you right. It's formalizing what we what people who that's what richard Paris intense formalizing people who have been succeeding. He tried to figure out how do they succeed? Okay, I can give you a I can give you a proof of that and I can give you a formal way of achieving that.
00:28:21
Speaker
You still have to, at the end of it, use these things like yeah know ATAM and Vermeer and go to the business and figure out what that architecture looks like, but at least it formalizes that stress part of the process without you having to experience it. and One of the things I find interesting about it is, um and we're just starting to see it's use pick an environment, which is why.
00:28:39
Speaker
I think we're both struggling somewhat to describe it because it's still I think his PhD. Is it the next type? so's It's the next type. Awesome. You heard it here first. I think his book is number three on the pub at the moment. So yeah um it's it's it's getting the hype already. But I think i think what it's it's it's it's interesting for me because one of the problems we've got when we're building software in teams of people in organisations is you can't solve for time. right So i mean any decisions you make about the structure of the teams, the software that you're building, each line of code that you write, um the impact of that is only going to be felt in the future. And and the only way the only way of knowing what the impact is, is to go to the future and find it. right so um So simulations, are i mean this is how you would solve for this in you know for things like I want to build a new
00:29:24
Speaker
a new wing for an aircraft.
Feasibility of System Simulations
00:29:26
Speaker
You don't just build the wing and then put it on an airplane and see if the airplane spirals down. You simulate it, you put it in wind tunnels, you build mathematical models, you do all these things. And I think there is an opportunity for us in the industry to do a bit more of that actually. Because then you can run it, you can run the simulation many times, much, much quicker than it takes to experience it.
00:29:47
Speaker
in five years, if you see what I mean. You can have to make that more concrete for me, because it sounds like we should build some software that simulates the software we're going to build. And that sounds like another 18 month project before we even get started. Yeah, well well, I mean, I would start with by building the simulation of the simulation of the simulation to build the simulator, yeah the software that we're going to simulate. But you you build by the hour, so that will work. yeah Day, day. But but are we how how can you feasibly build a simulation of the system you're trying to build when software is inherently a simulation of the thing we're trying to solve in the real world.
00:30:26
Speaker
Yes, you're simulating effectively how how things change over time, right? So you're simulating how a structure reacts to forces over time. But honestly, I mean, you can do with this spreadsheet pretty much.
Balancing Dependencies with Predictive Techniques
00:30:37
Speaker
But um I mean, you know, if we were in a complex financial model and we'd be running multi-kilometer simulations all the time, it would be bread and bar to us. The fact that that's what you do, you you say, what happens under all these variable conditions? What are the outcome, all right? And so I think the the question is, if we go back to the original point about microservice, which is,
00:30:56
Speaker
there's effectively this kind of Goldilocks point. And the difficulty getting the Goldilocks point is, can I predict where effectively this balance comes from in between intra-module and inter-module coupling? And the current best way we know of finding that is to suck and see. The question is, is there a more, is there is there slightly its a slightly less painful way of doing that? And maybe that, yeah, we could probably get closer. Like anything, I suspect it probably will be a good approximation, but it's probably a better starting point to be there or close to there than it
Goldilocks Zone in Microservices
00:31:29
Speaker
is right now.
00:31:30
Speaker
The problem is it's skill and judgment, right? Yeah. And I think basically, I mean, you we talked about the year curve earlier, so you've got this year curve, right? Which is, you know, there's some point in the middle, there's an optimum. You don't necessarily have to be at the optimum at that exact point. there's like But there's a Goldilocks zone, I guess. You can use the bore of the kind of distance from of the earth from the sun, yeah where water can exist in in life as as liquid water. We've all worked with companies that are definitely not in the Goldilocks zone. Right, right. yeah yeah yeah well the wolf Where have you been working? It's Mars, which is pretty cool, but no, I didn't want to see. yeah You have this, you could, but is the fact is you can be either side right by margin. What you don't want to be, because it's you, you don't want to be at the point where it suddenly shoots off into um into infinity, right? yeah um So if you can, if there's a better way,
00:32:16
Speaker
I've described a technique I would use to try and identify where you are in that you curve. And potentially this is another way of ah of more accurately then. Okay. Yeah. Yeah. Feeling out where basically that zone may be, what you might be trying to target.
00:32:32
Speaker
um And so there's a better chance that you have a good landing, right zeroing in on it early. Yeah. Okay. I can see that. Then let me ask you if that model allows for something else that's always bothered me with microservices, which is let's say you use this model and you get roughly at your correct boundaries and everything's good.
00:32:50
Speaker
You always have concerns that cut across all boundaries. Like the first one is always auditing. you've got to You've got to see an entire holistic picture of the system. And that is always a requirement. Have you either of you got any solutions for that while retaining your nice microservice boundaries? I've got an example, which is the judgeje the sort of degenerate case, I guess, for that, which is um an organization where they have of about 2000 microservices.
00:33:18
Speaker
And it's but it's a financial institution. And they decided that for audit reasons, they were going to use the event log, each of the the message logs, that each of the event each of the microservices were producing and and and the logs as well. um And they were going to use that as the audit trial.
00:33:31
Speaker
which meant they've had to, for each of these microservices, which is producing how many yeah megabytes of logs every hour or whatever it is, um they have to keep those and archive them. And they've ended up with a like login and order infrastructure that is literally two orders of magnitude bigger than the actual production infrastructure that they have. It's It's an interesting like decision that was made. It's like we got all this information. It's brilliant. We use that as the audit log. Oh wait, T is down the line. you know It's another one of those T is down the line. We didn't realize things. I mean i think there are i mean there whilst there are ways of getting around, not getting around, off north.
00:34:09
Speaker
of not doing that. I mean, one example would be data mesh, which is a, that's, I think, I see data mesh as the sort of um the last piece, if you like, in the sort of puzzle of how we are able to build independent or systems that are independent from one another. because yeah Obviously, we have like, we've had data lakes, we've had enterprise data warehouses, we've had all these sort of things, they've all had similar sets of problems associated with change. If you change schemas here, you've got to change schemas there. If youve you can change all the ETL jobs, you can change all that sort of stuff. Do you want to give you a quick definition of data mesh? Oh, it's going to be terrible. um My former colleague, Jamak Dalghani, wrote the book date on on data mesh. um But the the the basic idea is that each each of the domains that you're interested in um
00:34:54
Speaker
essentially serves or ah advertises and serves its own data set. Data is a product, essentially. And so then if you want to compose, for example, reports from different places or you want to you want to yeah do whatever you would do normally with a data warehouse, ah rather than go to one specific location.
00:35:11
Speaker
you have the ability to just scatter where that data is and then query that data from essentially source. I mean, that's the kind of idea. Again, it's an organizational problem as much as anything. It's about the other teams building that we building those that software, offering those contracts up and blah, blah, blah, blah, blah. So um yeah, but but that's an interesting that's an interesting way of of of solving for that in a sense. Can I ask a deliberately liberally pointed question? yeah Back in the old days, we just used to call that a database and there'd be all the tables in the whole system and we just look at those. Why is what you're describing better?
00:35:49
Speaker
Well, so it's a this question of scare, right? So say you're, ah say, I don't know, say you're a, let's see, I don't know, say you're an insurance company, right? You might be wanting to run queries that are across not just a particular system, which had a, of which was a, which was previously a single represented in a single database schema. You might be wanting to create, cause otherwise you wouldn't have enterprise data warehouses, right? Cause you'd have an enterprise data warehouse, which you'd have, which would be fed by all the different source systems that you've got in the organization.
00:36:19
Speaker
So you're not going to have one database ever. You're always going to have lots of databases feeding a single big thing. And then you're running reports on that or generation reports of that. So so I fundamentally disagree with your assertion there. Sorry. but i'm just I'm just for the sake of those that I know some people will wander out this, right? why we it's It can seem like we've split our system into many microservices. And now we're just trying to put it all back into one piece.
00:36:48
Speaker
Right. we we we We've correctly decoupled our all our boundaries and um there aren't shared databases because that's one of the golden rules of microservices. so We now have 2000 microservices with 2000 databases, and we now find we need to query across the whole thing, so we invent the idea of data mesh to pretend it's all one big database again. I think one of the things we learn is the difference between the needs of transactional and analytical data. yes So, I mean, typically, say for us, ah in a in in the transactional kind of data world, we can use at-scale databases like Dynamo, because generally what we need to do is store something with an identifier.
00:37:26
Speaker
Maybe two identifiers, but it's just very easy. It's a key value look up, right? So if I want to look at your shopping car or I want to look at your customer record or where the restaurant currently is with my current place of work, very easy. Just look up on a key value store. I can find that, right?
00:37:42
Speaker
But obviously, if I then want to basically slice and dice that data in new and interesting ways, I need a different kind of data store, right? So I need some kind of ability, potentially relational, or potentially other graph. I mean, all sorts of ways I might want to slice that data. So as soon as you get into that word of kind of polyglot views of your data, you kind of also begin to move away from this idea of I could have one single database that would serve all my needs. now The reality is and so chapters before people start doing the wrong thing at the wrong scales again, it is a scale problem, right? If essentially there are 40 people that use my system,
00:38:16
Speaker
Sure, use a relational store and query it, right? Once you get basically the the numbers begin to get bigger and the way that you want and you you outgrow your relational data store. I mean, I think it's public knowledge, so I'm pretty sure I can mention about before my time in 2016, Just Eat famously sponsored X Factor and during X Factor's kind of like saturday evening break fell over because it was trying to build on a relational data store that i've actually just could no longer cope it was about the biggest you know versed in that data store they could get and just no longer could cope so that's a lot what drives you towards that kind of problem of i need to break things off i need to have smaller stores i need to use a kind of at scale store like dynamo
00:39:00
Speaker
because it's not cheap, right? But as soon as you do that, you're then like, okay, but now I've specialized my database to be able to cope with the kind of volume that I'm seeing. I now need more databases because they're all filling quite specialist niches in what I do. um I mean, I think, to answer a question about sort of order, etc.
00:39:19
Speaker
So we definitely see some of that in almost like a semi-mesh sense where things like we need to know where all the j GDPR data is right across this thing if someone wants to cancel it. So we actually have things that the services tell another service, or I have some GDPR data. So you can go to this central service and go, where is the GDPR data? And it tells you, OK, I can tell you you who owns the GDPR data for this customer because they've all told me that they own some of it.
00:39:48
Speaker
i've got I've got a good time on the database thing that split them up. I was at an airline some years, many years ago now, and they had one giant database that served everything, essentially. So it did both the airside operations, so you know check-in and um you all the green screen stuff. They had green screen interface too. It's a SQL Server database. But it also served on the retail and fly availability and all that. So it did both sides, everything. And they had, the I think if you remember,
00:40:16
Speaker
Many years ago, there was the previous Icelandic volcano problem, which grounded grounded loads of flights. um And they ended up they ended pretty much being grounded, even though they weren't flying routes through that went near the volcano. They were flying they were mainly flying south. And the reason was because people got worried about whether their flights were going to be canceled, hit the website en masse, like thundering herd problem.
00:40:42
Speaker
um And of course, it it's because it was a single database. It started slowing down. um And then this web page isn't loading. Refresh, refresh, refresh. Even more load. And then the whole thing just went fell over. And they ended up having to go back to pen and paper around the boring planes. It was a complete catastrophe. And that was literally around because they had this single this single database serving all the purposes. So ultimately, your answer to that is we just can't go back.
00:41:09
Speaker
It's the age our problem, isn't it? right Which is it depends but then I think the thing is what is your context? How big and but that's the useful thing I think about residuality theory is it would tell you okay These are the states you move into if these things happen all these things going to happen Well, maybe if we're successful in two years that happens Okay, then we'll come back and look at this and we'll figure out we'll buy all those that complexity but i I think it's the age your problem where you know, so so I mean the reality of most
00:41:37
Speaker
big companies I work for, where effectively is that we come to conferences, we talk about exciting things with what we're doing, but the reality is that's the teams that we're willing to talk about doing the nice, exciting thing. We don't want to talk to you about the other teams who basically are in hell. The problem is that people go to that conference and they take away, oh, you know, it was doing this or Microsoft's doing this and I must do this, right? effectively But that area is quite often thought about that context and has that said, we have this context to do this. There are the whole parts of the business that are just doing nothing like that. Right. And see so that.
00:42:16
Speaker
This notion quite often of context, I sometimes feel that talk should come with a health warning, which says, this is my context. yeah And if you have context like ours, the ideas I'm about to express my help you, if you don't have my context, I can't tell you. Right. There's a reason patterns come with context and forces, right? Yeah. is that It reminds me of exactly like one day, it seemed like one day Microsoft announced they were doing microservices and every size of company now tries to do microservices.
00:42:44
Speaker
Yeah, well, I think the problem is as soon as you, ah I mean, partially also, I think there's an industry problem, right? And the industry problem is that we don't hire people as software engineers. We hire people as a set of skills and keywords on a CV that gets fed into some recruitment machine algorithm that pops out the relevant CVs that match. So we incentivize people to basically have microservices on their CV or Kubernetes on their CV rust on their CV, right? Rather than saying, well, you're a software engineer, I guess, probably, you know, we give you the skills to solve problems and you solve the appropriate skills problems. If it turns out there's a particular technology stat that you're lacking, well well, just teach you, right? It'd be a much better world. Unfortunately, we people will always react to what you incentivize them for. So if you say, I'm going to pay you 20,000 more,
00:43:35
Speaker
If you know the microservices, everyone's going to want to say, I'm going to do microservices at my current company so I can go to my next. I was at a, um I was at a government department. Uh, we wish that should remain nameless. And then suddenly they had all this, everyone was a security architect because also I was genuinely, everyone was just because we' we're all the other ones. Well, they, they used to be these other architects, but now they're security architect. How did that, well, they get paid 20 grand more because this pay scale of civil service pay scale is more for the for security architect.
00:44:04
Speaker
yeah yeah So they all just went boom, went over all security architecture. The other really famous case of that is when .NET came out, they basically had Visual Basic .NET and C-sharp .NET. Effectively, they both because they're both trying to represent the underlying assembly ah model in the and the runtime, were very, very similar. it was just like a It's like what's the difference of dialect. right no but C sharp developers were paid like twenty percent more so overnight there were no more VB developers because everyone just said what's pretty much the same language so i'd like to declare myself a C sharp developer and then there's no more VB developers right i mean but why wouldn't you do that it seems utterly predictable in hindsight and it seems like when we're not learning so i'm gonna ask
00:44:49
Speaker
Because we're at time, I'm going to ask you to leave me with the name of that book again, because it sounds like the one place we might learn to predict some pathologies here, and it's Barry O'Reilly. And there's some great talks on YouTube if you want to start getting into the idea. Cool. James, Ian, thank you very much. You're welcome. Thank you.
00:45:09
Speaker
Thank you, Ian. And thank you, James. And thank you to Goto Copenhagen for hosting and recording that one. Thank you everyone in the audience. I really enjoyed being a part of that, and I hope I make it back there next year. I will say before we go, there's one idea that didn't come up in that conversation, but Ian ended up saying it in his talk at the same conference, and I really liked it, so I'm going to repeat it here. He was talking about Amazon's two-pizza rule, which you may have heard of. No team should be larger than you can feed with two pizzas.
00:45:42
Speaker
It's designed to be a good rule of thumb for sensible, cohesive team sizes. And the thing I mentioned is it's a bit of a trap if you apply it blindly, because it lacks one really important piece of context.
00:45:58
Speaker
Two pizzas in America can be radically different sizes to two pizzas in Europe. And if you apply that sensible-sounding American rule blindly in Europe without considering the context, you might end up with very different team sizes and very different results. And I will be thinking about the two pizza rule the next time someone says to me, it's best practice, without telling me what context it was best practice in.
00:46:28
Speaker
I leave you with that thought. However, one thing that's always best practice in all contexts is, of course, if you enjoyed this episode, please take a moment to like it, rate it, maybe share it with a friend. If you want to support future episodes, please head over to Patreon, link in the show notes, and thank you to all of you who are currently supporting this little endeavour. Until next time, I've been your host, Chris Jenkins. This has been Developer Voices with Ian Cooper and James Lewis. Thanks for listening.