Transcript
Speaker: Hello and welcome to another episode of the Artistry of Code podcast.
Speaker: Your hosts for today are Grzegorz Godlewski,
Speaker: And that's the one.
Speaker: So today we've met up to talk about what went wrong with microservices.
Speaker: And this episode is inspired by our friend Grazjan, who was curious how we are going to uncover the subject of the microservices.
Speaker: So Grazjan, that's for you and for all of our listeners.
Speaker: Just before we start, guys, make sure that you follow the show on Apple Podcasts, Spotify, and subscribe to our channel on YouTube.
Speaker: Welcome to the Artistry of Code podcast, where we, together with our guests, are seeking ways of doing things right and having fun with it.
Speaker: Software, architecture, soft skills, teamwork and other crazy buzzwords.
Speaker: Are you ready?
Speaker: Let's bring more value to the business.
Speaker: And we wanted to start off by bringing up a couple of stories, which are going to explain what we mean by what went wrong.
Speaker: So what we've prepared for today is like three small talks about those stories.
Speaker: And then we wanted to cover sort of like
Speaker: benefits and drawbacks which you can have from microservices, including challenges, like always, which come across with them.
Speaker: And we also try to provide some recommendations at the very end.
Speaker: So please make sure that you stay till the very end so that you can get all the knowledge from this episode.
Speaker: And just to say it's not that we don't like microservices.
Speaker: It's just this episode that we want to focus on the failure stories.
Speaker: So in most cases, microservices won't fail you.
Speaker: But just listen to what we want to share with you, our experience.
Speaker: And let's see how we can make it better.
Speaker: So Grzegorz, you have the first story from our list.
Speaker: So please tell us more.
Speaker: Yeah, so it really comes to a quite recent experience when it comes to making bad decisions about microservices.
Speaker: And before I describe what was the decision, I just need to point out to one bit of a recommendation that you can frequently come across when reading about microservices is that you can structure your system, you can structure your microservices in a way that they are going to depict
Speaker: particular departments or teams around your organization.
Speaker: So the classic example, which is brought in many books or articles, is that if your organization consists of like a payroll division or accounting division, or perhaps there's like a shipment division, you typically are going to create microservices reflecting their behavior or modeling some processes which are running in those departments.
Speaker: Now,
Speaker: When I was reading this for the very first time, it really made a lot of sense for me.
Speaker: So when I was working at a company which was having very distinct divisions, sometimes even not operating with one another correctly, I followed this recommendation blindly.
Speaker: And this led to a situation that I've designed the system and there were those two microservices in that system which were reflecting...
Speaker: two divisions within that company.
Speaker: And it was all based on the assumption that at some point we are going to have two teams working with those services.
Speaker: Meaning if you also come across recommendations, you will also see that recommendation that each and every microservice should have only one team who's going to maintain it and specifically
Speaker: you kind of set on like an equation that you have one microservices, one team.
Speaker: And on paper, the whole thing looked quite nicely.
Speaker: Everybody really agreed to that when I was consulting this idea with many people.
Speaker: But there was that problem that the basic assumption taken there that microservice A is going to be maintained by team A and microservice B is going to be maintained by team B turned out to be wrong.
Speaker: And we ended up with a situation that we've created a system which had those microservices defined separately with the idea that they're going to be developed and maintained separately, but it wasn't the case.
Speaker: So then the team A really ended up with having the microservice A plus microservice B together in their code base.
Speaker: The problem which immediately emerged is that
Speaker: those microservices, they were also frequently changing together.
Speaker: So if you wanted to develop a feature, like a new functionality, then you had to modify both services.
Speaker: And this already rings a bell.
Speaker: If you are coming from the solid world, this is already ringing a bell and lighting a red light that probably the design here went wrong.
Speaker: And it really did.
Speaker: Because later on, we had to revert that decision and merge those services in order for the team A to be really performant.
Speaker: And probably at this point, those services, even though they were merged, they were like correctly separated into modules.
Speaker: So that's still some improvement.
Speaker: Also, in defense of the microservices, I would say that it's it wasn't for me, it doesn't sound like a problem with the microservices itself.
Speaker: But what I find very demanding when actually designing the system based on microservices is a correct identification of bound contexts.
Speaker: So in this case, it might have been that those two services was actually a single bound context, even though they were separately developed and on some
Speaker: One was influencing the other one and they were dependent on each other very strongly.
Speaker: So it might be the case that just the bound context was not identified as it should.
Speaker: And that might have been the problem that you spot on.
Speaker: Yeah, so starting with monolith in this case is usually much better because you don't have all those possibilities to make it wrong.
Speaker: Yeah, of course, you can tangle it too much, the code between the layers, especially when you're implementing some kind of pipeline.
Speaker: it might be very difficult then to untangle it when the scale comes in.
Speaker: But especially at the beginning of the project, it's better to focus on the functionality and prove it's working rather than trying to make that pipe shiny and frictionless at the beginning.
Speaker: Yeah, so definitely something which was a game changer for me when I'm looking into the past was that I came across domain driven design and those concepts of bounded contexts and aggregates, things like that too late, I would say, because if I would be having that knowledge prior to making those decisions, I probably would have made those decisions better.
Speaker: The other option is that if I wouldn't fail with those microservices, I probably wouldn't see the value in DDD that much, right?
Speaker: Because sometimes it's your experience telling you, oh, so this is how you do it, right?
Speaker: And not the other way around, because going into DDD directly without having some bumps earlier on in the past may not give you that wow effect, because then you're actually going to understand the problem.
Speaker: which is about to be solved by DDD.
Speaker: But then again, when we talk about structuring things, Artur, you also have a story slash problem related to microservices.
Speaker: Yes, yes.
Speaker: I also encountered one actually in more than one place.
Speaker: And the problem is that maybe it's not a big problem because that's what we do, solving problem.
Speaker: But we used to focus on the bond context so much when designing microservices that we don't think about technical problems that might occur
Speaker: in later developments.
Speaker: So let's say that we start with some simple microservice to solve problems of some small bounded context, and that's the MVP.
Speaker: So you have some kind of API, it's serving data, it's starting something in database, very simple example.
Speaker: Then your project is developing new requirements come from the product side, and
Speaker: maybe the scale is increasing and it might happen that you need to introduce the asynchronous communication, which is very common in the microservice world.
Speaker: So in this case, let's say that your microservice now needs to read also some asynchronous messages coming from other places in the system and react to them.
Speaker: And the common mistake that I saw in my experience is that
Speaker: Usually developers together with architects, the code creators decide to go an easy path and merge the consumption of those messages into the code base that serves the API as well.
Speaker: And when the system behaves like we expected, so we don't have big spikes of usage, we don't have any DDoS attacks, et cetera, we don't see the problem.
Speaker: But when the problem comes, when the load is very high, it might happen that the asynchronous consumption will affect the cooperation with your API.
Speaker: And depending on what is your core business value, if the core business value is, for example, taking requests from your users,
Speaker: then you have a problem because your application becomes unresponsive.
Speaker: So your availability drops.
Speaker: You have contracts probably for your contractors if they're using your API.
Speaker: And this begins to be not a technical issue, but like general company issue that can be connected with cache.
Speaker: So my recommendation here is always think about it.
Speaker: Always think what is your core value and try to predict the problems.
Speaker: So what I would only add here is that this rings a bell in regards to the scaling recommendations coming from the 12 factor application so that
Speaker: in that way of implementing apps, you're going to define tasks per process type.
Speaker: Essentially, it goes out to scale using the process model, which is you're going to define separate tasks, really separate processes, depending on the task which is to be handled by that particular code.
Speaker: So if you're having
Speaker: Let's say, let's put this, draw this image.
Speaker: You're going to have like a microservice, which is going to serve REST requests.
Speaker: It's going to have a gRPC endpoint.
Speaker: It's going to have a couple of workers.
Speaker: You don't put this into a single node app in terms of on the same event loop.
Speaker: Instead, you can have this in the same code base, but you define multiple entry points, which you scale separately so that if you need to scale the worker a little bit more, even in the same app, you're just going to scale the worker process multiple times.
Speaker: And you don't have to have distinct code bases really for those.
Speaker: You still can have a single repo.
Speaker: You just define those tasks and the roles inside the application.
Speaker: Yeah, that's right.
Speaker: And you have many good points that you can benefit from using the same repo.
Speaker: You have the same code base.
Speaker: You don't need to worry about other repositories or the libraries.
Speaker: You don't need to bump them if there is a bug.
Speaker: You just use the same code base by exposing different artifacts or one artifact with many entry points, like you said.
Speaker: that can be configured as different microservices in Kubernetes, for example.
Speaker: So this is how it should be done.
Speaker: And we know that we have done it right once in our career together, Grzegorz.
Speaker: But the temptation usually is there to, you know, when you are about to focus on
Speaker: on the business value, especially in startup-like companies, then the thing to actually introduce the separation to different processes is another cost that needs to be laid on the scale.
Speaker: So you need to think about it.
Speaker: But I wanted to emphasize that it might give you problems with production.
Speaker: in the future.
Speaker: So yeah, think about the weight of this decision.
Speaker: So let's go to our last example.
Speaker: Yeah.
Speaker: And that's something which I'm bringing, I guess, in the last year.
Speaker: I'm bringing this one too much, but I'm going to bring it up once again.
Speaker: I think that for me, that's the biggest failure when it comes to using microservices, which I was participating in.
Speaker: But I wouldn't say this is due to
Speaker: microservices themselves, that's more about the hype and bad organizational decisions.
Speaker: So once upon a time, not so long ago, I had the possibility to work on a team where everybody was senior and everybody was microservices.
Speaker: And that combo turned out very, very wrong for a number of reasons.
Speaker: Having a team composed only of seniors is essentially giving you dysfunctional teams this way.
Speaker: Full stop.
Speaker: I'm just not going to comment on that.
Speaker: You can just imagine what can possibly happen.
Speaker: It's not that everybody argues because seniors
Speaker: Normally they don't argue, they're being polite and politically correct and don't get anywhere with this.
Speaker: Yeah, but they're boiling inside.
Speaker: Yeah.
Speaker: And they tend to isolate and this leads to strange behaviors.
Speaker: or situations like if you have people who are not able to talk to one another and get things solved on a design level and you have microservices which creates escape rooms for those people, everybody can go to their own escape room and then you start to feel what's going on, right?
Speaker: This isn't going to go into the right direction.
Speaker: So if you have the low bus factor and
Speaker: Not even that.
Speaker: So the thing is that with the micros, even if even if those people would go to different rooms in regards to the microservices, everybody's going to have their own comfort zone within those microservices.
Speaker: If they struggle to collaborate,
Speaker: those microservices won't collaborate either because that's, again, Conway's law and that's going to reflect the organization.
Speaker: And if the team is dysfunctional in that regard, the resulting microservices will be also.
Speaker: And the problem now, so that was the decision making or the organizational problem part.
Speaker: It then broadcasted towards the technology so that
Speaker: you had microservices in from day one.
Speaker: And the problem here, putting aside that, you know, like that note about people isolating themselves within their own context and doing their own piece of work, the problem was that from day one,
Speaker: You invited a heavy technological stack of asynchronous communication over rabbit rest, things which may fail, stuff like that.
Speaker: So at the end, and that was a Greenfield project.
Speaker: That's very important to state.
Speaker: So how the story ended up was that
Speaker: The team was building up more of like a payroll system using microservices, but all of the technical details which were necessary to get through pretty much stopped development of the business features inside those services.
Speaker: So there was a lot of hassle when it comes to building it technically and eventually
Speaker: The project run out of money.
Speaker: So you had a senior team, which was very expensive.
Speaker: You had microservice architecture, which was very expensive and little outcomes because of the complexity of that situation, which ultimately doomed the whole project.
Speaker: biggest failure that I was able to participate in.
Speaker: And as you can see, I only smile when I talk about that.
Speaker: What a beautiful disaster, Grzegorz.
Speaker: Sounds for me like, you know, the orchestra on Titanic.
Speaker: They were like,
Speaker: going down, but the orchestra was still playing.
Speaker: Yeah.
Speaker: And I learned something from this, which is do not go early into the microservices, really.
Speaker: And that's why I, based on that history, I cherish the idea of modular monoliths
Speaker: to start from as really the way to go.
Speaker: Even if I'm building my stuff, I always start with the same single application.
Speaker: I do that task dividing for workers and web servers inside of that app, but I'm not running into microservices immediately.
Speaker: This is something which we're going to talk about in a minute.
Speaker: So with this, Artur, I guess we've went close to the challenges, right?
Speaker: And the benefits and the drawbacks.
Speaker: Okay, so let's go to pros and cons, right?
Speaker: Of the microservices versus Monolith.
Speaker: So I feel, Artur, that you would like to start.
Speaker: Maybe, maybe I can.
Speaker: So, yeah, the first cons of microservices is that you have so much extra work to start with to make it work.
Speaker: So now everyone works with Kubernetes.
Speaker: It's another buzzword, another honeypot, I believe, or developers.
Speaker: So to make it running, you either need to have
Speaker: developers that are able to configure your cloud and configure deployments for your application.
Speaker: And each new microservice needs to have its own deployment configured.
Speaker: Of course, it can be somehow automatized or at least using some templates, but it's another work that you need to do.
Speaker: Creating a template is the work itself that needs to be done, even though it saves us
Speaker: Sometimes someone needs to do it.
Speaker: And I see that also DevOps engineers aren't the group that is overflown on the market.
Speaker: So usually you have like two, three DevOps, and usually they are shared across many teams, so they have lots of work already.
Speaker: So it might be a wise decision to not add the work for them.
Speaker: Just try to reduce it as possible.
Speaker: Do you have, Grzegorz, some other drawbacks of microservices?
Speaker: So I guess we've covered this during the show mostly as I'm just going to do a quick recap.
Speaker: That's a complexity exploder, really.
Speaker: And if I would say that, if I were to answer your question, what are developers responsible for is like implementing solutions to real business problems and keeping the complexity controlled.
Speaker: And if you go with microservices for the start, you're going to have a big problem very soon if you didn't, you know, like have prior experience with this.
Speaker: So just don't do it.
Speaker: That's what I would say when it comes to microservices and dealing with the complexity of like repositories, synchronizing versions,
Speaker: doing big releases.
Speaker: I mean, the big promise was to be able to release things separately.
Speaker: Okay, you can release things separately if they are engineered correctly and you didn't go across bounded contexts.
Speaker: Otherwise, you will have to synchronize changes and releases.
Speaker: And that's always going to be a big problem.
Speaker: I, in general, came to the conclusion over the years that you need to do an exercise before you go into microservices.
Speaker: And even if you start with solid principles and object oriented programming, really, try to design an application with classes, with modules.
Speaker: and try to throw changes at this application.
Speaker: So let's assume you've created an app which is going to, I don't know, like take food orders and then you're going to be requested to add some sort of like a payment scheme to this or some sort of like a discounting scheme or perhaps some extras.
Speaker: And now
Speaker: within the same app, within the same repository, file next to the one which you've just created, you're going to see if your design thinking was correct.
Speaker: Because if you are going to end up modifying a lot of classes or a lot of objects in the same application, that means the way you design
Speaker: is not fitting microservices complexity, right?
Speaker: So you can do that exercise earlier without Kubernetes, without having all of that hassle of deploying microservices just to check if you did them right.
Speaker: Now, it starts with finding the areas of responsibility.
Speaker: It starts with finding the bounded contexts.
Speaker: And then you can test that model
Speaker: even on a number of files instead of building the whole system and then figuring out that you did it wrong.
Speaker: So I think this is one of the biggest challenges when it comes to going with microservices.
Speaker: This is learning how to design
Speaker: software in a way that it can really leverage the benefits of microservices because they have those benefits.
Speaker: The horizontal scalability, right?
Speaker: You can pin up as many instances of the services which you have, but this is requiring you to have some basic knowledge.
Speaker: And well, basic is like
Speaker: really underestimated here.
Speaker: It's not basic.
Speaker: It requires you to have solid knowledge about software design to get it right before it's there in production.
Speaker: And that's the biggest thing, because then you have all the technicals like
Speaker: If you have remote calls, they can fail.
Speaker: The server on the other side can become unavailable.
Speaker: It can be rate limiting you.
Speaker: Lots of branches of code.
Speaker: But then again, different effects come in, like back pressuring, for instance.
Speaker: If you have multiple services and the one at the end,
Speaker: is struggling.
Speaker: So you need to back pressure.
Speaker: So put that pressure back to the beginning of the system so that the users at the very beginning, they're not going to destroy the initial service, issue as many requests as possible so that the service at the very bottom of the stack can really have a break and recover from the surge of the traffic.
Speaker: And if you have the chain of services long,
Speaker: this is only getting worse and worse because you have to solve it on each and every chunk of the system.
Speaker: So, yeah, that's a lot of things to take care of, really.
Speaker: Not in a greenfield project, not day one.
Speaker: Yeah, that's true.
Speaker: And except for the design complexity, you also need to introduce the communication between them.
Speaker: And that's some kind of organizational complexity for me.
Speaker: Because when you start with microservices, and let's say that you will distribute it to some different companies,
Speaker: teams, then adding any cross-team functionality, cross-micro-service functionality will cost you more.
Speaker: And reacting quickly, especially in this startup environment, is very difficult because first you need to talk to the other team, try to predict the
Speaker: communication protocol, you need to design APIs, you need to wait for them to implement it.
Speaker: They might have their own problems with priorities because they may have been given some other very important tasks to do.
Speaker: Like I said, you need to, in the end, deploy it sometimes in a synchronized matter, so both of them at the same time.
Speaker: Or when you start implementing some feature flags that
Speaker: It might not be needed for you, but now the requirement may come, but another extra
Speaker: work needs to be done.
Speaker: So there's a lot of complexity while you don't benefit the main benefit that gives you the microservice design, which is the scalability.
Speaker: When you don't have clients, then you don't need to care.
Speaker: And I've heard it in a recent podcast that the only moment to consider microservices is when you're scaling
Speaker: is not sufficient.
Speaker: So if you see that your application is suffering from load, then you need to stop with implementing new features and start refining your design.
Speaker: And refining your design doesn't mean immediately going to microservices, especially that going with microservices, on the other hand, with the micro...
Speaker: prefix is something to go.
Speaker: Please raise a hand or write a comment to whoever who works with really microservices.
Speaker: In my case, it was usually just services.
Speaker: So the bounded context wasn't micro.
Speaker: It was like some starting context plus everything that fits the best when we needed to add it.
Speaker: So that's how usually it's implemented.
Speaker: It just reminded me about another promise which was there is that if you have different microservices, you can run different technologies.
Speaker: So you can have one microservice in Node.js, other in Python, the third one in Rust, the fourth one in C++.
Speaker: Who sold that?
Speaker: I mean, who sold that?
Speaker: Because there are so many developers on the market available, so you can pick
Speaker: One, yeah, for whatever technology you would like to.
Speaker: We even have buckets for Java developers, right?
Speaker: So it's like there's a market service for you in this company.
Speaker: Oh, yeah.
Speaker: So looking at this from, you know, like perspective, we can only laugh.
Speaker: We've learned a lot about this and we need to admit
Speaker: I guess me and you have thought that we were hype driven in those situations.
Speaker: But I think it's really very hard to avoid this, especially that if you don't have prior knowledge, you can't really assess
Speaker: that conference talk, whether that's bull crap or not.
Speaker: So it all comes with experience, really.
Speaker: OK, so since we are slowly getting to the end of the show, I think we can quickly form a general recommendation.
Speaker: So Artur, would you be so kind and give yours and then I can provide mine?
Speaker: I think I already told it, but I will try to conclude it in a few words.
Speaker: So go with microservices when you have problems with scale.
Speaker: When you're beginning your project, focus on the most valuable product you can deliver from the user perspective.
Speaker: So think of spending your time on things that will actually bring your clients, bring you money,
Speaker: bring you this economical fluidity.
Speaker: Then you can think about refining your product.
Speaker: So if you really want to be smart before you start, just when implementing the monolith, just follow some rules that applies to microservices, but on the level of modules in the monolith architecture.
Speaker: So when you create the feature,
Speaker: create modules that would be otherwise the microservices, be strict when it comes to reusing the internal parts of the other module, always try to provide entry points to your module and use only them.
Speaker: This will give you a good starting point for future improvements when the scale will be rising.
Speaker: Yeah, and tracking this on the module level, like employing things like clean architecture and segregating the interface.
Speaker: So that's only one service dealing with the module, like being at the front end of the module for the other modules.
Speaker: Then you can simply, simply, that's the trap.
Speaker: You can simply swap
Speaker: the module with the microservice, changing everything to asynchronous and that's it.
Speaker: And that's on the consultant talk, right?
Speaker: So you can simply switch it.
Speaker: No, it's not simple, you bastard.
Speaker: It's always complex.
Speaker: So it's not going to be easier.
Speaker: Yeah.
Speaker: So at least when it comes to those interfaces.
Speaker: Okay.
Speaker: So I totally agree, Artur, with this.
Speaker: So starting with monoliths, preferably modular monoliths, when you can
Speaker: apply the design thinking about how to slice things.
Speaker: You can test early if your structure, if how you've reflected the business is working fine.
Speaker: You don't sacrifice the speed of delivery.
Speaker: You actually gain it because you're doing this inside of a monolith.
Speaker: And you scale when you know which part is really having the issue.
Speaker: And essentially, the path of
Speaker: being a module in a monolith and becoming a service, I think this is the natural and the only way how microservices should emerge from an existing code base.
Speaker: But also that's the way how to get things back if you integrate services.
Speaker: There's also one recommendation which I would like to add on top, especially in the startup context.
Speaker: And this is related to bounded contexts.
Speaker: I had the pleasure to be on a very nice training when it comes to event storming and domain-driven design.
Speaker: And the lecturer there, and I'm going to greet Sลawik Sobรณtka here,
Speaker: where he mentioned a very important note about where you when you really shouldn't go to a certain level of event storming and then by consequence going into ddd this is when you're working with a customer client or you know like a company who doesn't know their business and if you're in a startup
Speaker: The startups, typically they're trying to find their business model.
Speaker: They're trying to identify their customer.
Speaker: They're trying to identify the problem.
Speaker: And they have a theory that they have a solution, but they don't know whether the problem exists, whether that's the solution for the problem.
Speaker: So there's a lot of evaluation there.
Speaker: And then bringing, you know, like heavy weapons in such a
Speaker: places like bounded context, DDD, doing all of that engineering, building this all up.
Speaker: That's a big of a hassle.
Speaker: And then it might be the case that in two weeks, the hypothesis about the customers turns out wrong, and then you need to throw everything away, or at least the majority of it, and then microservices won't help.
Speaker: So before you apply the design thinking, before you employ things like event storming, like DDD, like microservices, make sure that the people
Speaker: and the company which you're working with knows their business and knows the problem.
Speaker: Because if you're going to build up a big technical solution, invest all of that effort in making this happen, and then it turns out that the business goes the other way, you just created so-called technical depth.
Speaker: So the technical depth defined as
Speaker: the amount of work which you need to do in order to get back in line with the business.
Speaker: Right?
Speaker: So your architecture should always be flexible to follow the shaky business.
Speaker: And if you over engineer and over invest, that's going to be a big, big problem.
Speaker: So I guess that's going to be all for today.
Speaker: So once again, this is the point where we are finishing the episode.
Speaker: It took us a little bit longer than we initially expected, but I really enjoyed it and I really enjoy ranting on microservices specifically.
Speaker: So yes.
Speaker: Yes, and we are willing to hear your examples of misuse of microservice concept.
Speaker: Share with us using the comments or send us a message.
Speaker: We will be very grateful.
Speaker: Maybe we can even talk about your examples and comment it.
Speaker: That would be very helpful.
Speaker: nice idea.
Speaker: I even think that Grazie, and if you disagree with us on this episode, you're going to get invited as a guest to the show so that we can discuss this on air.
Speaker: But as for now, guys, thank you very much for staying with us.
Speaker: Please make sure that you follow the show on your podcasting platform and one of our social media pages.
Speaker: Have a nice rest of the day.
Speaker: See you.
Speaker: It was a pleasure.
Speaker: Bye bye.
Speaker: That's all for today.
Speaker: Thanks for spending your time with us.
Speaker: Visit our page on Facebook and Twitter, leave us a comment under the episode, subscribe to the updates and share it with others.
Speaker: We would like to hear your feedback so that we can prepare more interesting content for you to enjoy.
Speaker: Hear you next time.
