Transcript
Speaker: Hello everyone and welcome to another episode of the Artistry of Code podcast.
Speaker: I'm Mark Urbanovic and I'm with Artur Wolny and Grzegorz Godlewski.
Speaker: Today we are going to touch the hot topic of monorepos.
Speaker: But before we start, let's make sure you follow us on the streaming platform you are listening to us like Apple Podcasts, Spotify.
Speaker: or subscribe us on YouTube if you are watching the video of this episode.
Speaker: You can also follow us on the social media platforms like Facebook, Twitter or LinkedIn.
Speaker: So having that said, let's start.
Speaker: Welcome to the Artistry of Code podcast where we, together with our guests, are seeking ways of doing things right and having fun with it.
Speaker: Software, architecture, soft skills, teamwork and other crazy buzzwords.
Speaker: Are you ready?
Speaker: Let's bring more value to the business.
Speaker: So today we want to talk a bit about the monorepos and share with you our thoughts, some experience we have and share our point of views on this subject.
Speaker: As most of the stuff in the programming world, everything is more or less opinionated.
Speaker: I would like you guys to share your opinions and initial thoughts about monorepos.
Speaker: What do you think?
Speaker: So when it comes to me, I'm personally not a fan of monorepos.
Speaker: I mean, all my professional life I've used to work with
Speaker: multi repo setups recently i had the opportunity to work on two projects which were using mono repos so i i can see the benefits coming from it but for me essentially that's a like a complexity explorer to some extent at least when i'm used to you know like cloud native uh way of doing things where you rather had one app one
Speaker: repo and then one deployment or eventually multiple deployments coming from the 12 factor application mode.
Speaker: So it's for me understandable why people are using Monorepos, but I'm not the fan really.
Speaker: Me either.
Speaker: I also can answer that it depends.
Speaker: If everything goes well with monorepos, then I kind of like it.
Speaker: But usually from time to time, I encounter some problem that is so hard to tackle that I remind myself that I'm not a big fan of monorepos anymore.
Speaker: So yeah, I also, if I am about to pick one solution, I would...
Speaker: usually prefer to go with single repos, with single CIs attached instead of monorepo setup.
Speaker: When it comes to me, it's really hard to... And I wouldn't say, basically, I would pick any of the sites.
Speaker: I really somewhere in the middle.
Speaker: In the past, I wouldn't go with monorepo at all because everything like tooling was far behind the monorepos and wasn't adjusted.
Speaker: So everything had to be hacked in one way or another.
Speaker: But these days, I'm somewhere in the middle and I would go with it depends.
Speaker: Yeah, so this is the bit where we try to be a little bit more subjective.
Speaker: Now, if you know our initial thoughts, you probably see that we are a little bit polarized towards the discussion.
Speaker: But then again, we will strive to be as objective as possible in order to discuss the matter.
Speaker: So we wanted to take it from the angle of finding what are the benefits and the trade offs.
Speaker: of going with a monorepo specifically.
Speaker: Right.
Speaker: So guys, what kind of benefits or tradeoffs and or tradeoffs do you see?
Speaker: Despite my other negative attitude to it, I can see a big benefit.
Speaker: And first I see in the CI.
Speaker: So even though it is more complicated, you can do it once and maintain like a one CI process for
Speaker: Actually, all the projects that you keep in the monorepo, second big benefit that I see is that when you have code
Speaker: that is maybe not tangled but much dependent on each other and you need to update something it's much easier to work with on the daily basis with monorepo because then when you rebuild those projects it's the the results are visible immediately yeah you don't need to do anything
Speaker: extra like build, publish packages, publish alpha packages just to test how it will work eventually.
Speaker: You just have it given on the plate.
Speaker: I really would, in terms of like alpha packages and all this crazy stuff, when you want to test or preview something with the multi repo setup,
Speaker: And this is one case where the monorepo really shine to me.
Speaker: It's super easy with monorepo to set up, let's say, preview environments, for example, because basically you have a commit and you are checking out the repo at this commit.
Speaker: From the PR end, you are good to go.
Speaker: You can build all the stuff from this commit and everything will be really up to date as you want.
Speaker: So whatever adjustment you've made, it's there and it's super easy.
Speaker: There are approaches to this topic in the tooling.
Speaker: For example, if I recall correctly, Quaveri,
Speaker: doing that in the way that with multi repo setup, that with the preview environment, they will pick the branch you have on this PR and they will look for the same branch on the other repos or if the same branch does not exist in the other repos,
Speaker: as they will pick main or master, whatever.
Speaker: So this can be tackled, but it's obviously more complex, in my opinion.
Speaker: And especially with the CI and this previous stuff all around this topic, to me, it's super good benefit of the monorepo that you can do all this stuff just with the bash script, for example, like a bunch of if statements, for example,
Speaker: or using some diff, etc., compared to trying to communicate between repos and different CI pipelines or doing the multi-repo CI pipeline.
Speaker: This can be easily super complex and really becoming rocket science.
Speaker: Yeah, so this is partially about the trade offs.
Speaker: Like, if I were to point some benefits, I definitely agree for you know, like the code reuse, it makes it super easy to some extent to import types from
Speaker: If you're in the Node.js space with TypeScript, for instance, it's making it super easy to import types from other places.
Speaker: Well, super easy.
Speaker: Okay, that could be like an overstatement, but it's generally easier than in a multi-repo scenario to import types from the other project where you have some dependencies.
Speaker: So what is really nice is that when you have multiple teams working on different projects which are residing in the same monorepo,
Speaker: most likely during the CI pipeline, your changes, if they are going to break the other team's project, you will know about that, right?
Speaker: So this is something, and that assumes you're having like a monogamous language-wise setup so that the tooling is reused.
Speaker: Because if you're going to go into monorepo microservices with multiple languages, this is only...
Speaker: getting more and more dirty, so to speak.
Speaker: If you're going to have a Python dependency or a C Sharp microservice or things like that, and your Node.js app, that's just going to grow in terms of complexity.
Speaker: So there's some spacing here.
Speaker: And I would like to mention only a couple of trade-offs, which I also have on my mind.
Speaker: And first of all, these are...
Speaker: code specific so i'm coming from the world where of extreme programming where there was the advice that you need to have a build being done under 10 minutes i can imagine it very quickly coming or becoming a fact that those mono repo builds if you have multiple projects they're going to take
Speaker: way much more than that at some point.
Speaker: This is something which makes me to be a little bit against.
Speaker: So if you like to have fast CI, fast feedback, we talked about this on the show a number of times, to have a very short feedback loop time wise, that might not be the best idea.
Speaker: The other thing which I see from organizational perspective
Speaker: is that if you have teams in your company and they are developing different pieces of that project inside a Mono repo, it's generally okay if they're going to see one another's changes, that things are happening in the company, that's fine.
Speaker: But the problem starts if you try to hire an external company, like an offshore company and an offshore team, and you need their help for a number of months in order just to make something happen.
Speaker: And now you're faced with the question, should they have the access to the full monorepo?
Speaker: Yes or no.
Speaker: What about the business intelligence?
Speaker: So there are different drivers for the decision of monorepo or not.
Speaker: which are relevant in certain contexts.
Speaker: So if you're a product company that doesn't hire too much and you cannot afford exposing business intelligence, probably that's not the best setup.
Speaker: But if you don't really care, you can do a lot of NDAs really.
Speaker: So you can say, we did the paperwork.
Speaker: So the external team is not allowed to look in that directory.
Speaker: They won't.
Speaker: Certainly they won't because there's an NDA for this one.
Speaker: So there are some aspects which we don't consider on the code level, which are also relevant in that context.
Speaker: In regards to long CI time, yeah, it's a challenge to me or something which has to be addressed wisely because basically with the monorepo, you have to find a way to run what is really required for the given committee.
Speaker: So in old time, when I was working with the monorepo, in much simpler way than we have
Speaker: We are doing that in the community these days.
Speaker: We were using just, for example, the GitLab only on change or whatever it was called.
Speaker: So if the CI picked up the changes in the folder, it was running some tasks and it was working better or worse, but in general, it was quite okay.
Speaker: These days we have the tools like NX, for example, or TurboRepo recently.
Speaker: And the tools are helping us to pick up the tasks which should run.
Speaker: But it's the same risk as with, for example, using Jest functionality of running tests only changed, for example, when it tries to compare the current state of the code with the main branch.
Speaker: and run only affected tests because the tool cannot pick up the changes because of some quirks or some stuff can be cached in the wrong way.
Speaker: So you either go with running too much or which is, I would say, worse.
Speaker: You end up with running not everything which should run on the given change.
Speaker: Yeah, that's true.
Speaker: I consider switching to Monorepo as some kind of moving the complexity from the organizational level, from the personal level of developers to technical level.
Speaker: So the complexity pretty much stays the same, but there's other actor who is doing the job.
Speaker: And the problem that I see with Monorepo, so the challenge is
Speaker: Actually, covering all the edge cases.
Speaker: So you Marek told just a few minutes ago that tooling is picking up fast and it's much better than it was a few years ago.
Speaker: But in my opinion, it's still not covering all the edge cases.
Speaker: And I stumbled upon a few of them and it was like a very...
Speaker: painful process to get rid of those problems.
Speaker: And there's also something that is actually addressed, but
Speaker: in my experience, hasn't been implemented correctly.
Speaker: And I see like Monorepo as a factor that increases the risk to not make it correctly.
Speaker: And this is versioning, versioning of packages inside the Monorepo.
Speaker: In general, it's much easier to follow the Semver rules when you have separate repositories.
Speaker: It's easier to remember about it.
Speaker: Yeah, so I consider it much easier to break the versioning in Monorepo and I've seen it multiple times already.
Speaker: And I can totally sustain that.
Speaker: So I just recently was facing the very same issue where we have a Monorepo setup, which is based on NPM workspaces.
Speaker: There's even a bug in NPM, which is related to the NPM version bumping functionality.
Speaker: Because then if you do a bump, including the workspaces themselves, you know that behavior that you do like the NPM version patch.
Speaker: So it's going to bump up the patch version.
Speaker: And then it's going to create a git commit for it and also automatically tag it.
Speaker: Right.
Speaker: And now the bug is that if you do it, it's going to bump the versions on the main package JSON and the ones in the workspaces.
Speaker: But it's not going to commit the change in the workspaces.
Speaker: So it only commits the one at the top and the ones in the packages remain uncommitted.
Speaker: And the reason is that or the conversation and the ticket ended up in a way that it's not easily solvable because when you do NPM versioning, you version the whole monorepo really.
Speaker: The NPM version patch doesn't know how to version your packages and if any of them should be bumped at all.
Speaker: So this isn't really a pending issue in the community.
Speaker: Yeah, but there's also an issue when you change one package, which is a dependency of another one.
Speaker: So if in the one that you change, for example, you bump up the minor version, but for the dependency, that change might be only a patch upgrade.
Speaker: So now you need to manage two packages in one commit, one change set,
Speaker: You need to cover two different levels of upgrades for those packages.
Speaker: And of course, there is tooling, but like I said, tooling needs to pick up to all the edge cases.
Speaker: And I consider people to be better at it, at least for now.
Speaker: But maybe we have AI, maybe AI enhanced.
Speaker: Monorepos will change the world, who knows?
Speaker: Yeah, I think that's a good niche to use AI to support Mono Repos in a better way.
Speaker: We may work on this project maybe.
Speaker: But in regards to that cloud tooling picking up, I was thinking more about stuff like platforms for the deployments or like one of our beloved
Speaker: Sonar Cloud, for example, when in the past you couldn't really attach the monorepo project there because it was basically not working because the scripts couldn't pick up the projects inside.
Speaker: While these days often you can even select which part of the monorepo should be analyzed or something like that.
Speaker: So this is the good part.
Speaker: In regards to the versioning, this is the question if you really, in the given case, if you really need to version the packages inside there.
Speaker: Because if, for example, at Flick, we have mostly deploying everything together because we don't really need to, at least most of the stuff here.
Speaker: So we don't need like a super granular version.
Speaker: So for example, in our case, we don't really see the issue with that.
Speaker: So it all depends on the use case.
Speaker: If you have like clients or whoever using the packages like granular from this model repo, yeah, that's definitely a challenge.
Speaker: Yeah, so in my case, it's also the case that I'm rather deploying all the things together.
Speaker: So it's okay for me to version all the packages the same.
Speaker: They're not real packages, so to speak, because they are just versions of the code.
Speaker: I need a tag, which I'm then using to issue a deployment.
Speaker: So it's fine for me.
Speaker: But then again, when we're talking about where things are missing, it's like you said, the tooling is picking it up.
Speaker: But I think what's really missing on the market when it comes to monorepos is like a monorepo-centric solutions because all the things like Turborepo and these kinds of things, they are there to help you to deal with complexity in form of like a patch, really.
Speaker: It's not like a healing process.
Speaker: solution, it's really a patching solution.
Speaker: So we see that there's that niche that people are using monorepos and they're having some problems.
Speaker: Let's try to provide something or we solve this for ourselves and then let's make a product out of it because we already mastered this particular bit.
Speaker: But there are other tools not related to versioning which kind of fail to keep up.
Speaker: And this is something which we talked about before the show.
Speaker: It's like our IDEs working on the projects which are in a mono repo.
Speaker: are actually killing, those projects are killing the IDE.
Speaker: So myself, I'm having like 32 gigs of RAM, 12 cores system.
Speaker: And it's difficult for WebStorm to pull off a project which is having like two services inside.
Speaker: That's an NPM based project.
Speaker: So I assume most of the things are coming from the libraries and the amount of libraries which are there.
Speaker: So in order to be able to work with such a repo, in my IDE, I had to enable the power save mode, which is disabling backgrounds, checks, some inspections, these kind of things, which for me is a loss.
Speaker: Essentially, it's a loss because I could have feedback from the IDE sooner, but due to the fact that the IDE is not able to keep up, I had to take a pass on this one.
Speaker: I totally support this.
Speaker: And in the end, it doesn't matter if you really use WebStorm or you use VS Code.
Speaker: Every IDE has challenges with the performance in the monorepos.
Speaker: But I would say that in my opinion and in my observations, what I found is that in Node.js world, at least, the core of the issue is not in the IDEs, but in the JS tooling itself.
Speaker: So the most challenging stuff, which is slowing down the
Speaker: The IDEs are, in my observations, our TypeScript compiler, the TypeScript language service, and the SLint, basically.
Speaker: So I often hit that the IDE warning with couldn't get the response from SLint service in 20 seconds, which is ridiculous.
Speaker: basically, but it's somehow connected because one of my colleagues at Flick did a good investigation on this topic.
Speaker: And it's caused often by the fact that as with multiple projects, the more projects, the TypeScript projects you have in the code base,
Speaker: the slower Eslint goes, basically, because it tries to, like, it can't, it's related to what Grzegorz said, that it's like we are trying to patch the stuff instead of, like, maybe reinventing it from the, from the,
Speaker: bottom up.
Speaker: So the S-Link is trying to compile every project separately, going through all the type stuff again and again and again.
Speaker: So if you have 10 projects, including one library, it's trying to compile the same library 10 times and it bubbles up with more dependencies.
Speaker: And this is what is slowing down really.
Speaker: So I think this is a really good statement that we are still in the phase of patching instead of getting something which is really monorepo-centric.
Speaker: Yeah, but tooling is not only the CACD and EDE.
Speaker: I, for example, had quite a big issue with the SNCC.
Speaker: That was about the log file being out of sync.
Speaker: But the problem was deeper in the implementation.
Speaker: In general, I think that, you know, when Monorepo started to become some kind of buzzword and it's been used in the newer projects run by, I believe, younger developers from the new generation, I feel like the old tools that we all know and like and managed to get used to,
Speaker: now need to catch up because the approach has changed.
Speaker: At least I believe that the NPM a few years ago was designed not to go into the...
Speaker: into the monorepos, rather one project per repository, and then one package out of this project.
Speaker: And lots of tools has been built in JavaScript and in TypeScript in the way that it actually needs to understand the internals of node execution,
Speaker: internals of how JS and TypeScript is actually handled.
Speaker: And now it might happen, and I suspect it happened in that case, that some tools just need to still observe all the changes in the tooling like NX, like Lerna,
Speaker: and other stuff just to make sure that the validation that they do is correct.
Speaker: Because in my case, I get like an alert that actually blocked me from pushing my pull request, which was fine, to production.
Speaker: And when you don't have access to overcome this warning, then actually you're stopped.
Speaker: So the problem was big and I couldn't solve it on my way because when I used the right tooling that actually set up the workspaces in the monorepo, everything seems to be fine.
Speaker: No changes was needed to make it work.
Speaker: Still hasn't been accepted.
Speaker: So there's lots of tools that we need to
Speaker: evaluate if you consider switching to Monorepo, if they will still work.
Speaker: And this is, I believe, lots of work and it's like with the Spareto rule.
Speaker: It's easy to make it work in the positive paths, but it's super hard.
Speaker: It takes 80% of time to solve those 20% of the problems that you are seeing rarely.
Speaker: Yeah.
Speaker: So when you described the way we approach this, this made me think that we might sound like multi-repo boomers to the younger generation, as you call it, which I would really love to refrain from, if possible, because we are sending the message to you guys there as well.
Speaker: Single repo die, though.
Speaker: Yeah, yeah, yeah.
Speaker: But then again, just to quickly go back to one point which you guys made about the complexity is that I think we need to understand that the complexity of software only grows.
Speaker: And if we are going to push it out from our code base and just think that things are going to be easier if we have monorepo, it's like you, Arthur, said, somebody else is going to have to deal with the complexity coming out of it.
Speaker: So,
Speaker: I kind of urge everybody to consider the macro.
Speaker: So when you're thinking about making the decision, and that's really an architectural decision.
Speaker: And due to that fact, it's a game of trade-offs to pay attention to the trade-offs.
Speaker: And some of those effects of those trade-offs may not be on your desk, but that might be on the DevOps side, that might be on the infrastructure side, that might be on the deployment side.
Speaker: So understanding the macro,
Speaker: Here is a key thing, really, to make an educated decision about this.
Speaker: Also considering the organizational aspects, which I've mentioned.
Speaker: Whether we're going to open up this to somebody, will all the teams be okay with working on the same repo?
Speaker: Consider those drivers, because I find this as the most important thing to consider before going into that particular direction or the other one.
Speaker: Yeah, that's right.
Speaker: I also see that it's worth to try new things and maybe if you consider switching to Monorepo, just carefully pick the use case and start with something small.
Speaker: Verify if it works for you.
Speaker: Solve more problems that brings.
Speaker: So this is the thing and don't be like...
Speaker: Don't let yourself be focused too much on how fun it is to do a single rep or be a dinosaur that does not want to change because just because it's not like it was used to.
Speaker: So, yeah, that's my final thoughts on this matter.
Speaker: Yeah.
Speaker: So it's like Kent Beck once said, if I hate it, I need to try it.
Speaker: So that's the approach.
Speaker: As developers, I believe we enjoy trying new things, but we often tend to put ourselves in really shitty situations often because we don't really think enough before implementing the given tool.
Speaker: in the project but yeah we all have to deal with that fighting with the hype and going purely with the hype yeah so i don't think we can give any like solid recommendations on this topic but i think we share enough experience and some some point of views on on this topic so as with
Speaker: almost everything in the software development.
Speaker: It depends if you should go with or not with the monorepo.
Speaker: Always think about all the pros and cons in your given use case, because it can be completely different from our cases.
Speaker: Thank you guys for this episode.
Speaker: We are really curious about your thoughts and your experience with the monorepo.
Speaker: So please share
Speaker: with us on comments under the YouTube episode or the social platforms posts.
Speaker: And we want to wish you lovely springtime and hear you next time.
Speaker: Bye bye.
Speaker: Bye bye.
Speaker: Take care.
Speaker: Bye bye.
Speaker: That's all for today.
Speaker: Thanks for spending your time with us.
Speaker: Visit our page on Facebook and Twitter, leave us a comment under the episode, subscribe to the updates and share it with others.
Speaker: We would like to hear your feedback so that we can prepare more interesting content for you to enjoy.
Speaker: Hear you next time!
