Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Nix, The Build-Everything Language (with Julian Arni)
3.7k Plays27 days ago
Ever since we invented makefiles, the programming world has been wrestling with the problem of building software stacks reliably. This week we’re going to look at one of the most ambitious solutions available - Nix. Nix tries to do everything from invoking your compiler to installing your language, and even providing your operating system. But how does it work in theory, and how well does it work in practice?
Joining me to discuss is Julian Arni, a Nix-enthusiast and creator of a build/test/deploy service called Garnix.
Nix has been one of my go-to tools for years - I hope it’ll find its way into your stack.
–
Nix Overview: https://nixos.org/explore/
Nix Tutorial: https://nix.dev/tutorials/first-steps/
Nix Flakes: https://nixos.wiki/wiki/Flakes
The Nix Package List: https://search.nixos.org/packages
Garnix.IO: https://garnix.io/
Julian’s NixCon Talk, Call by Hash: https://www.youtube.com/watch?v=fU9ogB9hZZA
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Recommended
Transcript
The Problem with Inconsistent Development Environments
00:00:00
Speaker
It works on my machine. Have you ever heard those words? It works on my computer over here, but not on your computer over there. And the reason will turn out to be some subtle difference in the way we've got our machine set up.
00:00:14
Speaker
And I don't know what it is, but apparently it's important. So we have to dig down and find out. And eventually this is going to end with someone updating the readme to say, you must have OpenSSL version 3.2 installed first.
00:00:31
Speaker
And no one actually ends up reading that readme. So the problem keeps coming up. But at least you can say it's there in the docs. To be fair, that's probably the worst case.
Docker's Approach to Environment Consistency
00:00:40
Speaker
Things have got better since those dark days because generally now we solve the different environment problem by shipping the entire operating system to everyone that needs to use it as defined by a Docker file.
00:00:56
Speaker
Docker has become the tool of choice for reproducible environments and it's been really great. In practice, it has worked out terrifically well. I personally think it works a bit better in production than in development.
Introducing Nix: A New Way to Ensure Consistency
00:01:08
Speaker
For production, it's generally quite reasonable to ship a few different virtual machines. It's never sat entirely comfortably with me in dev because if I need access to three or four quite different programs, I shouldn't really have to run three or four different virtual operating systems in Docker Compose. That always seemed weird to me.
00:01:29
Speaker
And generally, i don't actually, there's a different approach I use that isn't Docker, but that solves the reproducible environment problem and ends up feeling a lot more like you've just got everything installed locally.
00:01:43
Speaker
So this week, I'm joined by Julian Arney, and we're going to talk about Nix, which started life as a PhD thesis in how you make builds reliable and has grown into this tool that can define and build just about any programming environment you need.
00:02:01
Speaker
I use it, I use it basically every day on every project I have. And I find it's particularly good if you end up jumping between a few different projects that all need different environments.
00:02:13
Speaker
So this week, Julian's going to unpack what k Nix is, how it works, and why you might want to add it to your toolbox.
Julian Arney's Experience with Nix
00:02:20
Speaker
Or actually more specifically, why you might want Nix to become the box you hold your tools in.
00:02:27
Speaker
I'm your host Chris Jenkins, this is Developer Voices, and today's voice is Julian Arney.
00:02:45
Speaker
I'm joined today by Julian Arney. Julian, are you doing out there? I'm doing great. Thanks for asking. Very excited to be here.
Comparing Docker and Nix
00:02:53
Speaker
You're over in Basel, in Germany. Basel. Basel. It's pronounced Basel.
00:03:00
Speaker
Basel. It's in Switzerland, on the border, exactly on the border with ah France and switzerlandland and Germany. So if I go for a run, and it doesn't have to be a long run. It can be a 5k run.
00:03:12
Speaker
ah I can cross into the three different countries, right? Oh, wow. That's cool. Do you need take your passport? Yeah. Officially, yes, but I don't think anyone does and I i never have.
00:03:26
Speaker
The freedom of movement that I've lost, but let's not get into Brexit. That's a different podcast. So you're here to talk about building things, DevOps, a topic sometimes I shy away from because I found it so painful in the past.
00:03:42
Speaker
Understandably. Yeah. but But there's... ah So maybe I should tell you my background just for a change. i worked at a company that used Nix many years ago as a contractor, and they basically paid me to learn Nix because I had to, right?
00:04:00
Speaker
So I've always felt that I've come at Nick's from a strange angle. the The initial hump being paid rather than voluntary. Yeah, yeah. so yeah And mandatory. That was the other thing. I never had the here's why it's good. I just got told by Dictat I should use it.
00:04:16
Speaker
So I'm hoping you can start by filling in the why is Nick's good question. ah Exactly. So that I can convince all the other people who are not being forced and paid to do it, to take a look at it. Yeah. um I think, ah you know, the sort of space that it's operating in is the the space of sort of reproducibility, right? Making sure that works on my computer works on yours as well. Yeah.
00:04:42
Speaker
And that's a space that we as developers are very familiar with, and we have our techniques to solve it. right like I think ah Docker is is is maybe by far the most popular one. ah But ah and Nix is coming at the same problem from a very different angle.
Nix's Deterministic Build Processes
00:05:00
Speaker
um and And as a consequence, it doesn't have a lot of the problems that Docker has, which is actually, you know to some extent, most obviously, that it's not that reproducible. If you want to run the same Docker file twice and it's not cached, you could get very different results.
00:05:17
Speaker
But then additionally, a number of other things, it's hard to sort of abstract over them. It's hard to connect multiple Docker files together, combine them. um You end end up spending so much time um when you have Docker images so sort of poking holes at the isolation so that they can talk to one another.
00:05:35
Speaker
It's very different than the way in which you program locally. right like Usually, you know if you think about the Unix philosophy, you have a pipe operator between two different programs and that's how you compose them.
00:05:46
Speaker
Whereas if you have Docker images, you suddenly have to have HTTP ports and make them into web applications or whatever it whatever mechanism it is. It's very different than than how you traditionally locally think of programs.
00:05:58
Speaker
And so there are all these costs that are associated with with the Docker model and that aren't there in Nix. um And additionally, there are lots of things that it enables that you wouldn't even think are possible. right So I think um some of the things that are quite phenomenal is are just the sort of the sort of incredibly sophisticated build caching that it has.
00:06:21
Speaker
um and um And, you know, with with once you sort of start using NixOS, which hopefully we'll have time to get into. it Definitely.
00:06:33
Speaker
It gets even more exciting, I think. Like you can sort of see rollbacks sort of get your machine to any any any state very precisely um and know exactly what state that is, be able to diff machines and in in incredible detail um with your usual sort of version control tools.
00:06:54
Speaker
Yeah, because definitely Dockerfiles sometimes feel like a bit of a black hole. Yeah, they you know in some sense, it's a <unk> a very nice interface. The Dockerfile, it's a very familiar interface on the one hand. right like You write essentially a bash script. It's a little bit different than a bash script, but you write a bash script, and and then you know it's like it's cached and you know kind of isolated.
00:07:19
Speaker
um And that that sort of initial experience is very, very nice, but then But then as you start progressing beyond that first you know those first five minutes, it it becomes cumbersome in a variety of different ways.
00:07:34
Speaker
Yeah, I think one of my least favorite is that if you add a command to a Docker file, everything's groovy. If you want to go back and change an earlier one, suddenly you're in a real build time problem, right?
00:07:46
Speaker
Yeah, that linearity is very strange, right? Like in some sense, the the caching is is linear, right? And so ah ah you have to think very carefully about the order in which you do things. And as you said, if you change something early on, it might have downstream consequences because, you know, maybe you're downloading something from the internet and the cached version that you had so far changed.
00:08:08
Speaker
and now ah And now everything else is going to be different as well, right? So you're kind of in this... yeah, yeah. So yeah I think we should build up the picture of how this all works, and then we can really talk about it.
00:08:22
Speaker
So what's it doing that Docker isn't? What's the what's the mental model for Nix? Yeah. ah Docker, in some sense, is you know taking a snapshot of the file system at a moment in time. right You just take a picture of whatever it is whatever state it is that you got got into after running a few commands, and you save that.
00:08:43
Speaker
um And there's a story about layered file system and whatnot, but that's that's essentially the the approach, right? Yeah, snapshot the whole OS, call that a base, and then keep adding snapshot changes on yeah top.
00:08:56
Speaker
Yes, exactly, yeah. um Nix is much closer to a different technique that is also very familiar, ah which is the package lock files. right and so we we What I mean this but by this is you know you have language-specific tooling such as cargo or NPM ah that nowadays are Python-based.
00:09:20
Speaker
pip or different various other Python tools um that take you know your project and maybe some some version bounds of dependencies that you specified and talk to the internet, find out what versions are available that match match what you requested, ah pick those versions, and then output that into a lock file, what versions it's picked.
00:09:46
Speaker
now Yeah, to pin the specific binary or the specific source code. Exactly. Exactly. So you can think of this as what's happening is you after you've pinned this, um the process of dependency resolution with that lock file is deterministic, meaning it will always be the same because you've pinned it you very,
Nix's Hashing System for Reproducible Builds
00:10:05
Speaker
very precisely. Not just a version number because sometimes ah you can change the meaning of a version number, ah but also the hash very often, right like it's the specific source hash. of
00:10:17
Speaker
So then after that, if anyone else tries installing it, they'll get exactly the same version. Yeah, because it's usually pi it's often pinned off git, which means it's cryptographically secure all the way back through the source code. So you get byte for byte exactly the same.
00:10:31
Speaker
Exactly the same. Yeah. And so this is kind of you know that that lock file. um is if you take that lock file and extend it in two different ways, yeah you get something that's that's essentially the next idea. right um The first one is that you extend it to everything. right like It's not just for a specific ah language language dependencies. right like It's not just for your Go dependencies or just for your NPM dependencies.
00:11:00
Speaker
It's for everything. It's for all your different language dependencies, all parts of your projects, your system libraries, um your various, you know, like maybe images or whatever you're downloading from the internet, everything that, you know, is speak is outside of this isolated sandbox that you need to fetch ahead of time, including, for instance, you know, the the UTC time or or all of those things, everything that could go matter to your build, you're pinning.
00:11:30
Speaker
um Yeah, because if like you and I build the same Rust project because of the lock file, we get exactly the same can cargo dependencies. We might have a different version of lib OpenSSL.
00:11:43
Speaker
Exactly, or zlib, or whatever else it is, right? Or yeah or a runtime ah binary that you use, right, that you're expecting to be there that's not a Rust ah project, right? Yeah, yeah. Maybe you use core utils or something like that, right?
00:11:58
Speaker
And... and Yeah, and often with languages, you even sort of you know you pin all the versions down, but you don't pin exactly but compiler version or something like that. right like Somehow, the ah domain in which these log files operate is just the library versions.
00:12:17
Speaker
The call someone says, it doesn't build on my machine. And you say, well, which OS are you on? Which version of Rust have you got? Yeah, and then you start wading through those reads. even And that what you mentioned is is a really good reminder. right Like even another thing that's pinned down and is the OS version, or whether your architecture is 64 bits or 32 bits, or whether it's an Intel or ARM micro Linux, right?
00:12:41
Speaker
All of those things. are pinned down um so that you know like really, conceptually, if you build it twice, even without cat the even without caching, you'll get exactly the same thing.
00:12:55
Speaker
Now, I say conceptually because there are small exceptions that maybe we can get into at some point. and but that's But that's number one. right you You start with lock files, but you're going to pin down everything. right um
00:13:10
Speaker
And then the other thing that I think is ah is a very, very big deal is once you've done this, you can use um you can use when you build your project. right So you pin down not just the Rust dependencies, not just the Go dependencies. Maybe you have a multi-language project that involves various different languages.
00:13:32
Speaker
You pin all of this down, and you ended up with a binary. The cool thing here is that you can give it, usually, you know, you you you might end up with out or some some binary version like that, or my project, and that's the name of the binary.
00:13:48
Speaker
But what you could also do is name that binary, the hash of all of those inputs, your lock file plus your source code. and The lock file that includes everything else that's not your source code and your source code.
00:14:01
Speaker
Right, yeah. So you get a cryptographic signature of that specific binary. Yes, yes. And so now you have a ah very good name to refer to that binary, and you can have various different versions of the binary coexist. right So you build that, for instance, and then you check out a different branch and you build it. And now you'll have two different binaries with two different names, and they won't overwrite one another. Usually when you do a cargo build or NPM build or whatever it is, you you you will overwrite whatever binary you had previously built.
00:14:34
Speaker
Because there's a specific name there that that keeps changing. And that has, you know, if anyone was using that binary, now they' have they'll have a different version, right? Yeah.
00:14:44
Speaker
Whereas with Mix, you tend to use these names. And there's there's there's a lot of infrastructure around making it easier to refer to software without having to explicitly mention these hashes.
00:14:56
Speaker
ah But that's, you know, you're going to name everything this way. um And so various different versions of of of binaries, of artifacts can coexist without a problem, right?
00:15:08
Speaker
ah You do this for directories as well. so Yeah, yeah. When you do a system upgrade in Debian or whatever operating system it is that you have, you usually you'll overwrite the version of Python that's in user bin or wherever it is.
00:15:20
Speaker
Yeah, sometimes with immediate regret. Exactly. You need to go back and use it. Yeah. whereas Whereas here, you know my project can say, oh, I want exactly this version of Python that's known by this hash.
00:15:32
Speaker
And then it'll never never, as long as someone's using that version, it'll never get deleted. But you can upgrade your OS, and it'll just install a different version of Python alongside it with a different hash name. Yeah. You know, I've used this to great advantage specifically when I've been contracting for different clients.
00:15:49
Speaker
And you've got like one client that's on this version of the JVM and another that's on that version. And you want like maybe they have different underlying C libraries and you want both versions to coexist on your laptop so you can work for both clients without going insane. yeah becomes trivial, right?
00:16:04
Speaker
yeah Yeah, and this is this is where one of the superpowers of Nix that it's quite famous for and rightly so comes from, right? This this notion of dev shells. we have we have a little bit of this so and We have a little bit of this with them with virtual lens and things like that that allow you to sort of enter, in again, in very limited ways, enter ah specific Python environment with a particular version of Python installed and whatnot.
00:16:32
Speaker
But Nix, again, does this for everything. right So you can now create development shells that have all of your all the versions that you want installed.
00:16:44
Speaker
And what that does is it just symlinks into your path, the specific Python version. you know They have a hash, Python slash hash or whatever, dash hash. um And you just symlink that as Python into your dev environment um when you enter it and then un-symlink that or remove it from your
Nix vs Docker in Development Environments
00:17:02
Speaker
path.
00:17:02
Speaker
um But you're never deleting anything or overwriting it. You're just sort of entering an environment that exposes specific version of each of those things under the path so that then you can develop with the version of Python that you want, with a version of Cargo and Rust-C or whatever it is that you want.
00:17:22
Speaker
um and And this is one of the great joys is you can create a project um and at work or in your open source project make a dev shelf, like specify the details of a dev shelf for it in Nix.
00:17:38
Speaker
And people will come along, instead of needing to sort of read through a readme um and carefully try to emulate everything and just do a Nix develop and enter that dev shell with all the tools that they need to build and test. and Yeah, yeah. i I have thought in the past when you've got like installation instructions that are more than just build the language project, you've also got to install these dependencies first. yes yeah That's where like that should have been Nix or something very similar, right?
00:18:06
Speaker
Yeah, yeah. Also because... it's And it's not just hard to sometimes hard to follow those instructions, but it's also sometimes, i mean i think very often, hard to write them correctly. right Because you don't know.
00:18:20
Speaker
dependencies of your project are very often implicit. You don't know whether your project is, or you've forgotten whether you're using zlib or whether a particular version of Bash has to be installed versus another because you have some particular syntax that's only available in Bash 4 or up or whatever is.
00:18:38
Speaker
Yeah, yeah, this is another thing you get with Nix, which is the isolation, which gives you an environment that doesn't include the things you're used to having on your laptop. Yeah, yeah, exactly. So that you know exactly what it is that matters, right? Like it's ah being able to empty empty your environment is really important to understanding what it is that you actually need, right?
00:18:57
Speaker
Yeah, yeah. So I'm sure I've hit that with something like image magic. It's like, oh, yeah, you also need to install image magic. I hadn't put that in the instructions because I've always got it installed. Yeah, yeah, yeah, exactly. And I hadn't thought about it, yeah.
00:19:10
Speaker
Okay, so I think i think there's if you if someone has bought into the argument that, like, cargo.lock file or a package.lock file or something like that is a good idea, we could go deeper and apply that to all the surrounding stuff for the language.
00:19:29
Speaker
The question is how? How does Nix do it, and what will it cost me to start using it? Yeah. ah you know Fundamentally, ah what what what the experience of, you know I think the the fundamental sort of unit of Nix is a build, a derivation. i won't go into details about that terminology yet, but fundamentally, you know, a recipe for how to build a project is, is and then actually building it is what Nix cares about.
00:20:02
Speaker
And the way in which it does that is by isolating the build environment and sandboxing it so that you don't have um don't have ah anything that you didn't specify. right so And then there you specify usually something like a bash. And this way it's you know somewhat similar to to Docker, you specify a little script that says, this is how you will build my project.
00:20:28
Speaker
um you know Maybe it's cd into this directory and and do a cargo build or make or whatever it is. right yeah And there's a little recipe there. But it's running in ah and without network asset access, um without access to the rest of the file system that's not your repository or main directory.
00:20:47
Speaker
So it's not accidentally picking up hidden inputs that you hadn't thought about. Exactly. yeah You won't have a version of Python at all unless you say, I want a version of Python. And then you have to say what version this is. right yeah yeah um and And so then you have this recipe that you know can be run by someone else that next will again um to do exactly the same thing, isolate you know create an isolated sandbox, um and then run this recipe range, making sure that any dependencies that you specified are, in fact, available to it. right So if you said, I want this version of Python and this version of Cargo, then those things will be available, but nothing else will.
00:21:28
Speaker
yeah that's that's That's how it kind of operates. but ah you know There's also this sort of surface level language. And the same way with the went to Docker file, there's there's this semi-bash syntax. you know um and next There's also next to language, that's the equivalent. right So so you you write these little data structures that explain your build recipe. right And most obviously obviously what that contains is the script that will do the building, yeahp and then also the dependencies. right So you you specify packages.bash or whatever.
00:22:05
Speaker
um And that will add those things too to your dependencies. So one thing I've always wondered and never actually looked into, maybe you know the answer to this, but if I want to build my Rust project, I will specify one of my dependencies is Rust the language.
00:22:23
Speaker
yeah So Nix will go off to the recipe to build Rust the language and it will find that one of the dependencies is GCC, let's say. yeah and goes off for the recipe for GCC, it finds one of the dependencies to that is a specific kernel.
00:22:37
Speaker
How far back does it go? What's the... If it's turtles all the way down, what's the base turtle of this build? ah There is, you know, there's work to sort of have a minimal sort of...
00:22:47
Speaker
ah
00:22:50
Speaker
minimal base and so that you know he's usually with these things, it's a little bit like these compiler versions. right like with compilers Very often, compilers are written in the language that they compile. yeah i don't know An example is, I think, Rust, but also Haskell.
00:23:06
Speaker
And so how do you get to how do you get to produce produce the current compiler version? Well, you get an earlier one, and you use it to compile the source code. right And ah yeah know this is a little bit annoying for reproducibility and for for making sure So so that there's work to to to make the space a little bit simpler.
00:23:24
Speaker
ah But in practical terms, it won't go that far down, right? Because when you need a dependency, such as Rust, So someone in the Nix world had to ah build that Rust compiler.
00:23:42
Speaker
ah But there's a NixOS organization that um for hundred thousand nearly 100,000 different packages does the builds already. And this gets into a different thing that um is very, very nice about Nix, which is binary substitution. right so If you're asking for ah thing by this very specific name, right if you're asking for cargo with a specific hash, because everything specified in that hash already, everything that could possibly matter is in that hash, if anyone that you trust has already built that, you can sort of just ask the internet. you know
00:24:19
Speaker
You have to be careful that you trust these people, but you can ask the internet, to has anyone built this? and And if they say yes, you can just use that instead. right So you won't be building when When you build a cargo project, you won't be building all these things. You'll be downloading a binary ah that is the already built... Because Nix can calculate that it must have come from the same tree of dependencies, the exact same tree of dependencies. Exactly. In order to have the same output hash name. Yeah, yes same hash. Yeah.
00:24:52
Speaker
Yeah. And that's really cool. It's the same trick that Git uses. yeah Yeah. Hash it, build a tree of hashes.
NixOS and Declarative Server Setup
00:25:00
Speaker
Yeah, yeah, that's right.
00:25:03
Speaker
Yeah, hashes are, you know, and in some sense, the and the the deeper lesson is just that hashes are really good names for things. Yeah. Content, what's it? Content addressable file systems. We have to get that terminology in.
00:25:18
Speaker
Yeah, yeah. Yeah, yeah. Yeah. Perhaps that's the solution to one of the two hard problems in programming, right? Naming things. Naming things. name it by the hash. ah Yeah, exactly. Which, to be fair, also, you know, Docker to some extent does as well, right? But it hashes the output, to the result of what it built, ah rather than the inputs that went into it.
00:25:40
Speaker
Yes. Yeah.
00:25:43
Speaker
That makes me just want to say, I think this is true. You can confirm for me. You can also use Nix to build Docker images, but with better hashing, right? With better hashing.
00:25:55
Speaker
Like you get a Nix quality input hash for your Docker output. that's it That's right. there There is a product that's quite widely used and in Nix packages itself that's a way of building a layered Docker image that where the layers are quite minimal.
00:26:09
Speaker
um So you know it's easy to, hey if you think if you think about the way Nix operates right with these hashes, in some sense, it's solving um solving the the problem that Docker has that we mentioned before with linearity. right like With Docker, it's like if you change something earlier on, you don't know what how that's going to impact the rest of the thing. So you have to go back and start building from that from that line that you changed, essentially.
00:26:40
Speaker
But with Nix, we know that ah when you build a dependency, when you add something to the file system, it'll only ah influence. ah Nothing else will influence it. really like the that When you build a pipeline with a particular hash version, it'll be exactly that.
00:26:57
Speaker
um So you can really sort of make very, very fine-grained layers, and you can also rearrange them somewhat more safely. um And that's that That is something that you people use a lot, right?
00:27:09
Speaker
Yeah, yeah. I've definitely found the experience of refactoring and rebuilding a Nix file to be much faster yeah than that a Docker file. Yeah, yeah.
00:27:20
Speaker
and And these things start to matter once it becomes large enough that it is something you want to maintain, right? Yeah, yeah. I think that that that's true. The other thing that's a bit harder to sort of give a sense for, and I recommend people just try out,
00:27:36
Speaker
is that, you know, it has these ability to create, a you know, what I always say is, you mentioned the beginning of this podcast, you know, like, oh, like DevOps is something that I often shy away from.
00:27:49
Speaker
And I think that that's a totally comprehensible perspective because, you know, when we... live in within that programming language, there's very clear semantics, and we kind of know how everything works, and we can kind of abstract things in exactly the right way and capture them and you know really find nice ways of expressing things. And the language doesn't work against us, and there's so many different languages to choose from that you know really almost capture the language of thought itself. madam And then you have these libraries that abstract over bits of functionality and you can combine the libraries and whatnot.
00:28:23
Speaker
But then when it comes to to infrastructure, to DevOps, right like it it really is like gnarly and feels like duct tape all all over. so i can yeah
00:28:35
Speaker
Whereas Nix goes ah much further in the direction of of making ah DevOps feel like programming again. right like You can abstract over these. If you have multiple different servers talking to one another, ah not only can you describe builds, where you can describe with entire servers.
00:28:52
Speaker
And not only entire servers, but entire sets of servers. right And so you can really... really
00:29:00
Speaker
Abstrack over the idea of, for instance, having all of your infrastructure servers talk to Agrafana, Prometheus, export things to Agrafana and Prometheus, and that just becomes a flag, right? Like something that you can just enable um ah in all of your servers.
00:29:17
Speaker
Yeah, yeah, like you can start to, because the description language, the equivalent of the Docker file is a lot more like a programming language. Yes. You can start supplying arguments to your built-ins. Exactly, yeah, yeah.
00:29:31
Speaker
Supplying arguments and writing functions that, you know, abstract over... Yeah, exactly. Yeah, yeah. I'm defining a build that builds several different versions because it's effectively just a for loop or a map.
00:29:43
Speaker
Yeah, yeah. Over build files. Yeah, yeah. Yeah. So this sort of gets us into the space of, because we talked about
Infrastructure as Code with Nix
00:29:52
Speaker
Nix. the that There are three parts to Nix, possibly more. Yeah.
00:29:56
Speaker
Nix, the tool for building stuff with this dependency tree we've talked about. Nix, the language for specifying those recipes. Yeah. You've hinted at Nix OS. Do we have to buy into a whole other OS?
00:30:09
Speaker
We do not have to buy into a whole other OS. This is an entirely optional. and Entirely optional. you know people I, for instance, run Nix OS, and I think a lot of people who ah are given the chance and the opportunity to spend time learning Nix often will, because there's a lot of joy out to begin from it, but it's it's completely optional.
00:30:33
Speaker
yeah So what does it get you? Why would you make that option? Yeah. I think you know when you think about um what getting what getting a server to be to be ready to do to to do the functionality that you wanted to do, maybe run your web server or whatever it is, you know your website,
00:30:58
Speaker
um there's usually like a lot of very sort of imperative instructions, such as sudo apt get this, and sudo apt get that, and then you know move this file here, and whatnot. right and And ah and and as we generally know, that this this can be really, really dangerous as well because it's a very stateable stateful system.
00:31:23
Speaker
um if we If someone ah logs in because they notice that there's a problem, maybe with a missing version or whatever, missing the binary, maybe they do a pseudo apt-get there um while sshed in.
00:31:36
Speaker
And that's you never know that that that was part of what makes this machine tick, what what makes it go in the right way. um
00:31:47
Speaker
And that's a big problem, because if you need to migrate it somewhere else or if you need to you know redundancy or whatever, you won't ah you won't have captured that. And you'll forget about it the next time around. Next, as we've been talking about, you know like it's very much about sort of really isolating ah uh, isolating your system from all of these, stateful interactions, right?
00:32:09
Speaker
You really have a build recipe that you specify once declaratively, and that's it, right? Like this, the state that happens afterwards, the accumulates, the cruft that accumulates in your, in your, uh,
00:32:22
Speaker
your own ah computer, for instance, is is ignored during a Nix build, for instance. right like It's all all about this isolation that ah captures that makes makes the recipe capture everything there is about a build.
00:32:36
Speaker
And it's the same with the OS. So yeah you can specify an entire OS that tells you exactly which files you need ah to have installed, in which locations, which systemd services. It's a Linux-based distribution.
00:32:48
Speaker
Which systemd services are there And um and then you have all this information. right So you can really, if you want to spin up a new server, you can very easily do that um ah by just reusing this this recipe.
00:33:07
Speaker
um Yeah, it's it's part of this, I think, is again that that moving things from a readme with a list of commands to type at the command line into an actual programming language where it's repeatable, right?
00:33:23
Speaker
Yeah, yeah, absolutely. yeah If we're describing it in prose repeatedly across the whole industry, we ought to be describing it in a programming language. Yeah, yeah, absolutely. Yeah, and you know, there is a...
00:33:37
Speaker
People say, you know like if if you're doing the same action multiple times, often the technique that I heard that I like a lot is write a little bash script and that at first you know just prints out and the different sections, the steps of the readme. And you say, like have you done this? Yes or no?
00:33:53
Speaker
But then you can slowly start abstracting that. right like If there's a step that ah is you can do automatically, you can change that instruction line for instead of just echoing the instruction, you actually do it.
00:34:06
Speaker
So this is you know this is a ah this doesn't solve the problem of reproducibility, but it does automate something, this this shell shell-based system.
00:34:17
Speaker
Whereas Nix really solves the problem of reproducibility as well, right so that you don't need the readme's anymore. right You just have one command that does everything for you. Yeah, yeah. And it also takes care of some of the sequencing, right?
00:34:29
Speaker
If it doesn't matter which order you run the commands in, then you don't need to worry about that. And if it does, you can specify it as an actual sequence dependency, right? Right, right, right, yeah.
00:34:41
Speaker
And you can build all you know all these dependencies that we were talking about. right like We can build them in parallel. right Let's say that you need both a Python version and a Rust version. Usually you won't build that locally yourself because, as I mentioned, you'll be using a cache that's already built it for you.
00:34:54
Speaker
Let's say that for some reason you know you have a patch to Python and you want to build it yourself in the same for and the Rust compiler. with nix you know like Because these these are very isolated, you can safely run these in parallel.
00:35:07
Speaker
right And so this this sequentializing this this annoying sequentializing of of Docker is something that happens a a lot less. Yeah. And that specific example I've seen play play out two ways.
00:35:22
Speaker
The first is that you can build each language on different threads. yeah And the second is, like if you wanted to do that with Docker, would you get a Python-based image and add in Rust? Or would you get a Rust-based image and in install Python?
00:35:36
Speaker
yeah yeah It's probably just a question of which one's harder to inst install first. And that's not the best answer I can imagine. yeah Yeah, it's just the wrong structure in some sense, right? like A list.
00:35:50
Speaker
yeah Yeah, append only is probably not the right data structure for build tools. Yeah, yeah. Okay, so from there, I think one thing we haven't talked about is time, the effect of time on this.
Long-term Consistency with Nix
00:36:09
Speaker
Because I have found Nix very interesting when I go back to old projects.
00:36:16
Speaker
It's like i Especially when, again, I come back to the time when I was doing a lot of contracting for different companies.
00:36:26
Speaker
Yeah. And I would like build a project and I'd accumulate this cruft from old projects and things I had to install for old projects, run out of disk space, clear it up, go back to the other client, and then find I've got to reinstall all the old dependencies for the previous project.
00:36:42
Speaker
Which might not even be possible or easy, right, to find those dependencies anymore. Yeah, yeah. And then you get into like, oh, now I've got the latest version instead of the version I installed when I was working there six months ago.
00:36:55
Speaker
yeah. Yeah, it's one of those one of those nice advantages. Like once you've actually specified all this in code, either it's in the cache or you can go and grab it from a remote cache and rebuild it exactly as it was, all right?
00:37:07
Speaker
ye Yeah, yeah. Yeah, it feels like one thing people say a lot is that it's it's it's a little bit or to a lot more upfront cost. um Because, you know, let's say that you're building a project. You already have a version of of of ah Python that you already have installed if you're a Python programmer.
00:37:27
Speaker
and And it's very easy to just be like, oh, like... Python so-and-so, like put in a readme, Python, this file is all you need to do. and Because for you, it's no extra work, right? Like you already have the Python new version installed. It seems like it's no extra work.
00:37:42
Speaker
You might maybe even have the dependencies installed somehow and globally that he didn't notice. Whereas Nick's forces you to be upfront about that cost, right? Like you really have to... yeah You really have to ah think about what Python version it is that you're doing.
00:37:57
Speaker
And you have specify those things. But once you've paid that cost up front, it continues. It's a gift that keeps on giving for years in the future, as you as you as you mentioned. Because if you come back to a project from four years ago or five years ago, still works.
00:38:10
Speaker
Yes. yes you a You've got a good reason to believe that it will still be able to build that entire source tree again. yeah build Some of it may have aged out of the remote cache, but it's still possible.
00:38:23
Speaker
Yeah. And and the cache has... a So far, the policy with the cache, with the sort of NixOS, the Nix.org cache, of the sort of official cache, has been to keep things indefinitely.
00:38:36
Speaker
no That'll probably change, but but it's still... ah It's still the case now, even after a number of years. And but that's really exciting. like Not only can you build it again, but you can build it it without doing a lot of rebuilds.
Challenges with Nix: Disk Space and Compatibility
00:38:52
Speaker
Yeah, yeah. having I suppose that must play into the amount of disk space it uses. Because if you wanted ah to store the binary for it um every Docker image, that would be a different thing because that's going to not just have Python 3.7.6, but it's going to have Python 3.7.6 for every different Docker file that uses it, right?
00:39:14
Speaker
Yeah, yeah. Yeah, that's that's... So the disk space issue is a serious one, both with Nix and with Docker, and might be more serious with Nix than with Docker, right? like Oh, okay. Tell me.
00:39:26
Speaker
ah and As you mentioned, you know, like... ah With but Docker, you have ah if you have if you in in one Docker file, you have a ah Python version installed. And another Docker file, you have a different Python version installed.
00:39:41
Speaker
And they don't share ah layers ah because of different images. um or you did something earlier on that's different. then you'll have that Python version, that same exact, supposedly exactly the same Python version installed twice.
00:39:55
Speaker
That's the thing that Nix solves. If you have the same Python version that you're using in two different projects, will it will be stored in your Nix store only once. So on the one hand, there's an advantage here.
00:40:08
Speaker
But um ah because every time you do a build with Nix, and you know if you're working with Nix, this can happen a lot, Whatever you change, an environment variable here or something else there or a GCC version, all of these things will change the hash. you know Anything that could possibly affect um your project will change its name, which in turn means that it will be stored twice.
00:40:35
Speaker
um and So let's say you're building a Rust project and you have you want to add a comment in your source code. now And you built built it once with Nix and then again with Nix.
00:40:47
Speaker
ah Because the source code is now different, the hash of the resulting artifact, the resulting binary, will also be different, which means that they'll be stored in and ah twice over, with some exceptions and small optimizations here and there.
00:41:03
Speaker
But essentially, that's the idea, right? So you end up using lot of disk space. I think I can accept that. Yeah, I think I can accept that if I've changed the source code to my project, I've got a different output.
00:41:15
Speaker
But there's a there's an almost philosophical question here. If you have version, Python 3.7.6, built with this version of C, and Python with not. Nick says no.
00:41:29
Speaker
but with a different version of c are they the same python three seven six no nix say not like says no yes And you know that's the philosophically, I think that's the principled answer, right? Because in fact, it could be that there's a bug in GCC or ah or whatever a C compiler you're using. Slightly different behavior, yeah. Yeah, exactly.
00:41:53
Speaker
yeah they I think the realistic answer is, yes, they're the same until you're the poor poor muppet that's trying to debug the difference. Exactly, yeah yeah yeah. You don't want to be that guy.
00:42:07
Speaker
yeah Yeah, that's right. So this i mean this raises... One question I wanted to get in from that is, because I don't know what the current state of this is, I remember...
00:42:23
Speaker
five, six years ago with Nix there being a slight problem in that it worked brilliantly on Nix OS and Linux. It didn't work as well on Macs because the funding for build machines, because it's an open source project, right?
00:42:37
Speaker
And you need a whole farm of Macs to build the caches up. Yeah. What's the state, but now that's, I should say for the record, now that's much better and it works perfectly for me on OS X. What's the state of Nix on Windows?
00:42:52
Speaker
i think i I think there's like very... and but To be honest, I don't have a very good sense. and The truth is neither I nor anyone I know and develop tunnel and on Windows. We'll have leave that for the comment section then. Yes.
00:43:09
Speaker
there There is, I think, you know, efforts to support Windows and and there's always stubbdu what is Windows subsystem for Linux and whatnot that ah but do allow you to use Nix via WSL.
00:43:26
Speaker
um But I don't know what the experience is like, right? I think that that's really a very important question is like what does it feel like when you actually try using it? how many And I have no idea.
00:43:37
Speaker
yeah Yeah, actually that, so we'll we'll have to we'll have to find another way to get information, but that does lead into something I've experienced. Using Nix, it feels a lot more like I'm running software on my computer as opposed to I'm running a virtual machine on my computer.
00:43:53
Speaker
Yeah, yeah, that's right. and I mentioned a lot, in you know, these the sandboxing, right? And and that's... in the In the build process, you don't you do want to sandbox things and kind of isolate them from the rest rest of your computer.
00:44:08
Speaker
But once you have that binary, you can now safely use it as if it were any other binary in your computer. Whereas with Docker, usually you know you build something and then you continue use running it, you know if you have a Python version or whatever it is with certain dependencies, and it's still running inside the Docker container.
00:44:23
Speaker
right Yeah. It's not just the it's not not just getting the Python version. so Even after that, you have to continue using Docker. and so And then when you you know when you want something, when you want to communicate with that, it's always a bit more of a hassle or a lot more of a hassle. Whereas with Nix, you really can just pipe things, call them in a bash script or whatever it is. right Yeah, yeah, yeah. and You don't end up with... Nix doesn't have an equivalent of Docker Compose, right?
00:44:53
Speaker
No, no. and Maybe I should get you to explain why it doesn't need one. ah ah Fundamentally, it it really is because, you know, in some sense, ah these...
00:45:05
Speaker
um Again, you know when you're thinking about infrastructure, when you're thinking of large scale infrastructure and deploying it to multiple servers, then of course, there's a question of like how do you how you or na how do you organize, capture that infrastructure ah declaratively? right And that's one thing that people use Docker Compose for.
00:45:24
Speaker
But they also use it locally. right um And that part is necessary, Index, because you don't need to be spinning up several Docker images to run specific versions, to your and you run your back end, or on your database, or whatever it is. right You can just run them in the same ambient environment.
00:45:43
Speaker
um yeah Yeah, you're running Postgres as though it were a local binary rather than a separate virtual virtual machine that connects to your source code's virtual machine. and Right, yeah.
00:45:57
Speaker
so I mean, i've but I was sold into Nix, but I've been a convert for years and I yeah use it for all my projects. But you have gone further. you've been ah You've been sold into it to the extent that you want to build a whole service around Nix.
00:46:15
Speaker
That's right. Yes. So what's missing in the world of Nix that you felt the urge to build? I think actually quite a lot. um In some sense, it's what's missing in Nix to make the experience of Nix users ah better. and But in some sense, it's also you know like It kind of feels like when I think about Nix, right? like It's already a relatively old project. But when I think about Nix, the adoption isn't it's not the the industry standard yet, and way the way that Docker is.
00:46:46
Speaker
No. um And it feels a bit like discovering Nix feels a bit like, you know, discovering a garden ah where there hasn't been anyone you know like and ah full of apple trees. that that There hasn't been anyone who so saw this. And you know this is not like, it's like somewhere off in the corner and not the main thoroughfare. So when you go there, all the trees are really laden with fruit and you can really pluck the low hanging fruits without any effort. right There's just so many benefits that but people haven't yet ah explored and investigated to their full extent in in that sort of paradigm shift.
00:47:23
Speaker
and And that's one of the things that really excites me. like I think and an example, so two two big examples is you know like how much better a CI can be if we base it on Nix instead of anything else.
00:47:41
Speaker
How much faster, how much easier, how much more reliable. But then another is... ah you know how how much better our deployment infrastructure can be, right so that we can really and abstract away ideas such as you know like having three servers, three Postgres servers servers with redundancy and failover or whatnot, how we can get zero downtime deployments really easily, atomic upgrades so that different web servers don't need to um Worry about being back staying backwards compatible or forwards compatibility, which is a huge deal when developing.
Garnix: Simplifying Nix for CI/CD
00:48:16
Speaker
um ah All of these things, you know like once you sort of start thinking in terms of the basic ah conceptual infrastructure of Nix, all of these things become possible. and so So partly it's solving an existing problem for Nix users, namely um the situation is a little bit like Git before GitHub. right like github came along made made get much more accessible. right Previously, you had to you know you were a Linux kn but kernel developer, probably, if you're using it. And you had to sort of format patches and attach them to your mail client and then also verify the signatures and use the GPG and whatever it was. it was you know it's so It's a workflow that, for a lot of people, is actually quite enjoyable, ah but definitely not for everyone. right Whereas yeah GitHub comes along and makes this really clear that this is
00:49:10
Speaker
this you know A, that version control is a really good idea, and that Git is a good implementation of it. But then also you know makes this accessible to people who aren't developers. ah And they can use this interface and see their change directly in the UI to make changes.
00:49:27
Speaker
And I think like this is something that Nix still needs. It needs company or several companies, hopefully, ah to come along and say, hey, ah we can all of the accidental complexity we can take care of. right like if you If you want to enable CI, it's just one button. If you want to deploy you know seven different servers in two different environments, that's also you know like two or three lines of code.
00:49:54
Speaker
And that will really um potentially but potentially even abstracting away some of the experience of using Nix itself so that you know um this this becomes but so that we can benefit from the conceptual advances that Nix brings without the admittedly quite high um barrier to entry range.
00:50:15
Speaker
Give me an example of how that works. Because it's like feel i remember there being a project called NixOps, which was attempting to say, we can describe infrastructure using this same language we're using to describe builds.
00:50:30
Speaker
Yeah. But that project, I think, died for lack of support. I don't think it's still going. That's right. It's not fully dead, but it's nearly.
00:50:43
Speaker
but you Are you taking up the mantle of that core idea, which is we shouldn't have a separate language for builds and a separate language for deployment infrastructure? ah somewhat I think that that idea has already pervaded some of the ways in which ah people use Nix. Nix has this addenda, let's call it, called Flakes, which is ah sort of a standardized way oh um of organizing your project.
00:51:12
Speaker
and And that means Nix-to-Language allows you to to build things, but you know probably you probably want to separate you know the packages from the tests and make clear to to people what it is that and those are and have a standardized way of saying, I want to run the Mac test because I'm on a Mac rather than the Linux ah Intel ones.
00:51:38
Speaker
And and you know sort of specify again in a sort of like very simple key value way almost, what the dev shells of this project are, right? So, so fix is, is, is, um,
00:51:52
Speaker
has become um a somewhat controversial, but also de facto standard um for how you organize your your your projects.
00:52:03
Speaker
Yeah, and you've raced past that one point. i'm going to emphasize it. It's very nice to have, as well as like ah ah script that defines how to build it, also says, these are the tools you'll want available in the shell when you're developing.
00:52:18
Speaker
Yeah, yeah. That's a really nice angle of that. Yeah. Yeah. The dev shell, you mentioned. Sorry, carry on. yeah ah so So there, and so they yeah i think the and the majority of of new projects by far use this this sort of, ah use Flakes.
00:52:40
Speaker
And they allow you to specify also NixOS machines, which, as we mentioned, you can have entire NixOS systems built on Nix, which you can run locally in VMs or which you can deploy. And so what we do is say, OK, well, given that you probably are already writing all of these things, if you're Nix savvy, are options if you're Nix savvy,
00:53:07
Speaker
there are options if you're not savvy but If you've already written this, what we say is, okay, we'll just build all these things without you having to specify any more details. We'll just build all your packages and and and checks on CI.
00:53:19
Speaker
And if you say, i want to deploy these seven servers or these five servers, we'll also deploy those servers for you. right so So we're not coming up with a new ah idea about how to...
00:53:31
Speaker
um but how to ah this ah how to deploy and and specify ah large-scale infrastructure Nix because this has already been, there are some already accept accepted practices in that regard.
00:53:47
Speaker
We're just making it automatic, right? so it's a ah platform as a service, right? Like you push to to GitHub and we run all your tests and we deploy um automatically. So kind of the ease that you might expect from from Heroku or something like that, but with a nix an entirely Nix-based system.
Future of Infrastructure Management with Nix
00:54:05
Speaker
is the trick here that it's the trick that a Docker file is more like a bash script than anything. Right. And you, you can't,
00:54:19
Speaker
you can't really like build the syntax tree and do compile time analysis in the same way in a Docker file that you could unnix the language. yeah And what you're doing is saying, well, we can peer into this language and peer into these specifications and figure out what they're trying to do.
00:54:36
Speaker
yeah you Reuse that machine language for a different purpose, the purpose of deployment. Absolutely. and and And it's beautiful. Like I, I, I really get a lot of joy from,
00:54:49
Speaker
thinking through the various ways in which this this is such a beautiful idea and concept. right like One of the things that's really great is you can actually see, you know because because you can peer into the structure, as you say, um you can say things like, oh what what is my network topology? Which machines are talking to which machines?
00:55:07
Speaker
um Or what are the versions? you know Let's say that ah ah we know about a CVE that just came out in Python affecting ah you know this version range.
00:55:20
Speaker
You can say, OK, which of the machines have this version of Python installed? right Anywhere, anywhere at all. right um So you can really peer into like into with a remarkable detail like and into ah your entire ah ah infrastructure and really start programming with it, like ah coming up with ah yeah coming up with questions to ask or or ways of automating systems and whatnot that that really...
00:55:50
Speaker
yeah Yeah. we We talk a lot about infrastructure as code. We don't then make the leap to say code is source of information we can automate further, right? Yeah. Yeah. Yeah. There is a, yeah, yeah, absolutely.
00:56:06
Speaker
and There is this, there's this nascent idea in, in things like Pulumi, for instance, right? Like we, let's use the data structures of, of a proper programming language so that, but there are problems with, with,
00:56:21
Speaker
with using an existing language like that to do these things. right like um there isn't There isn't really clarity about about these structures and running the same script twice might result in different structures and you never know. and um so So it feels again a bit of ad hoc even though it recognizes that the importance of being able to manipulate ah your your infrastructure as a data structure.
00:56:50
Speaker
Yeah, because you want... It does sort of push... If you want to really unlock that, it pushes you away from languages that are inherently ah imperative. You want to get more towards a language that's more like just a big data structure.
00:57:05
Speaker
Right, right, yeah. Which I guess is kind of a trick that Lisp figured out back in the 60s and we keep relearning, right? Yeah, yeah, yeah. That's right, yeah. Okay, so...
00:57:17
Speaker
tell Tell me, so your service for this is called Garnix, right? Yes. And I do want to know what state it's in and what, because I'm always interested when programmers get so obsessed, they become like builders of services.
00:57:30
Speaker
yeah yeah what's that road like and is the is there much competition is there much uptake how is your life now as not just a fan but a seller of mix yeah yeah to come yeah i think uh you know we've again the way we've been thinking about this is um you know again like building a platform as a service based on fundamentally different approach based on Nix, um, has a lot of advantages, has enormous advantages.
00:58:02
Speaker
But one of the things that we also sort of, uh, took for or a conclusion that we came to is also that ah to do a pass, to do the CI CD pipeline well, you should really do the CI pipeline very well.
00:58:15
Speaker
um And the reason here is, you know, when you see when you see all these other sort of ah companies such as Render or Fly or whatever, there's a ah The technique is something like, oh, if you want to try so try us out, we'll build your project using build packs or whatever it is, um and then deploy it.
00:58:35
Speaker
But usually at some point you'll switch to GitHub Actions for running the tests and whatnot, and then you have to call in to us and to let us know... um to let us know what it is as you built, push the binaries out, and push the Docker images or whatever it is to us.
00:58:51
Speaker
But this is kind of a narrow waste, right? like You lose a lot of information. There's there's one component, GitHub Actions say, that's doing the build, and another component that's hosting. And as we mentioned, you know there's ah there's a lot of information.
00:59:03
Speaker
It's really cool to have a lot of information about your infrastructure, to know exactly what's deployed, what's talking to what, how it's structured, what versions you're using. um and And if you have that narrow waist, not only do you have does the user have to deal with two different things, but you lose a lot of information in that communication process.
00:59:23
Speaker
So we focused initially on CI, and we have like a decent number of companies now sort of using this as their main CI. um And I would consider that very stable. um the The hosting part is also something that we have more and more companies adopting.
00:59:40
Speaker
ah but it still has some rough edges and some many features missing. And then another component that we recently released is what we call Garnix modules. right And the idea here is you know the reason why nixus is ah Part of the reason why and part of the reason why nick's adoption is is is being less than stellar is because of this big, big steep learning curve. right So what we do is, we okay, here's here's a way of generating the base Nix configuration for various different types of projects. If you tell us, hey, i have a Rust project, um and it uses this version of of Cargo and Rust, and
01:00:22
Speaker
then we can generate something and create a PR in your in your code that captures exactly what it is that you need to do so that you don't have to learn Nix at first, right? So that that initial hump isn't there. henry we Yeah, there does need to be a kind of his ah here's your first recipe automatically written for you Exactly, yeah yeah. So then you can get to, you know, in three minutes or five minutes, you can get to CI and deployments working and dev shells and whatnot working ah without having learned Nix. And then if you need to change the cargo version, it's relatively intuitive what to do. And you can learn gradually
01:01:02
Speaker
um on a per need basis rather than, you know, spending so much time and effort, um, upfront before seeing any serious results. Which is hopefully, I guess you're hoping is why you won't need everyone who learns k Nix to be a paid contractor.
01:01:17
Speaker
Exactly. Yeah. Yeah. Yeah. yeah You can think of it, know this last component, current xmodules, you can kind of think of it like Ansible, I think. right like Ansible has this YAML syntax where you if you have Ansible recipes, I forget what they're called, you can expose a way of calling those arguments via YAML. So for most users, you can just learn learn how to use the YAML component of of Ansible.
01:01:47
Speaker
But for users who um are more familiar with Ruby and the Ansible internals, they can write these recipes. right and so so it's And maybe at some point you sort of migrate slowly from being one type of user to another, but but you're you' ah maybe amortizing that cost ah over the lifetime of of your interaction, your usage of Ansible or of Garnix modules.
01:02:15
Speaker
yeah Yeah. I've never quite been, i I see the logic of that. I've never quite been fond of that approach because it means that as you become more advanced a user, there's inevitably going to be that point where you have to switch languages.
01:02:29
Speaker
Yes. Yes. And presumably you're avoiding that. ah We'll still see, right I think to some extent, uh, part of, uh,
01:02:41
Speaker
Part of what we think is like we you know we have to gather a lot of a lot of experience from various different users to see... um ah It's similar actually with build packs, right? Like Heroku and I think Render as well use these use use use the same technique, which is like you don't have to learn exactly what script is being run in order to build your project and deploy your project.
01:03:04
Speaker
and you can just ah be largely ignorant of that or have a tiny bit of configuration. But if it gets more complex, you're going to have to into those build packs, right?
01:03:15
Speaker
um Yes, but is your configuration still in Nix the language? Yes, it is. So there won't come a time where I need to switch language to actually use all the features?
01:03:26
Speaker
Right, that's right, yeah yeah. And sort of having poking holes or sort of... making that transition be nice is something that involves careful design.
01:03:37
Speaker
There's no sort of ah you know let's let's say that you you know you have you want to override a particular version of a dependency. Maybe you have a field that does that easily.
01:03:48
Speaker
Or maybe this is a time where you have to learn about FetchKit, a built-in FetchKit, or something like this. like You have to decide very carefully when that happens and how it is that the user experiences that transition and how do they learn about it and whatnot. And and I think it's just...
01:04:04
Speaker
A lot like designing a library or a programming language. It's just something that the nitty-gritty details have to be considered and done with love. right yeah Yeah, and it's one of those design decisions that will eventually shape and limit the size of your user base, right?
01:04:21
Speaker
Right, yeah. are you Are you just going to appeal to Nix experts or are you going to try and take it to the next level where anyone can use it? Yeah, exactly. And we hope that the answer is latter, right? I think that's our ambition.
01:04:33
Speaker
Yeah, absolutely. So I want to go back and pick you up on three features you said of using Nix for um deployment, because I think this is worth you unpacking.
01:04:46
Speaker
um Atomic update, like atomic deployments. Yeah. Why does Nix enable that? Because that sounds like a useful thing to move everything to the new version all at once.
01:04:57
Speaker
Yeah, yeah, yeah. ah Yeah, so so atomic ah deployments are something that you can see in two ways, right? Like one one question is the zero downtime question, right? Like, and this is something that's familiar to everyone.
01:05:12
Speaker
as soon as you have, you know, something that's a little bit more than your personal website, yeah you probably worry about the amount of downtime that you have, if any, when you upgrade the version of...
01:05:26
Speaker
of your OS, of your dependencies, or of your application, right? yeah Do you stop that application and start a new version? Do you have a gradual rollout as with Kubernetes? um and ah And what we do, um i think, you know, the the What I think is exciting that we do is we we extend the idea of Nix to the cloud by using these hash-based names.
01:05:57
Speaker
We use them as ah domain names for your web servers. That's a bit of a crazy idea. ah But when you deploy a web... you know I mentioned that when you build ah a version of your...
01:06:11
Speaker
of your package, it gets a hash name that uniquely identifies it. right and And if anything has that name, it's the same thing. Now, when you have a recipe for how to build ah an entire OS, you can also hash that in all of its inputs. And that it also gives you a name that uniquely identifies that behavior.
01:06:30
Speaker
Modulo state, which we can talk about later, no the volumes that you have attached and whatnot. I think I see where you're going with this. So you would be able to to build the new version entirely, get it up and running under a yeah new domain name hash, and then your atomic update is switch the load balancer.
01:06:51
Speaker
Exactly. Exactly. Yeah, yeah, yeah. Okay. So you switch the load balancer, and that gives you the atomic question. Yeah. So then we have to talk about state because that I can see that working perfectly until my database files also need to move over.
01:07:06
Speaker
Yeah, yeah, yeah. So let's let's let's maybe just elucidate a little bit more of the picture so it's clearer what the advantages are and how it works. right So we have these ah hashes, and as you mentioned, we switch atomically the load balancer.
01:07:20
Speaker
But the way in which each server is talking to one another is using those hashes as well, right? ah So the really nice thing is when you have ah when you have an upgrade um and you only change one server that's...
01:07:33
Speaker
thats ah that ah isn't talking to any other servers but is receiving requests from them, um then what happens is you know now, in theory, all the servers that are talking to it also ah also change because they're talking to a new version. right And so their behavior is different. right like The same way that if I have a function foo and it calls a function bar and bar changes, then in some sense the behavior of foo also changed.
01:07:58
Speaker
yeah So all of those get new hashes as well, and they all get deployed. But if it's the other way around, where you have a server that isn't being queried, but it's talking to various different servers, you just have to deploy that new version.
01:08:12
Speaker
And it can talk to all the old infrastructure. Yes, so when I deploy my new version, you can say for definite, you haven't changed the version of Postgres. That doesn't get new hash, so we just yeah the web servers just keep talking to the same old database.
01:08:26
Speaker
Exactly, exactly. So this is a minimal upgrade. right you you You spin up only the things that really changed, but it's also atomic, both in the sense of zero downtime, but just as importantly and even more excitingly, I think,
01:08:40
Speaker
ah the version upgrades happen atomically. You're never going to be talking, your new version ah ah of your application is never going to be talking to an old version of, of ah but your new version of the front end is never going to be talking to an old version of the backend, right? um Or of server one versus server two.
01:08:59
Speaker
And so you never have to worry. and one One thing that occupies, a you know, occupies developers a lot and it is really annoying to think about. It's like, oh, like always maintain backwards compatibility in all of your APIs because nothing is ever atomic, right? Like you're always going to have different version mismatches as you upgrade and roll out things.
01:09:15
Speaker
Whereas here they're not, they're atomic, everything's atomic. So you really can look at your code as a point in time thing, right? You don't have to worry about what what it looked like yesterday before you made this change.
01:09:27
Speaker
You can develop in a very yeah clean way. So that's another big advantage of this hash-based. This also makes it easy if I do want to maintain two different versions.
01:09:42
Speaker
Like if I've promised to keep the old JSON API running for exactly that client. Yeah. Yeah. The one technique that we haven't implemented yet, but ah we're excited about is, you know,
01:09:54
Speaker
ah
01:09:57
Speaker
leaving versions on for external users so that they also don't have to worry about this. right like If they use a if they use ah this hash domains, um you can just keep track of, you can keep upgrading without but backward compatibility changes or concerns.
01:10:12
Speaker
ah And then just keep track in the load balancer level of how many requests are coming in from from the external world for for those old versions. And again And, it ah you know, maybe you have a threshold that says, you know, like if there's fewer than one request a month, and you can then it's safe to spin this down.
01:10:31
Speaker
Yeah. Unless that request is coming from our biggest client, in which case we couldn't negotiate with them. Bet that happens. I bet you. Right. right ah Or alternatively, you know, like there are even weirder ways of thinking, right? Which is like ah ah kind of you can develop as a A-B b testing, sort of. like You can do deployments and then just see see what versions people like the most. the most like Oh, yeah. ah maybe you you know
01:11:03
Speaker
Maybe you have a branch where you're doing some work and you just deploy that. um And it lives as a fork or as a separate branch for a long time. um We are getting into the world where this one language which describes my dependencies and my language dependencies and my out-of-language dependencies and my deployment infrastructure also describes my A-B testing yeah and my feature flags.
01:11:27
Speaker
Yeah, yeah. Exactly, yeah. That's exactly right. And not having separate languages and separate configuration files for all those things sounds like where we should be. Yeah, yeah.
01:11:39
Speaker
i agree. Yeah. So this suggests that most of your future is going to be figuring out comfortable ways to describe this and then building out all those tools that introspect on these descriptions and surface things like here's ah here's a CVE you've got to worry about.
01:11:58
Speaker
yeah Yeah, absolutely. You must be hiring UI people. Or you will be if you're not currently.
Getting Started with Nix and Garnix
01:12:06
Speaker
ah Yeah, so far we've, you know, because the majority of our customers until two weeks ago, all of our customers were Nix users.
01:12:16
Speaker
I think... they're already kind of familiar with these ways of thinking. And so it's been relatively easy. But now as we sort of start to start to expand this, it really ah it really does involve thinking very hard about it about how to how to present this in a way that makes sense, that feels intuitive.
01:12:36
Speaker
Yeah, yeah. I'm probably not a great barometer of this because I'm an existing Nix user who is who was forced slash paid into it but actually became a convert.
01:12:47
Speaker
But there are probably people listening to this who are tempted to try Nix. And perhaps they can become a small flood of testbed users for you. So the question is, do you get started? Where would you recommend someone who's not being paid for it to get started yeah with Nix?
01:13:05
Speaker
Yeah, so this is bit self-serving, but I think you know our new modules project, garnix.io slash modules, ah can ah ah can help you know with Rust or Haskell or NPM projects, for instance, can can get you...
01:13:21
Speaker
um a lot of the way or the entire way into having, hopefully, the entire way into having project that compiles and builds with Nix and CI. um And obviously, the way i would like for this to work is that this becomes independent of Garnix altogether. you know in some sense, you I think ah vendor lock-in is a terrible thing. And so to some extent, I hope that you know people consider this, even if they don't want to use Garnix for anything else, for CI, or they generate this file and then go off and and you know build their project. and
01:13:57
Speaker
themselves in different CI system and deploy it however they want. um But I think that that's a very good way of sort of, you know, seeing that initial experience, right? Like seeing, okay, this really works and I can sort of understand the package infrastructure and I can look at the syntax and then see how it reflects facts about my project that I specified um and to understand a little bit about Nix and get a lot of the benefits, right? Like already get a dev shell that works for my project.
01:14:25
Speaker
yeah yeah yeah Yeah. Which languages do you support? Because I've tried this with one my Rust projects and I thought, okay, if I'm a Rust user, this is pretty straightforward. yeah Yeah. But presumably you have to write recipes for lots of different languages to make this universal.
01:14:40
Speaker
Yeah, and it it ah right now it's it's only Node.js projects, Haskell projects, Rust projects, and then also ah you know Postgres databases and Linux users as configuration for your ah for your web service if you choose to deploy web service as well.
01:15:00
Speaker
um But we want to ah we want to expand that a lot. And I think you know if there are people out there who are really excited about trying this out and want to reach out and say, like this is the product that I want, I think for us, it's very ah very helpful to have that sort of tight feedback loop so that we can see, OK, what is working, what isn't working for you. So do feel free to reach out if if these aren't ah if what you're looking for isn't supported yet.
01:15:29
Speaker
yeah But so far, it's it's those three tech stacks. mean OK. I'm quite surprised to hear Python isn't one of your early adopting languages.
01:15:39
Speaker
Yeah. I think ah and think Python would is the natural next one. and And yeah, that raises a question I haven't asked, but i'm just going to dip into it quickly.
01:15:51
Speaker
What language are you building this in? Presumably it's one of those three you've already mentioned. Do have doing this in Node, Haskell or Rust? ah Tell me. I reckon I can make a guess, but tell me. and Actually, full answer is all three. Our front end is a Node.js project, and we have some DNS servers and load balancers and whatnot that are in Rust, some small projects that are in Rust. But the majority of the codebase is in Haskell. Right. So what you actually need is a dependency that's built in Python for that script to get to production quality. Yeah, exactly. And this is why I say that having contact with someone who is so useful at this stage, because so far it's been very easy because we say, OK, let's try this out for, we're familiar with these stacks. Let's try it out for our own projects. Let's try it out for couple of friends' projects and whatnot. But as we move towards Ruby, for instance, the language that don't have very much familiarity with,
01:16:49
Speaker
um I think it's good to to to to be talking to people who do. Yeah, yeah. it's awesome I'm not sure you'll ever get there, but Nix is one of the few languages that um at least make makes the environment of Perl feel sane, because someone's had to go through all the pain of writing Nix recipes to build Perl reliably.
01:17:12
Speaker
That makes Perl less of a pain, you see. Yeah, yeah, yeah. yeah I don't think you'll be deploying Perl anytime soon, but you could. But I also think, you know, Zig, for instance, or g Gleam or things like, you know, these like new languages that could also be exciting, right? like Yeah, and those are ones that maybe haven't got a fully realized infrastructure story yet.
01:17:34
Speaker
Yeah, yeah, exactly. Yeah. Yeah. It's difficult to know whether you should ah go after the biggest players or the most hungry for new services players.
01:17:44
Speaker
Right, right. and Yeah, and it's also, you know, some extent... ah not only are these technologies newer and therefore maybe have less support, but also the projects in these languages are themselves newer.
01:18:01
Speaker
So it's more likely that you haven't yet solved or the big proportion of those projects haven't yet solved the question of how you're going to get deployed. right Whereas if you're running a Java project, those those are mostly, I would think, you know like proportionally more more legacy projects. might be more objectively, but they probably the majority of them have already solved these questions in one way or another. Yeah. yeah If you're so spending serious money on Java, you probably already know what your deployment stack is. Yeah.
01:18:30
Speaker
Yeah. Yeah. Well, in that case, it sounds like you've got a lot of work to do and a lot of market research to do. Yes, absolutely. So I should probably leave you to go and do it. Well, hopefully a couple of the listeners give you some more data to play with.
01:18:43
Speaker
And that more data will force you to do a data mining project in Python and we complete the loop. Perfect. And on that note, Julian, thank you very much for joining me.
01:18:54
Speaker
and Thank you as well. It was a pleasure. Thank you, Julian. So as always, take a look in the show notes for links. I've put a link to the Nix Getting Started guide in there.
01:19:05
Speaker
If you're using one of the languages that Julian mentioned, Garnix is actually looking to be a pretty good way to get started with Nix quickly.
Nix's Dependency Tree and Perl's Ubiquity
01:19:13
Speaker
So you might want to try there. Before we go, i'm going to leave you with the single most terrifying thing that Nix has taught me over the years.
01:19:22
Speaker
So Nix is a tree of dependencies and it goes all the way down to the kernel, as we've discussed. And when you're building something, you can look at the logs and see that what you're building depends on these things and those things depend on these things and so on all the way down. You get the whole dependency tree staring back at you.
01:19:41
Speaker
And I promise you, no matter what stack you're building, no matter what language you're using, somewhere in your dependency tree, you're depending on Perl. Absolutely promise you.
01:19:53
Speaker
So huge respect to Larry Wall for creating one of the foundation stones of everything we use, but also you might want to take two Valium before bedtime. Before you go and get that prescription filled, if you've enjoyed this episode, please take a moment to like it, rate it, share it with a friend.
01:20:11
Speaker
Make sure you're subscribed for the next episode. And if you're really enjoying the episodes, consider going to Patreon and supporting us. But for now, I've been your host, Chris Jenkins. This has been Developer Voices with Julian Arney.
01:20:24
Speaker
Thanks for listening.