Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Unison: A Programming Language for Distributed Computing
1.1k Plays1 year ago
“Software development has not caught up with the internet age.” So says this week’s guest, Rúnar Bjarnason. But what does that mean? What would a programming language for the internet age look like?
Rúnar’s answer is Unison. A language that completely rethinks the way distributing computing can work, from the source code up. Borrowing some key ideas from git, it challenges the way we think about code-sharing, compilation, versioning and more.
--
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Rúnar on Twitter: https://twitter.com/runarorama
Rúnar’s book, Function Programming in Scala: https://amzn.to/46I9jew
Unison website: https://unison-lang.org
Complete and Easy Bidirectional Typechecking for Higher-Rank Polymorphism (pdf): https://www.cl.cam.ac.uk/~nk480/bidir.pdf
Do Be Do Be Do (pdf): https://arxiv.org/pdf/1611.09259.pdf
Rúnar’s Øredev conference talk: https://youtu.be/EgIVzOobD48
Cloud icons created by Freepik - Flaticon: https://www.flaticon.com/free-icons/cloud
Computer icons created by xnimrodx - Flaticon: https://www.flaticon.com/free-icons/computer
Recommended
Transcript
Introduction to Developer Voices and Unison
00:00:00
Speaker
This week on Developer Voices we're talking about a new programming language and some new ideas for how programming languages should work by looking at the language Unison. Now Unison's been built specifically to tackle the problems of distributed computing and that might seem a bit niche but it's much more general than that because from a certain point of view when you think about it
00:00:24
Speaker
Pretty much all computing is distributed these days. We don't run code on one machine anymore. There's the obvious kind of distributed computing when you push your code out to a cluster of servers and they're all working together on the same task. But even in the more humdrum settings, I write some code on my laptop. I share it with the team. They share it back with me. We put it out to a CI server and to test and prod.
00:00:51
Speaker
And even though they're running different instances in different environments, isn't that all distributed code across a distributed code base? From a certain point of view, I think it is. And that's the core thing about Unison. It's got some ideas for how we can unify all those different kinds of distributed code and make the experience better, whether it's just you working on your laptop with a friend or a mega cluster.
Inspiration and Founding of Unison
00:01:20
Speaker
With one simple trick, as they say in the clickbait, I'm not going to give the game away, but I'll give you a hint. You know how Git has changed the way we share source code? And it did it with an immutable append-only database that was cheap and easy to copy, and just enough cryptography to verify that database as we pass it around, and be absolutely sure that if I've got the same hash as you, then we're working on the same source tree.
00:01:47
Speaker
I've always thought Git was a really smart approach. It's worked really well in practice, and yet we only apply it to source code. Can we take that idea further? Well, let's find out. I'm your host, Chris Jenkins. This is Developer Voices, and today's voice is Runar Bjornsson.
00:02:20
Speaker
I'm joined today by Runa Bjarnesen. Runa, how are you doing? Doing great. How are you? I'm very well. You're coming to us live from Boston, right? That's right. I'm just outside of Boston. But from Icelandic roots? Yeah, I'm originally from Akurere, Iceland.
00:02:44
Speaker
But I've come a long way since then. You really have. And you've got a career to match a long journey, I think, from what I gather. It's been a long, strange trip. Yeah. Do you have an academic background for starters? I do not. I went to a technical college in Reykjavik, but I dropped out because I got a job cataloging computer viruses.
00:03:13
Speaker
And yet somehow the academic stuff has caught you up, in a way. Yeah, you know, I've always been a reader, always sort of an autodidact. I taught myself how to program. And on the Sinclair Spectrum, ZX. Oh, which one did you have? Back in the day, I had the ZX48K, ZX Plus. The ZX Plus.
00:03:39
Speaker
A neighbour of mine had that and I have never been more envious of a human being before or since. I eventually got the next version, the plus two. But that's going back too far into history. We need to get a bit more up to date more quickly. To ground this, in 2014, you released a book which you co-authored called Functional Programming in Scala.
00:04:03
Speaker
That's right. And you clearly had got the functional programming in a statically typed language bug, but that hadn't gone far enough. So what did you do next? What did I do next? Well, uh, you know, I started with my coauthor Paul, uh, started developing the, the unison language. So Paul had actually been developing unison, uh, like after,
00:04:31
Speaker
We parted ways after the book and so he started to develop Unison. I was doing Haskell programming for various clients and working in industry. Then he got a little bit along with this Unison language and asked me to join and form a company. We founded a company called Unison Computing.
00:04:57
Speaker
And you know, we're developing this language. And that's been going for nearly 10 years now?
Unison's Features and Innovations
00:05:04
Speaker
The language? The language and the company. So we started the company in 2018. Okay. And I think Paul started working on it maybe 2016, something like that. Okay. But the sort of genesis of it is that we were actually working together. So when we started writing the book,
00:05:25
Speaker
we were working together writing Scala, purely functional Scala. We actually developed a small language for our employer at the time. We got the language development bug from there. Hang on. Why is a client asking you to write a language from scratch?
00:05:52
Speaker
So this was for doing financial reports and like sort of doing quantitative analysis on portfolios and stuff like that. And so it needed to be very expressive and they had this sort of visual language where you could like assemble sort of blocks of code together and kind of evolved into like a whole language. It was pretty cool.
00:06:21
Speaker
But you didn't get the visual programming language bug. You went back to text. Oh, yeah. I think visual programming is kind of a dead end in that it doesn't really allow for programs that are as expressive. No, and it doesn't manage complexity that well, I think. No, that's true. It doesn't hide the complexity very well.
00:06:46
Speaker
No, no. And it makes it a rat's nest to debug. Anyway, but you have some other ideas about how programming should work. And this is, I often think you can categorize languages as what they're like that's gone before. Um, what's brand new about them that we should pay attention to. So sort of. Yeah. Can you categorize unison that way?
00:07:14
Speaker
Sure. Unison is a statically typed, purely functional language similar to Haskell in that sense. But it is also a sort of image-based language similar to Smalltalk. That is, the code base is not a collection of mutable text files. It's an actual database that has your code in it.
00:07:42
Speaker
So that's rather similar to what languages like Smalltalk do. So that's how it's similar to other languages. But the way it's different is that all the code in your code base is referenced by hash. So that's really the main differentiator. So every definition has a unique, well,
00:08:13
Speaker
you know, most mostly unique and sort of eternal address, which is the hash of its implementation. And so you can unambiguously refer to any expression or type using its hash. So you're doing this trick that git does, but they do it like hash a file and store under the hash, you're doing it hash a function definition.
00:08:38
Speaker
Yes. So we hash a function definition, we hash types, we also hash namespaces, you know, that are collections of hashes. Right. Why? What does that get you? Yeah, why? So the main, for me, the main benefit of this
00:09:00
Speaker
is that it allows you to unambiguously communicate code over time and space, basically. So over space, meaning that you can communicate over the network. You can unambiguously communicate code over the network. So whereas if you're developing a distributed system in a traditional language, you have to do something at network boundaries. For instance, you'll take your data structures,
00:09:30
Speaker
serialize them to XML or JSON or some custom format, and then send it to the other machine. And then on the other side, you have to have some code that deserializes that and turns it into whatever data structures you have. Or you can do remote procedure calls, but then the locations will have to agree on what the code is beforehand, and they can't get out of sync and stuff like that. But in Unison, we can
00:10:00
Speaker
send the hash over the network and say, hey, here's the code I want to run, that I want you to run. It has this hash. If it has the hash, then it just goes and runs it. But if it doesn't have the hash, then we actually serialize the code. We capture the continuation of the program and we serialize that along with all of its dependencies up to some agreed upon depth. Then the remote location continues to execute the program.
00:10:30
Speaker
Does that include the state of the computation? It does. So I can get halfway through a function, suspend it and ship it over the network and have it run elsewhere. Yes. That kind of, okay, so that remote kind of remote code execution raises a lot of questions. The first one is security. Right. So you, so for, for one thing, you wouldn't want to, you know, accept this kind of call from
00:11:00
Speaker
the Internet. So you wouldn't want to expose your Unison node to the Internet and say like, hey, send me whatever code you want. So that's the first thing. The second thing is that since Unison is strongly and statically typed, and we have effect types, so Unison calls these things abilities. So different locations have different abilities.
00:11:28
Speaker
And when you expose a location or a unison node to the rest of your cloud, then you specify what abilities it has. And so you can't, for instance, send a computation that does some IO, for instance, unless the location specifies in its type that it is allowed to perform IO.
00:11:53
Speaker
So I could, uh, I could build a service that allows you to send me code and I run it provided that code is only either pure functions or gets and sets on your personal key value database.
Unison's Applications and Tools
00:12:10
Speaker
And it can't do anything security dangerous outside of that. Right. That's interesting.
00:12:18
Speaker
Yeah, because I remember saying because I saw you give a talk about unison at a dev a few years back, which is a great conference in Sweden. And you said something which relates to this you said languages have not caught up with the internet age. Which is a good quote related here and I want you to unpack it. So well, that was you know, that was a few years ago. But I think what I meant by that was that
00:12:46
Speaker
Languages mostly talk about what one CPU process is doing, the one OS process. It might have multiple threads and like doing lots of different things at the same time, but it's mainly, there's a single OS thread and it's doing some stuff. And if you want other OS threads or other machines to do stuff, you now need to break out of your programming language.
00:13:13
Speaker
there usually aren't any facilities in the languages for talking about what other machines or other OS processes are doing. So you need to get out proto buffs or some other non-programming language in order to bridge that gap.
00:13:41
Speaker
I can think of at least one attempt to do that, to build a language from the ground up that doesn't believe it's running on one node, and that would probably be Erlang. How is your solution to this problem different? Well, how is it different from Erlang? I don't know enough about Erlang in order to talk about it intelligently, but I
00:14:12
Speaker
I believe that the main difference is that we have this notion of hashing, is that we can unambiguously talk about code over the network. So I think that if you are sending a message to an Erlang location or node, you need to know something about what code it's executing.
00:14:41
Speaker
Like there's, there, there's going to be some kind of like version, version mismatch or there, there's some, um, impedance mismatch there that I believe they probably can't bridge without this kind of unambiguous addressing. Because your system is going to be able to say, if I start this running on one node, it can branch itself out to other nodes because it can deploy itself. Right.
00:15:10
Speaker
It can say to another machine, run this. And by the way, if you need it, I can send it to you. Yes. And I can imagine the program just spreading out in its own self-managed rolling upgrade. Yeah, that's right. That's pretty cool. And it's all done purely in the language. So there's an ability called remote. And it gives you this ability to send a computation. You basically have this notion of an abstract location.
00:15:41
Speaker
And there's a function called, I believe it's called fork at. And it takes a lazy computation, sort of like a captured continuation and the location, and it'll send the code to that location. And it's sending the data too, right? Yeah, it's sending the whole closure. Right. So everything that the computation depends on as well. OK.
00:16:11
Speaker
It could not only map-reduce data over a cluster, it could map-reduce the map-reduce computation as well. I have an image of a benevolent virus in my head, spreading itself across the cluster. Benevolent virus is a good way of describing it. We're calling it just-in-time deployment, but benevolent virus is a cool way of saying it.
00:16:38
Speaker
depends on whether we want to sound safe or dramatic, I guess. It's a lot like just-in-time compilation, where translating the program to machine code to run fast happens just-in-time, and in unison, translating to network calls happens just-in-time.
00:17:05
Speaker
Okay, this raises a couple of things. Let's go for the first one. It must raise the question, are you planning some like global Unison cluster where I just ship my code to that? Yes. That's in fact, the Unison computing's flagship product is Unison cloud, where we basically provide you with a library that you just put in your code base.
00:17:33
Speaker
And then you have an API key. And when you call this library, you are basically deploying your computation on our managed infrastructure. And you can source as much computational power as you require. That feels like
00:17:57
Speaker
It's bridging the gap or eliminating the gap between something like you can get an EC2 machine on AWS, or you can get a Lambda, which is running a single function. Right. It is like that, but I think for one, it's a lot simpler. And for another thing, you're doing it in your language. So it's really that you write your code, and then to run the program is to deploy it.
00:18:26
Speaker
It deploys itself on the managed infrastructure. You don't have to write YAML files or some kind of configuration. You don't have to hire armies of DevOps people to manage your clusters and manage your EC2 instances and stuff like that. The unison cloud is doing all that for you. All you have to do is just run your program and that deploys it.
00:18:56
Speaker
So I've got, I've got a hint of this from a talk I saw you give, but you would, um, you literally, you load your code into like a repl and run it and it works. And so then you say run it at, um, Boston compute dot unison. Yeah, basically. So that there's a local interpreter for the, for the remote ability. So you, you know, you can write your, your program.
00:19:26
Speaker
as a distributed thing, using the remote ability. And you can run it locally using this local interpreter. And there are other interpreters where you can diagram your system so you can see how it's distributing itself over the network. And that's all done locally. But then there's the unison cloud interpreter, which goes and actually deploys it on managed infrastructure in the cloud. And it's live.
00:19:56
Speaker
So how does that deal with, I guess these are related things, but the other thing you need as well as deploying code for a remote managed service like that is you need persistence of data and you need migration of data. Yes. How are you dealing with those things? So that's, you know, that's early days so far. So right now the things that you can do, for instance, is
00:20:26
Speaker
You can talk to S3 instance right now. In Unison Cloud, it's close to wherever your S3 objects might be. But we're also developing a
00:20:49
Speaker
managed storage system. So it's a typed storage that's basically just a unison library. And then you can take any unison object and you can persist it in the cloud. And then you get basically typed access to it back. And often there are problems with these kinds of things where like, oh, I can persist any object and then serialize it back.
00:21:19
Speaker
But if you try to do this with, I don't know, I've done this in languages like Java, where you have a library that can persist any object to disk or to a database.
00:21:35
Speaker
then you run into the issue that as the code evolves, like the data types change, you know, and then you end up with a version mismatch. And then it's very difficult to get like old versions of stuff like out of storage. Yeah. But unison doesn't have this problem because you can, you know, the, the, the storage knows what type, uh, the objects is. And so,
00:22:02
Speaker
the hash of the type is available, and you can load that type. And so you'll always have an object of the type that has that hash. And you can recover the code for that type as well. OK, so even if you update the definition of the type, you can still deserialize the old types because they're kind of stored in git-ishness. Right. That's the one sort of Merkle tree
00:22:32
Speaker
Yeah, but then you've got the problem that your new code you've updated expects the new hash of the data type. Yeah, but then you'll just get an ordinary type error. And so we all you have to do is basically convert the data that you've restored to the new type.
00:22:53
Speaker
And then you can store it again or do whatever. But it's all still very much in your language. And you get the benefit of normal things, like type errors, rather than a runtime error that says, oh, version mismatch. Right. Yeah. So you're pushing that kind of data migration problem to compile time. Yes, basically.
00:23:16
Speaker
That's okay. I like that because it's going to, that kind of area is going to come up and you have to deal with it anyway, but you see, you might as well find out sooner and with better error messages. Right. And then you can migrate things programmatically. Obviously you can write a program that says like, grab all this stuff and like convert it to the new version and serialize it again. You know, you can do all the sort of normal things that you would otherwise do. You'll end up writing functions, which take
00:23:45
Speaker
user.hash1 and return to user.hash2. That kind of thing. And you can do that at compile time, and it will check it for you and then just work. Great. That's an interesting promise for long-running computation, long-running services. So I best ask you this question.
Development Stage and Community Involvement
00:24:10
Speaker
What state is unison in? Is it alpha, beta, research, production?
00:24:16
Speaker
I would say sort of late beta stage at this point. So I wouldn't say anyone is using unison for mission critical stuff in production at this moment. And you probably shouldn't. But it's already a pretty capable language and has good library support already.
00:24:45
Speaker
and, you know, constantly new libraries being developed as an active and friendly community. And, you know, there are all these things that are sort of converging this year, which is, you know, we're getting, uh, unison cloud is coming out of, out of beta. It's, so it's in sort of a private beta at the moment. So you have to like get on a wait list and stuff, but so we're bringing it out of beta this year. Uh, we're, we're adding this storage layer.
00:25:15
Speaker
And Unison is also getting native compilation very soon. So you will be able to run your code in the cloud very fast. OK. What's the current state of it? So currently, it is an interpreter language. It's a bytecode interpreter. So it sort of compiles to a Unison bytecode. And then there's an interpreter written in Haskell that interprets that bytecode.
00:25:45
Speaker
And so we have a just-in-time compiler that's being developed by the community and by unison computing. And that's all being developed in the open. And so it compiles the scheme now. And the scheme compiles a very fast machine code. What's the other language that's Idris? Doesn't that use scheme to compile? Yes, I believe so. I believe Idris compiles the Chez scheme.
00:26:12
Speaker
Yeah. I wonder what it is about scheme that makes it a nice compilation target. I don't know. I mean, it's very comprehensible, for one thing. It's got good performance. I think the compiler produces really good machine code. And we found that the code that's generated by the new just-in-time compiler is
00:26:40
Speaker
between 400 and 700 times faster. So that's pretty good. That's cool. And that's coming out this year. Yeah, hopefully. OK. So we talked about this Git-like thing. I could come back to Git as my reference point for hash it and ship it. Sure.
00:27:03
Speaker
Um, the other thing, which you've touched on a bit, we should go deeper in is, is abilities. We had, uh, Louis Pilfold, the author of gleam on a few weeks ago, and he said he really wanted to bring effects, managed effects to the language, but couldn't find a good way to do it. Okay. What are managed effects for the listeners and have you found a good way to make it usable?
00:27:32
Speaker
Yeah, so I don't know exactly what he means by manage effect, but I imagine that he means that the effects are tracked in the type. Is it a type language? Yeah, I think he's going in the Haskell type side effects direction.
00:27:55
Speaker
Yeah, so there are a number of different ways that you can do this, like manage effects in the type system. And so the way that Haskell has gone is to do this with monads. So basically you just have a
00:28:08
Speaker
a data type that's parameterized, and the data type represents what kind of effects you can do. For instance, there's an IO, famously, there's an IO data type. So instead of just having a string, you read a line from the console, and you don't get a string, you get an IO string. So you get this data type called IO, which has a parameter, which is string. So it's an IO containing a string eventually.
00:28:40
Speaker
And that's a perfectly cromulent way of managing effects in the language. But it has certain drawbacks. For one thing, monads have a syntactic overhead in the language. So programs that are pure, that don't use any effects, they are syntactically different from programs that do use monads.
00:29:10
Speaker
So you have these sort of two programming modes. Yeah, one of the first things you have to learn when you're learning Haskell is how to flip between those two modes. Yeah, and then composing monads is, you know, a sort of an evergreen topic.
00:29:26
Speaker
Yeah. So one of the things that people do is monad transformers. So you want to compose monads F and G. And so you need an FT monad transformer, which is parameterized by another monad G. And that combines them into this sort of composite monad, where you want to do, say, state and IO, for instance. Yeah.
00:29:55
Speaker
And so then you get this like state T of IO or whatever. Yeah, you can end up with like a program that has the side effect of accessing the database and doing logging is different to a program that has a side effect of doing logging and accessing a database. Yeah, that's true. Logically, they should just have those two side effects. Yes.
00:30:18
Speaker
So what we're doing is a little bit different. So we're taking the approach of algebraic effects. We're calling these abilities, which is a term that we got from a paper called Doobie Doobie Doo, which is the best type of paper ever. It's about a language called Frank, which is by Conor McBride, I believe. Yeah. I've seen Conor in Amsterdam. He's a great, great thinker. Awesome. Yeah.
00:30:47
Speaker
Yeah, so we got these ideas from there. And so algebraic effects are a little bit different in that you are able to combine effects sort of side by side. So instead of pressing them together using like a transformer, you have this just sort of set of abilities on your functions. So a function will be
00:31:15
Speaker
It will require the abilities to do IO and state, for instance. That's just basically a set on the function arrow. Combining these things is sort of trivial. If you invoke an IO thing, that gets added to the set. If you invoke a thing that acts as a database, you get that added to the set.
00:31:40
Speaker
So when you're looking at the type signature, you see the sum total of all the effects that particular function has demanded in order to run. That's right. Yeah. And then presumably you write some code that dispatches each of those effects. So you see in your set, it says I need logging. So you wrap that function call in with standard out logging. Right. And that takes logging out of the set and eventually you get down to just a computation.
00:32:06
Speaker
Yeah, that's exactly how it works. And you can write custom handlers for these things. Unison has a keyword called a handle, where you can supply a handler for any effect. And you basically adjust pattern match on the constructors of the data type that represents the ability. And also importantly, I think it doesn't have any syntactic overhead, or it has very minimal syntactic overhead.
00:32:34
Speaker
A program that uses abilities looks a lot like an imperative program that doesn't use abilities. So there's no monadic syntax involved. We're not switching between do and bind syntax and regular pure function syntax. Right. So you're always just writing a block. And so you can just write, basically, a let block. And you can do effects there. And the whole block will take on
00:33:02
Speaker
the type of having or requiring those abilities. That sounds a lot like the upside of Haskell, you know, tracked, affects tracked in the type system, I think is a really good idea. But it's hard to teach and it's a bit of a learning curve. And it sounds like you may have solved that second problem, if you're to be believed.
00:33:26
Speaker
I don't know if we've solved it, but I think it makes it a little bit easier to think about, at least for people who are not versed in monads. You can still have monads in unison, and you can have monad transformers and all that stuff. That's perfectly possible. But as you know, it's not directly encouraged by the standard library and things like that.
00:33:53
Speaker
Another thing is that there are, of course, downsides to this. So one of the downsides is that algebraic effects, at least in the form that they exist in unison currently, are not as expressive as monads and monad transformers. And so you can't necessarily combine things in ways that you would otherwise do in Haskell. Can you give me an example?
00:34:21
Speaker
Can I give you an example? So for example, in Haskell, you might be able to do a higher order effect where you might abstract over monads that have a particular capability. So what's an example of this?
00:34:50
Speaker
there's something like monad state, for instance. So that's like a class that represents all the monads that could possibly be interpreted to track some state. So that's very expressive to be able to abstract over all effect types that track state.
00:35:12
Speaker
But the Unison's type system, Unison's ability system doesn't allow you to talk about effects that abstract over the capabilities of other effects at this moment. But then they come later. OK. I think that's probably, for the majority of programmers, this is already astronaut-y enough for that to be OK. Yeah, and it's a trade-off.
00:35:40
Speaker
For a lot of programmers, seeing a library that is very abstract while using it might be pretty straightforward. Once you get into it, the compiler will take care of mapping your concrete data types to their abstract types. But seeing a library that is very abstract, you may struggle to understand how you're supposed to actually use it. Yeah.
00:36:06
Speaker
Yeah, that all depends on like compiler support. Like, um, there are functions in Haskell that are terribly useful, but like have so many type variables, I can't keep track. Right. And the more type variables you have, the worse the error messages get. That is also true. If you'd done anything to solve that problem. Well, you know, one of the ways we solve that problem is, is the,
00:36:33
Speaker
You're encouraged to write things in a very direct style, and so you don't end up necessarily with these type astronaut-y things. I can think of a couple of libraries for Haskell, for instance, where people really struggle with error messages, and one of them is like Lens, where it's super useful. I love the Lens library, and I use it all the time.
00:37:03
Speaker
often error messages are really abstract. You have to be well versed in how to use Lens in order to understand it. Another one is servant, which allows you to basically describe an HTTP server as a type. You get this typed communication over HTTP, which is super awesome. It basically just generates all of the boilerplate for you. But the error messages it gives you are just
00:37:31
Speaker
It might be 20 pages long or something if you get one type variable wrong. It can be really difficult to decipher a type that describes an entire HTTP server. We're taking a lot of cues from languages like Python, for instance. The culture in
00:37:59
Speaker
in the Python community, generally, is that you write libraries that work in the obvious way, and you sit down with a Python library, and you can sort of generally assume that it's going to be written in a kind of obvious way, that how you're supposed to use this library is going to be straightforward, and it's going to write it in a very direct style. That's easier said than done, though.
00:38:29
Speaker
I mean, you've got to kind of agree on what counts as obvious. Yeah, that's true. Perhaps you should say, who's your target audience for Unison?
Target Audience and Adoption Strategy
00:38:41
Speaker
Are you expecting Haskell programmers to switch to it eventually? Or is it Python programmers? Who's going to be adopting Unison if you get your dream world? Well, in the dream world, basically everyone who wants to develop distributed systems, which is really everybody.
00:38:59
Speaker
Because there aren't very many systems these days that aren't distributed. Even things like systems that have a front end and a back end, the communication between the browser and your back end, if you're writing a web application, that's a distributed system. And one of the things that we hope to develop in the future is JavaScript.
00:39:29
Speaker
compiler for Unison. You should be able to write a distributed system that runs in your browser and deploys itself to the cloud. Will we be getting a future in which the browser checks out the hashes of the functions that have changed since the last time you checked the website?
00:39:49
Speaker
Yeah, that would be amazing. That reminds me of something I also remember you saying in your talk way back when, which I'll let you explain it, perfect compilation, perfect incremental compilation. Oh, yes. That's a big one. Yeah, so perfect incremental compilation. So in Unison, your code base is sort of always live.
00:40:16
Speaker
So it lives in a database, and it's always compiled. So let me contrast that with, so let's do the jennison differentiator thing. So contrasting that with the way things work in most programming languages, like let's take Haskell, for instance. So in Haskell, you write a bunch of text files. You proceed to mutate those text files. And then you run the compiler, and the compiler generates the code.
00:40:46
Speaker
And then if you've done something wrong, you get a long list of error messages, and you've got to go mutate the text file some more, and then run the compiler. And then whenever you change something, you've got to run the compiler and build everything. So the way that Unison works is that whenever you write a definition or a type or something, it gets compiled basically immediately.
00:41:09
Speaker
you submit it to the Unison code base manager, and you can add it to your code base right there and then, and it lives there in a compiled state. Then when you write some more code, it compiles that code, and then you can add that to your code base. So it's very incremental, and your code base is never in a state where it's broken.
00:41:36
Speaker
you know, mutate your, your code base in such a way that it is, uh, that it doesn't compile. Right. Right. Okay. And you've also got like your, you've got, if you've got like a large old code base where the majority of code there you don't actually touch. Yeah. That never needs recompiling. That's right.
00:42:03
Speaker
What if you get a new employee come along and they let's exaggerate and say we've got like Google or Facebook size Mono Reaper. They want to check that out and start working. Do they get everyone else's pre-compiled shared code base for free? Yes, they do. The way that the Unison code base manager works is that everything is stored in its compiled state.
00:42:32
Speaker
And then you can push that to other locations. And the one that we provide is called Unison Share. So it's sort of like our version of GitHub in a sense. And so you push your code to share. And then the other developers that you're working with will pull that. And what they're pulling isn't the text of the program. They're pulling the compiled version down.
00:42:59
Speaker
And then they can use the code base manager to view the source code or edit the source code. But the source code is not stored anywhere. OK, so how does that work for, like, if you and I have two functionally identical functions, but I've put comments in mine? OK, so that really depends on how you've added comments in it. So there's one kind of comment that's completely ignored.
00:43:29
Speaker
So then it just gets thrown away when you compile your function. And if you delete the source code, it's gone. The comment is gone. It'll never come back. It's sort of like a note to self. Another way you can write comments is to basically just put strings in line in the program. And there's a little function called ignore, which will allow, just tells the program to ignore this.
00:43:57
Speaker
that will change the hash of the program. And so if you have identical functions that have different comments, they do get treated as different functions. So we've toyed with the idea of adding a third kind of comment, which is a comment that lives in the compiled form but doesn't contribute to the hash. But that has the downside that if you ever want to change the comment, you would have to
00:44:27
Speaker
basically delete that hash from your code base and add it back with the updated comment. That's nasty. Yeah. Yeah. Uh, hash the metadata separately. Yeah, that's, that's another thing, you know, we could do is to basically put, uh, and, and unison, unison, the code base format has this facility for metadata. Now we include things like who wrote this and what's the license. Um, okay.
00:44:57
Speaker
and stuff like that. So every definition has, you know, some metadata hanging on it, like author and license and maybe, uh, whether or not it's a test. Um, where's the documentation, things like that. Could you theoretically then say, I want to make sure that in production, I'm not running any proprietary code. Yes, you can. And you can computationally do that. You can ask.
00:45:24
Speaker
You can basically write a program that takes a unison expression and at least theoretically, you could check the license, walk the tree, and ask it, does this use any GPL? Yeah, or can I make sure that none of my test code makes it into production? Yeah, yeah. Yeah, that's pretty cool. OK, one more thing on that, then. So you're saying I write a function.
00:45:53
Speaker
It gets hashed and the compiled version gets shipped over to the central repository. Someone else checks it out. They can't get my syntactic sugar. So there must be some official code formatting for unison. Yeah. So, uh, unison will format the source code. So, you know, there's a, it currently has one surface syntax, but theoretically there could be many.
00:46:20
Speaker
We've toyed with the idea of having a Lisp syntax for Unison. Oh, I could write mine in Lisp, but my colleagues would never know. Your colleagues would just see, they might see the Haskell syntax that it currently has. But yeah, the source code is thrown away, so you're never faffing about with formatting or whatever, you just write your code.
00:46:43
Speaker
submit it and you can just trust that it'll look nice when it gets rendered. Or, you know, if it doesn't look nice, at least it will look the way that unison code is supposed to look. Right. Yeah. You have to agree with the one true way of formatting code, but I'm kind of proud of that. Yeah. Yeah. I like that too. I don't like to have to think about, you know, formatting my, my code. That's just like not part of the job. Yeah.
00:47:10
Speaker
Yeah, I'd rather the entire team agreed on one code formatting and we all got it for free. Right. Yeah. Rather than debating and hand crafting to fit, to match the blessed guide. Right. I mean, we have, you know, unison is mostly written in Haskell. And so we have in our Haskell code base, you know, there's, there's this sort of, uh, blessed, uh, format, which is, which is, uh, generated by Ormolu.
00:47:41
Speaker
And so we will have these commit triggers and whatever, so that whenever you submit new Haskell code, it all gets formatted in the approved way and whatever. But even that can be contentious, like, oh, maybe you have different versions of the formatter, or maybe you don't like the way this is, and so you kind of want to tweak it.
00:48:03
Speaker
You may have to form a committee about, oh, do we want to tweak this parameter here? So even that leads to bike shedding. But in Unison, that is just completely out the window. There is no tweaking the formatting at all. You get what the code base manager prints out, and that's it. But in theory, you could have your preferred formatting that no one ever needed to worry about.
00:48:31
Speaker
That's true. And in theory, you could do that. Because I've worked on teams where every single person just wanted the unified formatting, except one guy who was absolutely news. That's actually, that's actually a good point. You know, you could, we could introduce facility so that you could tweak the formatting for, for like just your instance of, of the code base manager. So locally you're developing in, you know, beautiful, you know, some code that is exactly the way you want it.
00:48:59
Speaker
But then once you submit it to the code base, everybody else sees it the way they want it. But then you see their code the way you want it as well. Yeah. Specifically, this is very, it seems simple, but very different that we're, we're still dealing with text files, but we're not really.
Unison's Unique Perspective on Code and UI
00:49:18
Speaker
Text files are just a projection of what we're really dealing with. Yeah. Yeah. That's right.
00:49:24
Speaker
I think that code as text is really just a user interface onto like what you're actually doing. It's a really good user interface. Like it's very expressive, highly flexible. You know, we've toyed with things like structured editors. And so where you're like directly editing the tree and like, you can't compile errors and stuff like that. And I think that ideally that would be awesome. Like, but the, the user experience of it,
00:49:53
Speaker
I've never found it to be as good as just writing text. Yeah, yeah, I tend to agree. So, I mean, that sounds like something maybe for the next version or a few versions down the line. Yeah, a few versions down the line, maybe we'll have a structured editor. Yeah. And custom formatters. Yeah, I can believe. But where are we today? If I go and check out Unison, what kind of experience can I expect to have? What kind of libraries am I going to wish
00:50:23
Speaker
were there but aren't? Oh, that's a good question. What kind of libraries would you wish to worry about aren't? Well, so when you, when you install unison, you know, if you're, if you have like brew, you just say like brew, install unison, Lang or use language. And you know, you can go to the, to the website and like download the binary if you're on like Linux or whatever. And,
00:50:51
Speaker
It's really straight for a setup. There's one executable. You run the executable. It's called UCM. You just run the executable. And it's listening to changes to .U files in the current directory. So then you open your text, your favorite text editor. You open a buffer named .U, like something .U. And you just edit. You can write unison code directly in there. And whenever you save it, UCM will pick that up.
00:51:21
Speaker
and type check it and give you some feedback about it. Then you can interact with the UCM to add it to your code base or ship it to Unison Share or stuff like that or run it. So yeah, it's a very minimal setup. There's not a lot of configuration or
00:51:51
Speaker
installation that needs to happen in order to get going. You can really just start writing code right away. There's nothing to set up. I'll tell you a way that this would instantly interest me. I can imagine going to a hackathon
00:52:13
Speaker
with unison. I run this unison code manager. What I'd like to do quickly and easily is connect it to my teammates laptops in such that if I write a new function, they can instantly pull it in. Is that something I will need unison share for? Or can we just do like a local GitHub?
00:52:37
Speaker
You can use Git in order to push code around. It does support that. But it supports it as binary files. So you push, basically, you push your code base as a binary file to a shared Git repository, and they can pull it from there. But the nice way of doing things is just to use Unition Share, because you're not pushing these binary blobs around, really. OK.
00:53:07
Speaker
I saw, so another thing we had recently on the podcast was we were talking about Advent of Code as like a coding competition for different programming puzzles that goes on once a year. I saw you had a unison challenge around Advent of Code. Yeah, we did. That must have turned up a few like holes in the standard library, that kind of thing. Did it? Oh, absolutely. Yeah, that was fantastic. They've seen people
00:53:36
Speaker
solving these sort of simple programming puzzles, it turned up a lot of useful functions that weren't in the standard library, maybe whole libraries that didn't exist but should have. But I actually would love to know the answer to this question. When somebody sits down with Unison and wants to get started, what libraries are they finding aren't there? Because I don't know the answer to that.
00:54:03
Speaker
And I would love to know, because I want to write those libraries that people want. And I farm them out to the community. The community is hungry to write useful libraries. Cool. I'll give you three at the top of my head, because these are the first three I always look for in any language. Can I do a web server, a web socket, and connect to, let's say, Postgres?
00:54:28
Speaker
You can. So the first one, yes. So there's a web server library. There's also another library that I'm currently working on that creates a very ergonomic API on top of that low-level web server. There's not currently a WebSockets library. But that's a good one. We could definitely do that.
00:54:57
Speaker
And you can connect to Postgres. There is a little library that allows you to connect to the simple Postgres protocol. But currently underlies development of a library that allows you to use the more advanced Postgres protocol, do things like prepared statements and things like that. And is there like an interop? Have you got any way of like
00:55:22
Speaker
creating a new library quickly by reusing a Java C library, something like that? No, there isn't anything like that. Yeah, it's got to be written in pure unison, or it has to be added to the unison runtime, like through Haskell. That's currently the way things are. But that won't be that way forever. I envision a future where you'll be able to make calls into
00:55:53
Speaker
uh, foreign functions, you know, using the scheme interface. Yeah. And speaking of the future, perhaps to end on, do you think you've got the big idea you seem to have here? I mean, you've got a few big ideas for mainstream programming, but the big one is this idea of hashing code and, and storing sharing hash code. Do you think, do you think that will start to percolate into other languages? And is that something you're trying to do?
00:56:24
Speaker
Yeah, I think so. I think we're kind of already seeing that a little bit in various places. I mean, there's things like Nix, for instance, is very much this idea that everything is hashed. But the granularity is different, obviously. I saw some work under where it lasts a year with Haskell where they can do
00:56:55
Speaker
term equivalence up to hash. So basically inside the compiler, you can decide whether two things are the same based on the hash. But that's not yet communicating between machines or anything like that. But I see this idea of hashing come up all over the place.
00:57:22
Speaker
like languages that are based on like, I don't know, blockchain or whatever. I mean, the basic idea is that you have some like eternal address for an object and then you can like always refer to that, you know, that address on the, on the blockchain or whatever. Yeah. Yeah. Yeah. Yeah. They're, they're encoding code on a hash and like you send money to a hash, which turns out to be an executable contract, that kind of thing. Yeah, basically. Yeah.
00:57:52
Speaker
And then there are distributed hash tables. What's it called? HSFS? I forget what it's called. But there's a number of implementations of distributed hash tables where you can store data in the ether, basically, by hash. This big content addressable storage super network.
00:58:19
Speaker
Right. So what we're saying is this idea is already percolating into the world and a great place to discover it would be unison. Yeah, I think it's sort of an idea whose time has come. And it has a lot of benefits. It comes with a lot of
00:58:43
Speaker
implications that changes the way things are done, you know, like things like builds and like dependency management and internode communication. Like basically everything changes when you, when you start to refer to things by, you know, by a deterministic name rather than some invented name. Yeah. Yeah. And some invented and commonly reused name we like function name tends to apply to all the versions of that function we've ever written. Right.
00:59:12
Speaker
Yeah. And you know, you'll have a data type called list in a standard library or whatever. And like, well, do you mean list version one? Do you mean list version two? Like who knows? And like you will have different dependencies that might make different assumptions about what a particular name refers to. And if they don't agree, then you'll get like, uh, you know, in the best case, you will get a dependency conflict at compile time or at build time. In the worst case, you'll get a runtime error.
00:59:41
Speaker
And so we avoid that. All based on this problem we know exists, but we kind of hand wave over because you've never had a good way of solving it. Right. I think so. You might be a good way of getting to the next level of programming. I certainly hope so. Yeah. I mean, I definitely want to encourage people to just like,
01:00:02
Speaker
come to the Unison Slack, come to the Unison website, download Unison, install it, play with it, and let us know what's missing, what could be better, or let us know what's great and what's cool. We really, really want to
Conclusion and Reflections on Unison
01:00:20
Speaker
know. That'd be nice, too, on the internet, right? Yeah. Yeah. Cool. So unison-lang.org, is it? Yes, that's right. We'll link to it in the show notes, but yeah.
01:00:31
Speaker
For now, I think we should leave people to go and check it out. But for now, Runa, thank you very much for joining us. Thank you so much, Chris. Cheers. Thank you, Runa. You know, it's my ambition for this podcast that we'll be always looking to where we hope our industry will be in five or ten years. And if we just manage to get one or two of Unison's ideas into the mainstream, I think we'd be in such a better place. I hope Unison does it. I hope someone does it.
01:01:00
Speaker
You know, even the approach to incremental compilation would change our working lives, change the speed at which we build and deploy. I worked at a place once where their CI server was so log jammed that you would wait overnight to see if a branch would build. I don't think that's so uncommon. Incremental compilation would have made that as simple as compiling a few functions. I really hope it goes mainstream. I look to the future as ever with optimism.
01:01:29
Speaker
But that's all for now until we reach the glorious future of next week's episode. Take a look in the show notes for links to Unison and how to get started. I'll put in a link to that wonderfully titled research paper, dooby dooby doo. Check out the like and subscribe and share buttons if you'd like to show your appreciation. And I think that's all until next time. I've been your host, Chris Jenkins. This has been Developer Voices with Runar Bjornsson. Thanks for listening.