Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Creating and Evolving Elixir (with José Valim)
2.9k Plays5 months ago
Back in 2012, José Valim started building Elixir to as a way to have his ideal programming language running on the same platform as Erlang. Fast-forward 12 years and it’s become build anything from distributed infrastructure to notebooks and websites.
In this week’s Developer Voices, José joins us to tell the history of Elixir in a series of design choices. Which features mattered to him in the early days, and which ones excite him most now. What’s going on under the hood to make Elixir tick, and what does its future hold?
–
Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices
Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join
Elixir Homepage: https://elixir-lang.org/
Elixir Docs: https://elixir-lang.org/docs.html
Numerical Elixir: https://github.com/elixir-nx
Phoenix: https://phoenixframework.org/
Livebook: https://livebook.dev/
José’s Livebook & Elixir Presentation: https://www.youtube.com/watch?v=pas9WdWIBHs
Comparing Elixir & Erlang Variables: https://dashbit.co/blog/comparing-elixir-and-erlang-variables
Gleam on the BEAM: https://youtu.be/RntfkL8lUY4
José on Github: https://github.com/josevalim
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Kris on Twitter: https://twitter.com/krisajenkins
Recommended
Transcript
Introduction to Elixir's Evolution
00:00:00
Speaker
On Developer Voices this week, I'm joined by Jose Valim, the creator of Elixir. And we're going to take a look at the evolution of the language, from what he wanted Elixir to be in the early days, to what's happening right now. And it begins with a classic scratch your own itch story. He just wanted a language that worked on the Beam, the same distributed platform as Erlang,
00:00:22
Speaker
with a few extra features that he was missing from the likes of ruby enclosure and we talk extensively about which features they were and how they gradually made elixir into the language he wanted to use a language that has been very well received it's been used in some seriously large real world services you can find traces of elixir in companies from spotify to pepsi
00:00:47
Speaker
For a long time, it was the backbone of WhatsApp.
Elixir's Popularity and Type System
00:00:50
Speaker
It's currently a very popular way to build websites with the Phoenix framework. Elixir has been a success. But all that kind of condenses a long journey, a journey of about 12 years. And you have to wonder, is Jose still as enthusiastic about developing Elixir as he was back in 2012?
00:01:10
Speaker
Talking to him, I got the sense there was a slight lull in those middle years, as the language stabilized, as it could be considered done. But if there was a lull, it's abundantly clear that right now he's as enthusiastic as he's ever been, as he's working on bringing a gradual type system to Elixir. Opt-in, gradual typing, added to an established, well-received, dynamically typed language. How's that going to go?
00:01:40
Speaker
I try not to take sides in the type system wars, but I do think that static typing is one of those features that gets more and more valuable, the larger and older and more mature the code base gets. So the thing that's got Jose fired up about Elixir development might well take its user base to a new level too.
00:02:00
Speaker
Either way, it's a fascinating technical journey to hear about, so let's hear how it's done.
Language Design and Syntax Inspirations
00:02:06
Speaker
I'm your host, Chris Jenkins. This is Developer Voices, and today's voice is Jose Valin. I'm joined today by Jose Valin. How are you doing?
00:02:27
Speaker
I'm doing great. Thanks for having me. Thanks for being here. You've been a language designer for 10 years, which kind of puts you younger than Donald Knuth, but more an elder statesman than some. You're the kind of middle-aged language designer. Yeah. I'm wondering which kind of midlife crisis comes with this particular package.
00:02:55
Speaker
You'll have to let us know in hindsight. I think we should go back 10 years, 12 years, 2012 you created Elixir. I've always had the impression from the outside that the primary motivation was, love Erlang, don't like the syntax, let's put a normal syntax on it. Is that true, close to fair?
00:03:21
Speaker
Halfway there. I think a lot of people think about the syntax. It's the first thing that comes to their mind because Elixir and Erlang syntax, they are so different.
00:03:38
Speaker
Right? And Erlang syntax is for rain because it comes from prologue, it comes from that particular tree, the sentiment tree. Yeah. And Elixir is closer to other dynamic languages, like mainly Ruby, but also some from Python as well.
00:04:03
Speaker
And I think syntax is the first thing that pops up to a lot of people's head. But for me, I can't remember if... I don't think syntax was a big concern at the point because I actually... There are things that I really like about Erlang syntax because it is...
00:04:21
Speaker
very concise. It's also noisy in the sense it uses a lot of punctuation characters, which is common for Prolog and many functional programming languages. But it's extremely concise and there are things that I feel Erlang in its syntax does, they do more elegantly than Elixir.
00:04:41
Speaker
Do you have an example? For example, if you take a function definitions, like in Elixir and Erlang, a function can have multiple clauses. And in Erlang, I feel like, for example, if you have to write several clauses side by side, Erlang is going to be much more concise than the Elixir one. And I actually prefer Erlang one, right?
00:05:07
Speaker
And then, yeah, it's something that when you see side by side, you'll be like, oh, yeah, the Erling one, there is less.
Metaprogramming and Encoding Decisions
00:05:14
Speaker
There are definitely, you have to type less, let's say, right? It's more concise. And so for me, it was more like it's love Erling.
00:05:26
Speaker
And it was more about there were some features that I felt was missing as part of Erlang, but not necessarily as part of. There were some that are part of the language. And we can talk about them. So like everything being UTF-8 encoded by default. When I started doing Erlang,
00:05:46
Speaker
Erlang was one particular kind of letting, which I believe I could not even write my name correctly without adding special notations because I have an air with accent, right? Yeah. So there were some things at the syntax level, but it was mostly about like as a
00:06:05
Speaker
abstraction level, right? And there were two things in particular. So one was metaprogramming, and the other one was some sort forms of polymorphism. And the thing was that, so Elixir actually, if you go back to the repository,
00:06:23
Speaker
It started in 2011 because I was mostly playing with ideas. So the first version of Elixir was actually object-oriented because I was coming from a Ruby background mainly. And I was like, well, I'm missing those things. And then I was like, well, I think what I'm missing is objects. And it turned out, spoiler alert, it was not objects.
00:06:46
Speaker
But i felt like this is what i'm missing so and then i tried explore a couple different things and then like a year later i landed on the design but one of the things in this process that like look i want metaprogramming and there are several ways of doing metaprogramming and throughout the journey i was like look.
00:07:04
Speaker
I got to the conclusion that I want a Lisp-style metaprogramming. And the idea is that you can treat code as data. So, macros mean different things in different programming languages, but in Lisp and in Elixir, in Macros, I can actually receive the, when you call a macro, the macro can actually
00:07:27
Speaker
access the representation of the code, the codes represented as data, and then the macro can go and manipulate that code as data. And I felt that was really good for Elixir, because Elixir runs on the Erlang virtual machine, and Erlang virtual machine we have the code is compiled, right? So I think this kind of pre-processing happening during compilation through macros, it's a really nice fit to the Erlang virtual machine and how it compiles and runs code.
00:07:54
Speaker
So I landed there, like, okay, I want a meta-programming and I want Lisp-style meta-programming. And then there is the obvious question, why not Lisp them, right? Like, why not create a Lisp that runs on the virtual machine? And when I started Elixir,
00:08:11
Speaker
I am for sure there were two lisps, but potentially three. There was LFE, which is lisp-flavored Erlang, and it's maintained by one of the creators of Erlang. So you mean three lisps that were available for Erlang's virtual machine?
00:08:29
Speaker
Yes, at the time. And there was one called Joxa. And later on, I know there is a closure, which is a closure running on the virtual machine. So people have been trying LISPs on their virtual machine. And then I was like, look, I'm not going to do another LISP. Take the pros, take the cons. That's a whole separate discussion. But look, I don't want to have a LISP. I want to try something different.
00:08:56
Speaker
Yeah. So my idea, which I thought was novel. Okay. Okay. My idea was, look, I'm going to define like some basic syntax and I'm going to add like some
00:09:09
Speaker
some convenience construct. So, for example, in Lisp, you don't have operators, right? If you want to add two numbers, you do plus, three, and five, right? So you put the plus first, then the three, and the five. Everything is written in the same notation in the Lisp. Yeah, so prefix-driven syntax, right? Yes.
00:09:27
Speaker
Imagine that I have this prefix notation, but I'm going to allow some affordances. I'm going to have operators. I'm going to have this and that. I started with a basic syntax and added some affordances on top of that.
00:09:47
Speaker
And while I was doing that, I was like, OK, I don't want it to look like it's not going to be a list of what it's going to look like. And because I had a lot of experience with Ruby, I ended up going making like some Ruby like decisions in that part. But I feel like I also feel like it's kind of over like.
00:10:11
Speaker
overstated, like if you get Elixir and you replace all, so we use do and end, we use words to delimit blocks, right? And one of the reasons I wanted to do that is exactly because I was coming from Erlang, where it used a lot of punctuation characters. So as I said, it's noisy, like the ratio of like punctuation to words, it's kind of high. Yeah, it's a little bit squiggly.
00:10:36
Speaker
Yes, and it's common to functional programming languages. And again, I'm not saying this is bad, I'm just saying that's how it is, right? Some people like it, some people don't like it. And I was like, okay, so what I'm trying to do is that I'm going to try to balance and use do-end blocks to delimit blocks of code. And that's pretty much it. I think you forget Elixir, and you replace all the do-end blocks for curly brackets.
00:11:03
Speaker
It's not going to feel pretty much like Ruby anymore. So I think that's the part that makes it look the most, and optional parentheses, which is another discussion. Anyway, I'm going off. But to summarize, it was not really about the syntax. It was more like, look, I want these particular features. I want meta-programming. I want polymorphism. And then in order to implement them,
00:11:29
Speaker
I had to make a bunch of decisions, and I was like, okay, I don't want a list. So how do I want to look like? And those decisions ended up having a different syntax, but the syntax was not the mean, it was an end.
Syntax and Semantic Separation Benefits
00:11:46
Speaker
In order to get there, I ended up doing that.
00:11:49
Speaker
And the thing that I said that I thought I was coming up with a novel idea, later I learned... So in Elixir, we have this syntax that has a direct translation to AST, to its natural representation. And then I learned that in the paper by John McCarthy that introduces Lisp, the paper that introduced Lisp,
00:12:13
Speaker
It actually has the same idea. So in the paper, he's like, look, we are going to have Lisp, and there it was called S-Lisp for symbolic Lisp. And then McCarthy says, look, but this representation, the Lisp that we use, like that most people understand as Lisp today, he was like, this representation is too low level. We need to have a higher level representation that is going to translate to that one. And I believe it was called M-Lisp.
00:12:41
Speaker
So, so it's funny that later I was like, I paused and I was like, nope, it's there since the beginning. Since Lisp exists. Yeah, yeah, yeah. Lisp rings a vague bell. I remember reading somewhere that Lisp was always supposed to have a different surface syntax, but never really got round to it. Yeah, I believe, I believe that's exactly it. Everybody just went with the S Lisp. Nobody really went with the M Lisp.
00:13:06
Speaker
Yeah. So, so how does that change metaprogramming? I mean, lisp style metaprogramming, because if you've got a lisp, there is almost no difference between the code you write and the data structure you get when you're doing metaprogramming, right? They're almost identical, but you must have a translation.
00:13:23
Speaker
Yeah, so you kind of learn that, look, this code is going to be represented as this code. So this code is going to become this particular daily structure. So there is a translation layer, which, to be honest, it's how most programming languages work. Most programming languages, they have the syntax, and the syntax becomes our presentation.
00:13:48
Speaker
The challenge in Elixir was exactly... And Erlang works like that, for example. It has the syntax and then you have the representation. So the challenge we had for Elixir is what is the minimum language that we need to have
00:14:04
Speaker
So the syntax translation is straightforward. So we are not asking developers to learn 20 different translation rules. So that was the trade-off. We need to keep the surface as small as we can possibly can. Right, yeah. Presumably, one of those rules is
00:14:32
Speaker
If the syntax is infixed with operators, it's going to be changed to a tree, something like that. Yes, exactly. An operator and a function call, even for syntactically, they are different because the operator is in the middle, the function call has the arguments. In the ST, they are represented in the same way. They are all considered a call.
00:14:54
Speaker
So for Elixir, for example, an operator is a function call as any other. While other programming languages, they actually have a distinction between what is an operator and what is a function call. And those are two different things. In Elixir, it's the same. It translates to the same AST. It's just syntactically, it's something else.
00:15:16
Speaker
Yeah, that makes sense to me. That kind of distinction only exists at the syntax level. By the time you get into what's it actually trying to achieve, that distinction is gone.
00:15:29
Speaker
Yeah, and it leads to very interesting things because in Elixir, the syntax and the semantics, which is exactly what you have in Lisp, the syntax and the semantics, they are somewhat divorced from each other. It's not 100% divorce. Of course, they are
00:15:47
Speaker
You need to type things together at some level, but they are divorce. And that brings great features. So for example, one of the things that I like to say, today, there's a library called NX, numerical elixir, that gets elixir and compiles it to run on the GPU. And it's a library. It didn't require any language changes. It's just a library implemented using
00:16:14
Speaker
Alexia existing features. But on the other hand, it leads to some complications, for example. Imagine that in my parser, I find something about
00:16:29
Speaker
like an invalid syntax. There are examples, I can't recall, but let's imagine that I find something like it's an invalid syntax. And then I want to tell the user, look, maybe you wanted to do this and this and that. But because the syntax and semantics, they are divorced from each other, I cannot in the syntax assume that the user wants to do something in particular.
00:16:50
Speaker
I have to assume that look, this is syntax, the user can get these and compile to the GPU, the user can get these and compile to SQL query. So all I can assume are things about the syntax and not much more. Right. So this must play in at the same time to polymorphism.
Protocols and Polymorphism in Elixir
00:17:12
Speaker
Because you've got, especially in a dynamically type language, right? You end up, you're writing a macro. I don't know if you use that term macros for these. Yeah, we do. So you're writing a macro, you get some arbitrary syntax tree that you have to process and you have to say, well, okay, I'm dispatching on, if this is a function, I'm going to do this kind of thing. It becomes inherently polymorphic because you've got this syntax tree, which is carrying lots of different things that you need to walk down.
00:17:39
Speaker
So how does elixir flavored polymorphism look? Yeah, exactly that. That's an excellent question. So there are two things I want to say here. So first one is that what I'm going to say, and a lot of elixir features, they really come from Erlang. So and Erlang, yes, it's a dynamic
00:18:03
Speaker
programming language, as is Elixir, but they are much more assertive than what we usually consider to be dynamic programming languages. Because when you take up general dynamic programming language like Ruby, Python, JavaScript, they're usually object-oriented. Or in general, we can say,
00:18:26
Speaker
For example, you are calling things in objects, right? So you have an object, you say a true number or true string, right? And you don't necessarily know what is that thing. We say things like duck typing, like, oh, I actually don't care what is this object, as long as it
00:18:42
Speaker
implements that choice string function, or method, and that's the only thing you care about. In Elixir, we don't have objects, nor in Erlang, and we have some predefined data types, and we are often just pattern matching on them. For example, Elixir represents its main AST node, it represents it as a tuple with three elements, the function name,
00:19:11
Speaker
metadata and the arguments to that function. So, sorry, a function call, a function call. We represent as a tuple with three elements. And then you can go and pattern match on this. You can say, look, I'm expecting you write literally the syntax for tuples and you pattern match on each exact element. And we have, which is very similar to, you know, for functional programmers when you think like,
00:19:37
Speaker
abstract data types when you create those data types that you define. And then you have to match on them exactly. That's pretty much how it works in Erlang and Elixir, except that you match on any existing shape there is.
00:19:53
Speaker
So you can match on topos, you can match on lists, you can match on our dictionaries. There's a pattern matching syntax for pretty much everything, and that came from Erlang. So we have this polymorphism, right? But the issue with this polymorphism when I was starting using Erlang is that similar to other functional programming languages, so
00:20:17
Speaker
Well, we can pick any other functional programming languages, and I'm going to talk about how we solve this problem. So that's definitely a form of polymorphism, where we're like, look, this function accepts a parameter of different types, and then you can pattern match on those types. If I was using a statically typed functional programming language, what I would do is that I would define a type for the AST, and then for each node in the AST, I would describe all the different nodes individually.
00:20:46
Speaker
I would be able to do that. I will define a general type and define each possible node as part of that type. The problem with this code is it doesn't happen for AST, but imagine that, well, what if you want to add a new node representation to that type thing?
00:21:09
Speaker
It's not really possible, right? But with AST is weird. If you're going to add a new node, a new AST node, you end up adding a new AST node
Challenges of Implementing Gradual Typing
00:21:19
Speaker
to the language. But think about, for example, I like to use JSON as in coding because that's a very continuous case.
00:21:26
Speaker
Imagine that you are building a web application and you have your business, you have your domain type, like imagine this ad commerce, you're going to have a type for representing a cart.
00:21:42
Speaker
a type for representing the shopping items, everything that you model in your business domain. Now, imagine that in order to expose that from an API, you have to convert all those things to JSON. In order to do that, you most likely create a function called true JSON in your application that receives all the possible types that your application knows about,
00:22:12
Speaker
It's going to go. So it's exactly the polymorphies that we're talking about. It's parametric polymorphism because the types they are coming as the possible arguments there. So you may need to have some wrap, you may need to wrap things around, maybe not, but you have to define this huge type that receives this huge function to JSON that receives all the types on your system. So you receive those things as parameters and convert them. That's one of the ways that you can do it.
00:22:38
Speaker
Every time there is a new type, somebody needs to go. If I'm another member of my team,
00:22:44
Speaker
they are adding a new type, they have to go and change that to JSON, right? Which for an application, it works fine, but for an ecosystem, it's kind of tricky because it means now, imagine that I'm working on this application and I need to use a library that introduces like a money type, right? Because you don't want to model money with float, right? You want to have a specific type that models money, currency, this kind of stuff, right?
00:23:11
Speaker
But now that library, when I bring that into my application,
00:23:15
Speaker
I have to handle that type also in that huge JSON function, right? I have to handle it there. But imagine that I also bring a... Imagine my standard library doesn't have a set, right? It doesn't have a set data type. So I bring another library, and I may have now that money type inside the set library. And now I have to go... So it gets to some point where if you want to compose, if you want to say,
00:23:45
Speaker
I'm just going to bring this set and I'm going to provide a way for you to handle JSON.
00:23:51
Speaker
and allow everybody to compose those things together. And you just render JSON. That's the polymorphism that I was missing, right? So in order to recap, Erlang always had its dynamic, but it always had this kind of parametric polymorphism, but it was not extensible. The only way for me to teach a function how to handle my data type
00:24:18
Speaker
would be by forking that code or changing that code. I had to go there and change it. And that limits the creation of contracts because I cannot have a library that says, look, give me any data type that is convertible to JSON and I'm going to work because you don't have this extensible mechanism. You don't have a mechanism where everybody can say,
00:24:38
Speaker
Oh, I'm going to implement this. You cannot define those contracts. So we had the parametric polymorphism, but it's there. It's in a single place. You can add as many clauses as you want to that thing, but you have to change. And the feature that we added to Elixir, it's something that it's very popular today.
00:24:58
Speaker
because it's equivalent to Go interfaces, it's equivalent to type classes in Haskell, it's equivalent to protocols in Clojure. It's something that is an extensible polymorphism. So I can say things, look, give me any data structure that can do this and I'll be fine. I'll be able to figure out the rest. And we use this kind of everywhere. So for example, even simple things like if our programming language has a repo, has a terminal, right?
00:25:27
Speaker
How do you know how to convert a data structure to print in the terminal? Because if you have a data structure and you want to print in the terminal, you can leave everything hard-coded, but it's a great use case for a protocol. You can say, oh, you had this data structure. You tell me how you want to represent it, and then we can provide
00:25:48
Speaker
higher-level representations of internal data types. We've gone through both of them. Those are the two things that I really wanted to add in Elixir. It was meta-programming and this sort of extensible polymorphism. There are multiple different implementations, but in Elixir, they are called protocols. They bring this sort of polymorphism where I can define
00:26:15
Speaker
data contracts and say, look, you can give me anything as long as you implement this, I'll be fine. Okay. So someone can write a protocol like show on the terminal and the person writing the data structure can implement it or the person writing the protocol can implement it. Yes. Yes. And you're not tied to owning one or other code. You're not tied to owning a specific side of the relationship.
00:26:38
Speaker
Exactly. Also like trait in Rust. And just to wrap this up, it goes back to the beginning when I was thinking like, oh, when I was starting to use Erlang and I felt some things were missing, I was like, I think I was missing objects. I was not really missing objects. I was really missing this particular form of polymorphism. That's what I was missing.
00:26:59
Speaker
Yeah, yeah, yeah. And you have that one, like, I think I'm right in saying like in Java, you have interfaces, but the problem is that it's the object that has to implement it, right? You have to have access to the object source code to create, to use a new interface. I'm honestly not, I'm honestly not sure with 100%, but I mean,
00:27:24
Speaker
I think you should be able to implement an interface for another object, but the issue is, I mean, this also happens in Elixir, is that if you are implementing an interface for an object that you don't own, that object needs to expose the functionality necessary for you to implement that interface, right? Because otherwise, you cannot implement it. You see what I mean?
00:27:48
Speaker
Yeah. And I think in Java, you always end up with, if you don't have access to the object, you have to create a wrapper object that just exists to implement the interface that you own. You own the wrapper so that you can add the interface. But you're saying a protocol-like thing splits that up, so it doesn't matter which side you own. Yes, exactly. But you still need to rely on the data structure or the module that works with the data structure.
00:28:16
Speaker
providing enough information for her to implement the interface, right? Because you cannot, you should not access the internals of the data type. So there is still like some, some boundary, but there is no language limitation is a software design limitation. Right. Yeah. Yeah.
00:28:31
Speaker
So I know that there was some influence on elixir from closure. So I'm going to ask this because closure has this thing called multi methods, right? Yeah, where you can do polymorphic dispatch, but you can also include logic in it.
00:28:47
Speaker
Like if the first argument is less than five, then I am the method that you're going to run. Otherwise look elsewhere. Do you have that? Yeah. So yes and no. So the way that perm, like I was saying, like that to JSON function, the way it works, it works pretty similar to how a.
00:29:05
Speaker
a multi-method works in Clojure, but it's exactly that it's not extensible, right? So you define it once. So we have like, you can create dispatching rules, like, look, if this argument is this and the other one is that we call them patterns and guards. So you can write code like that, but it's not extensible. And I believe Clojure multi-methods, they are extensible. I think I can add
00:29:31
Speaker
new rules to a multi-method anytime. I think you don't need to own them. You don't need to own the definition or the object. You can be a third party. It feels almost a little bit like monkey patching. Yeah, but there's the big thing, which is like namespace clashing. That monkey patch can make it very common.
00:29:57
Speaker
that I don't feel multi-method has. There is another language worth saying here that has these that we are describing, which is Julia. I believe Julia, everything is a multi-method in the sense that if you have a function, you can define new clauses that dispatches on any type at any moment.
00:30:20
Speaker
So that's also very interesting. And closure was one of the influences. The fact that we call our polymer phase protocols is exactly because of closure, because our design is the closest to closure. And multi-methods, they are more general.
00:30:41
Speaker
than protocols, right? So for me, it's like, ideally, I would prefer multi-methods, like in an ideal world, right? I would prefer multi-methods because they are more general, so I would be able to express more things with them.
00:30:58
Speaker
And I think there are programming languages that really do interesting things with them, like Julia. I don't know if this is still accurate, but I think at the time when I was talking to closure developers, they were saying that multi-methods, they are rarely used. That for some reason, protocols, they were preferred.
00:31:18
Speaker
And but the biggest thing was really so that was mostly when I heard that was mostly for peace of mind. Because the biggest thing is that I don't think I can I could implement multi methods efficiently in the early virtual machine. Maybe I could, but it's
00:31:41
Speaker
It's not easy. It's not easy. Maybe today, with 10 years, 12 years of experience in the back, in my back, I can go back and like, oh, maybe I can try to design multi-methods now. But I don't even know if Closure solves it, because that's the other thing. You can have something that is...
00:32:04
Speaker
that does one thing and does not do a lot, but it can have something that is really flexible and then people can use a lot of things and then that may lead to code that is harder to understand or hard to maintain. So how do I ensure that a multi-method is not going to clobber the other one?
00:32:24
Speaker
If anybody can register a multi-method at any moment, how do I ensure that there are not going to be conflicts? Solving those problems, it depends.
00:32:39
Speaker
in which way you allow you to guard those multi-methods because if I allow people to write arbitrary expressions or even comparison operators, if x is more than 5, how I'm going to implement the code that is going to say, look, you are overriding something that the user wrote.
00:33:01
Speaker
and that's going to lead to issues that are going to be very hard to debug because you have library C defining something that library B was expecting to work and now there is one library very far away that is not working as you expect. Those are trade-offs and I think having a more general mechanism can be a powerful feature to have.
00:33:30
Speaker
But I think in order to add something like that, I would also like to have some balance, some checks in there to make sure that users are not going to make obvious mistakes or hard to debug mistakes. Yeah, I totally get this. Yeah, and that's hard work. It's actually hard work to guarantee those properties.
00:33:51
Speaker
Yeah, you open up all those features and then you have to design useful and usable ways to close them back down again, right? Yeah. This is always something that interests me about language designers, like where do you sit on that power spectrum? Because it's been a while since I've been a professional closure programmer, but multi-methods were used sparingly in my time because they're really powerful. You can do anything with them, but they also run the risk of the actual behavior you get is magic because you didn't know there was another definition over there.
00:34:22
Speaker
And it's like pinning down what's your sweet spot for power versus usability. Exactly. And I don't think there is...
00:34:37
Speaker
You know, like the I said joke in the beginning about like midlife crisis. And I think for me, that's exactly that's like a hard part of it because there is no answer. And and then sometimes I'll be like, well, what if I did this thing differently? And then it opens up like it closes the door on so many other things that I actually enjoy. And then I would be like,
00:35:06
Speaker
I wouldn't make the choice. I would still keep things as is. I think the Lisp-style metaprogramming is a really good example because when you have Lisp-style metaprogramming, we're talking about syntax and semantics being divorced from each other. And then there are places like if you are implementing a language server, for example,
00:35:32
Speaker
When I see a plus in 99.9% of Elixir code, the plus means the plus, but somebody can go and actually replace that plus to mean something else, and that makes those tools harder.
00:35:51
Speaker
We're talking about balance. What I did in Elixir is that, look, you can replace plus. But in Elixir, everything is lexical. If you want to replace what plus means, you need to have something in the code that says, look,
00:36:05
Speaker
I am replacing plus. So I can never do things globally. I cannot change the meaning of plus in Elixir globally. There is always going to be a line in there saying, because plus in Elixir is a function, I can say, look, I don't want to import plus from kernel, which is where everything is imported by default. So I can say, look, I don't want the plus from kernel. I want to bring this other plus. So it's there in the code. But now it means that a language server
00:36:34
Speaker
needs to go through the code, process these directives saying, I want you to remove this, I want to add this, and it's more work, right? And then I could say, well, I could
00:36:46
Speaker
I could not have those features and then it would require less work on the language server side. But this feature, the plus, it's a very small feature. It's a plus operator, right? It's what allows us to write Elixir code that runs on the GPU. I can write an Elixir code.
00:37:05
Speaker
that is using plus division multiplication, because that thing replaces the plus. There's explicit calls saying, look, I wanted to replace the numerical Elixir operations by the GPU Elixir operations. It's very exclusive. It replaces that, which for me, it's an amazing feature. It's a very exciting feature to have.
00:37:24
Speaker
And then I'm like, yeah, this is a really tough trade-off. So for me, these checks and balances, look, I want to have this thing be expressive as a programming language, but I don't want to
00:37:44
Speaker
allow somebody to change the meaning, which would be monkey patching, right? Like I know some code and just by loading that code or by adding a dependency, that thing would immediately change what plus means that I don't want, right? But if I say, look, I want to change, I should be able to do that. Yeah. Yeah. It's funny how like once you open the door to meta programming, you've reignited that global variable, global variable versus local variable things.
00:38:14
Speaker
in the global variable, what does plus mean? Yeah. Yeah. And that's why for us, it's like, look, all the macros. So like, for example, you don't even have to, in order for you to call a macro, you have to require it. I cannot call a macro without saying, so in Elixir, like, I believe closures like that, closure Ruby, well,
00:38:39
Speaker
But in Elixir, when I'm going to call another module, many programming languages are like that. When I'm going to call another module, I don't need to explicitly require it. It's not required for me to have an import or something like that. Many languages require you to do that, right? So it's convenient. I don't have code because sometimes they're going to work on projects that have explicit imports. The first 20 lines are verified. It's like all the imports that you have to do.
00:39:04
Speaker
In the leaks, you don't have to do that. But we kept this for macros, because look, I don't want like, if you're going to call a macro, you have to say like, so we have like all those signals of like, look, I am bringing something that is going to potentially change the code. Yeah, if you're going to change the rules of the game, you have to be explicit about it. Yeah, Russ does something similar where I think all macros, they end with an exclamation mark as well.
00:39:34
Speaker
just like being there, like, look, you know, things, things can happen differently than you expect. So it's always, it's always, uh, explicit. And, um, yeah, the option is there, but you have to say, look, I want this. Yeah. Yeah. I want to get into the specifics of, um, numeric elixir in a moment, but there's one more core topic, which we've touched on, which relates to this whole, what do we do at compile time question?
00:40:04
Speaker
Which is like optimization, compiling. For starters, does Alexia compile to Erlang or does it compile to the same bytecode that Erlang compiles to?
00:40:16
Speaker
So Elixir, by design, we compile to, we get the Elixir AST and we convert that to the Erlang AST. So most compilers, they have multiple representations and multiple passes, right? So it's like, it starts with the high-level AST and then it's at some point transforms to something else that is better to do some optimizations. And then at the end you have bytecode,
00:40:48
Speaker
I think in Erlang, you would have... If you want to target the Erlang virtual machine, you realistically have three options, the Erlang AST, something that they call the core Erlang AST and the bytecode. And I chose to target... And you could actually compile to Erlang source. That's what Glyn does, I believe. They actually compile to...
00:41:12
Speaker
through a textual Erlang file and not AST. But that has some very big limitations, like your lines are going to be off in stack traces, right? So when you get an exception in Gleam, I believe, the stack traces, they do not match your actual code.
00:41:30
Speaker
Which is something that, yeah, like the JavaScript community, they had to solve those issues. That's why you have things like source maps. Yeah, so I think those are the four options. And we chose their link abstract format because it allows us to have like proper stack traces and everything, proper messages, but still share as much as possible, as much as possible tooling as air link.
00:41:59
Speaker
And the reason we did that at the beginning was because there were like, so Erlang has a tool called Cover, which is for test coverage. And the test coverage maybe changed today, but the test coverage, at least back then, it would analyze the Erlang AST and add annotations for test coverage. And if I didn't target the Erlang AST,
00:42:27
Speaker
I would have to implement my own cover version. There's also a tool called Dialyzer, which is a discrepancy analyzer. It's kind of a type system, not really, but it aims to find bugs in your Erlang and Elixir code. And Dialyzer as well, I believe
00:42:50
Speaker
I believe at the beginning it required the Erlang abstract format. So in order to maximize the tool, what we could use from Erlang, we chose the Erlang abstract format. But later on with time, I wanted to have the option of like, well, maybe I want to compile to core Erlang in the future. And then I would be able to do things that I cannot do today.
00:43:17
Speaker
So I would get a little bit more expressive power. So with time, I actually made a bunch of improvements. I submitted a bunch of improvements to their link compiler tooling. So it allows programming languages to say, look, I'm going to keep my own representation of things.
00:43:36
Speaker
And if you want something else, I'm going to give you that something else. So I came up with a contract between programming languages and tools. So we have the flexibility to choose, but we never really ended up making the change. We still target Erlang abstract format, but we have done work to be able to say, oh, if we want to go lower level in the future, we can as well. Okay.
00:44:04
Speaker
I suppose one huge advantage to that is just like you were saying with elixir macros, right? If you can turn, if you know how, if you've got an intuition for how the syntax of elixir turns into an AST, you can write macros. I guess you develop the same intuition of here's the Erlang source code I want to ape. I know what AST that will represent, so I know how to compile it.
00:44:28
Speaker
Yes, the translation layer, it's like it's really straightforward between one and the other. Honestly, the hard, the only tricky thing in the whole translation layer is the difference between how Erlang and Elixir, they do variables. That's potentially the biggest difference between the languages. So in Erlang, the variables, they are
00:44:57
Speaker
you can only assign them once, right? So if you do x equals 1,
00:45:01
Speaker
you cannot do X equals show later, right? And that's good. There's a long blog post I wrote, like comparing the variables and the pros and the cons. Alex here does allow rebinding. And one of the issues why we actually allow rebinding is because of macros, but it's a separate conversation. But you can think that a macro may want to pass some information around using the same variable. So you need to rebind it and you cannot
00:45:30
Speaker
You cannot generate a new name every time, because then you have to find a place to track the new name. There are some complexities there. But it was one of the reasons. But also because I saw that in a lot of places in early in code, because again, everything is immutable. So you are transforming everything. And sometimes you are transforming the things, but it's still ultimately the same. So in early in code, you would have things like
00:45:57
Speaker
post zero, post one, post two, post three, because you have to version the variables because you cannot reassign them. So that's a very big difference between the languages, between the two. And the other one is that the
00:46:17
Speaker
I don't like the variable scoping in Erlang. If I have a case or an if, and if I change a variable inside that scope, that variable may actually be available outside of that scope as long as all clauses assign the same variable.
00:46:40
Speaker
If not all clauses assign the same variable, then you get a compilation error. So there is this thing like, I can be on the inner scope, and changes to that scope may affect the parent one, and Elixir doesn't have that. So in Elixir, if you are inside a conditional, any change in the variable that you want to make inside that conditional,
00:47:03
Speaker
you need to return as a result of that conditional. So like if K is like sweet statements, all those things in Elixir, they define new scopes. And I think this is, and not many languages do this, by the way, like very, very few of them, functional programming languages, they tend to do that. And I think it's a great feature because it makes refactoring so easy. If like, if I have like some piece of code, right? And if I have a conditional,
00:47:32
Speaker
and I want to move parts of the conditional to a separate code, I can just get that code and move it and pass the same arguments in. If I have a conditional that is setting a variable that is changing the outside scope, right? It's changing the current scope. Now you have to come up with all, now you're like, oh, but this variable was here. Now I have to pass it back. You have to do all the work. And in Elixir, it's like, oh, you want to move code around. It's always inputs and outputs. That's the only thing you worry about.
00:48:01
Speaker
There is no scope changes. Kind of what conditionals and control flow to have the same semantics as functions, right? Yes, exactly. Yeah. Cool. So before we leave the foundations of Elixir, you're kind of making it seem like all of your early day work was in the parsing and translation world. Is that what makes a project like Elixir hard?
00:48:32
Speaker
I think the early days, yes. I think the answer is yes, like the early days. I was trying to think what is early days. But if we take the early days, really about like for me, I would say like that was the most intensive period of like language design, let's say. It was exactly like figuring out what the language is going to look like.
00:49:01
Speaker
and which features that I want and how those features map to the Erlang virtual machine. And how can I implement those features in a way? Because the reason for Elixir to exist is the Erlang virtual machine. So I want the features to be done in a way that is
00:49:21
Speaker
Possibly, at least as far as I know, the best way to implement those features, right? And I think generally speaking, we're able to achieve that. So I was saying about UTF-8, like today, Erlang, so Elixir had since very early on, I can uppercase my name and it will uppercase the accent in my name correctly. It's going to change the ad with the accent. It does a bunch of Unicode things correctly.
00:49:51
Speaker
Later on, Erlang pretty much got our implementation of Unicode and added it to their standard library, implemented differently because in Elixir, we can use meta programming. Erlang, they do it differently, but the idea is the same. That was the hard at the beginning was, look, I want to have these properties.
00:50:15
Speaker
I want to have metaprogramming, and the reason why I want to have metaprogramming in polymorph is because I want to have an extensible programming language, and an extensible programming language is going to be good because people are going to be able to get the programming language and run to solve a bunch of different domains.
00:50:35
Speaker
They're going to use it for web, they're going to use for GPU compilation, they're going to use for data processing. That was the idea. And then figuring this stuff out, it was a lot about the parsing, figuring out the syntax, what it works and what doesn't. But from there, after this initial period, let's say the first two years, it was exactly because the language was designed to be extensible.
00:51:01
Speaker
It's like now the huge majority of the work is actually in the language and in the ecosystem. So I spend most of my time actually working on Elixir libraries and frameworks than Elixir itself. Because the language is meant to be accessible, the language is not meant to be a bottleneck. I do not need to ask permission, right? I do not need to ask around. I can go and extend the language, bring it to different places,
00:51:30
Speaker
and explore different ideas with it. And so at some point when the foundation was there, most of the work is in like, then I was mostly, let's say I was writing Elixir code. Like most of this in the library, it's implemented in Elixir. Most of the tooling is implemented in Elixir. And from that moment on, I am mostly writing Elixir code.
00:51:58
Speaker
That's a good endorsement because, I mean, if your motivation is to write the language that you want to use and you moved relatively quickly to actually using it rather than tinkering around with the project, you know, you can so easily go down a rabbit hole of language design, but you sound like you fairly quickly popped back up to the point where it's your day-to-day tool. Yeah. And there are things also, like there are some people, I remember, you know, in the early days when trying to
00:52:28
Speaker
promote a programming language. There are many pointless discussions. And there is one where somebody was saying, I could push this further. I could say, look, I could write all of Elixir in Elixir itself, which we call, I could bootstrap the language. But to me, Elixir is not bootstrapped. And then some people would say,
00:52:55
Speaker
I don't think any, like some people say, I don't think a programming language that is not bootstrapped is worth using, right? And then I'm like, well, we did like, I don't know what is the proportion, but it's probably like 95% of the Elixir repository is Elixir files, right? The rest is like shell files.
00:53:20
Speaker
Erlang files and so on. We could have pushed it forward, but for us, it's like, look, I want the language to be extensible. Those are the things that I need. From there, I'm like, okay, I can go now and I can build the rest. That's perfectly fine for me. We don't have to rewrite everything else in Elixir. I actually enjoy writing Erlang as well, so I kept doing a little bit of that.
00:53:46
Speaker
Yeah, you don't want to lose the ability to write it when you've got to be able to translate to its source tree, right? Yeah. Yeah. Okay. So that leads into the thought that, I mean, you said to me before we recorded this, that you think like about five years ago, Elixir was basically done.
00:54:05
Speaker
Yes, but you're still working on it. So where's the work happening today? Yeah, so I did a presentation saying exactly like I think Elixir as a programming language is done and kind of hinting that most of the work now is going to happen.
00:54:25
Speaker
in the community, in the ecosystem. And this was already happening when I did this thing. It's like, I don't think it would really have caught people by surprise when I announced that, in the sense that it's like, that's how things have been doing. And like, I feel like since Elixir 1.6, which at this point was probably
00:54:50
Speaker
seven years ago or something like that has been quite some time. It was already clear that what we were doing at that moment is that when we were improving Elixir, it's not because of like, oh,
00:55:04
Speaker
We want the direction to go there, like the direction was the, sorry, we want the language to go to a particular direction. Like the language already was where we want it to be. But now the issues we're running to where like people saying, look, we need a code for matter because they're starting to have more and more teams using Elixir and we are going to pointless like.
00:55:24
Speaker
formatting discussions we want to have a code for matter and people and people are like well we want we want we want to just like there was like we want the elixir core team to write a guide telling us exactly how to write elixir code you know and then i was like well
00:55:38
Speaker
I would rather just automate them. If everybody's fine with saying, look, we accept somebody telling us what to do, I was like, okay, let's accept that and have a tool do that for us. I was not convinced at the beginning, but I'm fully convinced now. Then people are like, well, we need to deploy the Elixir experience.
00:56:06
Speaker
And we want to improve the deployment experience in Elixir. And then we worked on improving that experience as well. So it has been quite a while that we're like, look, there are things that we want to make that need to be part of the language.
00:56:29
Speaker
We need to define a rule. What is part of the language and what is not part of the language? We keep in mind that we want the language to be accessible. The rule that we came up with is that we only add things to the language if we can
00:56:50
Speaker
if we can show that the only way for that feature or to be implemented or to be maximized is by making it part of the language. Making it part of the language is going to bring special benefits for the feature. Because again, it's meant to be extensible, there are very few things that
00:57:16
Speaker
fit this rule. For example, we don't have a JSON library in Elixir because I can just add a JSON package and it's the same. Me adding a line that brings a package and me using the package from Elixir, it's going to be the same. The user experience is going to be 100% the same. The difference is one line of code that I add for dependency.
00:57:38
Speaker
And that's how we have approached most things. But for a thing like a formatter, it makes sense for the language to say, that's how it should work. Because if the goal is to have a consensus, a unique view, like everybody formats code this way, comes from the language.
00:57:57
Speaker
is good. Then the other things that we add are things that exist in R-Link, and it may make sense for us to expose in Elixir because it already exists in R-Link, so it's already there. The hard work is done, so it doesn't make sense for us to expose it in Elixir. Most things we say, it doesn't make sense. You can just use the R-Link version directly. But some other things, it makes sense. When talking about deployment,
00:58:25
Speaker
AirLang already has this feature called releases, which it gets your application, the runtime, everything, puts everything into a single directory that you can ship to production. You're like, well, AirLang already has that, so let's build on top of that and provide the same experience. I was talking about JSON. AirLang is actually got JSON a month ago.
00:58:47
Speaker
And now, we are probably going to expose that in Elixir using protocols. So now we are getting JSON, but the motivation is not... The motivation is simply because, well, now Erlang has it, so it makes sense for us to have it. And the reason why Erlang has it is because one of the protocols used for telecommunication, because Erlang has the root in telecommunication... Yeah.
00:59:10
Speaker
they added a JSON encoding in the coding, like a very old standard. I think it's ASN. They added JSON support and our language is like, well, we need JSON support because I don't know, 5G or 6G somewhere is using JSON in there. And now it's getting to our language. And then we're like, okay, now we're going to have it. But we were able to define the rules of what should be part of the language. And then we're like, well, we tackled
00:59:40
Speaker
we really tackled most problems that we were supposed to tackle, and it makes sense to be part of the language. The language, it's close to done.
00:59:52
Speaker
And most of the work should be in the ecosystem. And that was like five years ago, and I was 100% sorry, and I was 50% right and 50% wrong. And I was right because I feel like most of the exciting things that have happened
01:00:12
Speaker
in the community, with Elixir, happened in the community. The whole machine learning effort, Elixir compiles to the GPU, live view, we have audio and video streaming. There are so many things happening, and those are all happening in the community. We are not changing the language to make it happen. This is all work happening around Elixir, and that's fantastic.
01:00:35
Speaker
But the reason why I got it 50% wrong is because maybe a year and a half after I said that,
01:00:48
Speaker
Um, so I have like, so one of the things that people always told me, not always told me, but as we solve like those problems with elixir, when you would ask somebody like, Hey, what needs to improve in elixir? A lot of people would say we need typing. Right. And, and like really majority of people. So, uh, and majority in the sense of like.
01:01:09
Speaker
If I would go to an Elixir event and I would ask who wants Elixir to have more type... I did not say specifically static type system, but I want Elixir to go more towards the direction of types and explore something in that direction and have better tools for developers.
01:01:26
Speaker
Like really, like the more than 50% of the room would have their hands up. That surprises me. Because it's like one of those big dividing lines that you decide, a lot of people decide which side of that line they're on by the time they've chosen the language. Right. Right. And I think, yeah, that's a very good point. And I think there are a couple of things that make
01:01:51
Speaker
And okay, I have not thought about that. And I think one reason why I would speculate on that, why it's so big for Elixir, it's true reasons because I like to say that Elixir code is already assertive. So as I was saying, we do not have objects where I'm calling true string.
01:02:10
Speaker
We do have that as protocols, but protocols, they are explicitly defined, it's like an explicit contract. And when we have the other polymorphies, like the JSON with multiple clauses, you know the type of the things that you are receiving are explicitly saying, look, I'm going to have a user, I'm going to have a customer. So we don't have types, we don't have static typing, but we are matching on the types all the time.
01:02:37
Speaker
So the concept of types is very much there. It's the checking that's happening. Exactly. So the code is so assertive that the tool that I was mentioning, like Dialyzer, we have a tool called Dialyzer, and it stands for discrepancy analyzer. The way that it works is that it looks for your code and tries to find things that it can prove
01:03:01
Speaker
that can fail, right? So if this tool finds, and this is just looking at the code, it doesn't use the notations. It just looks at your code and say like, oh, from looking at this code, I can find like, because we are putting the types everywhere, our code is assertive, we can actually have tools that look at the code and say like,
01:03:18
Speaker
I can see here that these cannot possibly be true. So I'm going to to emit a warning, right? While type systems, they're like, well, these may not be true, right? Like, so what is like the types is like, oh, these may not be true. So I cannot prove it. So I'm going to emit a warning. And this too is like, look, I can prove that there is 100% an error. And that's when this tool emits the warning. Right. Yeah. Yeah.
01:03:45
Speaker
So we have this tool, but this tool has some issues. So Dialyzer, it has some issues. So one is that the error messages, they are really hard to understand. The error messages, they are not precise. And it's a separate tool that you need to run.
01:04:04
Speaker
And it takes a while to run, right? So I like to say that, I think, I think that if we got this tool, we made it part of the compiler, made it fast, and with good error messages, I think like my gut feeling is that the majority of the community would be happy with that, right? Right, yeah.
01:04:25
Speaker
So I think the situation where a lot of people want it is exactly because the types are everywhere. And there is a tool that when we look at this tool, we can say things could be better. We could improve this tool. It's like we are circling around the swimming pool and not jumping. So I feel like that's kind of where we were.
01:04:55
Speaker
And then two master's students in Uruguay, they sent me a paper because they were exploring gradual typing for Elixir. So I was interested in typing, but I did not know how to move forward. So I got this paper from those master's students, and I was like, oh, this is really exciting. And I was like, should we try to find a way for us to work together and bring this forward?
01:05:20
Speaker
And then they were like, nope, we are done with university. We don't want to. It's nothing about you. We're just done with university. They didn't want to become barely paid post grads working for you. Yes. So they're like, we are done. But then when I went through this paper, most of the references came to the work of Giuseppe Castello.
01:05:46
Speaker
which is a very known research in the era of stack typing. And so there was a lot of links to his work. And the thing is that when I did the announcement that Elixir is done, I actually tried to implement a type system for Elixir. And the hard thing about implementing a type system for an existing language is because type systems,
01:06:11
Speaker
inherently, they constrain the type of code that you can write. You can write code that you think, look, this feels valid, but the type system is like, I cannot prove it. I'm not going to type check it. You cannot run it. But when you have an existing language with all the semantics already in place,
01:06:29
Speaker
You do not have this option, right? The semantics are in place. So you have to find a type system that suits the language. And if the type system is all the time saying, oh, I can't do that, right? You're either going to end up with a type system that is not really doing a good job.
01:06:45
Speaker
Or you have to end up restricting the language and say, oh, you can no longer do that when you are using types. And then if you do that, if you restrict the language, is it still the same language, right? It's like, if I cannot use all the features that exist in Elixir, like what is the percentage, like which features, what is the percentage of things that I can do until I have to say, ha, this is no longer Elixir. This is something else. Like this is a subset of it.
01:07:13
Speaker
Yeah, it's that discrepancy between, there are rules to the language, and then there's the language that describes those rules, which is the type system. And if there's a, but if you start with the language that expresses the rules, you end up with a language that is restricted to conform to that type system. If you start with a language that just organically evolves its rules, is your rule writing system going to be expressive enough to describe what is?
01:07:41
Speaker
Yes. And, and so when I started, so when I did this prototype of a type system, um, and I was like completely fresh to this, I was like, and I talked to some people to some, uh, a type system research at the time. And then, and I found this thing called intersection types at the time.
01:07:59
Speaker
And then I was like, look, I did some things. And not only the type system there, it was very restrictive. It would have to restrict a lot of things. But if I wanted to have something like type inference, for example, it would be very expensive. I was like, I built a prototype. And then I went to talk to John Hu, because I showed him his prototype. I was like, look, I implemented this. And it's kind of working.
01:08:26
Speaker
And I implemented intersection types, which he was familiar with. And then I was like, it's kind of working, but it's super slow. And then he is like, yes, intersection types are known to be very slow for inference. And I was like, I've read so many papers, and I don't remember a single paper mentioning that. But anyway, we're like, look, so we did this whole thing. There was Eric from the Elixir team as well investigating those things.
01:08:55
Speaker
And we were running into issues. And then so I got this paper for the gradual typing. And then I read the references, right? And then it was Giuseppe Castaglia in the references. I was like, okay, let me read some of those papers. Let me learn more. And then it was hilarious because I started reading the paper. And the paper was like, look,
01:09:18
Speaker
In this paper, we are going to show how to solve problem X, Y, and Z. I remember exactly that X, Y, and Z was one of the problems that I had when trying to implement a type system. He described the problem perfectly. Then I got so excited. I was like, all right, this is it.
01:09:38
Speaker
We got this. And then the paper is like, well, so it introduced the problem, right? And then the paper is like, well, we can formalize this problem with these mathematical properties. And then it goes to mathematics. It designed the solution in the mathematical world. It proved everything is correct. And then it's like, here, we show it works. And it never went back. It never went back. Like, hey, here's how you implement this. This is how you're going to look. It's going to look important. Yeah, yeah, yeah.
01:10:07
Speaker
Right? So, I sent an email to Bepi, to Giuseppe, right? And telling exactly this story, he had like a good sense of humor about it. And he's like, yes, you know, it's like, you know, like, when we write those papers to publish at conferences, that's what they're interested in. They're not interested in the implementation. Yeah, yeah. But I do have a paper talking about implementation.
01:10:33
Speaker
And we started talking about, oh, maybe we should find somebody to work on this.
01:10:41
Speaker
And then we got a PhD student, Guillaume Duboque, to research, like, can we actually implement a type system for Elixir and what we can express and what we cannot express? What are the trade-offs? And we published the paper actually going through all those things. Like, look, we've mapped a huge part of the language because that was the biggest challenge. How can I map
01:11:06
Speaker
a huge part of the language to a type system. We were able to do that in parts where the theory was not there, like Guillaume and Beppe, they wrote a new theory. There's a new type system theory that exists because we want to have a type system for Elixir. And that's the 50% that I got wrong because now I've
01:11:30
Speaker
Possibly compared to the last five years, now is when I'm working the most on Elixir because we are implementing a type system for Elixir. And the Elixir release from three weeks ago was the first Elixir release to have the type system features.
01:11:49
Speaker
I say joking. We say that there are a lot of things that we can talk about the type system. One is that it's a gradual type system because it needs to interface the static world or the dynamic world, but it's not gradual-like type script. Because when I say a gradual type system, most people think about type script. It's actually quite different. But it's a gradual type system.
01:12:15
Speaker
But I say that we are implementing it gradually as part of the language. So it's like a gradual gradual type system. And I said this as a joke, but it caused a lot of confusion. So I'm, I'm not supposed to talk about it anymore, but here I am talking about it. And, but the goal is exactly that. Uh, so again, like going back to elixir cold, being assertive, we can actually get, can get a lot of type information.
01:12:40
Speaker
from the code that exists today. So the first two or three releases of the type system, we are just looking at the code you already wrote, gathering type information from it, and doing type checking.
01:12:52
Speaker
So what it means is that we are going to run static analyzer code. You don't have to change a single line. You don't have to add type annotations. You don't have to do anything. And we are going to try to find bugs in your code for you for free, right? And the idea is that if this works, then maybe we can think about introducing type annotations.
01:13:15
Speaker
But the idea is that we want to try the type system, to implement it, see the performance impact in large code bases. We want to work on the error messages and make sure that this initial experience that we are giving to users, that it is a really good experience. And then if it works, and then we will gradually introduce
01:13:42
Speaker
more features, type annotations, and so on to developers. So yeah, so it's a lot of work. We are going towards the direction of being a statically-typed programming language, because if you have a Gradual-type system and you don't use a dynamic type anywhere, it is a statically-typed programming language. But it's still
01:14:07
Speaker
a few of your ears of work. Right. Yeah, it sounds. But would you say you've already reached, I mean, if your goal was to be faster and more intelligent than Dialyzer, do you think you've reached that milestone? So, uh, yes. Okay. So the answer is, is, is no.
01:14:33
Speaker
Not right now. We are not there yet. When we released this new version, people were like, oh, Alex actually found bugs in my code. It's good. We are starting to find bugs in people code bases without dialyzer wood. That's good. I think we can do part of what dialyzer does, but we cannot do everything that dialyzer does.
01:15:03
Speaker
So I'm going to explain exactly what I mean. So the goal of Dialyzer is to, it's going to say, look, it's not a type system. It's like, if I can prove with 100%, if I'm 100% sure that there is a bug in here, I'm going to emit a warning. So they say that Dialyzer is never wrong. If Dialyzer has a warning, it's because there is something wrong with the code. As we said, a type system is not like that. A type system is going to say, look, I think there's something fishy going wrong here.
01:15:31
Speaker
Right and or I cannot prove this is correct. Maybe it is maybe it's going to be fine at runtime But I don't have enough evidence to say this will work. Yeah, so that's why we do we're actually working on a type system Which means that we will not be able to catch all the bugs that dialyzer catch. Okay, right? but it means that if you if we add type annotations and you have a static types that
01:15:58
Speaker
we are going to give you the guarantee that we can now prove that that code is free of some typing violations, which is not something that Dialyzer can prove. Dialyzer can prove that there is a type of violation there. Dialyzer cannot prove the code is free of typing violations.
01:16:16
Speaker
Right. While the type system can say, look, this code is free of type violations. So the way I like to say this is that it's like we got parts of Dialyzer. We cannot do everything that Dialyzer does, but we can get a reasonable amount of what Dialyzer does and embed that into a type system. So at some point, I can upgrade that to actually be a type system and get the benefits of a type system.
01:16:46
Speaker
The way it works, it's actually very interesting, or at least I find very interesting. One of the things that we usually have in most dynamic programming languages is that we have unions. We say, look, this code can accept an integer or a string. The way you handle this is that in JavaScript, you can do type off to check for something in particular. You can check if a method exists, but you can pass unions around.
01:17:16
Speaker
And in a type system, the way it works is that if I have a union, the type system, if I say, look, this thing can receive an integer or a string, you have to write your code in a way that you, so imagine that you have a function that says it returns integer or a string.
01:17:39
Speaker
The caller of that function needs to handle both return types. I need to say, look, if I got an integer back, I'm going to do it like this. If I got a string back, I have to do it like this. So the unions, I have to deal with both sides of the union. And even if at runtime, even if I look for the argument that I'm giving, that thing is never going to return a string. You are 100% sure.
01:18:09
Speaker
Yeah you have to but you have to say look i'm getting a string i may get a string here you can even make the thing raise and blow up like i was not expecting to reach here but you have to handle those things you are required to handle that.
01:18:21
Speaker
The way that gradual typing works in Elixir-type system is that dynamic, so we have a dynamic type. Usually, the way that dynamic type works in Gradually-type languages is that it's a wildcard. It's a joker. It's a way of saying, look, this thing can be anything. It's dynamic. I don't know what it is.
01:18:43
Speaker
It can be anything. In many cases, it is the top type. It can be anything. Anything is dynamic. The way that dynamic works in Elixir, the type system, is that dynamic is actually a range. Dynamic is not like the top type. Dynamic is a range. I can say, if I have an integer or a string, you said it has to handle both. But if I put dynamic on this union, it says, look,
01:19:12
Speaker
you don't have to handle both. You have to handle at least one of them. Okay. So my code, if I put dynamic around,
01:19:22
Speaker
My code is going to be like, look, if I deal with integer, it's going to be fine. If I deal with string, it's going to be fine. If I deal with both, it's going to be fine. But if I don't deal with any of them, then I'm going to emit an error because I know it's meant to be one of those two, right? So if it's not one of those two, I'm 100% sure that it is an error.
01:19:43
Speaker
And that's exactly how Dialyzer works. What Dialyzer does is like, oh, you're not handling those two possible things, right? So this was very exciting when we realized that like, wait, what?
01:19:59
Speaker
We can actually do these kind of things because exactly what I said. So if you want to have the static typing code without dynamic, it's going to work like a type system. But if we want to do things like inference, where I don't want to bother the user and say, oh, there may be a bug here that I cannot prove, I can have the dynamic going around, and that does not mean, oh, I'm giving up on everything.
01:20:24
Speaker
it means that i will still try to find some bugs if i can prove that the bug is there so i find like this very exciting because don't like
01:20:37
Speaker
I mean, I don't want to sound like, oh, you know, but like this, like we, there's nobody exploring this. Like we are the only ones that are exploring this right now, which can be really great, right? Because it's like, maybe we found something really great that is going to perfectly match Alex's semantics, right? And it's going to be very exciting. Might change the future of other systems. Yeah, but it can also be a disaster, right? It's like, oh, like we think that we have something great. And then like three years from now it's like, ah, like, you know,
01:21:06
Speaker
We reached the dead end. We were the only ones exploring this road. So there's nobody we can ask help for, right? Because we are the only ones here. But I think it's exciting in thinking about these problems and how we are going to communicate this with the developer.
01:21:29
Speaker
trying to get the right trade-offs in place. There are a lot of decisions to make here, but I think it's exciting. Yeah, you're kind of going from mathematical typing to
01:21:42
Speaker
Police typing, where you don't have proof, but you gather evidence as you go. Yeah. And you know, like the interesting thing about this, so the type system that we are doing, we are talking about mathematics and the type system that we are doing is, it's called set theoretic types. Okay.
01:22:02
Speaker
And what I really like about them, it's because everything is done with set operations, everything. In particular, TypeScript, it has unions, intersections, and negations. TypeScript has those things, right? And our type system has something like that as well. Other programming languages like Racket,
01:22:30
Speaker
type of racket have like union intersection negation as well. But the way that our type system is implemented differently, because it uses something called semantics of typing. And the idea is exactly like the whole type system, it's implemented on top of these operations.
01:22:49
Speaker
For example, if I want to check if two types are compatible, I'm calling a function. Does this function accept what I'm expecting? The way I do this is that I represent types of sets. I get the intersection of those sets. If the intersection is empty, then it means that type checking is not going to work. Those sets, they have nothing common. I like this because
01:23:18
Speaker
I feel it's more straightforward for us to implement and discuss the ideas because everything is about intersections, union, and negation. A simple theoretical model that obviously translates into code. Yes. There is something that is very complex, which is that I have to represent
01:23:42
Speaker
I have to represent all the Elixir types in this set and define... It needs to be mathematically proven when I do the union or the intersection between those things that needs to exhibit the correct mathematical properties. But that thing, once I implement that data structure, so that's where the mathematical is. It's implementing that data structure.
01:24:07
Speaker
We'll say implement that data structure the semantics, right? It's obvious semantics that we can all reason about its union intersection and negation which means like When so like Beppen Guillaume, they are the one working on the data structure so they kind of like give the data structure to me and then now I can go and start using that in the Elixir compiler and then I'm like I'm just asking like is this an intersection or not? Like it's really straightforward for me to
01:24:37
Speaker
To use it and that's another part that makes it exciting and it's exciting like I don't know I I don't know how to explain this right like but the dynamic thing that I was talking about is also defined as
01:24:51
Speaker
reveal that data structure with all the mathematical properties that you expect from it. It's really fascinating. I cannot understand how it all works and proves, but if you get any
01:25:10
Speaker
got a programming language that has a hash dictionary or a map. Internally, the representation of what that thing is doing in memory, it creates buckets. It uses bitwise operations. There are so many things happening there, but when we use, we don't care. The complexity is abstracted for us.
01:25:30
Speaker
It's exactly like that. And, and it makes me happy because, uh, if I had to do like, uh, uh, I would be like, you know, if I had to do like a PhD to understand the mathematics to do all those things, right. Uh, then it would slow everything down. But you've got the surface level tool that you can just use as a programmer. It's funny how you've gone from being a language designer to someone who thinks the job is done to someone using a different library to become a language designer again.
01:25:59
Speaker
Yeah. Yeah. And you seem fired up about it. Really fired up. Yeah. Yeah. I mean, I, I enjoy those things generally speaking. Like for me, it's like, um, uh, like it's because I'm learning a lot and I think like, that's what makes me, me, me excited about those things. It's like, in a way, like I always had like some concerns, like, you know, um, like, well, maybe,
01:26:29
Speaker
maybe like when I get older, like 12 years, right? I'll be like, oh, you know, like, get fed up with this. But I kind of like, it kind of has the opposite effect, because it's like now whenever I learn something,
01:26:45
Speaker
I am like, oh, it has been there for 12 years. And I did not know this, like the opposite effect, instead of being like jaded. It's like, you know, I'm like, oh, it has been there since like forever. Just waiting to be discovered. Yes, right. And so there is a nice project. So there is a project called Flame.
01:27:06
Speaker
from Chris McCord, the creator of the Phoenix Web Framework. And what it does is that the project, when he released it, it was like 200, 300 lines of code, something like that, maybe a bit more. Let's say less than 1,000, which is not a lot, with docs, right? And what it does is that it's outscaling on demand. You can easily, in your Elixir code, start a new node.
01:27:33
Speaker
send code to that new node to execute it, and it sends the result back. And the reason why this is very easy to implement in the Erlang virtual machine is because Erlang has always supported distributed execution. So distributed execution is part of Erlang, is part of Elixir, right? And then this library is like, well, I can use something like Firecracker, which is something that can start nodes and machines really quick.
01:28:03
Speaker
So you can use something like that to start a node really quick, have that thing do the work for you, like if you want to do something that is expensive and you don't want to do in your main machine.
Elixir's Operational Simplicity and Versatility
01:28:13
Speaker
So we spawn a machine really quickly, you execute something there, and it gives you the response to the machine shutdown.
01:28:19
Speaker
and you can have a pool, and then you can outscale easily. This is great because it gives you the level of where you want to always carry your code. Usually, it's like, oh, you want to outscale, then you have to do a Lambda. You can use Amazon Lambda, but then you have to talk to an API, so you have to serialize your request, put it on a queue or send it to Lambda. Lambda gets the result, puts on S3, you read back, there's so much that you want to do. Here's like, look,
01:28:47
Speaker
I'm going to start a node, I'm going to send the data structures that I have in memory because Erlang can do that. Everything, all the Erlang data structures are serializable.
01:28:55
Speaker
I get them back. I'm happy." And I think he released this last year. And then we were talking and I was like, look, we could actually have released this eight years ago. Like, there is nothing that we are doing here that did not exist for eight years, a decade already. It has always been here. And then we're like, you know, this is...
01:29:18
Speaker
Uh, you know, it's like, for me, it's like, it's fascinating. And then it keeps like this kind of like the spirit is like, what else do I have to discover? You know, like, what, what are the hidden things that are, are, are, are waiting for us? Yeah. Yeah. Yeah. This is, this is why I think you can have a whole career in programming that never gets old because there's always more to discover. Yeah. Great. If that's happening, we might have to get him on to discuss that system. That sounds cool.
01:29:45
Speaker
Yeah, he would definitely love to join and there are a lot of things that he would be able to share about Phoenix and Flame in general. Cool. Okay, so one final topic of conversation before I let you go back to all the things you have to work on and discover. Because we've talked about your journey from 2012 to today, but what's Elixir's current state of the journey? Who is Elixir for today? What status is it at?
01:30:14
Speaker
So, like the obvious person Elixir is for is for web developers. I think that's where most people are. And Elixir, the promise of Elixir is that, so today, I think I'm going to use my usual pitch, which is the promise of Elixir is operational simplicity. So today, like most people who are building web applications,
01:30:42
Speaker
Most of them, they are drawing in complexity because you start with your web application and then you add a database. You have to add a database. Then you want to do some background processing. You need to bring a library or maybe another system like RabbitMQ, SQS,
01:31:02
Speaker
to do that thing for you. And then you want to do some real time messaging. So you bring another service, right? Or maybe people they even like sometimes they want to bring authentication and then they have to pay for an external provider to give authentication to users. And like every time you're doing this, you are
01:31:22
Speaker
bringing operational complexity. And we are also like moving, keeping everything away from us, right? Like now I have those three services. So if I'm going to run that, those are things I have to deploy and manage in production. And if I want to run the tasks, I either need to have like a mock
01:31:44
Speaker
version of RabbitMQ or SQS. And then if there is a bug, you're going to find out in production, not in passing. So there are so many complexities. We are constantly relying on external things. We are bringing those external dependencies. We are giving up control. And that being complexity,
01:32:09
Speaker
to the development because now I need to use those three different things, four different sources to get anything up and running. Every project starts with a Docker Compose that is bringing four or five things up to do anything. There is a lot of complexity in there. I have to say that I think programmers, they are the only people who
01:32:33
Speaker
take the word dependency lightly. Like the only ones like everywhere else in life is like somebody says, Hey, do you want this dependency? Do you want this thing that if something goes wrong, you are responsible for it and you have to take care of it? And the program was like, yes, give me, give me a hundred of those, right? Like just right here. So, so I think, so I think that's where a lot of people,
01:32:59
Speaker
are that describes the day-to-day of many projects. The goal with Elixir is that you need Elixir and a database, and that's it. Sure, are you going to use Elixir libraries? Yes, you're using that. But with Elixir and Phoenix, for our web developer, that's what you need. Distributed PubSub, so if you want to do anything in real-time, this user, your friend, join. It's part of the box. Presence, you want to know who is connected,
01:33:29
Speaker
in a cluster and now, oh, your friend joined, your friend is online, send a message, comes out of the box. There's literally no other framework that comes with those things out of the box. It comes with internationalization out of the box. It comes with a dashboard, like a monitoring dashboard because the airline virtual machine is excellent for this. There is a dashboard where you can see all the code running your system.
01:33:53
Speaker
where things are in memory, how much memory is being used. It's all there. There's so much packed out of the box. And with things like there's a project called Live View, which allows you to create interactive, real-time web applications, controlling them from the server.
01:34:11
Speaker
And if you go to my company blog, like dashbit.co, or you go to the Phoenix Framework website, there are a lot of articles about those on how that solution works. Everything's out of the box. So ultimately, you get
01:34:26
Speaker
Control back of anything it's all part of your experience right and when you're going to run into production it's going to scale to multiple cores you can scale horizontally as well because distribution just works so it offers so much and it reduces so much complexity overall.
01:34:45
Speaker
And that's the pitch. And we're not saying, oh, you're getting all those things, but now you have to write low-level code or anything. No. Elixir is a high-level expressive language, as we have been talking about. So I think for the web, it's a really sweet spot in giving you everything that you need, but keeping you productive and being able to ship what you want.
01:35:13
Speaker
So that's the easiest pitch. That's why I attract most Elixir developers nowadays, most developers to Elixir. But I've always been like, we're talking about the language being extensible. It's because I never wanted Elixir
Expanding Elixir's Capabilities Beyond Web Development
01:35:32
Speaker
to be... I knew Elixir was going to be good for web because I came from Ruby.
01:35:37
Speaker
I worked on Rails for several years. I knew people who were following me, they had that background. I knew it was going to be good for web, but I was very careful. I don't want Alexa to be a web-only language. I don't want this to be tailored for web. I want it to be a general purpose programming language.
01:35:59
Speaker
And then with time, we saw many things happening in the Elixir community. So we have nerves, which is about building high-ended, embedded software as well. So there is a whole framework for doing those kind of stuff. There is audio and video streaming and building data-driven pipelines as well. There are libraries for tackling all those sort of problems. And the most recent one that we've kind of talked here and there today
01:36:28
Speaker
was the numerical Elixir effort, which is... Look, we have Elixir. We can actually implement machine learning, deep learning, neural networks in Elixir, get that, compile to the GPU, and run that. And then we have
01:36:48
Speaker
Livebook, which for those who have user Jupyter Notebooks, it's kind of like the Jupyter Notebooks version, but tailored to Elixir and leveraging Elixir Strengths. There is a lot that we have been working on this area, but it's a new area. There are things that I feel like we offer
01:37:14
Speaker
like unique abilities that you don't find anywhere. So I have a presentation called Elixir and Livebook where concurrency, where web concurrency and AI meet. If you put Livebook, Elixir, concurrency, AI, it's probably going to show up. I'll find a link and put it in the show notes.
01:37:35
Speaker
Yeah. And what I do in this talk is that I introduce Livebook. Like the first 20 minutes, I introduce Livebook, our coding notebook platform. And in the next 20 minutes, I live code a web application that is doing machine learning distributed in a cluster. In 20 minutes, I live code that. Oh, that's fun.
01:37:56
Speaker
Yes, and this is not like, oh, this works because I am bringing a separate solution here that is going to serve the machine learning model for me. No, it works because I don't have external dependencies. It's everything running in Elixir. I'm using Elixir to talk to other machines that are doing machine learning, and it's all
01:38:23
Speaker
there, right? Which I think goes directly to what I was saying about Phoenix. There is so much we can do and it's not even, most of it, the credit is not Alex's credit, it's really the airline credit. It's the platform that exists for like almost four decades and we keep discovering new things to do with it.
01:38:41
Speaker
And so that's a new, exciting area. But if you like that, go give it a try. But beware, we are still in the process, right? We are catching up to, I think, I don't know, two or three decades of Python still. So it's a lot of catching up. But I think it shows, there are parts where it already shows where Elixir and this platform can be so great and so unique.
Praise for Beam VM and Listener Engagement
01:39:09
Speaker
Yeah. Yeah. It's funny. I've never known any platform to attract as much love as the beam does. And putting a friendly user space language on top of that is full of potential. You've done your job. You've maybe want to go and watch that video and try it out for myself. So maybe we should wrap up there. Thank you very much for taking me through both the compiler and the language space today.
01:39:36
Speaker
Thank you for having me. I had tons of fun. I really enjoyed talking about this stuff, so I appreciate the opportunity. It shows. Thanks for joining me.
01:39:44
Speaker
Thank you, Jose. I have to say that was a longer episode than usual, and I still feel we could have talked for twice that time. So if you're feeling similar and you're left wanting more, I'm going to put a link in the show notes to our episode on Gleam, which will give you at least another hour's worth of typed programming on the Beam platform. And I'll put a link to everything else we discussed in the episode in the show notes, as usual.
01:40:08
Speaker
On the subject of podcast length, I'm going to have to run a poll, I think. How long is the right episode length? Should it be an hour? Longer? Shorter?
01:40:18
Speaker
I've always gone with the rule that it should be as long as the conversation is and no longer than that, but if you have thoughts, please share them. While you're sharing, if you enjoyed this episode, please leave a like, maybe share it with a friend if you think they'd find it interesting. And if your only feedback is please carry on, maybe consider supporting Developer Voices on Patreon, which we recently opened.
01:40:40
Speaker
You'll find a link to that in the show notes. Thank you very much to everyone who's signed up so far. Thanks to everyone who's persuaded their marketing department to sign up too. That helps. We'll be back next week with more, so make sure you're subscribed for that. And until next week, I've been your host, Chris Jenkins. This has been Developer Voices with Jose Valim. Thanks for listening.