Introduction and Episode Setup
00:00:15
Speaker
So, welcome to Deafen, episode 21. Ray in Belgium, Vijay in Holland. You are in Holland this week, aren't you, Vijay? Yes, I'm in Holland. And Mr James Reeves, over somewhere orbiting around London. Hello. We'll add better applause in the later post-processing. We won't.
00:00:43
Speaker
Don't give away how we produce the show. I mean, you're just giving it away that, okay, we don't do any post-processing. No, maybe we're going to use Wouter, the sound engineer, to put all kinds of cheers in now. Yeah. So James, hello. Yes, hello.
Show Production Humor and Sound Effects
00:01:01
Speaker
It's good to be here. Cheers, mate. Thank you for joining us.
00:01:05
Speaker
So let's dive right into it or I think we we did try dive right into it. We've got to calm down a bit here. Yes. That's why I was I was rethinking my strategy. I think we should start with the most boring shit again.
00:01:21
Speaker
Now, okay, you were telling me earlier on that you like to do Habitica. Yes. And you're trying to stop saying fuck all the time. Yes, I'm trying to not say the F word. Yes, the F word all the time. So I'm trying to do that. So I use Habitica, which is like a online gamification to-do list thingy. So I have a character there every time it loses 1.5 or whatever health, whenever I say F word.
00:01:50
Speaker
And yeah, I think I've been, I'm known to say this thing a lot. So my character is dying a lot, or at least losing a lot of health. So I'm trying to work around it now. But I think James had a good point though, which was you should hook it up to Amazon Echo or to Siri or something. Yeah, or Siri. Yeah. That's true. To keep you properly honest. Yeah, I think I need to buy one of those things. But I'm not sure then, like,
00:02:20
Speaker
The entire, that's a nice way to corrupt their big data. You just connect this to some people and then all they hear is that this kind of crap. That'd be nice. We've got to find somewhere to undermine this internet spying, that's for sure.
00:02:39
Speaker
Yeah, exactly. I think if there was some sort of a movement like that, right? I mean, you log into Facebook and then you just keep posting completely irrelevant stuff all the time just to confuse their... Well, the funny thing is, I mean, you know, I was telling you earlier on that I did a bit of comedy stuff yesterday. One of the things I was talking about was like the Bible as well, actually, and how, you know, to me, as I was growing up, I lost my belief in God because I just couldn't believe that
00:03:08
Speaker
That there would be such a such a possibility of knowing what everybody is thinking Or where everybody is any one time or what they're going to do next and it turns out that it's completely possible
00:03:23
Speaker
And we've got it today. And Google are now God, basically. Like the Christian version of God now exists in Mountain View. And that is pretty scary, you know? Because I always thought it was a bit scary that God knew what we were thinking. You know, that's a horrible concept to me. What
Habitica and Voice Assistants
00:03:42
Speaker
a horrible God that must be. We must be bored shitless as well. But yeah, what do you think then, James? Are you a fan of the internet spying thing?
00:03:52
Speaker
Not really. I don't even find many people who will say, yeah, I'm fine with it. So I am tempted by the Amazon have Edieko and Google have, is it Google Home or something like that? Google Now. So those little things you can put in your home are kind of tempting from a technological point of view.
00:04:20
Speaker
I have a Raspberry Pi on my desk right now, which I was planning on making my own one of these things. There's an open source project called Jasper, where you can kind of set up your own
00:04:36
Speaker
Amazon Echo or Google Now, and then kind of code it so that it works on the commands you want. But I haven't gotten around to it. So that's been sitting on my desk for a few months, waiting. Well, I see your one Raspberry Pi and I raise it with a cluster. Oh my God. Well, now you know your next project. You just have the cluster of Raspberry Pi trying to figure out when you swear.
Internet Spying and Tech Misinformation
00:05:03
Speaker
So this is a Raspberry Pi cluster of one, two, three, four, five, six Raspberry Pis. All I've done so far is running Cassandra on it. For no reason. By the way, just back to the Echo thing as well. Did you hear that Amazon put out this thing where they would take photographs of people and they would give fashion advice?
00:05:29
Speaker
Oh, so now you can be, you know, in your bedroom and you can get photographs of what you're wearing and it will tell you basically that you're an idiot. You should be wearing something better. Having this damn thing in the bedroom, you know, I mean, it's bad enough having it in the kitchen to time your eggs, but by having it to literally time your women's eggs. I mean, this is just, this is great. Why not?
00:05:57
Speaker
Why not? Because it's goddamn horrible. Anyway, James, let's get on to some more interesting shit. The concept of maybe the internet spying on us is probably really boring too because everyone's talking
Google's Omnipresence and Modern 'God' Comparison
00:06:10
Speaker
about it all the time. So let's move on to something more interesting.
00:06:16
Speaker
I think we need to, of course, I mean, James, I mean, most of the people who are listening to this this podcast, they don't need any introduction about you. But it would be nice to tell us how did you get into closure or what is your previous experience? Can I just interject one little possibility there? When we talked to Karen recently and also to Eric, they both ended up, they both had a history of really dealing with terrible, horrible languages in the past.
00:06:43
Speaker
You know, I think it was, was it Pascal or something like that? Yeah. And PHP. Yes. So I don't know. Do you have like a shameful background in some horrible language, James, that you want to admit on A? Well, I did dabble in Java at one time. Turbo language. Who'd host a language on that virtual machine? Oh my God.
00:07:12
Speaker
I'm proud of you James. I have a long list of languages because I kind of hopped. Before I came to Closure I tried to pick up a new one like every six months or so. Before I came to Closure I was learning Haskell which I stuck with for about a year.
00:07:38
Speaker
If you haven't learned Haskell, I'd recommend it because there's nothing that makes you feel stupider than having like five, 10 years of programming under your belt and coming to Haskell and realizing that you're a moron.
00:07:58
Speaker
So I did Haskell for about a year. For the first two months, I... It was undermining your ego so much that you just gave up again. It undermined my ego for about two months. I think that was quite good. It kind of cut down the hubris I was feeling before. I like sticking with things I don't understand, which maybe means I don't understand closure. So I don't know where you have me on.
00:08:25
Speaker
But yeah, so I stopped with Haskell for about a year. So the first couple of months I remember being completely flummoxed by everything and then things slowly started to click into place. And after I had learned Haskell, or at least as far as you can learn Haskell in a year, I think it's kind of one of these things that takes a lifetime to really master.
Programming Language Journey
00:08:53
Speaker
So you got to a point where you could actually print Hello World. By the end of the year, I could print Hello World. In fact, I was actually writing, because this was years ago, back in 2006, 2007, before GitHub, because I don't actually have any of my Haskell
00:09:13
Speaker
code up on GitHub. There was very little software in Haskell then for building web applications. My first kind of big project was to build a HTTP server in Haskell. I got to the point of it supporting HTTP 1.0.
00:09:34
Speaker
And I was working on 1.1 when I decided that I was a little bit bored of the language and I wanted to switch to something else. So after learning Haskell, which is the big, pure, functional, lazy language, I looked at my list of languages and went, well, what haven't I tried? And the thing I hadn't really tried was a lisp.
00:10:00
Speaker
So I went around looking for Lisp-like languages. Scheme was a candidate. Common Lisp was a candidate as well. At the time, Paul Graham had just come out with Arc, which was his Lisp, which was built on top of, I think it was implemented in Scheme, which is now called Racket. And that was interesting. It had like a dot operator for composing functions, which I thought was quite nice from coming from
00:10:32
Speaker
So I was all ready to learn Arc when at around about the same time, Rich Hickey came out with Closure and I worked with Ruby professionally. So I worked with Ruby on Rails and I transitioned from that from .NET and previous to that Java, I think previous to that PHP. So I'm kind of... I'm kind of going up in the world.
00:10:58
Speaker
So Ruby's quite a nice language. The problem is it's mutable and it's object-orientated. But it has a lot of nice syntax. When I looked at Clojure, I went, oh, it's got these nice Ruby-type maps. It's got these nice Ruby-type keywords. So the syntax kind of attracted me to it because it sort of felt like they took the best parts of Lisp, removed all of the cruft, which has accumulated over the past
00:11:25
Speaker
70 years now. It came out in like the 56 or something like that. So he removed all the cruft, added some Ruby-like data structures, and of course it was immutable and functional.
00:11:41
Speaker
So this ticked all of my boxes, right? It was something I was familiar with in terms of, professionally, like Ruby, it had the immutability, which I was now used to from Haskell, and it removed all of the cruft. It was kind of like a clean slate. So it seemed like an actual language to use and I started using it. And since then, I kind of stopped looking at other languages. It sort of became the language I stuck with. No man's squeeze.
00:12:09
Speaker
yes but do you think is there any reason because usually people go in the other direction or at least that's what i thought you know the general programmer's evolution is that you start with a language that that starts with a p you know
00:12:25
Speaker
and something like that. That's probably insulting most of the languages, but then you move into gradually more towards the languages which make you feel stupider and stupider, and then you end up with Haskell or something like that. Because I've seen enough people who got into Clojure, did something with it, and wrote a lot of code, and then decided, oh, wait a minute, static typing can help me more as well than this, so I'm going to switch to Haskell, for example.
00:12:55
Speaker
So how did this transition happen for you from static to closure and then deciding that closure is my thing now? It might have been that I did Haskell first and then closure second, which is why I'm seeing closure. But I think it's more to do with how Haskell and closure
00:13:15
Speaker
approach the problem of stopping things going wrong, right? So I think the evolution of programming languages or what makes a programming language good is how fast you can program complicated stuff in it, but more how do you stop things going wrong?
00:13:34
Speaker
So if you're programming something like assembly or C, you have a lot of things which can go wrong. And as you kind of get to languages that are higher and higher level, they have more and more tools to stop things going wrong. Programming at a higher level itself, like writing less code, can stop things going wrong. There was, I think there's a, it was in the mythical man month where they did some experimentation with
00:14:04
Speaker
what causes bugs. And they found that there was a more or less, it was more or less independent of language, but they found that there was a fixed number of bugs per line of code. So the fewer lines of code you had, or the more expressive your language, the fewer bugs you'd create. That's maybe because you're reusing things, right? You don't need to, if you're in a sort of more primitive language, which says loops, if you're building a sum function, you've got to sort of create it manually.
00:14:33
Speaker
Whereas in Clojure and other functional languages, you use something like reduce. And that eliminates error because you're using a tool which has already been created, which has been tested multiple times, which is part of the core library, rather than writing it yourself each time.
00:14:51
Speaker
So languages are about stopping human beings from doing the wrong thing, right? They're incidentally for computers to execute, but languages are mostly developed for human beings to express problems in ways that stop things from going wrong. And Haskell and Clojure have two different ideas about how to stop things from going wrong.
00:15:17
Speaker
Haskell's idea is we're just going to build this framework of types. And if you stick within this framework of types, then you're limiting what can go wrong, right? You can't return the wrong result if the function says int. You can't access IO if it's a pure function.
00:15:38
Speaker
The type system is not just an aid for the compiler, it's also a way of structuring your thinking. In Haskell, when you're creating a program, you're typically starting off with the types. You map it out with this exoskeleton, which you construct first, and then you fill in all of the organs later on. Maybe the problem with that is that you can't expand out of the skeleton you've created.
00:16:06
Speaker
Closure has a different idea and that's that things go wrong because they're too interconnected. The more things that are interconnected, the harder it is for our brains to figure out what all the consequences of actions are. And this struck me as a more
00:16:28
Speaker
accurate or perhaps based on my, I mean, this is all subjective, but based on my experience, it wasn't so much the types or the structure that caused me a problem. It was the interactions between things. So when Rich Hickey came out with his talk, Simple Made Easy, that struck a real note with me, right? The closures structured around this idea of reducing interactions means reducing the amount of things that can go wrong.
Clojure vs Haskell Philosophies
00:16:56
Speaker
It's the only language I'm aware that really pursues this as a philosophy, and it's something I've really bought into. And maybe that's why I've snuck with Clojure so long, because it's not the language so much that I fall in love with, but the philosophy that a language is developed around.
00:17:15
Speaker
Right. So it's a kind of culture, isn't it? Really? Cause that's what you're saying, isn't it? That, uh, with, with these strongly typed systems and higher type, higher kind of type systems that it's the algebra is trying to, you know, trying to do good things like through mathematics, basically. Whereas with things like dynamically typed languages, like closure or even Ruby as well, it's more social, you know, it's more kind of cultural and you kind of, you kind of have to follow the, the, the way.
00:17:44
Speaker
that there's nothing inherently in the language which makes it like that, actually? Yes, maybe. I think there might be a way of creating a statically-typed language that follows closures, conventions. Actually, there is a language called LUX. I think someone from probably Argentina or somewhere in Latin America, someone is developing right now.
00:18:09
Speaker
That is LISP, but essentially statically typed Haskell-based sort of thingy. I'll send you a link on Slack later. But that's the opposite of what he's saying now. Using closure, but ML-like closure. Yeah, it's more like...
00:18:31
Speaker
It's more like closures about reducing the interactions between components. And this philosophy results in pure functions because pure functions only rely on their arguments and their outputs. But it also encourages data which only relies on itself. Whereas Haskell
00:18:50
Speaker
is more about restricting what you can do with types, right? It's not about restricting the interactions per se, it's about restricting what you can do in the moment. So I guess it's more like whether you restrict the individual or whether you stop the...
00:19:11
Speaker
I mean, going for a kind of police state analogy. If you're a dictator and you're running a police state and you want to crack down on the dissidents, then you've got two options, right? You can imprison the individuals, you can take your troublemakers and stuff them in a hole somewhere.
00:19:35
Speaker
Um, or you can, and, and, and, and that's, that's my analogy for Haskell. Um, yes. Whereas, whereas closure takes us all more, perhaps a more subtle approach and says, well, we can stop distance by cutting the social ties between troublesome groups. So maybe that's like the, um, uh, uh, a dictatorship, which like uses like Facebook or, or sort of algorithms to like,
00:20:04
Speaker
manipulate people through the mainstream media and make them cut their ties to their troublesome neighbours. It's like America. Yeah. Well, maybe where America will go if Trump remains. I'm going to stop using closure now. Forget it. What you're basically saying is that Haskell is not the Korea of programming languages.
00:20:30
Speaker
And Closure is the US of the programming languages or something like that. Yes, maybe. I'm paraphrasing, you know. It's more about restricting the individual components. So Haskell likes to sort of have everything kind of in its particular box, whereas Closure says, okay, well, the individual things can do what they want, but what we want is to restrict how the group interacts.
00:20:53
Speaker
with each other. So it's about cutting ties, about isolating people or isolating functions individually in isolating components rather than restricting them.
00:21:06
Speaker
Yeah. See, I see it as a bit like things like Git as well because Git is about, you know, there are tools there to, yeah, to version your code, et cetera. But most problems with code are social problems, you know, rather than necessarily than tool-based. Tools help to organize things and make the visibility of these changes more obvious.
00:21:32
Speaker
But really, most coding is a social thing. That's why GitHub is so popular, isn't it?
00:21:39
Speaker
Yeah, and maybe it's the case that Haskell actually has an edge on Clojure in that respect. Because if you create something in a statically-type language, you're saying to the people who can maintain you, okay, well, this is the structure which I've created the program around. And you can take a look at the structure, but it's all typed. And if you want to add stuff to my software, you need to adhere to this structure. And if you don't, the compiler will complain.
00:22:06
Speaker
Whereas I think with closure, it's maybe easier to run away with, to add things in which the original creator wasn't kind of thinking of. So this can be a good thing or a bad thing. But I've heard people say that when they create closure projects, that they don't turn out like you have two separate teams developing something which is very similar. And the end result turns out to be very, very different.
00:22:35
Speaker
And maybe that's why we need something like a framework, like either arachne or integrant slash duct, which I've been developing. Yes.
00:22:51
Speaker
We'll get into them. I think. Yeah. Yeah. Yeah. Who can I definitely get onto that? Yeah. Exactly. That's a good background then, isn't it? Yeah. I think to sum up that.
00:23:07
Speaker
Closure stops you from doing wrong things through isolation, whereas Haskell stops you from doing wrong things by being restrictive.
Open-Source Projects and Simplicity
00:23:16
Speaker
I think you've got restriction versus isolation as your two main schools of thought in how to stop things from going wrong. Having tried the restriction school of things, I'm preferring the isolation school of things, which closure promotes.
00:23:35
Speaker
What do you think about this other argument as well, that it's about kind of, like you said at the beginning about the kind of organs and the exoskeleton, what do you think about the experimental nature of closure and lisps in general, where you can do this kind of exploration through the grapple of your data structures and the kind of functions that you want to
00:23:59
Speaker
Stitch together which is which is more tricky to do when you're kind of having to stitch all the types together first That that's to me. That's a really appealing thing about closure Yeah, and I think I agree with you there. I'm much more into
00:24:15
Speaker
I mean, that's why I've never really gone on with test trim development. I kind of prefer to take an exploratory approach and then develop the tests afterwards. And Ruby had, or at least when I was developing, had this culture of test trim development, which I never really got on with because it felt like putting the cart before the horse that you've got to have your design fixed before you actually start developing. And it seemed like that was maybe the wrong way to go about it.
00:24:45
Speaker
Yeah. So right now, I'm curious about your process of developing software because, of course, used Haskell, then in Haskell, it's a completely different way. As you said, first you start with the types and then you fill in the types and then define everything to be undefined and then figure out the whole interaction between the types, then start putting in the code. So what is your, if you start with a problem or a thing that you need to build with Clojure, how does your development cycle look like?
00:25:15
Speaker
probably the opposite of what you described. So absolutely nothing like I do it in Haskell, which may be why I switched to Closure, because I didn't like that approach. So I like to take an exploratory approach, taking a small function, first of all, writing it out.
00:25:36
Speaker
having a REPL open so I can test it and figuring out getting something minimal working. And if it's in the wrong direction, just delete some code, add in some new things. There's more exploring what I can do. And then gradually something will take shape and I'll have some idea of where to go. But a lot of the initial part of writing a library is as much deleting code as adding code.
00:26:05
Speaker
So, on that note, we're talking about the development cycle. So, Emacs or something else. Oh, God. Emacs, I'm afraid. I did use Vim when I was developing Ruby. Yeah. I tried to use Vim... This is basically... Yeah, I tried to use Vim a little bit with Closure. And by a light learning thing, so I went, okay, well, this is my opportunity to learn not just Closure, but Emacs as well. So, I found out that there was...
00:26:34
Speaker
There was another like VIM project, which I can't remember the name of, but I quickly switched to evil mode on Emacs, which is almost like a better VIM. The only thing I can complain about with Emacs is that it has a really Byzantine way of updating its screen. So it kind of pretends that its screen is a terminal and yeah.
00:27:02
Speaker
It keeps redrawing. Yeah, it's a complete mess, basically. But that's the only thing I complain about. It means that smooth scrolling doesn't really work on eMacs, even with the Mac OS kind of built version of it.
00:27:17
Speaker
Yeah, yeah. Did you use the railway cat version? Yes, yes, I do. So I use the railway cat version of Emacs for Mac, which has like smooth scrolling, but the CPU just spikes whenever you try it because it's just, it's redrawing the same thing over and over again, rather than caching any of it. And I sent the email to the creator as well saying, is there anything we can do about this? And his answer was, well, no, because Emacs draw code sucks.
00:27:47
Speaker
Yeah, so that's the point. Yeah. I have CPU. Yeah, yeah. Yeah, eight makes and constantly swapping was the old. I think I think that nowadays eight gigabytes and yeah, we moved to the next level, eight terabytes and it's a big data editor now. Your Mac boots directly into a max, doesn't it, Vijay?
00:28:15
Speaker
Pretty much. There's a full screen. I have been interested in Atom recently because that has like VIM plugins and it has a very nice like proto-reple is looking nice each day and the VIM stuff is looking quite nice as well but it's not on the same level as evil mode or cider. Yeah.
00:28:37
Speaker
Yeah, that's true. I mean, I used Atom as well and I like the concept of it, but it's so slow. Just typing stuff is slow and that is, you know, you can't have that. You cannot have that. You cannot have it where you're typing stuff and it's not coming on the screen from an editor, you know, that's not allowed. That is a fundamental thing an editor should do. I'm just typing something, let's put it in there. Yeah, it has to be on the screen. Come on. Anyway, yeah, right, fine.
00:29:05
Speaker
So you're using Emacs, and obviously that is the main reason for your extreme productivity of almost 90 closure libraries. I don't know how you write this stuff. That's what we were just discussing before the show, that you almost have 100 closure libraries. Of course, you're using Emacs. That's the reason why you're producing it, but what is the real reason? What is the auxiliary reason? So a lot of them are very small, like a few hundred lines are carried at most.
00:29:31
Speaker
I think my smallest library is like three lines of code and that was a library for doing a constant time equality check for cryptography purposes which I thought was probably something I should put in a library because I don't want to keep writing this code and make a mistake because if there's any code you shouldn't make mistakes in it's cryptography or any code to do with security or cryptography.
00:29:56
Speaker
So a lot of libraries are quite small, like a few hundred lines of code. The other thing is I quite like writing open source libraries. And I'm a contractor, which means I don't have a nine to five job that I have to work with as well. So I have a lot of time between contracts, which I can use to work on open source projects. Yeah. So you've got quite a few equivalents of left pad in the, in the chlorogen.
00:30:25
Speaker
Oh, yeah. No, no, that's a crypto equality is my only left pad equivalent.
Integrant and Duct Frameworks
00:30:33
Speaker
And hopefully that's justified because cryptography kind of has its own rules. Yeah, you want you want a lot of testing on that thing. Yeah. But but do you know chronologically, what is the first library that you started writing enclosure? You remember? Yes, it was some composure. And I started writing that in
00:30:50
Speaker
I think April 2008, I'll need to check the history. But the original composer looked very different. So I started off with Sinatra, which is a Ruby library. And the first, because Closet didn't have any web stuff at all then,
00:31:14
Speaker
Composer was effectively like a batteries included thing. So it was based on top of servlets. It would actually generate a servlet directly. It had like cookie code and parameter parsing code and all of this. It was essentially based directly off the Java servlet stack.
00:31:33
Speaker
And then a chap called Mark McGremihan contacted me and said, I have this, I've been thinking about this abstraction layer for closure, which I'm calling ring. And at first I was skeptical because I went, okay, well, you're adding an abstraction layer on top of the abstraction layer, but
00:31:51
Speaker
Java has like we're just piling abstraction. I've already got this abstraction later. It's called Java serverless. They work fine, right? He goes no, no, they're rubbish. I kind of go when I actually looked at some ring. I went yeah, you're right. This I can see the possibilities here. So I moved like
00:32:09
Speaker
90% of the code I've written for composure into ring. So all of the sort of cookie parsing and all of the parameter parsing and everything like that. I just shoved into ring and composure was left with like 400 lines of code, which essentially just made a routing library.
00:32:29
Speaker
But that's actually got more stars, more GitHub stars than Ring has. Yes. It's longevity, I guess. Yeah, so longevity. And the other thing is that we moved Ring from Mark McGranahan's repository into its own team repository. So there's actually two
00:32:49
Speaker
There's still two repositories and Mark McGranahan's ring has probably got a few stars that weren't transferred across. You can't tell GitHub to do that. You can now. I don't think you could at the time.
00:33:07
Speaker
So in terms of stars, you're like internet billionaire and GitHub is the only thing available and stars are the currency. I'm pretty sure that there are people in more mainstream languages. There's far more stars than I have. JavaScript, yes. I'm pretty sure left part is like the peak and the star overflow. Now everybody keeps monitoring it that it shouldn't die now.
00:33:33
Speaker
So you're talking about composure and then obviously you're maintaining it forever in the internet years, almost nine years, ten years. And now you're also really stuck and integrant. Yes. So what is the difference between these things?
00:33:57
Speaker
Duct was a... Maybe it's just to start by saying what the kind of purpose of duct and integrant are, what's driving you to kind of to produce these things. Yes. Some of these people don't know what they are. Sorry. So as I said, I do a lot of, well, I do some contracting work and some of the contracting work involves web stuff because I developed Ring and Composer so people kind of go, well, can you help this Closure team build a website for us?
00:34:26
Speaker
So I've been doing a lot of and pretty much all of my previous jobs as well were to do with designing websites. So this has been my sort of full kind of job history. Yeah. And Duct started out as a template for what I thought a closure web
00:34:51
Speaker
site or web application should be developed as. So originally it was a template similar to Luminous, but it was very component based. So it used to use Stewart Sierra's component library, which is a micro framework for structuring applications.
00:35:14
Speaker
So it's not really a, I think component is a few hundred lines of code as well. It's not like a framework with a very specific task like,
00:35:26
Speaker
Rubon Rails and it's not a framework that has basically everything like Java Spring but it's a micro framework which allows you to structure an app in a certain way and it provides a life cycle so you can start and stop things. It provides some introspection into what's running on the system so you can inspect systems say okay what's running these components here.
00:35:51
Speaker
It provides dependency resolution, so before starting the web server, it might start the database which is needed by the web server to serve things.
00:36:04
Speaker
And it also allows you to have more than one instance of a system running at one time. So your system isn't limited to your application. You can be running multiple ones at once, which gets useful for testing, right? So you can have a system running, which you're currently developing on, and then you kind of go, actually, I want to test some stuff. I'm just going to pull up another system to test it in some way.
00:36:29
Speaker
So those things quite appealed to me. And then after watching some of Stuart Sierra's talks, I liked the idea of components. And then that led to me writing a template around it, which was Duct. And then as I started to write Duct, I think the first thing was I was working with a chap called James Conroy Finn, and he
00:36:58
Speaker
mentioned to me that, wouldn't it be nice if, although he said that the problem with templates was that after you had created them, you were kind of stuck with whatever they were, right? There was like a static thing. You create your project from a template.
00:37:21
Speaker
and then you're stuck with whatever the template has. You can modify the code, but there's no easy way of saying, oh, the template's updated. How are we going to update it? How are we going to pull it in? How are we going to update our code base?
00:37:33
Speaker
So I was starting to thinking of ways of using generators, which are from Rails, where in Rails you create your framework, your new project, and then you can use generators to create new database migrations or new controllers and new models. So I was taking that kind of approach in a kind of close-up way.
00:37:55
Speaker
And then I think Henry Garner mentioned later on that he quite liked having configuration in a file, which he could take a look at. Up until that point, I was used pulling in configuration for environment variables. I kind of went, well, actually the configuration I have is very limited. I didn't have a lot of configuration in my application. I just need a port and maybe a few other things. And Henry said, well,
00:38:25
Speaker
That's true now, but maybe you want to alter things that could be configurable, like the number of threads in your server or various concurrency settings. If you have a very large configuration, you can tweak all of these individual settings.
00:38:43
Speaker
And that appealed to me well. The more stuff you can put in a data structure, the less stuff you have in code, and that's good for isolation because a data structure can't be altered, especially a mutual data structure. It only depends on itself, whereas code is sort of interconnected everywhere. So the more stuff I could pull out of code and into a data structure, the better. So then I created a configuration for ducts.
00:39:12
Speaker
From there, I started running from this component because I'd have a configuration and then I'd try to produce from that configuration a structure, a system of components which I could then run as an application. The problems I had with getting component to work with that led me to develop Duct, not Duct, let me develop Integrant.
00:39:36
Speaker
An integrated essentially another micro framework, it operates in more or less the same space as component does, but it has the advantage of building a system from a configuration. And it separates out
00:39:57
Speaker
the configuration from the implementation. Whereas in component, they're kind of the same thing. Your implementation is built out of records which you can then introspect when you can start and stop them. Whereas with
00:40:13
Speaker
You start off with a purely data-driven configuration. You initiate that into an implementation, and then when you're done with the implementation, you can halt it in a side-effectual way and throw it away. Once you get into the mess, it doesn't try and make the implementation nice. It just says,
00:40:33
Speaker
start off with the configuration, do as much as you can with the configuration. When we initiate it into actual code, that's kind of something that is effectively a black box. The only thing you can do with it is press the stop button and throw it away.
Opinions on Pedestal and Arachne
00:40:49
Speaker
Okay. So what is the... Yeah, sorry, go ahead. No, you're gone.
00:40:56
Speaker
Okay, because I'm trying to see the Twitter questions and I wanted to adjust it. So there is, I think we should say who is asking this question. There is IkuruK from Japan. Wow, people from Japan asking questions on DeafN. My life is complete now. Which we've unlocked for sure, yeah. Exactly. So he or she is asking about your opinion about Arachne and Pedastal.
00:41:25
Speaker
What do you think of them? I mean, there are two, there are different kind of, I don't know, quote-unquote frameworks in Closure World, right? Yes. So what do you think of Arachne and Pedestal? I'll start with Pedestal, because that's the easiest one for me to get out of the way. Pedestal is similar to Ring, and it actually uses some of the
00:41:46
Speaker
functions in ring, so it depends on ring as dependency, but it doesn't use ring middleware, it has its own concept of interceptors, which are more flexible than ring's middleware, also maybe a little bit more complex. It's very much geared up to
00:42:05
Speaker
handle asynchronous connections. I mean, it can handle synchronous stuff, but a lot of the design, at least my impression of it, is that it's designed with asynchronous HTTP in mind, which is something that Ring hasn't been designed with. So Ring has kind of been designed from a sort of synchronous perspective. Pedestors come the other way and being designed from an asynchronous perspective.
00:42:34
Speaker
And in ring 1.6, we've added sort of asynchronicity, but it still has kind of roots in synchronous access of HTTP. So Pedestal fills a particular niche, but it's not something that I've ever actually had to use, because usually my bottleneck is not the HTTP server, it's my database, or it's worker processes, or something which I can't scale horizontally.
00:43:04
Speaker
I've heard from Cognitec and from others that Pedestal does fulfill the criteria they want. It's just a case that Pedestal probably has a use, but it's not something that I've ever had a need for. It's not something I've ever actually really experimented with.
Prog Rock Library and Music Preferences
00:43:24
Speaker
Arachne, on the other hand, is something I have a need for because it's currently built on or has components which link to Pedestal, but it's very neutral in that respect. Eventually, we could have it linked directly to Ring instead. Pedestal is a larger framework. It's not a micro framework, it's actually a real true framework. Maybe the first real true framework that Closure has had
00:43:54
Speaker
And that's very similar to the way integrant and duct currently work. But arachne is maybe a little bit more involved. So I haven't used arachne a lot. It's still in alpha. But the way I understand it works is you first create a configuration script, which is basically a bunch of closure, which
00:44:23
Speaker
which you run, and it's effectively side-effectful, and it generates an in-memory database, which is, I think, either in-memory Datomic or Datascript, one of the two. So then you have this in-memory database. Then we have another function which takes this in-memory database and produces from it a system of components.
00:44:50
Speaker
And then those system of components can then be executed and then you have your full application. So we kind of start off with data and then we go to implementation. We have a few steps in between.
00:45:03
Speaker
I looked at this and went, hey, that's a really good idea. But I don't like all of these steps in between. I don't like using component. I don't like using Datascript and Datomic. And I was kind of leaning towards having a database configuration anyway. So why don't I start off with a map, just a configuration map. And then I have something that turns it directly into an implementation. And then we can cut out the steps of having
00:45:33
Speaker
the in-memory database, and instead of having a script which creates the configuration, maybe we should have a bunch of, so Duct introduces this idea of modules, which are pure functions which transform the data structure. And because we're using a data structure rather than a database, we can use kind of pure functions to
00:45:59
Speaker
or we don't need to sort of lean into Tom X mechanisms for querying and transforming the data structure. We can do that with any pure function we want. Yeah.
00:46:15
Speaker
And of course there is a, we already discussed a bit about the differences or the duct and integrant and how it is working. I think it's asked by Michiel Borkent and Arnaud Bosch. And Michiel was also asking, there is a library by you called Prog Rock. So he is interested if you are in what kind of progressive rock band you are.
00:46:38
Speaker
you're listening to, or if you're into that thing. So I actually saw this tweet, and I went, do I listen to any for a rock? And then I went through my Spotify playlist. And no, no, I don't. It's just a coincidence.
00:47:00
Speaker
Can we start tonight? Yeah, so I apologize, but prog rock is not currently on my place. I'm not a huge fan. I actually went out and, before this podcast, put a prog rock playlist on Spotify on so I could listen
Real-Time Database for Gaming
00:47:18
Speaker
to it. I was just skipping through lots of the tracks.
00:47:22
Speaker
Sorry, I'm not a fan. I chose Progrock because it has the word Prog like is in progress in it. And I'd already created a library called Ragtime, which is another genre of music. And I chose that because Ragtime is an anagram of migrate.
00:47:42
Speaker
Okay. So you can kind of see we went from an anagram to a music style and then because I had missed that library and Progrock released them soon afterwards, I decided to go for another music style and
00:47:56
Speaker
But all the people that are eating the sausage now are very upset about the way it's made, you know? This is the bad thing. But what kind of music do you listen to when you're writing chord? I don't actually listen to much music. If I do, it's usually some kind of video game or kind of retro music, something that requires very little cognitive thoughts to appreciate. Something which is actually background noise, right? Yeah, yeah.
00:48:27
Speaker
Yeah, so there is one more. You're talking about like, you know, when you're explaining Pedestal, the major bottleneck for an application is the database and how do you know, that's where the scaling issues are coming from. And you're also working on some sort of a database, right? Yes, so it's unrelated to... By having working on sort of
00:48:52
Speaker
creating games in my spare time. When I'm not working on stuff which I can actually sell or market, I have soft spots for creating games or seeing if I can create a game in Clojure because Clojure is a functional language and there aren't many games which have been created in functional languages.
00:49:15
Speaker
I wanted a game which is multiplayer. If you have a multiplayer game, you're dealing with communications between different clients. Effectively, what you have is a distributed database. The server has a copy of the database and the clients have a copy of the database. You need some way of fuzzily keeping them in sync.
00:49:39
Speaker
And having them be real time is more important than accuracy. Because on the client, things can change around a little bit if they're things that don't matter. Like the positions of different players can shift around a little bit as long as nobody's directly shooting at them. So I created a distributed real-time database called Ition, which is an old name for a slower than light particle.
00:50:09
Speaker
And that was appropriate because when you're developing multiplayer networking code, you're essentially running up against the speed of light. Effectively, you're trying to break the rules of the universe, right? Because you're dealing with people who are very far away and they have a certain amount of latency. And you're trying to hide that latency and pretend they're actually much closer to each other.
00:50:34
Speaker
So you're just trying to make it seem like you're effectively having fast and light communication when you're not. That's the nature of computing though, isn't it? To lie and pretend and to cheat the universe. Does anything ever get saved to a disk, I wonder? That's true. Does it matter?
00:50:58
Speaker
Does it matter as long as the computer is on? Yeah. I mean, I guess that's what you want, right? You want to filter out on the stuff that doesn't matter and you want to sort of get rid of those things and say, well, maybe it doesn't matter if we save it immediately. But if you get it wrong, you might end up with something like MongoDB, which has a maybe undeserved reputation for losing data. Yeah.
00:51:26
Speaker
I think if mathematicians can pretend what the square root of minus one is for the win, then we can do that kind of shit as well. Just pretend what's inconvenient doesn't exist and just move forward in the hope that it actually won't really matter. And it turns out that's probably true. So you have the normal real database and imaginary databases.
00:51:49
Speaker
Ah, yeah. And you keep on looping around them. Exactly. Apart from writing closure code, what else do you do? I don't know if you can find some other time between writing so many parentheses all the time.
00:52:06
Speaker
Well, so a lot of... I like programming, so I do do that a lot. Other stuff I do do on the computer as well, so video games and things like that. Which maybe explains why I've kind of been creating some games in my spare time. What sort of games do you like, James? I have maybe a wide variety of genres which I...
00:52:33
Speaker
which I work on. So first-person shooters, real-time strategy, the kind of MOBA genre, which is out there. Anything multiplayer? Playing a lot on Tabletop Simulator recently, like the board games there are really quite nice. When I meet up with...
00:52:56
Speaker
Like every six months I meet up with a group of friends and we just play board games for about a week. So TableSoft Simulator has helped fill in the gaps between that. It's a program that allows you to effectively play like cardboard board games over the internet.
00:53:13
Speaker
So that's another thing I'm into.
Personal Interests and Hardware Projects
00:53:16
Speaker
Beat-em-ups like Street Fighter or more obscure ones. So yeah, that's outside of the computer. Maybe not so much.
00:53:31
Speaker
I probably should get out more. A few years back, I had a go at doing a climbing course. That went quite well. I have been trying to do more hardware stuff. So I've got a Raspberry Pi, which I will at some point install Jasper on, which is an open source echo slash Google Now sort of replacement.
00:53:58
Speaker
So I kind of want to try that as well. And at some point I need to replace the battery in my phone, so that's going to require a lot of fiddly hardware pulling out, especially since the battery isn't meant to be replaced.
00:54:12
Speaker
So hey, I'd like to thank you again for coming to the DCD, the Dutch Closure Days, and giving the keynote there. You're very welcome. It was a very nice keynote. You were talking about the data, and I think there were already 500 views on the YouTube.
00:54:31
Speaker
We hope more and more people get some understanding of what you're trying to explain. So let's see, do we have any other interesting questions that we need to ask?
00:54:45
Speaker
Just wondering. The one thing that I kind of mean, it's not really that interesting, but closure wise, screw it. Okay. It's a closure podcast. We're going to talk a little bit about closure. It's one last little thing. It is, uh, there's something that someone asked you at the, at the, at the, at the conference, actually at the Dutch closure days. Um,
00:55:07
Speaker
and something i was talking about with someone at work recently is these derived keywords and i think it's something which i think many people have kind of seen when they're reading about closure for the first time and then just treat them as a little novelty and a bit of a toy but you seem from your recent adventures to have got some real value out of them
00:55:28
Speaker
Maybe you could just talk quickly about that because I think that's really, that was a surprise to me. I've got to say, I know a few of the people in the room as well. Yeah. So for, I've been working with closure for about nine years, I think. And for the first seven years, I didn't use them at all. And I thought, hey, it's kind of neat that names based keywords can have, could be derived. So you can have an inheritance tree just involving keywords.
00:55:57
Speaker
And it struck me as something which I liked the separation of it, because in object orientated programming, you have an inheritance tree of objects. Whereas in Clojure, Clojure takes a very
00:56:12
Speaker
It's slightly different between a pen knife in the object-oriented world where the objects attempt to do everything. They have polymorphism, they have inheritance, they have encapsulation, they have data, they have functions, they're often used for namespacing as well, not in Java but in Ruby as modules and Python as well.
00:56:33
Speaker
The modules are objects. So objects are used for loads of things. Whereas in Clojure, we tend to have very specific tools. We have protocols for polymorphism. For inheritance, we have these derived keywords. So I quite like the way that all of these tools separated out and they were sort of their own thing. But I hadn't found the use for derived keywords for quite some time. I thought, well, that's probably because inheritance isn't that useful. Maybe there's some niche cases for it, but maybe not.
00:57:03
Speaker
But when I was developing Ition, the real-time database I was talking about earlier, I actually found a use for them because in games you often have events that inherit from other things, right? So you could have
00:57:22
Speaker
something that moves is like a sort of mobile NPC, right? So that might pull in a certain thing of inheritance and maybe it
Derived Keywords in Clojure
00:57:29
Speaker
can be damaged. That's another sort of structure. There's a game development sort of pattern called the entity component pattern, which effectively involves creating or pulling in a bunch of components into a single entity. So it's effectively multiple inheritance.
00:57:48
Speaker
So that seemed like a good thing for me to use with derived keywords. Once I'd used it there, I could then see other places. Once I'd actually used it in anger, I could see other places where it could be used. One of the interesting things about derived keywords is that the keywords are effectively
00:58:06
Speaker
Closure has symbols which have a definition outside of themselves, so a symbol will refer to something else in the code. A keyword has its own definition, it stands on its own, but we can describe keywords in various ways.
00:58:26
Speaker
Closure spec is one way to describe a keyword. So you say, well, this keyword can associate with this data and the data must be an integer or it must be something else. But we can also describe keywords in terms of other keywords. So we can use the inheritance tree to effectively act as like a dictionary.
00:58:49
Speaker
And this is something that I started using in Integrate, where I can say, okay, we have a key, which is, say, server slash dot HTTP slash jetty. So that's a jetty server. Maybe we can derive that from server slash HTTP. So if we want to ask... So you want to use aleph or something, you can have another one.
00:59:10
Speaker
Yeah, exactly. And then if we have an integrant configuration and we want to ask it what are the HTTP servers, we can say, well, what are all of the keys which are derived from server slash HTTP?
00:59:26
Speaker
And because it's multiple inheritance, we can add in multiple definitions. We can say, well, Jetty is both a HTTP server and it's a synchronous server as well. Mostly handles kind of like synchronous or it has support for synchronous as well as asynchronous. So we can say, well, this is both a HTTP server and it's a synchronous server. So we know that when we start it, it will be using up threads. And you can start
00:59:52
Speaker
adding more and more general keywords to describe more specific keywords. And then that gives you more ways of querying the configuration, like saying, okay, well, what are the servers that are currently active? Like what ports are they running on? Or what servers are synchronous and therefore going to be using up a lot of threads?
01:00:12
Speaker
or what are the background workers in our process, or what are the database migrations. And by using this inheritance tree, we can have specific versions without losing our ability to have general queries.
01:00:36
Speaker
What are you using those queries for in a reporting sense? What's the value you're getting out of that structure? It comes back to modules, which in duct are pure functions that transform a configuration. If you have a pure function that transforms a configuration, you need to have some way of
01:01:00
Speaker
understanding what you're transforming. If you don't understand what you're transforming, then you can transform it only in very limited ways. You can add stuff to a map, okay, but you can't modify stuff that's in that map because you don't understand what it's for. For instance, there's a module in Dux which adds all of the stuff you need to build a website.
01:01:23
Speaker
But what it does is it first queries the configuration to look for a HTTP server, which is already in the configuration. Because if the configuration has a HTTP server already, it can use that instead of adding its own. So it can kind of say, okay, well, you've already got a HTTP server, I'm going to add all of the middleware to that instead of
01:01:50
Speaker
instead of just creating my own blindly. So by allowing modules to query, they become smarter. They're not just blindly adding stuff, they can modify existing things as well. Okay, awesome. Yeah, sounds really good. I mean, like I say,
01:02:10
Speaker
I think there's probably much more use for this stuff. I think we, like you say, we forget all of these various tools that we've got around closure sometimes, and it's nice to see a fresh approach to something which I think everyone has just forgotten about. We don't see it much in the wild, so it's good to see it out there. Go on, sorry. What is the status that integrant is in right now? Integrant is pretty much stable, and it's not 1.0, but it's
01:02:40
Speaker
It's not 0.04 so there's probably going to be some there might be some breaking changes in the future, but if there are, they'll be very minor. It's pretty much stable.
Clojure Development Challenges
01:02:50
Speaker
Duct is currently in alpha, so it's in a position where you can try it out. There's no documentation written on it at the moment.
01:02:58
Speaker
I will be writing some soon. It's not production ready, but some brave individuals have said, OK, well, we want to build our next stuff around Duct. And if you happen to decide that you want to do that, then I will try and help you. And there will be more documentation in future for the alpha version of Duct, which is
01:03:21
Speaker
very different from the previous version, because the previous version used components, and now I'm switching over to integrant. A lot's changed between version 0.8 and the 0.9 alphas. Yeah. So maybe before we close off, I'd like to just ask one question. So we've been praising Clojure a lot. I mean, what is one thing that bothers you in Clojure? Startup times are not that great. I can't use Clojure for
01:03:50
Speaker
writing shell scripts or system files or things like that, because the startup time is just kind of bothersome. I mean, it's less the startup time of Clojure itself than it is of the loading in all of the libraries, because they've all got to be sort of loaded into memory each time. Maybe that's kind of solved by Clojure script and node. But yeah, that's kind of one of the things, the main thing that bothers me currently.
01:04:18
Speaker
But in the language itself, do you have any things that you think could be different a bit? So one problem is that Clojure tries to strike a balance between performance and
01:04:35
Speaker
Well, maybe Closure leans a little bit too far towards performance. So for instance, the keyword function in Closure Core doesn't have any checks on it. So you can create invalid keywords with spaces in and things like this. And because those
01:04:53
Speaker
I'm adding in the additional checks that would stop it would take extra time right to checking each time you create a keyword this one the problems dynamic languages have right if you're you got all your types available at runtime then all your checks on those types have to be at runtime and that means you're slowing down runtime.
01:05:14
Speaker
Is there a spec on keyword now? I don't know actually. Maybe there should be a spec on keyword. What might solve the problem is closed respect because with closed spec we can instrument functions which means we can add in conditions
01:05:30
Speaker
or checks onto functions dynamically so we can have them in development and we can turn them off in production. Maybe we could even have them in production if performance isn't like a huge, huge deal, which most of the time it's not. Otherwise we wouldn't be using Ruby and Python and other languages like that. We'd all be using C. So I think
01:05:55
Speaker
I think Closure Spec will solve that problem. Eventually, once all of Closure Core has been spec'd out, then the performance and error message and the checks will be solved. And also the stack traces will also be a lot better, because currently the stack traces just go into Java, right? Oh my god, yeah.
01:06:21
Speaker
Whereas what we ideally want them to do is we want them to stop at the closure layer and give us a nice exception. But again, because of performance concerns, they've just been let loose to kind of fall back to the Java layer. So with closure spec, maybe that'll be solved.
01:06:39
Speaker
And I think they seem to be taking the same attitude as you are with integrant. I noticed that the latest alpha of Closure 1.9, they've separated out the Closure spec into its own library. So everything is changing. I was hoping it would be settling down by now, but they're still obviously quite hard at work and there's still some big rocks to be moved around there.
01:07:06
Speaker
I think it's a smart move to move spec out because then we can actually have closure 1.9 out of the gate while still keeping the experimental aspect of spec because spec is still very much experimental and it's better to have it experimental for a while so we can have more time to think about it and work out what's best than to settle on something too soon and then be stuck with it for the next seven years.
01:07:32
Speaker
Yeah, yeah. All right. Okay, good. I think we're well past the hour mark, actually. So, I remember when we talked originally, James, you were thinking that you wouldn't have anything to say for more than two minutes, but it turns out that, well...
01:07:49
Speaker
I think we need to do another episode with James. I know you've got so much stuff. Exactly. I think I think we can, we can, you should, we will really, we would really like to have you again. And because there are a lot of other things that we want to talk about, obviously. I mean, if we, if we go by the libraries, I mean, we can, we can make 20 episodes. Yeah. I mean, I think we've covered three or four today, you know? Exactly.
01:08:13
Speaker
We probably have, we have to be one every like three months or something just to stop coding now, okay? Because we can't catch up, you know?
01:08:22
Speaker
We have given the most interesting libraries.
Twitter Handle Origin and Conclusion
01:08:25
Speaker
The other ones may be less interesting. I still think, honestly though James, I think you're doing some interesting things. I mean, I just noticed this last week you published something that visualized tests and obviously you're doing either this Chordox thing and Closure formatting. A lot of Closure tools that will help developers to, let's say, in terms of like
01:08:48
Speaker
huge value-wide, maybe he's not, but things which contribute towards the community and the tooling around closure. So I think you're doing great stuff. I really appreciate you coming on and talking about this kind of stuff and taking all these questions. So thank you very much. Really appreciate it. Okay. Thank you. Yeah. Thanks a lot.
01:09:08
Speaker
Thanks a lot for answering all sorts of questions that we're posing. One last question actually that I had as part of this thing was that you have a Twitter handle called Weave Jester and a little Jester, a little character. So how did you come about? What's the story behind that one?
01:09:27
Speaker
Well, it's anagrams again. So I was trying to figure out an anagram of my name. So my full name is James Matthew Reeves. And I plugged that into the internet anagram server. And I came up with two candidates that kind of made sense. One of them was them am weave jesters. And I quite like the word weave jester. Yeah. And the other one was meet Jew shave master.
01:09:58
Speaker
And when I checked the shoes between Reeve Jester or Jew Shavemaster, I decided that Reeve Jester was the better choice.
01:10:10
Speaker
Good choice, I think. Very good choice. Yes. I think we'll stop when I know what I think. Do you have a closure library for anagram generation? If not, there should be. No, no, I don't. I think that's... That should be under your name now. I think that niche, yeah, it could be called Jujave Master.
01:10:31
Speaker
Exactly. But yeah, I think there are plenty of places on the internet where you can... Check for... Check for anagrams. Okay. Okay. On that bombshell with Jewish Airmaster, it's time to roll the credits, I think. Yes.
01:10:50
Speaker
So this is episode number twenty one and yeah when it's been fantastic fun and thanks James for joining us enlightening us with some some deep knowledge of closure that you have already and we wish that you know you will you will find more time to
01:11:09
Speaker
put and maintain these libraries and there are more people joining in and helping you and then because these libraries are the ones that make the language, keep the language alive. Thank you very much. People use these tools and yeah.
01:11:24
Speaker
For the credits, again, this is the same music that we've been using and abusing from Pizzeri on Soundcloud. Actually, it's very much the kind of stuff that James likes by the sound of things. That kind of retro games music.
01:11:43
Speaker
And our fancy audio fixes are done by Wouter Dalit from Belgium. A big shout out to him. And did I miss anybody? The logo. Yes, the logo is designed by Louv of Sultan. She's the one who also designed our Dutch closure day branding as well. So a big shout out to her. And that's all for today, I think. Okay, thank you.
01:12:13
Speaker
See you again in the next episode. Bye!