Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#87 Jack Rusher
46 Plays1 year ago
We are back our (ir)regular schedule! Kickstarting 2023 with Jack Rusher https://jackrusher.com - talking about Clojure, exploratory programming, types, pull requests and more!
Recommended
Transcript
Introduction to Jack and His Background
00:00:15
Speaker
Anyway, welcome to DEFIN again. I think we spent enough time not doing Fortnite, so this time we have just a little bit of technical difficulty, if we can call that. So welcome, Jack, to the episode. Hey, PJ, nice to meet you. Yay! Nice to meet you. Yay!
00:00:36
Speaker
is the official introduction. But I think let's start with Jack. I mean, okay. So tell us about yourself, Jack. You have, I don't know, how many minutes left? 25 minutes. It's all yours for 25 minutes. I am an earthling. Despite any rumors to the contrast that you may have heard, I assure you that I am an earth man like yourselves.
00:01:04
Speaker
Hmm, that's exactly what an alien would say, right? To convince us. He's among us. Yeah, definitely. So what did you lads want to talk about
The Value of Types in Programming
00:01:16
Speaker
tonight? Here we are. You've invited me here to this thing. We've tried for months. Yes, we've been chasing you for months.
00:01:25
Speaker
Uh, well, let's talk about closure or well, maybe even, even more than closure actually, because I think, you know, you did a really, really, uh, interesting talk at, uh, was it strange loop? I did, I did a talk at strange loop, whether it's interesting is a question of opinion, of course.
00:01:40
Speaker
Well, I mean, interesting is not an objective quality, right? It depends on the person. So I'm saying it's very interesting to me, and I'm pretty sure a lot of people found it very interesting. Yeah, but I would say don't talk about that because you can just go and watch the talk. Yeah, it's on YouTube. You can find it easily. There's a subject.
Jack's Early Computing Experiences
00:02:00
Speaker
Come on. It's boring to talk about the same things all the time.
00:02:04
Speaker
No, we're not going to talk about the same thing where my question is like, where did you come from? That much of that much of experience about, you know, all the stuff because there is, you know, the computing is started almost like a punch card level stuff.
00:02:22
Speaker
Yeah, so what happened there biographically is that when I started at university, it was 1985, and punch cards were not the leading edge of technology at the time, as you can imagine, but they were still out there in industry. And so I had to take one Fortran class using punch cards to make sure that were I to end up in industry writing that kind of code on that kind of equipment, I would be able to do it.
00:02:46
Speaker
So, in that sense, I'm kind of a member of the last generation of people who were made to do that at university. And I'm happy to report that although I did write some FORTRAN for money, I did not have to do any punch card work for money. That was all class work.
00:03:03
Speaker
And then the big leap to closure from punch card. Yeah, pretty much directly. Yeah, because I had a time machine and so I was able to come forward. So the early part of my career was spent mostly doing very low level things in assembly and C.
00:03:21
Speaker
I did some embedded systems things. I was a Unix kernel hacker starting in the mid-80s. I was mainly like porting BSD to different hardware, working on file systems, blah, blah, blah, that kind of thing. But at the same time, I was very interested in first Lisp and then later functional, other functional programming languages. So, you know, we had Spice Lisp and we had Vax Lisp, which ran on our old Vax machines in the mid-80s.
00:03:47
Speaker
Of course, there was Scheme, which I encountered originally by way of a classic intro to computer science course for which sick pee is the manual.
Stack-Based Languages and Lisp
00:03:57
Speaker
And then in the sort of latter part of the 80s, there was Miranda, which is kind of Haskell's mother, which the author of Miranda tried to take it in the direction of being a commercial sort of language. And it didn't really take off in the research community, decided they needed something like it to carry on. And that was the birth of Haskell.
00:04:17
Speaker
So, I always really liked the pure functional stuff once I encountered it around 87 or so, but I also preferred S-expression syntax and did a lot of lisping on a wide variety of different environments. And the low-level stuff also brought me into contact, but I've written a bunch of fourths. Whenever I encounter a new piece of hardware, I like to write a fourth first to explore the hardware and then work from there.
00:04:41
Speaker
So what is the stack-based programming thing? Yeah, some people like to call them concatenative languages. They're an interesting kind of relationship to lisps where they sort of put the verb at the end instead of the beginning and they are based more on function composition rather than function application.
00:05:05
Speaker
But they are both languages that can, you can start with a very minimal core and build the DSL that you
Opinions on Typing Systems
00:05:11
Speaker
want. They both support a quoting mechanism that gives you something like macros and so on. They're really great languages. I find forth difficult to write large systems in, but it happens to have the property that it is a good and flexible interactive programming environment that you can fit on arbitrarily tiny crappy hardware.
00:05:29
Speaker
So it's a nice place to start. When I write a bootloader for some new thing I'm going to develop, I'll often use C and assembly to create a fourth and then do everything with fourth for a while until I'm comfortable with how the hardware works and then work my way up from there. No, so it's basically like the C for the hardware, like the first layer of high-level programming language that you build quickly and then, OK.
00:05:54
Speaker
Yeah, the amazing thing about it is how much leverage it gives you, how expressive it is while expressing also extreme machine sympathy. Because the machines we have, they all have a stack discipline on them already. So it maps very squarely to the kind of hardware that we typically deal with. Yeah, it's one of my sort of desert island programming languages. I would recommend everybody to learn whether they need it or not so that they can use the perspective that it provides in their other work.
00:06:24
Speaker
But because you started with, you mentioned Miranda and then a bit of a Haskell thing. But I know that you have, I think, strong opinions. Maybe it's my perception that strong opinions about the types and lack of it. So how do you find closure versus other stuff in that sense?
Programming Cultures and Types
00:06:46
Speaker
Well, so with the, and I do have to mention the talk boringly, a lot of people have an impression from the talk that I tried to deliver in a way that would hint towards how I really felt about it, but I only had so much time. You might notice that I'm speaking in fast forward without the fast forward button through that talk to try to get everything in.
00:07:05
Speaker
So with the transcript of the talk that I host on my own site, I include a bunch of notes, including some things that go into a bit more detail about that whole question. And it's not that I feel types are useless or harmful, it's that I feel like every other tool available to us, they have their benefits and their costs, and it's best to use them when the benefits outweigh the costs.
00:07:28
Speaker
So, for early exploratory programming, I find them actively frustrating. And I think it's very easy to over-tighten the specification of your program too early when you're not really sure what you're building yet. And certain kinds of software, especially the kind of software that, say, Rachinki spends most of his time writing,
00:07:45
Speaker
It can be actively unpleasant to have a type discipline that prevents you from kind of using the software in an open way. So moving maps around and leaving it open so the next person who comes along doesn't need to change a million call sites in order to enrich some piece of data flow is quite useful in that kind of software.
00:08:06
Speaker
And then there's other kinds of software where you really do want to tighten it up. And so my more nuanced position on types is that, for me, the future of type systems and programming languages is more gradual typing. So if you look at typed racket, you look at typescript, you look at what is sort of kind of but not quite happening in type closure, you see a kind of direction where you can play fast and loose in the early stages and then tighten things up once you know what you're actually doing and only in the places where it makes sense.
00:08:32
Speaker
Yeah, but I think the idea, I totally understand the idea that at the initial phase of it, you want to have as much freedom as you want to, as you'd like to have. But then the choice right now is that you have to rewrite in a different language, which is there is no easy way to switch from. I'm writing completely dynamic shit in Python or Clojure and then now I know the design more or less. Now I can tighten it up, but it's not possible that much with the languages that we have.
00:09:02
Speaker
Well, there are relatively few languages. This is what TypeScript does. TypeScript adds types as a layer on top of JavaScript. If you find, as I do, JavaScript somewhat unappealing to write, then it's unfortunate that that's the language that you're enriching with the types. And type bracket is actually very nice and is a great source of ideas in that broader context. You know, there's a whole research community there that's working on this problem and coming up with good ideas.
00:09:25
Speaker
I think one of the biggest issues that I find in this sense is there's this idea of tightening things up, which I
Thinking in Types vs Functions
00:09:34
Speaker
agree with that. Of course, it would be nice just to apply the minimum amount of types afterwards. The problem seems to be almost cultural, though. It's like whether you think in types or not, or whether you think in data. That seems to be a bigger cultural difference
00:09:50
Speaker
that maybe people struggle with, you know, because, you know, I come from a, like, IQ, I started with C, which was, you know, essentially untyped, you know, you can kind of type deaf things, et cetera. But, you know, it's fairly wide, you know, what's the pointers that can go wrong, opaque pointers everywhere. So you can definitely play fast and loose with opaque pointers. But the point is that when you think like that,
00:10:17
Speaker
you kind of are focusing on the data that's being passed around eventually. But then you go to Java or C++ and things like this or Haskell. Obviously, the type system is more powerful with Scala and Haskell and stuff like that. But you end up thinking in those types first and the data second. Do you think that matters?
00:10:40
Speaker
Well, you talk about the culture and I think there's a real thing there where the language communities have a culture that points in a certain direction, but I would say also that the culture is following to some extent the disposition of the people.
00:10:56
Speaker
And so I think some people just naturally think type-wise rather than thinking function-wise first. And I think some people naturally think better in, for example, objects than they do in functions and data. And so although I have my preferences, it would always seem the case to me that none of these domains is necessarily the ultimate one because different people, their brains work different ways and there should be mechanisms available that match their disposition.
00:11:23
Speaker
Wait, are we going to nature versus nurture
Challenges of Type Systems
00:11:25
Speaker
thing here? No, but isn't it a matter? I mean, you say it's like people. I think the problem you end up with is in a kind of like where you if you if you meet people where they are, which is I think what you're trying to do, that's fair enough. But but if you're sort of like designing like the future of computing or whatever, if you're thinking about how should we think, what are the best ways of thinking?
00:11:51
Speaker
objects and stuff like that don't seem to offer very much. They seem to just offer a kind of odd way of passing up data. For me anyway, having been through that, having been before it and through it and out the other side, it doesn't
00:12:09
Speaker
Well, there are certainly some type systems that I think are less good than others. So I would say that if a person, and I counter this very often in the Closure Community, so I'm going to point it, because I might mostly be closer to hear this, I'm going to point out this particular way of thinking that I think is something of a mistake, is that many people who arrive to the Closure Community's experience with, for instance, object-oriented programming, is specifically object-orientation as practiced in Java and or C++.
00:12:38
Speaker
And they both have actually very bad object systems, ultimately. And my apologies to the designers of languages who are very smart people who work very hard. But if you compare the experience of programming those languages to programming something like Smalltalk or using the Common Lisp object system, you have a completely different world. So I don't think dismissing objects entirely is necessarily as good of an idea as looking at aspects of specific type systems or object-oriented type systems that are a problem.
00:13:08
Speaker
So in the case of Java, for example, the entire culture of Java programming for the first 15 years of the language was based about building ridiculously convoluted patterns to get around the fact that you can't just pass a function, right? So you've got factory, factory, factories because you can't partially apply a function instead.
00:13:23
Speaker
And that's because that type system is really not as expressive as it should be. But if you look at things people get up to in small talk or with class, it's actually very good sometimes. There are situations where it makes sense. And just to give you one example of a place where I would tighten types up in Closure code, for example,
Appreciation for Closure and Digital Art
00:13:42
Speaker
In my early career, I would say the code I wrote the most often was like link lists and hash tables and stuff like that because I was always going to do hardware where I didn't have a good library. But once that stuff kind of evened out, I found myself rewriting essentially the same sort of linear algebra code over and over again. I have to manipulate these matrices.
00:14:01
Speaker
And in those cases, we kind of know what that code's going to look like and what it's going to do. And if I can give some type hints that prevent me from passing the wrong shaped thing in the wrong place and can allow the compiler to generate much better code and so on, then it's worth my while.
00:14:17
Speaker
Because you're mentioning like linear algebra stuff, so mainly are you working like what is the area of focus that that you're applying closure and all the other languages that you're using? Is it it doesn't sound like it's a it's a web programming kind of thing like cloud applications? I don't do much of that. I'll say that
00:14:39
Speaker
I have done some of that in my professional career to make money, but it's not the most fun that I have. But what I do find is that there are a tremendous number of things in my life that are easily expressed in the terms of a vector or of a matrix across.
00:14:55
Speaker
Maybe you need an adjacency matrix because you're doing some kind of path tracing thing and it's more efficient than building out a bunch of nodes with pointers or something. Even that, you need some way to efficiently deal with big boxes of things or whatever, shapes that have numbers in them.
00:15:10
Speaker
But there are also loads of things, simple mathematical things. If you want to do something with a complex number, you're probably going to use a two-vector to represent it. And you want to have it not mix up the types when you're making calls and so on. And Closure has some nice ways to do this. You can use def method to do some of this stuff like you would do in Common Lisp and so on. And those are useful. Those are actually object-oriented abstractions. That's not object-oriented. That's just a better kind of object-oriented.
00:15:39
Speaker
But so are using any libraries, because for this computing stuff, the stuff that is, I think, Dragon and other folks are building the old linear algebra libraries, or are you artisanally coding in Clojure?
00:15:55
Speaker
So I do have a private wrapper around BLAST, LA Pack and BLAST that I wrote in 2009 or so and have been carrying around with me. I have been meaning to update the code that depends on that to use Neanderthal or something instead because I would prefer to push the dependencies out to things that I can share with other people when possible.
00:16:19
Speaker
But then a lot of things are just running on the JVM. If I don't need that much performance, it can be very convenient to just hack it the other way. And in that situation, if you need a really flexible, well-designed API for doing those kinds of things, then Karsten Smith's Geom stuff and his mathematics library, that stuff is really good.
00:16:39
Speaker
He's moved on now to mainly working in TypeScript, but the code still runs fine. It's Closure code. It doesn't rot very fast. And it all runs equally well in ClosureScript and Closure, so you can write this code in a Closure C file and use it in both. So that's what I would recommend for most people who just need a good flexible API and don't care about the performance. If you need slightly better performance and don't care about ClosureScript, then you can use the FastMath library.
00:17:04
Speaker
which is written by someone whose public name is Generate Me blog. He was a very nice guy who's a member of the community and does a lot of art stuff. So yeah, I guess you guys are also where I do a lot of digital art and a lot of that stuff, of course, is quite mathematical in its underpinnings. And so yeah, I'm doing a lot of things with matrices and so on in that context.
AI-Generated Art and Satisfaction
00:17:28
Speaker
Yeah, this digital art thing, well, probably it's not related to closure, but hey, it's not a closure podcast anymore, so fuck it. And we're talking about everything here. So how do you feel about all this table diffusion and what is it, a mid happy face, hugging face thingy and all these things generating art? And then, you know, I know your art is more generative art, like it looks like more mathematical. Yeah, in a sense, everything is mathematical, right?
00:17:55
Speaker
Yeah, it's all, I mean that stuff's all linear algebra at the bottom as well. Yeah. Yeah, I have done work since an early stage using neural networks to produce art to experiment with it, but I have never found the practice very satisfying.
00:18:11
Speaker
So for me, there is a kind of flow state that one can get into where there is a connection with the work that's being done and you steer the work towards the part of latent space that you want by changing the code and it's kind of like a painter painting with a brush. I love that feeling.
00:18:28
Speaker
When I type a clever prompt into a model that generates a picture, often the output is very cool looking, but the actual process that I am using to do it is not at all gratifying. So I don't do that stuff because I don't enjoy it very much. I have colleagues who train their own models and do incredible work. There's a woman who goes by the name of Nural Bricolage.
00:18:49
Speaker
who is one of the best. There's also Mario Kligeman. Shout out to both of them as people who are doing real competent artistic work, not just using it as like a strange art slot machine. But for me, it's not gratifying, so I don't do it.
Tools for Coding and Art Creation
00:19:05
Speaker
But what is your process like then? Because you're using Closure plus your homegrown libraries? Yeah, I've got a bunch of my own libraries, some of which use some of Karsten stuff, some of which use some of Generate Me blog stuff, some of which uses stuff that is only my own. I have a bunch of things that allow me to live code the art as I'm going. So I'm always running in Emacs and evaluating code as changing the outcomes.
00:19:33
Speaker
And say it's a 3D thing, I can export that and render it either in my own ray tracer that I've written in C off to the side, or if I want some really elaborate material for which I've not coded up support, I might import it into Blender. I have a thing that transpiles my closure sketches into a Python dialect that Blender understands, so it emits Blender files that I can then load into Blender and adjust the camera and stuff. Yeah, I mean, it's pretty much what you'd expect.
00:20:01
Speaker
But you didn't try this unity stuff with closure yet. I couldn't get it to work within one hour of playing with it and like at all, and then I stopped. That's basically my policy on those kinds of things. I'll give it a second and if it's too much pain, then I'll just keep using my own stuff. So I haven't done anything with Arcadia yet, but it does. I like the people who made that project and I wish them the best and hopefully at some stage it'll work for me and I'll make some things with it.
00:20:27
Speaker
Yeah, but what do you what do you see like the positive stuff with closure doing all this kind of work and then the pain in the ass stuff in closure for you in the art context? Yeah. So plus other other work that you're doing, like a bit of mathematical work as well, everything.
00:20:44
Speaker
Sure. So for me, I, and I told Rich this the first time I met him at Euroclosure one year is that I think he has done a really fantastic job at curating a set of features that work well together for the kind of programming that I enjoy.
00:21:03
Speaker
So that is, I think, his approach to language design rather than coming up with many languages will be sort of one clever idea and then a bunch of junk, you know, bolted onto it and act and you like the clever idea, but using the language isn't very fun.
00:21:15
Speaker
But with closure, he was very careful. The sort of orthogonal set of things that he chose, all those decisions somehow hang together in a way that makes for a very pleasant experience writing code that then is very maintainable and continues to work for years and years. So I've got some code that I wrote in 2009 that's still happily in production on a VPS out in the cloud doing what it's supposed to do, and I've had to do nothing to that code.
00:21:38
Speaker
in the subsequent 14 years, and I like that very much. The only languages I have that are doing better than that, I have some scheme that I wrote 30 plus years ago that still works, but it's a rare quality in the world we live in where JavaScript libraries explode seven days. It's like one of those spy letters that says we'll explode after opening when you download the package for NPM. Yeah. It's an invisible ink, that one. Yeah.
Future of Programming and Interactivity
00:22:08
Speaker
But that is the positive part of it. So that's the positive part. For me, the negative bits are when I need to do something lower level. Obviously, a JVM-hosted language is not great for that. And so I might drop out and write some C. I have a scheme compiler of my own that's a private thing that I've been carrying around for 25-plus years that compiles to very tight code. So I use that for some things.
00:22:30
Speaker
So those situations where I need to get closer to the metal, obviously, closure doesn't work for me. It's a hosted language and I have to be in a situation where the hosted runtime is the right thing.
00:22:41
Speaker
So I was about to ask whether you use Emacs or some other shit, which is something I usually ask people. And I heard you use Emacs, or is it? Yeah, I've been using Emacs since 1985. My DNA and the DNA of Emacs are now entwined permanently. I will probably never escape. So it is burnt into your fingers.
00:23:04
Speaker
And for people who like to use kind of Apple Macintosh sort of computers and want an Emacs that cooperates with that environment a little better, I have a config, a very simple, small config up on GitHub that is less painful to start with if anyone wants to get involved with Emacs.
00:23:25
Speaker
So what do you see in terms of the future, like future of programming in general? You've got three minutes, Jack, by the way. All right. I'll try to answer the question. Define the entire agenda for the future programming in the next three minutes.
Exploratory Programming and Language Evolution
00:23:37
Speaker
For that, I think I would refer to the last 10 minutes or 15 minutes of my talk at Strange Loop to give you a sense of it.
00:23:47
Speaker
Yeah, so I like applicative languages in general. I like Closure in particular, and we're doing a bunch of experiments over at Next Journal to try to improve programming experience using Closure. One of those is Clark, or for Americans, Clark, which is firstly a computational notebook, but a lot of other things too. You should watch Philippa Markovitch's talk on this from a recent Closure conference where she gives many excellent examples of the kinds of things you can do with that software even today.
00:24:14
Speaker
And we are rapidly building new bits and pieces that will be kind of like the Iron Man super suit that any closure developer can put on and fly around with superpowers. Some of that... So it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it's basically, it
00:24:37
Speaker
Well, I think interactive programming is super important and I think it's an area that we've mostly been ignoring for the last 20 years or so in how we approach designing programming languages and programming environments. I like types and I like using them, but I do feel we've been obsessed with them in academia and it would be nice to work more on some of these user developer experience kind of ideas. We have less than a minute. Nice.
00:25:06
Speaker
a few moments later. So yeah, so you're talking about the exploratory thing that you think it's going to be like one of the things that we ignored. And then, well, kind of it's on the fringes of the programming, right? And then it became
00:25:23
Speaker
sort of mainstream with Clojure and not anymore. Probably in as mainstream as you can call Clojure, I suppose. You know, it's always been there in various languages, the Smalltalk people, the Lisp people. They have a long tradition of interactive programming and the fourth people, of course, fourth is interactive. It's one of its great strengths. And many of the other languages that are not interactive could be.
00:25:46
Speaker
So, for example, if I'm programming in Haskell, I tend to use the Haskell interpreter off to the side.
00:25:55
Speaker
that that binary crashes quite often. It's extremely fragile. So I suppose it's not completely correct by construction. But, you know, I do what I can. And the same if I'm using OCaml, I also, there I will always run the top level you top off to the side, because for me, I want to interact with the program as I'm running it. But because the cultures, and this is back to what Ray said earlier, because the cultures of those languages are not as interested in interactive programming, the tools are not as good for it.
00:26:24
Speaker
But did you, well, obviously you did, so I'm imagining. What other languages that you use? Like do you reach out to Closure as the default one for all the stuff? Or you already mentioned a bit of Scheme, any non-glispy languages that you're interested in or playing with, or you see some promise? I've used dozens of languages over the course of my career and there are things to like in many of them.
00:26:51
Speaker
and things to dislike in all of them. That's just the nature of programming languages. I would say that my default language for most sort of day-to-day programming tasks is Clojure at this stage. It's a nice language. I'm very familiar with the standard library and the surrounding pooling and all those other things that make it easier to get work done. But obviously, if I'm doing something lower level, then I'll reach out to other language. I've recently written, just as an example of some other languages, I've recently rewritten some things in
00:27:21
Speaker
Rust and in Go and a bit of assembly language I deport my scheme compiler to ARM when I got a new laptop, so that was a couple of days of fun. Yeah, so what I reach for depends. Oh yeah, and I use Python for some machine learning things sometimes because it's easier to participate in the ecosystem than to start over from scratch over in my own world. Not always, but sometimes.
Python in Machine Learning
00:27:45
Speaker
Yeah, actually, I was about to ask about Python because, as you mentioned, most of the machine learning stuff, I mean, it's kind of like took over, right? Python basically became the de facto.
00:27:56
Speaker
language for all the data science-y sort of stuff. But do you miss anything from there, like all the scikit-learn stuff or all the prophet and different libraries that Python people put out, mostly PyTorch and everything? Do those things, do you think those things are not missing in the enclosure world for you?
00:28:18
Speaker
Well, I would, so I'm in the weird position where I know all the stuff from the math side and all the stuff from the machine side. So when I need a thing, I generally just make the thing, but it's not practical for most people in that situation. They're either sort of more data sciencey people where they know the maths, but they're not really implementers and it would take them forever to build the libraries they would need or they're implementers, but they are not really strong on the other side. And so they wouldn't really know what to build and so on.
00:28:45
Speaker
So from that sense, I don't miss it so much. But I will say that because the ecosystem is so large and well-developed for things like machine learning, that it is pretty time-consuming to do it yourself, to recreate a lot of those things. On the other hand, I have found on the Python side that the library ecosystem is so enormously fragile. And fragile all the way down to like if you're doing some machine learning thing and you're using an NVIDIA video card,
00:29:14
Speaker
and now you're trying to run a library that was written a year, you know, something with a library that was written a year ago on a new piece of hardware, something will have changed in the API and the driver that will require you to refactor code at every layer up the stack where you can't even run the code. So the rate of bit rot there is just excruciating. Whereas in my own stuff, I have code, you know, 15 year old machine learning code that still does what it's supposed to do because the layers underneath are not quite so quicksandy.
00:29:41
Speaker
I think the way the dependencies fail in Python is crazy, or not change the levels of transitive dependencies. And then because I write some Python code, and then four levels deeper, some library changes something, and everything breaks because there is no constraint put in in one of the things. So this whole package management thing is a shit
Next Journal's Reproducibility Focus
00:30:05
Speaker
show.
00:30:05
Speaker
But I'm curious about your view on next journal versus, because obviously, that's the one that you're working on. But if you see the things like JupyterLab and these things, is there any difference that you see, or are they trying to solve the same problem like you guys?
00:30:25
Speaker
So in the case of Next Journal in particular, the emphasis was on reproducibility, on avoiding exactly the kind of thing I was just talking about. So that was one and the other emphasis was on being able to use multiple languages in the notebook in a very transparent way.
00:30:43
Speaker
So with Next Journal, you can go through and have Julia code cell here, and a Python one here, and an R one there. And they can all talk to each other through a kind of file system layer that's happening underneath. And for each of them, the code you've written, plus all of its dependencies, is stored permanently in a content addressed way in a Docker container that can be pulled up later.
00:31:07
Speaker
So the idea there is that if scientists do their work, we want to be able to rerun their analysis later. Otherwise, there's no reproducibility. Even if they did good work, we don't know what happened. We'll leave aside the problems of just getting scientists to post their data publicly. But assuming that they have posted their data in their code, a lot of time the code for Python reasons that we just discussed is useless within six to 12 months because the ecosystem has moved on.
00:31:31
Speaker
So, I was not involved in making the decision to build that project. That's really Martin's baby, and his thing was to try to make science work better by making better tools for scientists to use. So far, I think it's mainly turned out that the incentive structure for the actual scientists has not pushed them in the direction of caring about reproducibility over the long haul, which is a cultural problem rather than a technological one, and thus much harder to solve.
00:31:56
Speaker
What do you use to pass the data around? Because you hand wave, you set some file system type stuff, but I guess you need something to serialize and deserialize or... Yeah, so the bad news is I don't have like a, we just use Apache Arrow kind of story that unifies everything in a nice tidy way. Instead, you have to pay attention to what file formats you're using between the different things and so on.
00:32:19
Speaker
The code cells run top to bottom, and there's a kind of file system projected underneath that's shared among the different runtimes that are running the different languages. And so if you're emitting some format that you have a library for in your various languages, then you can read them back and forth between the languages.
00:32:36
Speaker
In an ideal world, we would probably have a dataflow architecture that just handles those things directly without a file system abstraction. But unfortunately for most of these languages, it would mean fairly radically changing the way the code works for the language and thus breaking compatibility with what I think of as copy paste compatibility. Yeah.
Good Practices in Code Documentation
00:32:55
Speaker
Yeah. So you get ended up with a Jason or Afro or portable for something. Yeah. Yeah. We would end up with something like that. And then run times that would look more like observable, but for the various languages, like that would be the nicest way to do it. But again, it would, it would break ecosystem things that people need because the whole reason they're using these languages is because of the, you know, MedX do what I want aspect of the libraries.
00:33:20
Speaker
Yeah. But if I understand correctly, because you can use multiple languages in the same flow, passing data between them must be pretty complicated, right? Well, it's not too bad. Everybody can read CSV, for example, and a bunch of other formats like this, some of which are packed binary formats that everybody has a library for. So if you have to do it, you can do it this way. And there's some other cleverness, but I'm not sure how much of it is shipped at the moment.
00:33:48
Speaker
Yeah, but I think the thing about notebooks in general, which I like, is this literary programming aspect of it, where you can communicate the growth of a program. Because I think that's one of the problems with programming, isn't it, is you end up just looking at the finished thing. And it's difficult to communicate what choices you made as you went along in that environment. That's the nice thing about a kind of literary style, where you can
00:34:17
Speaker
Yeah, like I said, you can kind of show the thoughts and the options and the choices that you made. And then people can review it in that sense, which I prefer than this bullshit we have with GitHub at the moment.
00:34:30
Speaker
Yeah, I would agree completely that literary programming is a very nice way to kind of explain what's going on. And I would say that we don't generally, as a culture in programming, we don't spend enough time thinking about explaining the code to people who didn't write it. And that person who, or maybe didn't write it isn't enough. Like that stranger who's looking at the code scratching his head six months from now might be you. So you really want to make sure that you've wrote down what you were thinking right now.
00:34:59
Speaker
So that that future person, that future version of you isn't just cursing your name. Yeah, I got I got very frustrated with this idea of just read the card, you know, it's like, well, yeah, okay.
00:35:11
Speaker
can do that, but it's not very efficient. And I would extend that even to code review. Other than very trivial things, I don't believe in asynchronous code review. I want to actually talk to someone and have them explain to me what they were trying to accomplish and what trade-offs they made. I don't want to just look at the code and make sure there are no typos. The compiler can do that. Yeah, exactly. Yeah, that's why I like pair programming, mob programming, things like that.
Inefficiencies in Code Review Practices
00:35:33
Speaker
And fully enough, I was talking to a colleague of mine the other day about, and he was saying that they do exactly this, where he used to work, where they would
00:35:40
Speaker
not do a PR in the traditional sense, they do a kind of walk through PR. And we're going to start doing that as well. Because we've mostly been pairing, which is great, because then you don't have to explain so much. But as we do less and less of that, then yeah, these walkthroughs sound ideal. Yeah, that's definitely. I haven't got much experience of it, to be honest. I don't know about you guys.
00:36:03
Speaker
So we pair a lot, at least in my team. The other part, though. The walkthrough PR is not something we've done. Unfortunately, we're mandated to have PRs and approvals on them from auditors. But I kind of agree that they suck in the sense that.
00:36:22
Speaker
If either the PR is trivial and then I can read it in five minutes and sign it off and it's fine, but then it probably didn't need to review or the PR is non-trivial and it's essentially going to take me more time than it took the person who wrote it to review it. And generally speaking, that doesn't happen. So you sort of get a rubber stamp.
00:36:45
Speaker
But yeah, for audit reasons, we're sort of forced into this process at the moment. But yeah, it's. You guys are using PRs. I mean, we were supposed to send a patch via email and attach it to Jira. Keep forgetting. Yeah, it can always be worse. That's true.
Tools and Teamwork in Coding
00:37:04
Speaker
So i would say also that not doing the walkthroughs the very thing that you're talking about about how you just get to the culture of lgtm right looks good to me because nobody has time to actually the thing and when i look at that combined with the style of management and this is from a particularly degraded form of agile.
00:37:23
Speaker
that works by making a zillion tiny little tickets and then one person takes a unit of work off and in isolation of the rest of the system works out how to kind of half-ass get the feature or the bug fix done and then somebody else rubber stamps it and it goes in and then you look up in six or twelve months and you realize that you have an unlimited quantity of architectural debt because no one has been zooming out and discussing the big picture as they've been going.
00:37:48
Speaker
And that's one of the things that I think is better when you do these walkthrough PRs and you talk about the motivation and you talk about the architectural consequences of the decisions you're making. How good is this API going to be? How much will it distort the rest of the system to have this API? This is the grown up stuff that you have to be able to do ultimately.
00:38:05
Speaker
Do you think there is a chance, or based on the literature, or the knowledge that you have and experience, so is there any technical solution to this problem? Or do you think it's purely people's problems, so you just need to slap on their fingers and then say, no, we're going to do this way?
00:38:22
Speaker
Well, I don't recommend finger slapping necessarily. We're going to do that part out. And then we say, Jack said. Now, what I mean is that organizational change is a different thing than if the technology for like type systems, like Haskell forcing you into a specific type of thinking. Do you think the tools can help here or the technology can help here or the language can help?
00:38:49
Speaker
Well, I think the tools can help in the sense that they can reduce the friction to doing what is the right thing in terms of they can help the people to work together better. And this is partly because of the most common disposition we have as people who fall into this trade.
00:39:06
Speaker
Are people who work very well alone most of us and so the tools are often designed with this idea that everyone is an island and they're sending notes in little message in a bottle style off to the other islands. When we should be doing more work together and are the tools that we use should be facilitating encouraging that kind of togetherness.
00:39:27
Speaker
and then good management, which is actually very rare and very difficult and which helps people to establish trust within teams to be able to do their best work. Because if you don't feel like you can say, I'm not sure if that's a good idea because this is the guy who blows up every time you criticize anything he does or whatever, then those things also will gradually destroy the quality of the work that comes out. The human factors are intensely intertwined with the technical factors. And if you try to work on one in isolation of the other, you will have problems.
00:39:57
Speaker
Yeah. I mean, if there is a great tool, and then that's going to solve all the problems, and then everybody will adapt it, and then we'll all be using Emacs anyway. I'm not sure if it is a tooling problem. Your mic's very quiet, by the way. I'll get a bit closer.
Importance of Documentation
00:40:14
Speaker
While you're doing that, let me interrupt.
00:40:18
Speaker
I've had it. Go on, go on. No, I don't think this is a tooling problem. I think we fall into the trap of thinking that stuff can be fixed with tooling. But other engineering disciplines have this as well. Like in electrical engineering, only for electrical engineers, you can't solve this with another tool. I mean, you're designing a CPU or a board, some documentation on the side is going to need to happen.
00:40:46
Speaker
And yeah, but why would you, you're an idiot, mate. You're not thinking about software. You know, software can fix everything. Remember that. Yeah. No, this is, this is, I mean, I'm joking. Of course you're definitely not an idiot, but, uh,
00:41:03
Speaker
But the thing is, this is the problem with the Silicon Valley mindset, is that we just need tools and we just need this idea that it's solvable via tooling is in itself a kind of illusion that they're very happy to sell.
00:41:19
Speaker
Yeah, I think certainly looking for some kind of absolute solution from a technical tool would be ridiculous. But I do think that we tend to follow the easy path of the tools that are presented to us. So in the GitHub PR situation, the easiest way to do that is you shoot the PR off, and somebody else, their easiest path is to type LGTM, and then it's done. So you need both social infrastructure, but also tooling infrastructure that encourages the good stuff.
00:41:48
Speaker
And so what we try to do is we have like a hierarchy of documentation. So there's stuff that goes into code comments, which is very much like this implementation is kind of iffy. Let's write down why it is iffy. So I understand next week.
00:42:07
Speaker
Then we've got this thing called ADRs, architectural decision records. They're essentially markdown files. We ship them in the same repo and they're the zoom out stuff you talked about where, you know, if you need to change an internal API or something around how the code is structured, you know, module wise and communicating.
00:42:30
Speaker
that will be captured there where we'll try to sort of address what problem this is solving, how is it evolving internal APIs, et cetera. And it's again mainly for us to capture the discussions we had with the team.
00:42:45
Speaker
And then we have what we call like zips, but there are other companies called them RFCs or whatever, which are like the really high level architecture of like, okay, we have this system. This is its purpose and this it's super high level, you know, clouds and arrows diagram. And yeah.
00:43:07
Speaker
which we tend to write before we even begin coding a system. But I think it's mostly discipline, right? Yeah, but it requires a lot of discipline to capture all of it and capture all of it greatly. And I think their tooling can help. But I haven't really seen anything that's better or saner for this type of stuff. I think it comes back to Jack's point, though, that a lot of it is
00:43:35
Speaker
management and leadership and having people that understand the ineffable nature of this problem. Yeah, integration goes probably. Speaking to the ADRs, what I find so important about documents of that kind is that when you look at the, you know, you bisect your repo and you look over it, you can see very quickly what was done.
00:44:00
Speaker
But without the ADRs, you don't know why it was done. And that's the part that if someone comes along later and they see something in the code that looks a bit fishy,
00:44:08
Speaker
They may start trying to refactor that code and then spend a couple of weeks to find out that they're breaking something at a distance because there's some reason or some dependency. There was a why involved with that decision. And so, yeah, that's I think super important that you take the time to write those documents that clarify those things.
Facilitating Better Programming Practices
00:44:24
Speaker
And it's also I think speaks very well of your team that you do it in the repo because that's where those things belong. And a lot of teams will split those things off and put them in Google Docs or somewhere far away from the code. And the farther away they are, the less likely it is anyone will look at them.
00:44:38
Speaker
And that's one of the kinds of things I mean when I talk about tooling decisions. Is your tooling decision to make somebody use a crappy web app that's far away from their editor? Or is your tooling decision to use something that feels very native to their workflow that will encourage them to touch those things more often?
00:44:54
Speaker
Yeah. Do you have to log in to understand the code? Yeah. It's a big problem. One of the main reasons we chose to commit them to the repo is actually because... Yeah, you're incredibly quiet. Well, for a sound man, come on, mate. I mean, it's like why we chose the repo also is because it does happen to help with git bisect like it may not happen at like the same git commit.
00:45:17
Speaker
But the ADR will have been committed either two commits earlier or four commits later. It ends up at roughly the same time period as when a particular code change has happened. It's not entirely perfect, but at least for me, I find that it helps grok things. You only have to roll time back or forward just a little bit to find the why.
00:45:45
Speaker
Yeah, now imagine that you had a tool for version control that allowed you to add annotations to the code and to that particular version of the code. And then you can scroll back and forth through it and see the things popping up and disappearing as it goes to get kind of a transcript of the decision making that went in. So this is what I mean by there can be better and worse ways of facilitating the kinds of things that you're talking about. Yeah, definitely.
00:46:12
Speaker
Cool. Yeah. So I think the whole idea of like having like language sensitive tooling as well is super important. You know, because there were certain trivial changes to types and to data and to functions or whatever. And there are more strategic changes and a tool could potentially find them.
00:46:33
Speaker
Nice. So I think we're almost like out of time probably, as promised. Let's be respectful. After hopping through like 17 different services and then going back to this one. Jack, just to wrap it up, what is next
Exciting Projects in Closure Community
00:46:54
Speaker
for you? What is something that you're looking forward to in general? Not just work, but any computing and
00:47:01
Speaker
you know, anything that you're super excited about or from all the way from the 80s on, we had an almost like 30 years and what is the next 30 years?
00:47:13
Speaker
30 years is a long time, but what I would say I'm excited about right now from us and from the wider Closure community is, well, there's a project called Tracy, which is a tracing debugger for Closure that we're building as a kind of Clerk add-on. And I think that's going to be really great. And we've built this sort of CI hosting thing for Clerk Notebooks called Garden that I think is very good, but which will grow into something much more ambitious that I'm not ready to talk about very much now, but I think could be used by many, many Closure
00:47:42
Speaker
programmers in the future. I'm pretty excited about EMI, which is a computer algebra system, by Sam Ritchie that you will see at the conge this year. There's also Borkdude's work. I mean, all of it is really, really great. But in particular, what he's working on now with Cherry and Squint, which are like alternative ClojureScript compilers, I think could really change the game in terms of using Clojure on the front end.
00:48:06
Speaker
And I think there's a lot of wonderful work being done in open source libraries for people like Matos and Chux. Now in the wider world, I would say that the uptick in interest in data log is really great to see. I've been using data log like databases and in fact building them since probably, well, 25 years now and it's fantastic to see that the atomic has finally gotten people to wake up to how good that model is.
00:48:30
Speaker
I'm enjoying seeing some backlash against React on the front end, I think for many things. It is sort of a performance sink and quite complicated. So hopefully we'll see more alternatives and move to things like HDMX.
00:48:46
Speaker
Optimistically, if Wasm goes in the right direction, I think we should see a flowering of interesting things there, but it'll have to get garbage collection and access to the DOM to be everything that it could be. Again, optimistically, I really like Nix as a package management strategy. It would be fantastic if it were to become easy enough to use that more people would in fact use it.
00:49:07
Speaker
And then very optimistically, I'm interested in seeing paid end-user storage as a utility so that more apps can actually be things that people own that store their data in a place they pay for and can move away from the kind of surveillance capitalism model that has taken over much of the internet.
00:49:23
Speaker
Nice. Hey, thanks a lot. Cool. Thanks for your patience and all the shitty technical things. But it's super exciting to finally talk to you. And I think we hopefully will bump into each other at some point, maybe somewhere in Europe, because I tend not to go to the US.
00:49:45
Speaker
Yes, I mostly try to avoid it myself. It's far, it's far from where I live. That's the main reason for me as well. Yes, wink, wink. But it is. It is a pleasure to meet you lads. And I'll see you on the closure and slack. Yes. Yeah, we'll see you on the conference circuit. No doubt. All right, take care. Nice. Yeah, see you. Have a nice day.
00:50:12
Speaker
Thank you for listening to this episode of Deaf Hand and the awesome vegetarian music on the track is Melon Hamburger by Pizzeri and the show's audio is mixed by Wouter Delert. I'm pretty sure I butchered his name. Maybe you should insert your own name here, Delert.
00:50:30
Speaker
If you'd like to support us, please do check out our Patreon page and you can show your appreciation to all the hard work or the lack of hard work that we're doing. And you can also catch up with either Ray with me for some unexplainable reason. You won't interact with us. Then do check us out on Slack, Closureion Slack or Closureverse or on Zulep or just at us at Deafened Podcast on Twitter. Enjoy your day and see you in the
00:51:26
Speaker
next episode.