Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#48 David Miller and Clojure on the CLR
81 Plays6 years ago
After 10 years of diligently maintaining the CLR version of Clojure, David discusses the project for the first time.
It was our absolute pleasure to speak with David and discover his motivations, ideas and plans for the next generation of the CLR port.
David on GitHub - https://github.com/dmiller
Clojure on the CLR - https://clojure.org/about/clojureclr
Recommended
Transcript
Introductions from Different Regions
00:00:15
Speaker
Welcome to DeafN episode number 48. This is Vijay from Holland. Reverend Belgium. David from Chicago. Hello David. Welcome to DeafN.
Introduction to Closure CLR and David's Background
00:00:29
Speaker
Welcome. Thanks for having me here.
00:00:32
Speaker
So I think we were trying to talk about different things that are happening in Closure World. And obviously, you've been working on Closure CLR. So we thought that'll be a nice topic for us to discuss today.
00:00:49
Speaker
So before we get into the whole discussion about CLR and all the things that the work that you're doing, can you give us a brief introduction about yourself and what you're working on and where did you come to? How did you end up in the closure world? Okay. I'm an academic by profession. I'm a working computer science though for the last 25 years. I've really been a college administrator.
00:01:18
Speaker
I currently serve as Dean of the College of Computing and Digital Media at DePaul University in Chicago. That's my day job. That's what pays the bills. I've been programming since, I don't maybe want to say how long,
00:01:37
Speaker
I think I started my first Fortran program in 1973. Well, right. Yeah. And I probably wrote my first Lisp program around 1978, I'm guessing, when I was a graduate student. You will. I win a lot of things in that regard. If it just comes to being old, I'm working on it every day. It works a little better. You're winning every day.
00:02:07
Speaker
Difficult to catch up. So that's what I do during the day.
DePaul University's Operations and David's Role
00:02:16
Speaker
It's a big place. We have computer science and design and actually film school in my unit.
00:02:25
Speaker
About a hundred and twenty full-time faculty five thousand students well when you said hundred and twenty i thought number of students. That's full-time faculty probably another two hundred part-time faculty fifty full-time staff it's a it's a big operation.
00:02:45
Speaker
Computer science is the biggest part of that, and that's the part I originally joined in when I started at DePaul. So that's what I do. I've never been a professional programmer. I have done consulting on the side, but it's never paid all of my bills by any means.
00:03:04
Speaker
But I've obviously had to stay up with computing technology over the years. So this was, Closure CLR was one of my ways to sort of keep my hand in things. The impetus, I've done lispy things for a very long time.
Academic Contributions and Lisp Programming
00:03:24
Speaker
As I said, I started, I think, my first lisp program in the late 70s. And actually, as a graduate student, I
00:03:33
Speaker
The university I was at didn't have much in the way of programming courses. So a friend and I created a programming sequence, an intro programming sequence that started off with one quarter of Pascal. You might remember that language. Second quarter was data structures. And the third quarter, just for fun, we tossed in symbolic programming and Lisp because it was, you know, no one told us what we could do. So we just did whatever we felt like doing.
00:04:02
Speaker
We feel exactly the same way. My background was in mathematics and in logic in particular, so the lambda calculus and stuff was dear to my heart anyway. So Lisp was an appropriate place to be. So I think I started a Lisp, there's probably some Mac Lisp variant of some type running on a
00:04:26
Speaker
deck 10 or something like that back in those days. And when I went to DePaul, I did a lot of artificial intelligence work. And so I did a lot of lisps and I cannot tell you how many lisps I've worked in. I worked in
00:04:42
Speaker
Lisp, I can't remember, but Franz Lisp was one. And nil, which was a predecessor to Common Lisp.
Symbolic Programming and AI Work
00:04:55
Speaker
It was called nil? Yeah, it was called nil. No one ever heard of it. But to give you an idea how long ago we're talking, I got nil because I sent off to MIT and asked him to send me a mag tape.
00:05:13
Speaker
That's how he got the source for that and got it up and running. It was on predecessors to Common Lisp when it was still being developed by the folks at Carnegie Mellon. I've had Lisp pitches of various types scratched by different things over the years.
00:05:34
Speaker
Is this something that's been common in the whole journey? For me, it's been there at various points along the way. As a practical matter, I've never actually taught it, for example, because the courses I taught never really required it. I've done a lot of other
00:05:55
Speaker
other things, some of which I can talk about, some of which I prefer not to remember. Particularly because I did some teaching and certificate programs within my unit, which were not degree bearing, they're just sort of like, you know, here's some skills for the classroom and they're for work and other things I can't even remember anymore.
00:06:19
Speaker
Don't talk to me about com programming and some other things because I have bad memories. Or Visual C++, which I also worked in, and the Microsoft Foundation classes. In fact, I'm still maintaining old code I did. Yeah, it gives me nightmares. It's all right.
00:06:43
Speaker
You know, with the Lisp thing, David, because you mentioned the lambda calculus and that kind of stuff. So was it like the functional programming stuff that got you into it? Or is that something which is a separate matter as far as you're concerned?
00:07:02
Speaker
Well, I mean, of course, all lists are some variation of functional depending on how you want to define that. So it was really more the symbolic programming aspects. When I was still doing some AI stuff, it was not the modern day. It was very much
00:07:21
Speaker
a much more focused on symbolic programming. I actually taught courses in practically how to use symbolic programming techniques in industry. I actually did some consulting work that delivered in prototypes in Lisp, actually. How would you describe symbolic programming to people that aren't familiar with it?
00:07:49
Speaker
I don't know, non-numeric, non-standard data structures. To give you an example of one thing I built, there was a scheduling engine for a steel mill that we defined a form of temporal calculus where we manipulated
00:08:08
Speaker
notions of intervals and time constraints on intervals and did that as a formal calculus that we implemented in Lisp and then plugged that into an engine that did some
00:08:24
Speaker
highly heuristic backtrack searches in order to develop schedules for the steel mill.
Evolution of Programming Tools and Editors
00:08:30
Speaker
Much easier to do that in Lisp than anything else in terms of both doing a REPL-style development of code, but also just the sheer data structures that were available to work with.
00:08:43
Speaker
You see that, of course, in enclosure these days because it is in that line of work. I re-implemented some of those things in C++ so you can be done. It's just not nearly as much fun to do it. I think, can we get to one of the burning questions that we have in this podcast regularly? I think I want to get that away first. So, Emacs or some other shit, I mean, what do you use?
00:09:09
Speaker
Oh man, I've used everything. Except editors that begin with VIM. I was going to say with V, but I've used Visual Studio a lot. So I took care of VI. I couldn't get it down to that. So I did Emacs for a long time. I've not been doing it that recently because of the kind of code I've been working on. I've been mostly programming in C sharp these days, which has landed me in the Visual Studio world. And I can live there. I can live anywhere.
00:09:39
Speaker
I started on punch cards, so nothing fixes me. Yeah, exactly. That's what my next question was. But that was my next question, actually, because how did you see the transition of the tools as well? Because it's been almost 30 years of programming or 40 years of programming. So how do you see the tooling and the IDEs and because starting from punch cards all the way into Visual Studio or Emacs?
00:10:08
Speaker
Um, well, I don't know that I can say anything tremendously insightful about it. I mean, obviously, uh, you know, moving up from punch cards to, you know, even just, you know, I had emacs some variation of emacs very, very early, actually in the late seventies, I guess I was started working in it in different places.
00:10:29
Speaker
But I got to tell you, for a lot of the work I did in various places, Visual Studio was having the goodies to do the visual environment work, to build the interfaces. That stuff was all fine for me. The modern world is just as interesting. I'm liking seeing a lot of the lighter weight alternatives.
00:10:51
Speaker
I haven't had time to play a lot of VS Code and some of the other stuff that's out there right now in that environment. I spend as much time in Notepad++ as other things. Whatever I got sitting on a machine, my biggest problem is I've used so many things.
00:11:11
Speaker
and switch around so much, it doesn't matter what tool I am, I'm always lost. You know, I used to be really good at all my emacs, you know, keyboards and it's not built into my fingers anymore like it was. I mean, I was one of the guys that, you know, started getting repetitive stress injuries in my little finger from, you know, holding down control keys and things like that. But yeah, yeah. Yeah. Watch out. I think
00:11:42
Speaker
I think now I have the caps lock syndrome instead of controls syndrome. Right. Yeah, you switched up. Yeah. And I think for better or worse, I switched to the WIM model editing in Emacs as well.
Motivation Behind Closure CLR Project
00:11:56
Speaker
Yeah. It helps a bit. Anyway, so Closure and CLR. So how did the project start, and how did you get into this thing? It started back around,
00:12:12
Speaker
just over a decade ago. It was in late 2008, actually, when I started thinking about it. I was bored, mostly, because I'd been doing a lot of, I'd been programming at .NET and playing some other games in terms of what I was teaching and a little bit of consulting work, but I was not feeling the love, particularly. I wanted something interesting to work on that would have a little intellectual challenge.
00:12:40
Speaker
And there were three things that came together for me in my head at that time. One was, it'd be nice to do some list beginnings. I hadn't done any real lists in any real sense for a decade, probably. I was interested in doing more work in functional programming also. I felt like there was a whole side of functional programming that had developed. While I wasn't paying a lot of attention, I was off doing other things. I wanted to put my face into that.
00:13:10
Speaker
And I was also interested because of some other projects that were going on with how some of these things were being implemented on runtimes like the JVM and the CLR. In particular, this was around the time when there were things like Iron Python and Iron Ruby being developed on the Microsoft side. And I was just curious as to how they were approaching it. I've taught a compiler class a long time ago, but I, you know,
00:13:40
Speaker
I never actually tried to develop that kind of thing. But I was just very interested in sort of what the techniques were. And as I was kind of playing things through in my mind, I just happened to run across an article about closure in a couple of different magazines or trade pubs. And it just struck me that that was a confluence that might bring all those things together.
00:14:06
Speaker
In particular, I knew that Rich Hickey was doing this mostly on the JVM at that point, but if you actually looked and read carefully, you would have seen that he had originally started the development jointly in the JVM and the CLR. So I said, well, if no one's doing it on the CLR, let me just play around with it.
00:14:35
Speaker
I don't know if you know, the real background is that I went back and did a little spelunking on the web, but Rich did things before a closure in various aspects of LIS before he settled on closure.
00:14:50
Speaker
There was one thing he announced back in 2002, which was .LISP, which was described as an interactive LISP-like language for .NET scripting and development, having deep .NET integration.
00:15:06
Speaker
He had ideas about that going back well before closure actually existed. I guess it must have been sometime around 2005. He took time off from consulting to start working on developing closure in the background. The initial public release was announced in October of 2007.
00:15:31
Speaker
It was kind of late, about a year after that, that I was just thinking about what I was going to do. And that's when I first ran it across the notion, I mentioned closure itself. So I wrote Rich in January of 2009 and asked him if he would maybe answer questions for me as I worked on this project. I went back and found the email and
00:15:53
Speaker
When I wrote to him, I sent him a screen capture of my initial attempt to close your CLR. I don't know if you remember one of the first things that he used. Did you pick up his code, by the way, David, or was it? Oh, yeah, absolutely. I mean, I wasn't smart enough to do it from scratch.
00:16:14
Speaker
So not the CLR code. I started off with what was the public code. That was all JVM. So I don't know what he wrote before that. This was just a straight out the JVM code. I just started rewriting it. Yeah. There was no CLR code in it. Did you find the CLR code and pick it up? No, I can't. No, no. Actually, I looked at it. And I had actually, if you went back around in that early history, the ants demo was a rather famous thing that he demoed. So I actually sent a screenshot of the ants demo playing.
00:16:43
Speaker
in my closure, CLR, which was the GUI on it was implemented in Windows Forms. And so when I first talked to him, I had that much running. It was incredibly slow at that time because of some of the techniques I was using. Certainly nothing had been optimized. But when I first wrote to him, I already had all the basic data structures and
00:17:06
Speaker
vars and refs and agents and transactions and multi-methods were all functional and I could do interop to do things like Windows forms programming. So, I was already looking at things like iron Python and iron Ruby which were based on the dynamic language runtime, the DLR.
00:17:28
Speaker
that provided some of the tools for trying to embed languages like this and provided some very nice tools I may talk about in a bit for doing this. So I wrote to him in mid-January 2009, he replied a couple of days later and said he was interested in having an official.NET port of closure based on original closure sources. I'm quoting here.
Challenges and Technical Aspects of Closure CLR
00:17:53
Speaker
There's definitely interest in the community as well.
00:17:57
Speaker
And then he said, what is important to me is that any effort I apply ends up in the closure project. Towards that end, I would ask you to consider making your efforts a contribution to closure under the contributor agreement.
00:18:09
Speaker
Then your .NET port could live in Closure Contrib alongside ClosureScript, which I didn't realize ClosureScript even went back that far, so that's longer ago than I thought. The long-term objective would be a set of core CLJ files that supported all ports without changes, all platform specifics isolated in the support libraries.
00:18:30
Speaker
That last part still hasn't been accomplished yet. We can talk more about that later. So I actually had no plan on initially doing anything public. To me, this was a side project I wanted to work on. And then if it got good, maybe I would think about opening it up. But Rich basically said,
00:18:50
Speaker
If you want to talk to me, open it from the start. So I was public rather earlier than I would have liked. So the first commit to the closure CLR repo was February 3rd, 2009.
00:19:08
Speaker
So we just passed our 10th anniversary. And in 2080 commits later, here we are. So it's been an interesting 10 years. So that's how it got started. Over the years, Rich answered a few questions. And other than that, I've been pretty much just doing it.
00:19:34
Speaker
And how is the design process? I mean, is it like because if I understand correctly at some point, the main closure was essentially on JVM. Yes. So what were the challenges or still open challenges that you solved or you're still working on to make it CLR compatible?
00:19:58
Speaker
The ones I solved are still the ones I'm working on, and I have some interesting plans for that. I mean, ClosureCLR has been functional and mapped the functionality of the mainline Closure. How should we refer to the other Closure, just Closure, I guess? Yeah, Closure. When I write about it, I sometimes refer to Closure JVM, but I'd be the only person to call it that. Yeah, fair enough.
00:20:21
Speaker
There were some real challenges based upon the difference in platforms. I mean, at one level, they're not so different that I had to do massive rewrites. So my original process
00:20:37
Speaker
when I started to build it was to run through all the support code, for example, the data structures. You have all these wonderful persistent immutable data structures floating around that really form the core of it.
00:20:52
Speaker
you've got things like vars and you've got namespaces and you've got these basic stuff that's the underlying infrastructure that's implemented in Java on enclosure JVM. I could translate relatively straightforwardly. Rich writes beautiful code, very clear. That translation process from Java to C-sharp is not exactly automated, but it was very easy and it didn't require that much thought.
00:21:24
Speaker
So, you know, I could port those over and get them running fairly quickly. And that was okay. The compilers where obviously you run into the problems because that's where you hit the actual underlying platform in a direct way. And the problems there are several.
00:21:47
Speaker
It's not even at the level of emitting the IL code that comes out because even that has got some similarities. I mean, a lot of the same op codes are there and do the same sorts of things. It's the same underlying model for the engine. But where you run into differences are the things that the CLR has added or done differently than the JVM. And some of those I don't handle well today and have some plans to do that. We can talk about that.
00:22:16
Speaker
Some of the things are like just assemblies. There's a whole different model for how you package IL code in the two things. You have class files over in
00:22:32
Speaker
the JVM world. Is that what they're called these days? Yeah, yeah. Okay, thank you. And, you know, everything is, the class files are all in individual at a class level, right? And you can just, you know, transport them on mass someplace. Everything is tied up in packaging into assemblies on the CLR. And so you really have to think a little bit differently about how that interacts. There are
00:23:04
Speaker
you know, different numerics that you have to deal with. And like, for example, there are no unsigned in JVM world, right? But there are unsigned integer data types in the CLR. That's not a big problem until you look at
00:23:20
Speaker
some of the assumptions that are made about how, for example, the integer types are handled within the language. For example, everything promotes up to long. That's okay. As long as we don't pretend there's an unsigned long around, you're just fine until you actually want to interoperate with the underlying platform that actually knows their unsigned logs, you need to deal with them.
00:23:38
Speaker
But probably the two biggest pieces that cause problems are value types and generics. Value types, they exist only in a very limited set of data types within the JVM, primarily the primitive numeric types. So ins and longs and doubles and those things are all handled as stack allocated.
00:24:07
Speaker
unless you need to pass them to things that don't know that that's what they're getting. So you need to, at that point, put them out on the heap and put a reference to them. So you basically box them, as it is said. So you move from a small L long type into a capital L long type, which is a boxed version of long. And, you know, that is handled in,
00:24:31
Speaker
even at the level of primitive numeric somewhat differently in between the two engines, and you have to figure out how to manage some of those differences. But more to the point is that it's an extensible class of types in the CLR world. So value types exist and can be defined by the user. I think Java is eventually going to get there, if I understand your developments. Maybe in the next version or something. They're not there yet.
00:25:00
Speaker
Right. So, you know, 20 years later is a good time to do it, I guess. But value types allow you to define your... A lot of users took about 20 minutes. Exactly. The...
00:25:17
Speaker
So value types allow users to define their own stack allocated types essentially. And you get incredible performance improvements with those types if you can avoid going through boxing. I kind of half asked the handling of value types because I was not trying to make significant changes to the compiler. Let me back up. One of the things I was trying to do
00:25:41
Speaker
In my initial approach, ClosureCLR version one, and we're still version one, was to not very much, to vary as little as possible from what the JVM was doing, what Rich's main line of code was doing, for multiple reasons. One, I wasn't necessarily, I can think of three reasons.
00:26:07
Speaker
One is if anybody else was ever gonna take over this project probably be good not to have a deviate too much. I'm so that they can be more easily track changes across there was also some hope that as rich said in his original comment to me that you know the long term objective is to have you know set of course the files that support all ports without changes all platform specifics isolated in the support libraries not achieved yet we come back to that.
00:26:36
Speaker
For example, when you look at the implementation of Closure JVM, there's a set of files that are defining Closure itself, right? There's CoreCLJ and some related files like that that define a lot of the runtime in defining Closure itself as it bootstraps itself up. If you go today into Closure CLR and look at CoreCLJ,
00:27:06
Speaker
all the line numbers are still preserved. You go to any line, you go to line 1482 in that file, it's the exact same function and exact same line in that function in there. The idea was to change as little as possible. And that included at that level, I couldn't do that for all files.
Community Involvement and Microsoft's Open-Source Shift
00:27:25
Speaker
Like, you know, there's some files where they embedded a lot of,
00:27:29
Speaker
They did straight out of calls to the ASM library that does their IL generation. I had to toss that stuff. I can't do line to line on that stuff. But as much as I could, I wanted to keep that. The same thing was true of the compiler, particularly because in the early years, things were changing too fast.
00:27:50
Speaker
You know would put in variations on the compiler and i don't know how long it took him but you know running backwards like i was doing a. It might take me several months of experimentation the background to try to figure out how to translate some of those changes over into the compiler.
00:28:06
Speaker
So I didn't want to make too many changes while it was more in flux because it's just too hard for me to track and make sure that I was doing semantically equivalent things. So I stayed very close, which meant there were things that could have been done in the Closure CLR compiler that just didn't get done in this first round.
00:28:28
Speaker
So value types for one are not as handled as well as they could. What you want to do is try to avoid. Boxing as much as possible and that requires a greater degree of analysis of the types that are being passed through in the flow of code in a function.
00:28:47
Speaker
But how does it, when you're talking about value types, because in Java, I think only the primitive things are the value types. Primitive numeric, yeah. So is the, does it, when I'm using ClosureCLR, is the syntax different to specify that, okay, this is different than JVM?
00:29:08
Speaker
No, because they still look like a class. It's just that as it's implemented and handled at runtime, the class is being known to be a value type when the when the is that's generated for it will keep them on the stack. So that. I guess it's not the is the actual machine code that gets generated off of the is.
00:29:32
Speaker
When you know you're dealing with a static type, it will maintain it on the stack unless it needs to be boxed out for some reason in order to pass it to something that's expecting a reference type. And that's when you have to box it.
00:29:47
Speaker
I don't have much, obviously, I mean, I'm completely stupid about this stuff. So if it is a stupid question, please shoot it down. So I'm thinking the persistent data structures that you have, like maps and sets and vectors, they are value types as well on CLR? No. Okay.
00:30:08
Speaker
No, I mean, that's there's no point to do that because those typically the things you do as value types tend to be small and because you're going to blow the stack out. I mean, it doesn't make any sense to do it. So you're typically doing it to give an example. And I'm sure we'll talk a little more about the unity and Arcadia guys later who were.
00:30:30
Speaker
giving all kinds of inspiration for new things to do. They're working in there with, let's say, a point type, a 3D point type. So it's basically three floats, let's say. Well, three floats is small, right? And so to pass those around on the stack is very efficient. To sit down there and box them where, you know, really all you need is three floats worth of space. And now you've got to have three floats worth of space plus pointers. It just doesn't make any sense to do it.
00:30:57
Speaker
The problem is that they ran into is there's not enough analysis of the flow of types through functions in the compiler as it is right now that there's boxing that happens that could be avoided.
00:31:14
Speaker
and they get really quite incredible speed ups. They've done some rewrites of the compiler to handle some of these things that they can get really tremendous speed ups just by avoiding the boxing. And so I'll come back later to how I might fix that and use some of their work. But the other major piece that causes headaches is generics, generic types. There are generics in Java,
00:31:44
Speaker
They're not real. I mean, they disappear. They are absolutely first-class citizens in .NET, but I think .NET put them in 2.0 or something like that, so they've been there for a very long time. It was a significant effort, and it's a very interesting history about where those ideas came from, but they stole a lot of ideas from, you know,
00:32:09
Speaker
other aspects of functional programming languages and some other things. And they are true generics. And there are places in there where I don't think I've handled them as well as they could be. And it comes up in here and there. I mean, you can work with them. But for example, I did not
00:32:32
Speaker
for example, try to build the immutable data types, let's say, persistent vector. Persistent vector in JVM, it's all objects. I mean, what you have is an array of objects. And, you know, conceivably, I could have made those actual generic types, which
00:32:55
Speaker
has its own set of problems I don't think I'll get into today, but you could sit down there and avoid boxing if you did just a persistent vector of ints. You could avoid boxing all the ints and save some space on that. You'd blow some of the
00:33:15
Speaker
Caching characteristics, if you've tried to do that with really large value types, but I don't want to get into those details. I think it would still be a win just because you'd avoid the external references or the boxing aspects of it.
00:33:33
Speaker
If we did that, I could play games where we brought in a persistent vector of ints and you could play with that as equally well as you played with the persistent vector of objects. But again, that would be non-trivial change potentially to a variety of things. So those are the things I had to play with and figure out and
00:33:55
Speaker
My goal was to keep up as best as I could with what was going on in the mainline JVR closure. And just make it so that as easily as possible, people could try to use it and also to port things across from Closure JVM like libraries and so forth.
00:34:20
Speaker
I think I've kept up mostly with less than a six month lag on the major releases in main line closure. I'm about two commits away from 1.10. I've got a couple other little fixes I need to put in, but 1.10 is really feature ready and could go out today probably if I wanted to.
00:34:44
Speaker
So I'm a couple of months behind them, but that's usually what happens. Yeah, that's awesome. Yeah, yeah. And what about the libraries like Co-Racing? Because I know C Sharp has, or at least they have channels, or is it the similar concept like Co-Racing or other libraries?
00:35:04
Speaker
Yeah, the problem is I have to port all those. And this is part of the problem of using this tool is that the tooling is just not there in the same degree it is around closure JVM. It's a very small community, not that many people contributing to it.
00:35:27
Speaker
Anybody who's developing a library for closure is not going to do the work to make a run under closure to CLR. Some things will work directly. There are some libraries, a few that I've brought over that work directly. It's because they have no interop in them. If it's a straight closure library, I'll run it. That's not going to be an issue. Interop obviously has to change. Even as something as simple as you're calling string.append,
00:35:56
Speaker
You got to change the capital A to a capital A. The string append is pretty much the same, except the name, but you still got to go through and change the source codes. Now, at one point, they introduced reader conditionalization. I can almost say that. Reader conditionalization. And as an attempt to be able to do that,
00:36:23
Speaker
And it certainly could be used for that, but of course, again, the guy who's off writing, let's pick something, nREPL, right? The Network REPL library. Who's gonna sit down there and every time he makes a change, you know, we have to make an equivalent change because the networking code is different. You know, it's non-trivial to make those changes. Now, if you sat down to design the library,
00:36:52
Speaker
or initially in the right way, which would be to try to figure out how to isolate Interop. You can do that in several ways, have either, you'd have to think about an Interop shim of some type potentially, right? And just write your own little API to do the networking calls you need to abstract those out and then implement those in a separate library on either side. If you did that, you can do that. But again,
00:37:21
Speaker
nobody wants to sit down and do that level of work because the easiest thing is just sit down there and just do straight out and interrupt to the JVM and be done with it. And that's one of the, of course, one of the wonderful things about working in Clojure is that that process of calling out to the underlying libraries is so easy. So I can go back through on any of these libraries and do the work, but it's only me, it's a lot of work. I maintain about,
00:37:51
Speaker
six or seven libraries that are essential. You asked about Core Async. I'm about to get back to that finally. Yeah, of course. So there are some that are absolutely essential, like the new spec libraries. I have to have those. I mean, that's just absolutely core to the whole endeavor.
00:38:10
Speaker
The testing libraries, data generators in those libraries are absolutely essential to be able to run all the tests that are written in Clojure. And I run all the relevant tests. The test suite all runs across the board, except for certain things that don't make any sense at all. And that's part of the way I know that I'm batching what's going on on the JVM side.
00:38:33
Speaker
But Core Async, I got 95% through with it and ran into a problem, then ran out of time. And it never quite got finished. Most of it works. But NREPL, it all works, or did work, except for interoperable operations.
00:38:54
Speaker
In NREPL, you can start an operation remotely. You're working across the network. We can also go signal that you want to interrupt that. And I screwed up something in how I configured my thread startups on the NREPL that they can't be interrupted. So now I got to go back and figure out every place I screwed that up. And I just haven't, my time is limited. I do have a full-time job. And my wife likes me to think I have a life too. I don't know what that is.
00:39:28
Speaker
So those are, Core Async was one of those that it's mostly done and I should get back to it, but just the press of trying to keep up with mainline closure, it's not 100% complete. And those things affect, if you don't have those extra libraries that bring people to closure and they're not there on the JV, on the CLR side, then that certainly affects the community.
00:39:51
Speaker
Now recently I've been getting a little more interest of people looking at things like this. Some fellow wrote me a couple of months ago saying his company was going to give him time off to complete the unreple port. So I was very happy to hear that.
00:40:08
Speaker
I just talked to a fellow in the Netherlands, I think. Yeah, yeah. I know one company that is doing closely along here. Is it Mediquest or is that? Mediquest, yeah, Mediquest. So C's, I think his name is, he's about to release some libraries that he did, which is
00:40:31
Speaker
to do some common work that would support both JVM and the CLR and getting more of this stuff out there would be nice. Community is definitely an issue. There's not that many people using Clojure CLR. I have no idea how many. I do know of a few interesting projects here and there. But it's tough.
00:40:55
Speaker
have the support of the mainline closure community, right? Either the people and the ancillary people
00:41:02
Speaker
who are developing all these wonderful libraries. And also Closure Core itself, the community right around Rich, the Cognitech folks and so forth that are involved in it, they're not interested because that's not how they make money. They do consulting work and they're very clear about that. And I would guess if they had suddenly massive clients interested in the CLR, maybe they would be,
00:41:29
Speaker
spending more time on it. So to commit to closure CLRs is tough. It's me and then some people who I appreciate very much who've done some additional work and helped me out on it, but that's where it is. Plus, I know you're about to ask a question, but plus the Microsoft community and the world
00:41:50
Speaker
particularly in the early days of this project, open source was not exactly the favorite thing in that community. If it wasn't invented at Microsoft, it didn't really count. Even things like Iron Ruby and Iron Python didn't really get significant traction, even though they were very interesting projects.
00:42:10
Speaker
So of course we've seen a massive change in the public stance towards open source in the Microsoft domain. So I'm hoping that will...
00:42:22
Speaker
My Microsoft owning GitHub was like, what the fuck? This is impossible to imagine. Because I remember 15 years ago or something, I was doing MSDN subscription and following through all that shit. Anyway, Ray. I think Steve Ballmer once called Linux a cancer.
00:42:44
Speaker
So definitely, you know, from cancer, it seems like the patient is cured and has made a lancerous like recovery. Yeah. So pretty good. No, it's gonna what? Ah, right. What was the question? Yeah, it was.
00:43:05
Speaker
I don't know if you took any kind of inspiration or cross-fertilisation from the Closure Script people.
Lisp's Popularity Among Developers and DLR in Closure CLR
00:43:13
Speaker
No, because I agree that, like, you know... Actually, no, forget that. We'll rub that out in a second. Or maybe it's not. Who cares? We've never heard of this thing. I'll come back to it. Apart from the open source thing,
00:43:30
Speaker
Why do you think the lisp, per se, lisps? Because lisps are kind of interesting. So why isn't lisp interesting to Windows developers? Because they haven't got another lisp, have they, as far as I know? Or maybe as they do, and I am not aware of it.
00:43:47
Speaker
I'm trying to think of anything that's going across my radar, not particularly. And I don't know why that is the case. I don't have any great insight.
00:44:02
Speaker
You can't get F sharp to gain significant mindshare over there either. I mean, I'm very interested in what they're doing. And that's something that actually is supported very well by Microsoft. It seems to be getting quite a bit more mindshare lately.
00:44:18
Speaker
So, but, you know, why that is the case, you know, I'm, you know, it's, it's like, cause you're implementing in C sharp, but did you consider implementing in F sharp? What was that not round at the time? Or is it still 10 years ago? No.
00:44:34
Speaker
Right. And moreover, put a pin on that because I'll come back to that in a second. No, but. We've got value types in F sharp. We've got them. We've got.
00:44:52
Speaker
My job would have been much harder if I had gone to F sharp A because it didn't know it would be because nobody knew it 10 years ago. And think about the translation from Java to C sharp is so easy. It's almost mindless. So I can do a lot of the work without even thinking very hard about it.
00:45:10
Speaker
And if I had gone to the F sharp side, I might as, you know, at that point I've been like translating, well, I don't know, into another language entirely. Going from Java to C sharp was pretty much rote. So I didn't think about it at that time. C sharp was the obvious thing to do.
00:45:29
Speaker
And coming back to the cross-platform thing, as you said, I mean, the community is not actually, or at least in the Closure world, everybody's focused on getting it up and running in JVM. But there are some libraries which are like CLJC, you know, they're both in JavaScript and Closure.
00:45:51
Speaker
So what do you think are the challenges to the first email from Rich that said, okay, I should be able to write one CLJ and then running it on all three platforms? Well, it's pretty much what I was describing.
00:46:07
Speaker
a moment ago is that it takes work and it takes coordination and it takes people who know both platforms. Most people are just trying to solve their own problems. If I'm working in closure, I'm running on the JVM, I know my JVM libraries very well. I'm going to design and write this thing. I probably don't know the.NET that well. I don't want to think about it. I probably would need somebody else to coordinate with on it.
00:46:34
Speaker
And to do it really well, like I say, if you thought about it for very long, you could sit down and say, well, okay, I'm working with threads here. The thread libraries are a little bit different. I could push everything off into a, think about the API need against threads, push those off into, you know, a side library and in my implementation enclosure, and then conditionalize those in terms of their implementation into interop calls. Not that hard to do.
00:47:05
Speaker
For the most part, you can, if you wanna get fancy, you can even, you could, if you just wrote an interop as a function call, you could turn it into a macro that expands differently depending on which platform you're on and you never see it, but it's very efficient and, you know, but who's gonna do that?
00:47:29
Speaker
Yeah, so it's basically the... If you take a look at N REPL, N REPL is very nice, but it's got things like data encoding library in there that you have to make changes to. There's different data types, how you interoperate with the IO changes. So it's not that hard to do.
00:47:52
Speaker
But who's going to do it? It's not a platform issue. It's a human problem. There are a couple of things that we pinned down apart from the one that, so the dynamic language runtime tooling, you're saying something about what kind of things are available on CLR to implement these things.
00:48:19
Speaker
Yeah, originally I actually used the DLR too much. The Python, Iron Python and I, Ruby, generate ASTs, and when they do their parsing of the language and they generate ASTs, abstract syntax tree, and then they actually use the DLR itself to do code generation into the IL.
00:48:40
Speaker
And I found that didn't work well for me and it was kind of slow for what I needed to do, particularly the way Closure, you know, you're doing everything on the fly. And sorry, IL is like intermediate language? Intermediate language.
00:48:59
Speaker
Yeah, bytecodes in JVM, right? Yeah, so that's IL. Microsoft never calls them bytecodes. Maybe it's a trademark issue. Probably, yeah. So the intermediate language for .NET.
00:49:21
Speaker
But there are pieces of it that actually work very well. And I use it for some things that give me actually occasionally some slight advantages over the JVM version. Though I've never tried to benchmark it to see how much. But there's some nice stuff in there for doing what in C sharp is called dynamic. I'm not sure when they introduced dynamic. People are probably more familiar with VAR.
00:49:50
Speaker
where you don't have to actually, you can declare a variable and never actually declare its actual data type. But with vars, you actually know what the data type is. You just didn't bother to write it down, right? It uses type inferencing and just moves the data type around with the variable. You get the type from how it was initialized and you move it on. Dynamic allows you, for example, to say, I'm gonna do an operation like call method foo on this thing. I don't even know what the type of the thing is.
00:50:18
Speaker
And this is not. I don't know. It's not inheritance based. It's literally I could send in two completely unrelated classes and a runtime dynamically have it figure out what it means to call foo on this object.
00:50:36
Speaker
Isn't it similar to invoke dynamic on JVM? Yes, which is a much more recent development. Yeah, of course. But the DLR itself supports what are known as call sites, which
00:50:52
Speaker
implement this, and dynamic call sites essentially sit down there and develop code on the fly to do the type discriminations necessary, let's say, for a method call. So when it comes in, it knows nothing. The first time you call it, it'll sit down there and say, what am I calling this on? I don't know how to do this. We'll go off and say, oh, what you are doing here is calling this method on this type.
00:51:13
Speaker
And it will generate a little hunk of code that says, if you have this type call, do this call. And if you then come into that same call site with a different type later on and it doesn't match the discrimination code that's sitting there to figure out what type it is, then it will add more code in and dynamically reconfigure that call site for you. So there's a bunch of support for doing that. So there's a bunch of places where
00:51:41
Speaker
In the CLR, you can get rid of a lot of reflection at compile time and closure if you put type hints in. And of course, a lot of people will code with type warnings on, so that reflection warnings on, so that you don't have to do runtime reflection.
00:52:02
Speaker
And so you try to take care of the compile time. But in those cases where you can't do that, you then have to sit down at runtime. You say, oh, I've got this type. Let me go off and retrieve the method and do the appropriate filtering on the methods to find the best match and everything else. And I can do that little, I can use this dynamic call site mechanism that C sharp now has. And the DLR is what originally provided that mechanism.
00:52:26
Speaker
And the overall process of just doing method matching, I'm using the same code that is used in these dynamic calls to do these discriminations. I can do that at compile time also and use that same set of code to do that. So that's primarily where I'm using the DLR is for these dynamic call outs at this point. So anyway. So that's the DLR tooling stuff that. Right. OK.
00:52:54
Speaker
So let's, let's talk about the unity and you know, the, how, how is, um, giving you new ideas in taking, taking CLR in a closer CLR in a, in a different direction, probably, or, or better direction, better, better direction. So, um,
00:53:13
Speaker
There's two fellows in New York City that contacted me at one point. Tim Gardner and actually Randy Nasser was the one who initially contacted me and saying that they were using Closure CLR. I'm always happy when somebody tells me they're actually using it, except for the pressure it puts on me.
00:53:34
Speaker
add that they were using it in unity on unity works on the platform and they were actually bringing closure in there so that they can have a running raffle inside the unity platform to do you know closure style lispy raffle programming on the fly in the unity framework.
00:53:53
Speaker
And what they were running into were things that impacted the performance of that, because they want to deliver games. I mean, game guys are serious. They like performance. Something about frame rates and things, they seem to care. I don't know. So...
00:54:13
Speaker
They started reworking parts of the compiler and scratching other issues that were caused by things that could have been done better. One of those was redoing some of the code generation aspects and sitting down there and doing a better job of tracking data types.
00:54:37
Speaker
through the functions so they could avoid boxing primarily. There is a little bit of work on trying to avoid boxing in this way beyond, you can pick up some of it by doing type hinting and so forth to avoid reflection and so on in enclosure JVM.
00:55:00
Speaker
But when it comes to the fact that they don't really support value types because all they have are the primitives and even there they kind of primitive numerics and even there they kind of punt on it. So one of the ways they got efficiency out in terms of handling primitive numerics was to introduce primitive interfaces where
00:55:20
Speaker
they introduced something like 350 new interfaces, which all have one method in them called, I think, invoke static or something like this, which why there are so many is because they ran through all of the type signatures for the parameters up to four parameters and the return type.
00:55:44
Speaker
So there is an interface whose name is OD, which is what I feel like when I look at 350 templates is OD. But that's one that takes, I forget which order the arguments in, an object parameter and a double return. So what happens is if you put, and so there's OD, there's O-O-D, there's L. They only do it for combinations of object, double and long.
00:56:12
Speaker
Okay. Okay. And so what happens is if you have an iPhone, a closure function that has type hints on the parameters and return type, and those type parameters say that, okay, I got along here, double here, and it returns along.
00:56:28
Speaker
then it will generate an additional interface beyond the regular iPhone interface that all the functions have enclosure. It will identify, it will do whatever I just said, an ODL interface on it. And then as you're compiling against it, other things against that function,
00:56:48
Speaker
And they can detect that it's got this primitive interface. And if you've got the right type sitting there, then you can pass the types in directly without going through boxing. So you can avoid the boxing steps on those calls. The problem is because
00:57:09
Speaker
they need all these interfaces ahead of time and they have to limit it. So it's only up to four parameters and return type all have to be object double or long. And so everything's forcing that. Well, I have an infinite number of types I have to work with potentially, right? And not known ahead of time.
00:57:25
Speaker
But of course, this is where generics come in, because we have real generics, right? Generics aren't type-erased away. They're real. I've got a function type. I can just say, I can support the function interface with a double, a double, and return a long. Yeah.
00:57:43
Speaker
And I don't do that right now because I track exactly what they're doing in the JVM. So future plans would be to sit down there and get rid of all their limitations on primitive type hinting. There are limitations. You cannot put primitive type hints other than double and long on things. So toss all that. So it's starting to diverge a little bit from what's possible on one platform versus the other. So I can start to do some of these things and then take advantage of what Timbs and Ramsey have been doing
00:58:12
Speaker
to take real advantage of that information. They've also played a lot of fun games in terms of
00:58:21
Speaker
Because they're working sort of after the fact, they basically have isolated the code generation out as a separate step for what they need to do. And so they play some interesting games, like rather than generating directly into IL, they generate into an intermediate symbolic representation of the IL that they can do some keyhole optimizations on.
00:58:49
Speaker
There are games there that could be played. That's a nice pun. Talking about Arcadia, you're introducing there are some games that can be played there. Right. You already, of course, did an interview with Tims and had some talk about that.
00:59:07
Speaker
I don't use Arcadia because I don't write Unity games, but the fact that they're doing that is, and needed to face these issues, has been significant and really quite an inspiration. And they've done a lot of work that I hope to steal, like everything else in Closure. Everything in Closure CLR is basically from somewhere else.
00:59:27
Speaker
So there's that aspect. The other aspect would be that there are things I did. I was the only person using Closure CLR as far as I know for years. And so there are things that probably aren't documented as well. So one of my thoughts is actually just go through and...
00:59:42
Speaker
I shouldn't say this, but rewrite the whole thing. Take note of every single thing I do that deviates from the JVM and get that properly documented so that the next, whoever follows me and the larger community really has a good sense of all the differences. I think documentation is one of the things that certainly is lacking. You can get it up and running. There's enough documentation on what to do,
01:00:08
Speaker
There are corners that probably I might be the only person who knows about. And that comes to the F sharp part because I get bored easily. And the notion of just doing that work and not doing something significantly different doesn't strike me as very interesting. So one of the thoughts I've had is actually doing a complete rewrite in F sharp. I figure it's not going to
01:00:35
Speaker
hurt the people who are helping me write Closure CLR because that's a vanishingly small set. So I won't be turning off any potential help that way. I might actually attract people who might be actually interested in it. And it's actually kind of a step towards a closure and closure thing because it'll be rewritten in a very functional style at that point.
01:00:57
Speaker
Um, and also I get to learn F sharp, which is, you know, I like to learn new things, but the, is something, uh, sorry to interrupt, but is it, is it the, when you say rewriting it, um, are you thinking like still keeping like closure in spirit, but actually implementation is completely different or thinking new features. It'll be,
01:01:23
Speaker
everything is much the same as possible. And in terms of, you know, the persistent vectors will still be there, right? You know, all the supporting stuff around the infrastructure that's needed. But to do the rewrites just so I can catch all the places where I make changes, like most of it's in the compiler. And to sit down there and really rethink the compiler and not slavishly follow
01:01:52
Speaker
their model. I mean, I don't know if you've ever looked at the Closure compiler itself, it's 9,000 lines in one file and it's complex and it works. I don't think anybody wants to touch it particularly, but I would redo it. And actually, if you go on to one of the Closure sites, I don't know if it's in Dev.Closure or where it is, there's a very, very old file
01:02:20
Speaker
out there that Rich wrote called, it's on dev.closure.org and it's called compiler enclosure. I think it was written about two years after closure was public, where he talks about all the things in the compiler he would change if he could. And it's kind of like, well, why don't I do that? So what are the advantages from your perspective, just to sort of, maybe as I sort of,
01:02:48
Speaker
In terms of Arcadia and your perspective, from your perspective, you get a new language to learn and you get potentially some brevity over C-sharp. But what are the other performance benefits or other affordances that F-sharp would give you over C-sharp?
01:03:09
Speaker
I don't know that would give me any affordances in this. I mean, as you say, less code noise, you know, a smaller code base. I don't know. You know, I'd like to use, I guess, in the design of the compiler, you know, a little more of the sort of what you see when you look at the toy examples of doing things like compilers in F sharp, where you can really take advantage of
01:03:33
Speaker
of the discriminated unions and so forth, the data types that they use, and just really write it very cleanly. And does it give you a better way to manage value types, for example?
01:03:46
Speaker
No, because either way, most of the work that's happening is what happens at runtime, right? It's doing the analysis. It's the kind of IEL you generate. I'm going to do that work in either language, but I think I'd rather think about it as I rework things in F sharp. I mean, if I'm going to sit down there and do a significant retooling of the compiler and change the kind of data structures that's churning out and sort of
01:04:13
Speaker
turning it into more of a multi-pass compiler that does a lot smaller work on each pass and should be much mentally easier to figure out what's going on. I think I'll just end up with a cleaner design. I could do that in either one. The F sharp thing is just me saying, it's partly discipline.
01:04:34
Speaker
If I re-implement the stuff in F sharp, I have to look at every single line of code and think about it and understand why it's there. Go back and look at the JVM and say what's different about it. And really, it's as besides my own interest in doing something different, it will force me
01:04:56
Speaker
as a discipline to sit down there and look at every single thing. I mean, I could translate stuff from Java to C-sharp all day. And rewriting it from C-sharp to C-sharp, it's easy just to cut and paste and say, well, I was good enough before. You get fresh eyes, basically. That's the interesting thing about it. Yes.
01:05:19
Speaker
It could be awesome as well because especially given that now .NET is on Linux, it's on every platform. You essentially have a cross-platform closure as well, but written in a better compiler rethought.
01:05:36
Speaker
The changes that are happening in the .NET world in terms of platform is extraordinarily exciting. I do not yet take advantage of any of the new stuff. I'm still delivering a .NET 3.5 version. Well, I kept it for years longer than I wanted to because of Ramsey and Tim's. Because Unity was stuck on 3.5. I wanted to dump that thing
01:06:06
Speaker
multiple years ago, but they were the only guys I knew who cared. They were very important to me. So I've kept the 3.5 around. So one of the things that's kept me from making a quick jump into .NET standard and .NET Core versus .NET Framework and some of these others is because initially, there was no support for system.reflection.emit.
01:06:35
Speaker
which is the stuff that does all the IO code generation because in the initial phases of that, they didn't know how to do that in a cross platform way.
Impact of .NET Evolution on Closure CLR
01:06:43
Speaker
And as a matter of fact, they are now putting that stuff. It's already been put in as an external library, I believe, in core. And of course, .NET Framework, the main line, has always had it. They are going to put it in .NET Standard 2.2, is my understanding. And it's put in there in an interesting way. For example, you can detect whether or not the platform you're on actually supports IEL generation or not because some platforms do not.
01:07:13
Speaker
Okay. Which means, of course, you could go to a more interpreted mode on those systems if you could detect that that was the case. And so that would present some interesting delivery issues onto platforms that don't support IO code generation. I don't know enough about the dotnet platform to know what like
01:07:37
Speaker
uh what core is and what standard is and and what and what these two point things mean so just just give us like a two minute run that one just so we understand what the uh evolution has been i know that this is a concept of cross-platforming now but you're going into the details of it and i like that but i don't understand where where these things vary like i don't know the difference between like
01:08:02
Speaker
I know 3.5 is old, but I don't know. And I know that core is meant to be everywhere, but I've never heard of standard to be honest. Okay. No, no one's listening, right? I'm hoping you aren't either actually. I'm likely to say things that aren't
01:08:26
Speaker
accurate because I'm not the greatest expert on all the distinctions right now because I'm still stuck in the old world. I'm still back in 4.0. But the basic idea was as Microsoft has looked to moving the .NET platform off into these other
01:08:42
Speaker
venues, these other situations, to be able to run on Linux, to be able to run on smaller devices, and a bunch of other places, they really want to infiltrate, right? And applauded, I mean that in a good sense. I mean, I applaud their notions of getting off of just Windows. I look at it from my viewpoint, there's be more opportunities for people to run and close your CLR in a lot of different platforms. It'd be nice to sit down there and run it on that phone, or wherever.
01:09:09
Speaker
So they started to rethink some of how that was working because not everything needed to go over. You don't need some of some of the support that's not done that framework is going forward is what you think of as.
01:09:25
Speaker
the old .NET, right? And that's designed to run the standard Windows workloads and be specific more to the Windows platform. And as such, it needs all the support for all the different kinds of project types that are out there. So, you know, ASP.NET, Windows.Forms, you know, all of the traditional stuff.
01:09:47
Speaker
But they wanted, I believe, a common set of interfaces and base classes and so forth to work on a more portable basis. And so that is what the .NET standard would provide. The .NET standard, then, you could have different implementations of that. So .NET framework could be an implementation of .NET standard.
01:10:10
Speaker
But then there's also .NET Core. My understanding is .NET Core is where they're doing a lot of work to develop new implementations completely because they don't have to support everything.
01:10:23
Speaker
And the plan is .NET Core is where they can experiment and move faster and I wouldn't say break things because Microsoft is pretty good about not breaking things, but move into new things quicker. And apparently, there's a lot of work on performance issues and new things. Apparently, it's quite nice and quite swift. A lot of people are very excited about it. And that's this thing that may move over to other platforms more.
01:10:50
Speaker
So, but I'm still confused about all the distinctions because I haven't had time to sit down and work through it. That's my, once I get 1.10 out the door, this is my next project is to sort through all of it. But I've stayed away from it partly because from my viewpoint, it wasn't yet viable because at core in enclosure, enclosure CLR is dynamic generation of intermediate code, IL, right? And then having the jitter compile that out for you.
01:11:20
Speaker
when you load a form into Closure or Closure CLR, it gets compiled down to IL and then the Jitter takes it the rest of the way and then that's what actually goes, right? And there was no support for the APIs needed for that in the original .NET Core and in .NET Standard. So I was stuck in .NET Framework land for the time being.
01:11:45
Speaker
Now, some people came back and did some libraries for.NET Core is my understanding that you can get off of NuGet and bring down to do reflection. The primary namespace is system.reflection.emit where a lot of this happens. NuGet is the distribution environment. NuGet is the distribution environment, yes, for libraries. I just read that the next release of.NET Standard, which is
01:12:14
Speaker
The base APIs that everybody's supposed to be supporting finally adds in system.reflection.emit. And .NET Core 3.0 when it comes out later this year will be supporting .NET Standard 2.2.
01:12:32
Speaker
OK, yeah, that's my that look on your faces, my faces. I'm working through this. I'm reading the announcements and then I find and then I find out that that dot net framework will not be supporting dot net standard two point two for the foreseeable future because there's too many changes in it. And that framework is the slower moving thing. Yeah. So now I'm back and looking at, OK, well, I can't.
01:12:56
Speaker
code maybe directly to .NET standards. So that means I have to have different builds for .NET Core and .NET Framework and maybe for Mono and maybe for UWP. So I don't know yet what all the implications of this is, but my plan going forward is certainly to move into that and maybe make
01:13:15
Speaker
I don't know. It depends on the user base. It depends on what I find out from the community. It's a really nice, interesting dichotomy. On the one hand, the Unity and Arcadia project is the one that is making you think ahead. And on the other hand, they are the one that are pulling you back on 3.5. They were. No more. Oh, no more. OK. No, no. The latest Unity has moved up to 4.5, 4.6. So I think you're OK.
01:13:41
Speaker
So, like I say, Tim's and Ramsey and Tim's have done some really great work and it's been very inspiring and shown what's possible and really inspired me to think about actually putting all that work in to do this.
01:14:03
Speaker
And rethink some things, for example, like I say, how I handle generics could be done better. Because there's no real notion of generics in a true sense. You don't have to even ignore them completely on the JVM side, essentially. You can code in it, but it disappears. So the runtime, which is what we're compiling to, doesn't really care that much. And there are just things there that I think we could do that would really enhance the utility. There's, for example, a lot of
01:14:37
Speaker
How should I say this? Let's say you build a protocol in Clojure, which is basically a Clojure-style interface that allows you, but it's designed to solve the expression problem, so I can say that this built-in class existed before this protocol implements the protocol, right? So it allows you to have this extensibility dynamically. Well,
01:15:04
Speaker
I can do that, let's say I take a, you know, closure generic list of some type, sorry, system generic, whatever, system collection generic list of some type. And I can say, well, I want to extend it if the type arguments are object and object, let's say on a hash table, for example. But that doesn't help me for all the other types. Maybe I should be sitting down there and saying, well, why can't I make this extension using type variables?
01:15:35
Speaker
So that I can, you know, you basically, you want to toss me a hash table that's keyed with int and has string values. Well, right now I can only handle object-object because I'm hard coding those protocols in one by one. Why can't I do it with generic variables, for example, put type variables in those generic calls? So there's things to think through that would fully extend
01:15:59
Speaker
some of the closer things like protocols fully into the.NET world. That would be some of the other things I would be doing. Because protocols, I guess the nice thing about protocols is you do declare the interfaces. You can know what's going on in the compiler. Right. Because obviously you can redefine them at runtime, but that would be weird.
01:16:24
Speaker
Yeah, but the point is, when you sit down and define a new piece of user code, for example, and you say, you know, I want to interoperate with all kinds of different things, so I can define a protocol, ifu, whatever, and you can say, traditionally with interfaces, anything defined after you've got the interface, ifu,
01:16:47
Speaker
You can say I implement ifu but with protocols you can come down and say well strings implement ifu as a protocol even though they pre-exist. So it's a way of doing extension types. And just to find out better ways to do that,
01:17:03
Speaker
that really fully incorporates things like true generics that we have would be, I think, useful. But one more thing on protocols is JavaScript uses protocol, sorry, a Clojure script uses protocols internally for a lot of their implementation because they're in a totally different land in terms of object orientation in there. And Rich has made the comment somewhere along the line that if he were to redo
01:17:32
Speaker
Closure he would think about getting there's if you ever looked at closure implementation, there's like, you know 90 interfaces, you know, I seek I I Persistent vector I I III and You might take a little performance hit but the extensibility Of those of those being implemented as protocols would give you some some handle. So I actually wrote back
01:17:56
Speaker
I usually talk to Alex Miller and then he asks Rich my questions. Alex is my filter. Rich's filter. He's Rich's filter. You have to speak to the Cardinal now. Yeah, pretty much. That works well. Same last name, so it works well.
01:18:25
Speaker
I said, does Rich still think that way? And he wrote back and says, yeah, he would still think about it. If he was redoing it completely from scratch today, he might think about getting rid of all the interfaces and going to protocols.
Vision and Future Improvements for Closure CLR
01:18:40
Speaker
So I won't commit to doing that, but I'm going to experiment with what the performance hit would be and think about what it would be like to redo that too.
01:18:49
Speaker
This this closure CLR dot next whatever you want to call it is a chance for me really to rethink everything from the ground up I You know to make it
01:19:00
Speaker
as performant, as minimal, at minimum with what's there to greatly increase the level of documentation, to increase performance in these key cases, to really make it live fully on the CLR platform. And that to me is kind of like, that may keep me interested for a few more years. Well, I still got a few brain cells left. That's fantastic, yeah, yeah, really good.
01:19:25
Speaker
I think we covered most of the topics that we pinned around. Obviously, I mean, we can dig into all your experience and then keep talking for another four hours or eight hours or even longer than that. But I think we are, at least for this episode, I think we reached some sort of a logical conclusion. A positive view towards the future. I think that's a good place to end. Yeah, that's really exciting. Exactly.
01:19:53
Speaker
But it sounds amazing. I mean, the amount of work that you've been putting in for more than 10 years now, obviously, and I mean, usually every new language, I mean, I do consider this to be more or less like a new language because you're essentially putting everything onto a different platform.
01:20:11
Speaker
It has a bump and then falls flat on the implementation. But you're doing amazing work and I hope once the .NET platform is becoming more and more accessible on different platforms.
01:20:26
Speaker
This could be a really nice, viable alternative. And I'm really curious about, I think once you start thinking about F-sharp thing, please do come back on the show and then we can discuss how you think about F-sharp driven compiler development. That'll be super cool to discuss. I think it'll be fun too because I'm looking forward to that particular intellectual journey.
01:20:49
Speaker
Of course, the fact that it's continuing is you have to give a shout out to Rich and all the other people who do this tremendous work. I don't have to think about closure. I mean, there are the guys thinking about closure. I'm not implementing enclosure. And that allows me to focus on the specific problem of what it means to have closure on the CLR.
01:21:12
Speaker
You know, I've done that pretty much keeping very close to the mothership and human to that. And I think the next real phase for this is to just continue to dance the closure dance to the tune they're playing, but add a few steps.
01:21:30
Speaker
Yeah, yeah. But this is, I understand you're not thinking about closure as in the feature level thing, but I can see that this is a, implementing it again as to spec is a different kind of work than implementing it based on the code because you don't have a spec that you can code to. You're essentially trying to reverse engineer
01:21:57
Speaker
how Closure is built and then re-implementing it. So that's a different kind of work. Yeah. Totally, yeah. Fortunately, a lot of work has also gone on and on the Closure side, there's a lot more documentation when I started out. I mean, there's a lot more explanation of this is how things are supposed to work, you know, and it's kind of like before that, in the very early days, I mean, 10 years ago, that documentation wasn't there. I basically, I had
01:22:22
Speaker
There's a couple of web pages and the code. And so now, it's a different world, too. And it gives me a little more. If I greenfield here, in that sense, there's a lot more to work from. Fantastic. Thanks a lot. Thanks a lot for your time and all the insights. And we would really love to have you back on the show again to discuss F-sharp driven and where you're taking closer CLR. And any last words, Ray?
01:22:50
Speaker
No, well, I think what would be really nice would be to maybe sometime like next year to think about having a roundtable,
Community Building and Future Aspirations
01:23:00
Speaker
maybe with Ramsey and Tims and David, because this seems like to me, I mean, you know, the kind of possibilities that performance for games that you can get in the CLR, it's pretty exciting to a large community, you know, and I think there's a lot of
01:23:17
Speaker
There's a lot of people out there that could take advantage of this. What I was going to ask actually was because you were saying, David, that there's a small community, but maybe it's this podcast, maybe it's a little bit, but maybe other things like Ramsey and David and Tim's that are doing to another building more of a community on the unity side.
01:23:39
Speaker
It would be nice to see a bigger community developing around the CLR version as well as Microsoft open up their arms a little bit. Nice to see Microsoft chipping a little bit in terms of one of their engineers. Again, I don't know, for instance, whether they do things like in Visual Studio, whether they have closure bindings and stuff like this.
01:24:03
Speaker
I don't know about those kind of affordances upstream. That would be quite nice, wouldn't it, to see those kind of things? Maybe they're already there. It could be done. There was a plug-in for Visual Studio at one point that hasn't been maintained to bring closure development onto Visual Studio.
01:24:20
Speaker
There's a whole side of this we didn't talk about, but in terms of community development, but tooling is extremely important. And I want to pay some more attention to that because it's a place where we're a little bit lacking. But that's probably longer conversations for another day. But it's really good. Yeah, it's fascinating stuff. I mean, definitely, we were talking before this about the fact that we kind of got the schedule in one way around a little bit because I should have been first then Tim's.
01:24:50
Speaker
But it's been, I mean, learning from both sides about the way the CLR operates and that it's really interesting from a games programmer perspective, which is a lot of closure people talk about performance, but I really have, like, eh, does it really matter? But I know, as you said, David, for an absolute fact that closure, performance really matters on a games environment.
01:25:15
Speaker
So it's kind of interesting that your platform, your kind of version of it is actually building out that niche quite heavily. So it could be an interesting place for people to go who were interested in that more generically, as you said, because the people that are doing the Arcadia stuff now, they're making very specific benefits for them. If you do these generics things, that would broaden the value of the platform more widely, I think. So I think there's some exciting times ahead by your description.
01:25:46
Speaker
Yes, I hope so. I mean, I'm excited by it, which is a good thing. A decade into this, I need all the stimulus I can get.
01:26:01
Speaker
So I think that's the that on that positive note, I think we'll close this episode for now. Hopefully one day I was thinking, you know, we can have you, David, of course, and then Rich and probably Michael David all in together and then see, you know, the script and then closure and then and then we just step back and then learn from you. That'll be an amazing panel.
01:26:28
Speaker
Well, actually, if you got those two in the room, I'll step back and listen too, because those are some really sharp guys there, and I'd love to hear more about what they have to say. But, you know, thank you very much for inviting me to join you today. This is actually...
01:26:46
Speaker
The only time I've ever publicly talked about Closure CLR in 10 years, so. Wow, okay. Oh, wow. Scoop, for different. That's amazing, yeah. Well, I mean, I was feeling absolute pleasure. You heard it right here. Yeah, it's been an absolute pleasure. I mean, I'm amazed because you're such a good talker and so passionate that, you know, I'm, well, I hope you're doing more of it. You know, I'm sure there are other podcasts in the Closure world and conferences and stuff that would love to have you, huh?
01:27:15
Speaker
So keep it up, man. It's good stuff. But put a pin in it too. We'll come back and circle around and we'll see if I've actually done anything. Well, you've already done the Lord, so. Exactly. The new things. The new things, yes. So thank you for that. Because I'm very interested in seeing that story.
01:27:35
Speaker
Okay, so that's it from us for episode number 48 with David Miller and discussing ClosureCLR. And if you're on Windows or if you're using .NET CLR, do check out the ClosureCLR stuff and reach out to David on Closure and Slack or something to see how you can contribute. And of course, there seems to be plenty of work that still can be done to make it awesome, awesomer rather.
01:28:05
Speaker
That's it from us. And Ray, do you have any Patreon lists today? I haven't been very well recently, so I haven't done any preparation. I've just walked into this. No problem. This is as good as it gets. Sorry. Exactly. Sorry for that. Next time we'll do a bit more effort.
01:28:21
Speaker
My apologies. Thanks a lot for all the people who are already subscribing or showing their patronage on Patreon. We would like to do something different. Maybe we'll discuss it in the next episode or something. We want to make some benefit for these guys and girls because they're really pretty phenomenal and very kind. Obviously, as far as we're concerned, this podcast is for everybody.
01:28:49
Speaker
But it would be nice to do something for those people as well. OK, so goodbye and see you in the next episode, episode number 49. Bye bye. Bye.
01:29:27
Speaker
you