Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#55 Learning Schafkopf with Martin Kavalar
127 Plays6 years ago
We talked to Martin (https://www.sauspiel.de/schafkopf-lernen fame) to learn about a famous Bavarian card game.
Oh, also we talk about some other stuff like Erlang, Clojure and ClojureScript :)
Recommended
Transcript
Introduction of Hosts and Special Guest
00:00:15
Speaker
Welcome to Deaf and episode number 55. I think we took a bit of a break for the, I don't know why, maybe it's some sort of a solstice break or whatnot. But we're back. Yeah, that's for sure. Yeah. Exactly. So we're back. This is Vijay from Holland. Raimondo from Belgium. And we have a special guest all the way from Germany. Hi, I'm Martin.
00:00:44
Speaker
Hello, Martin. Maybe you should introduce yourself, Martin. I'm Martin from Berlin, and I work on this thing called Next Journal, which is a computational notebook. We started out a bit differently, but these days it's all written in Clojure.
00:01:07
Speaker
Perfect. I think this is a Closure podcast, by the way, so I think we can talk about some Closure later. We'll always get to Closure eventually. I think the last five minutes are completely about Closure. So the rest of the stuff is mostly nonsense. The last time we met Martin was when you were in Belgium, wasn't it? When you were in Belgium for the Heart of Closure conference.
00:01:32
Speaker
Right. Yeah. That was in August. Yeah. I don't think it did. That was a really nice conference. I think we talked about it last time as well. It was super fun.
What is Next Journal?
00:01:44
Speaker
So Martin, I mean, you work at Next Journal in Berlin. So what is Next Journal? Yeah, we are building a computational notebook with a focus on reproducibility.
00:02:00
Speaker
There's lots of buzzwords. Yes. Can you explain to us what you mean by reproducibility? Why that focus? Reproducibility is about if you run something today, ideally it should also be running tomorrow or
00:02:23
Speaker
That seems like a goal for every program. Right. But also a problem that everybody who uses a computer has experienced. Exactly. What's it called? That thing they had in Heroku, the six-factor app or whatever it was. The 12-factor app. The 12-factor app. Is it the 12-factor app? You just ignored six of them. Yeah, well, I was rounding down.
00:02:53
Speaker
We had enough at six. Exactly. Yeah. Well, to be perfectly honest, I'm not sure how many people did really do 12 and some of the 12 were actually bullshit, but okay. It's a bit like the 10 commandments, you know, there's a lot of repetition there.
00:03:10
Speaker
Yeah, you know, we get it, we should worship you. Then there's about three left, I think, you know, anyway, we don't want to get into that. So Martin, yeah, the problem with Heroku, what they were saying was that if you make an app, then tomorrow, the operating systems change, the libraries change, the frameworks change, all these kind of things.
Challenges of Reproducibility in Computational Notebooks
00:03:33
Speaker
Is that is that essentially the problem you have with like computational notebooks as well?
00:03:39
Speaker
Um, well, the problem I think is mostly about like, yeah, when you, you discover something, um, like an interesting paper or something and you, when you try to run it, um, like I've been in this situation a couple of times and it almost never works. And, um, yeah, the older the paper gets, um, the more problems you run into.
00:04:05
Speaker
It's basically, yeah, pretty much impossible down the road to get old code to run. I'm confused by what you mean by an old paper. Is a paper a term you use for some sort of... Like a scientific paper. Okay. Oh, you mean the experimentation? Sorry? There's no code in the paper.
00:04:30
Speaker
Yeah, no, so like a paper linking to some some code on GitHub or yeah. Right. Yeah. Yeah. So it's it's it's also about. So now I know I get it, I think, because I was thinking next journal is basically like Jupiter or something. But it is more than that. It is more about the journal thing comes out of this scientific research reproducibility. Is that correct? Right. Wow. I feel smart already.
Features of Next Journal Compared to Jupyter
00:05:00
Speaker
Now I understand a bit more. How does it compare with Jupyter and other things? Jupyter is basically just a Python library that gives you this literate programming, so the ability to have cold cells and have narrative that goes along with them.
00:05:23
Speaker
But a lot of things are outside of Jupiter's domain, like setting up the computational environment, so the operating system, that's nothing that Jupiter takes care of, as well as how do you get to the data.
00:05:44
Speaker
And yeah, with Nexterna, we built a system that encompasses all these things and that uses immutability at all layers of the system. So when you work in the system, the idea is that without putting in extra effort or much extra effort, it's easily runnable in the future as well. So it is a bit like Heroku
Ensuring Reproducibility with Docker Images
00:06:11
Speaker
then for scientific papers.
00:06:14
Speaker
in a sense that it's like a platform as a service type thing that you can deploy your proofs into. Right. Yes. No, so next-generation, it's not something that I can run on my own hardware? You can't run it locally, currently. That's something I'm working on. Currently, it's only in the cloud.
00:06:36
Speaker
So how do you achieve this reproducibility at different layers? Because as Ray was saying, there are so many moving parts. Everything is fucking changing left and right. So how do you maintain this? So on the data layer, we use content addressing. So when you upload a file or when a cell produces a result, we put this into content address storage and version it for you.
00:07:05
Speaker
That's an append-only storage. Regarding the computational environment, you can save your whole operating system as a Docker image. Essentially, the entire operating system plus libraries plus your experimentation code plus the data is the whole thing that is versioned. Right.
00:07:31
Speaker
So do you provide, just a quickie, do you provide like the database operating systems in Martin? So you provide like some Linux version or other? Yes, these are also created in notebooks. What, you specify them in a Docker file or something? Like from your point of view? No, you just have bash cells that can perform whatever installations. And you can import Docker images into our system.
00:08:01
Speaker
So we're running our own Docker registry that's also immutable. So you can't delete stuff or rename stuff. And we never identify images by their tags, but always by their hashes. So there is no force push. Right. Isn't there a bit inconvenient in some ways if you can't, like a lot of these things, if you can't edit them, if you can't delete them,
00:08:30
Speaker
Isn't that a bit inconvenient sometimes if you actually do want to edit them and delete them? Well, yes, the lawyers don't like that very much.
00:08:44
Speaker
Currently, when stuff would end up there that shouldn't be there, this would be a manual intervention. People copying AWS keys and shit and whatnot and secrets and then you're pretty much screwed. But also, I guess you've got GDPR risks as well. People are putting personal data on there as part of their data set.
00:09:07
Speaker
Yes, I mean, we don't publish all this stuff by default. That's still like, yeah, even though the data would be in our system, then like, yeah, you still can't really guess the hash, right? But yeah, that's definitely something to consider and where we'd have to do manual intervention currently.
00:09:31
Speaker
Okay, I think we're being like proper software engineers. I mean, just talking about the problems. Maybe we should talk about the features first. Everybody who opens something, it's always like, how many ways this system can be bad? Go through all the edge cancers first, please.
00:09:46
Speaker
Exactly. Let's talk about everything. Oh, wait a minute. But what about if I'm running on Nix with whatever the fuck with some sort of a weird... Anyway, so let's get to the real meter, which is like proper features of this thing.
Multi-language Support in Next Journal
00:10:00
Speaker
So this is operating system level and like Jupyter, that means you can support any language, right?
00:10:09
Speaker
so you can just use any language you'd like on the journal page. So, yeah, we do support Jupyter kernels as a runtime protocol. So, yeah, you can basically use any language that has a Jupyter kernel. And, yeah, we also support other protocols, like P REPL. And, yeah, like our own protocol is a bit inspired by P REPL.
00:10:37
Speaker
Okay. And, um, so, uh, essentially you can mix and match the languages within as well. Like, so the data, for example, on in one cell, I produce something using Python and then the second cell, I can consume it in, in Julia or, or whatever, or is that possible? Yes, this is possible. Okay. So that could be.
00:11:01
Speaker
But how does the data exchange work then between them? This is then being exchanged through this content address storage. So sometimes can exchange files. Yeah, like we had this in an earlier version as well, and this will come back at some point at.
00:11:18
Speaker
Currently, it's just these file blobs are being exchanged and you have to do the parsing yourself in the language. But we'll bring that back to be automatic sometime in the future. When you pass transit between the cells or JSON.
00:11:38
Speaker
So how does the, like, is there a multi-user thing? Because it is hosted, right? So at the same time, you can edit that file, whatever you call it, like cell notebook, then I can edit the same thing. Do you handle the multi-user thing as well? How does it work? Yes, we have collaborative editing, like the operational transform.
00:12:03
Speaker
Yes, so you can work on the same notebook currently. I heard that there is something like CRDTs and whatnot for the editing, right? Or is it similar to that one? We're using operational transform, which is the simpler version. It needs a centralized authority, which CRDT does not. Then it doesn't come with the
00:12:32
Speaker
CRDT has this problem that the document only grows as you're editing it. And yeah, with OT, which Google Docs also uses, it's the central authority that decides what to keep. And it's the clients that rebase changes against this central authority. So your server doesn't need to keep all the elements that were ever made.
00:13:02
Speaker
So you were just, I think, right before we started the episode, you were talking about where did Next Channel come from. So maybe a bit of history. Where did you guys come from? And then can you give us some idea? Yeah, so the core team, we've actually been working together for 13 years now.
00:13:26
Speaker
And we started out building this traditional German card game. Which one? You need to explain the card game to us now in an audio fashion. This is going to be amazing. So yeah, the game that we've been running, it's called Chavkopf, and this is only played in southern Germany. This is a
00:13:50
Speaker
like a trick-taking card game. You might have heard of Oktoberfest, which is in Munich. Of course, everybody. I can imagine like four Bavarians sitting at the table in Lederhosen and Dindl.
00:14:12
Speaker
Yeah, traditional Bavarian outfit. And yeah, drinking beer and playing cards. OK. Sorry, what is the game called again? Shavkoff? Shavkoff. It's like sheep's head. Yeah, in German. That's sheep's head? Yes. OK. But is it like a card game with the playing cards, like the normal cards? Or is it special cards?
00:14:39
Speaker
Yeah, it's a Bavarian version of the normal cards you might know from poker. Yeah. Yeah. Yeah. Okay. And you guys made an online version of it. Right. Yes. Do you meet each other at Oktoberfest or?
00:14:57
Speaker
I can imagine all the 13 guys in leader hose and then sitting together having a beer and like, okay, let's, let's make an online. Isn't that how the community made its first money as well by selling virtual beers at the table? Wow. So what, what was that built?
00:15:20
Speaker
What is the stack that you guys use for that one? So in the very early days, this was like a Rails website and Java game server. Okay. And yeah, then it was like, yeah, as it picked up a bit in popularity, we pretty soon experienced all these like Java concurrency pains. Yeah. So yeah, it was like, we tried,
00:15:49
Speaker
tried to fix the Java server for a while, but didn't really get that to work. After a while, we really couldn't figure out how to fix this. What were the actual problems, Martin? Scaling issues that would get slow, like with a thousand concurrent players,
00:16:14
Speaker
And then also it wasn't stable. So in the end, it was like this. It would run for two weeks and there was some like race condition that occurred and the server would be gone like this. So every two weeks you have a crunge up to restart the server. You have a little docker wall up there. The fail wheel is up.
00:16:38
Speaker
Yeah, people were praying for our server. The sheep side was severed. Yeah, this was with a couple of days away from my job, like, or visiting RailsConf, I came back and decided to rewrite the thing in Erlang. And how did that go?
00:17:05
Speaker
That went really well. Like this was like three months. And yeah, then it went live and yeah, it's been serving us really well to this day. And it's really incredible there. Like the ability to like change the code while the server is running these hot code deployments that Erlang can do.
00:17:33
Speaker
This is, yeah, for game servers, I think it's really a great choice. Okay. So you went to RailsConf and then decided to rewrite it in airline. Right. Because everyone at RailsConf was basically saying, ah! We're all writing everything in airline again. Like, want to rewrite your code in airline?
Success Story: Rewriting Java Server in Erlang
00:17:59
Speaker
Come to RailsConf. That's their pitch.
00:18:03
Speaker
We tell you what not to do with Rails. I think this, yeah, was 2008 or something like where, yeah, there was quite a bit of talk about Erlang and Ruby. Yeah. Yeah. Yeah. It's true. I think 2008, 2009. Yeah. I remember having an Erlang course as well during that time.
00:18:26
Speaker
Yeah, I mean, I, I, I couldn't get the syntax. I mean, it was like super archaic way of writing things in a line, I think. Right. You kind of feel like you're doing this, like, yeah, why is there to, to end, um, yeah, to end those sentence with the full stop. Yeah. Call on nothing or full stop. Yeah. Exactly.
00:18:57
Speaker
Yeah, I was too illiterate for it. So you started with Java, and then you, the same team switched to Airline, pretty much, for the sheep's head. Shopkopf. Shopkopf, yeah. Shopkopf, yeah. So, and then you guys are like, okay, enough of the games, now let's do some serious shit. And then move to next journal.
00:19:24
Speaker
Right. This was a process that took like 10 years or so. But something like that. Was the idea that people were trying to somehow prove scientific papers via the sheep's head game?
00:19:46
Speaker
There were enough papers about that. And then they're like, okay, we're gonna try to reproduce the sheep's head. But I saw some recently talking about the papers, I saw there was somebody who wrote a paper about the fluid dynamics that caused the coffee to spill when you're walking with a mug. Like entire paper, like, you know, all the fluid dynamics and how you're supposed to have them. Then they found out the optimal way.
00:20:12
Speaker
One of the optimal way of having not to spill the coffee is to have the cup in a concentric circles of smaller sizes. So inside of the cup, then then you won't spill the coffee apparently. So I guess we need to use next to find out like is this true?
00:20:30
Speaker
Exactly. Somebody is funding this. Internet. Anyway, so why didn't you guys then, because you had plenty of experience with Erlang running this highly scalable sheep's head shop cop game. Yeah. So how did the next general start in terms of technology stack?
00:20:58
Speaker
This was, yeah, we were starting out with Elm and Elixir. So we had like, yeah, from on the backend side had kind of moved like from Rails and Erlang to Elixir and Phoenix is like a Rails framework there, I guess.
00:21:23
Speaker
And yeah, on the client side, really enjoyed Elm when we were starting out and did some experiments with that. And yeah, that's what we did. Did you just decide that everything had to begin with the letter E? Is that the idea? Yeah, definitely.
00:21:47
Speaker
I mean, it's not a bad metric, you know, it's good stuff there. So then it's just a prototype, or how did it scale? So, yeah, the own thing, like, this was pretty quickly that we realized, like, well, it had served as well on the things we did before, which were sort of isolated.
Development Challenges with Elm and Elixir
00:22:09
Speaker
On Nextro, we had to do like a lot of interop with existing JavaScript libraries.
00:22:17
Speaker
and code mirror for the code editing and a library for this content editable browser. This is really painful in Elm. You get this nice pure language and with these pretty amazing guarantees like when it compiles, it runs, which really is the case oftentimes.
00:22:43
Speaker
It's quite amazing, I think. But when you want to interrupt with other libraries, it just becomes a huge pain. You basically have to talk to this library as if it was an external JSON server somewhere. You have to use these ports. Right. And also, Elm was undergoing some pretty major changes at the time. So Elm 17 came out.
00:23:11
Speaker
like did away with like the signal graph, which yeah, was a pretty essential part until then. And yeah, this was quite disruptive to the ecosystem. And like, yeah, all libraries, like it took, I think months until all libraries would catch up and be compatible again. And yeah, so I figured we don't really want to go through this like once a year
00:23:44
Speaker
Yeah, I think especially coming from language like her language has been like solid and forever. Then you switch to this thing like every six months, every three months, you need to keep updating the stuff and rewriting the application. Well, for a brief period, we tried to go back to JavaScript, but there we realized this was definitely a worse idea even. Yeah.
00:24:11
Speaker
Did you, did you look at the other options as well, Martin, like pure script or, um, you know, things like, cause pure script is a bit more Haskelly, isn't it? Then, um, enclosure script is, so it's a bit more like from Elm, you know, and it's a bit more of what you say, friendly towards, um, friendly towards JavaScript. You can just run it.
00:24:30
Speaker
I think like basically like Reason, ML, OCaml, this was like a, yeah, it was a choice between ClosureScript and Reason at the time. But yeah, like Reason was just getting started like with their React wrappers. So yeah, we like talked to some people about this and yeah,
00:25:01
Speaker
they couldn't recommend it or said we should be ready for doing a lot of this early framework or library code, having to work on that ourselves. This made me really take a serious look at ClosureScript, which I had ignored for the longest time before that, not seeing through the parentheses.
00:25:30
Speaker
So did you have any idea about Closure or ClosureScript at point one? Well, it was recommended to me like a couple of times, I think. I think like when Stu's book came out, I got that as well, like recommended by a friend and like read it to the first page where I saw some code on it.
00:25:58
Speaker
And yeah, I really like, yeah. You've actually been through airline and you puked at the syntax of closure. Come on, man. Exactly. That's what I was about to say. I think you already lost that point already when you started with airline. And we had one of our, like one of the freelancers working for us. He, he was a fan, but yeah, it also, it didn't really get through to me. We were really faced with this problem, like,
00:26:28
Speaker
Yeah, we want the same functional language. And yeah, we need to go to enter up. And it was also like running into Matt Hubert at this conference. This was Curly on in 2016. I think David Nolan was also presenting there and seeing Matt Hubert. He also worked on this notebook prototype
00:26:58
Speaker
which kind of, like, cells it was called, which later kind of turned into Maria. Yeah, and seeing what he had done there was definitely quite impressive and made me take a serious look at ClosureScript. So how is the front-end built? So now you started picking up ClosureScript, so essentially all the front-end is now ClosureScript right now in production?
00:27:23
Speaker
Yes.
Transition to ClojureScript for Front-End
00:27:25
Speaker
Okay. So how is the architecture like which libraries that you use and how is it being built? Maybe some idea about the frameworks or architecture? Yeah, we're using reframe, use that from the start. And then what are we using library wise? We're using reted now for the client side routing.
00:27:50
Speaker
and basically sharing views between our Closure app to do server-side rendering on the JVM. And I have specs that convert our data from the database into view data, which the client requests.
00:28:21
Speaker
Okay, and there is in the back end now, is there still some AdLang Alexa running? No, the more we started getting closer and learning more about it, I think it took a year after we started with ClosureScript and we had to rewritten everything on the back end as well.
00:28:44
Speaker
How did that go then? I would still argue, we could have an interesting discussion actually about the model for closure on the backend versus Elixir. It seemed quite different. Yeah, but as I learned about Datomic,
00:29:07
Speaker
With the site focusing on reproducibility, I think the Atomic is really a great fit. For example, we don't want to break any links, but we also don't want to do
00:29:29
Speaker
All this housekeeping work ourselves from the start. With Datomic, we just know we can always recover information and set up redirects later, even if we don't have to do all this work from the start. Before switching to Datomic, what was the data storage? Postgres.
00:29:54
Speaker
Okay, so you were maintaining essentially, quote unquote, immutability in Postgres by just appending or copying the data. Right, using event sourcing basically for each notebook. Yes. So just so we get this straight then, so you were doing closure script in the client. Did you do closure on Postgres or did you say, okay, no, I'm just, now we know the atomic, we're going to rewrite the backend enclosure to serve the atomic class.
00:30:24
Speaker
Yes, that's what we did. It wasn't really that we wanted to use Closure from the start, but we really wanted to use Datomic, but it also didn't seem sensible yet to use Datomic with our Elixir backend, really. So at first we started swapping out just a part that's reading and writing data, and eventually also this code execution part.
00:30:54
Speaker
Okay. So how is that? Because I can imagine that, as you said, the entire, is that called a notebook? Like every, every project or what is the nomenclature there is one? The documenting is a notebook. Yes. Okay. Yeah. So one notebook is one Docker image. Does it, how does it work? No, so one notebook,
00:31:19
Speaker
is basically one VM instance that we need for you to execute code. And you can have more than one Docker image. Like if you use like three different libraries or you can have three different languages, sorry, or can also use like, yeah, for Python runtimes that use different Docker images. Yes. Yeah.
00:31:44
Speaker
And now how does the, because I can imagine Erlang and Elixir are more or less native because they have like, you can call all the operating system level things. So the VM management and Docker image management, this also happens via closure. Yes, we're using this Spotify Docker.
00:32:05
Speaker
So Java library, yeah, to do Docker stuff. Okay. And Docker is also just a HTTP demon basically. Yeah. Yeah. So you interact with Closure Java libraries and then Closure and then? Yes. Okay. Impressive. And where is your next general running? Is it on your own cloud stuff? We run on Google Cloud. So how does it compare with the Google Collab, by the way?
00:32:36
Speaker
Because Google had this Colab thingy. Yeah. Yeah, Colab is pretty much. That's just a Jupyter frontend, basically. Yeah, so you also can't store Docker images there. You can't store for data storage. It also then just uses what's it called, the Google
00:33:01
Speaker
Google Big Table and whatnot or Google Cloud Storage Spanner. No, Google Object Storage or something. Just a Google Drive or something. Oh, yeah, right. It stores the notebook into Google Drive. Yes. So I guess that one of the things about Google or
00:33:25
Speaker
One of the things about the atomic is that it has this cloud version that runs on Amazon, but doesn't run on Google. So I guess you have to run the atomic yourself in your Google cloud. Yes. We're, we're running the on-prem version.
00:33:38
Speaker
I always think it's funny to run the on-prem version in the cloud. Well, then your cloud essentially becomes your data center, right? It doesn't, it's not like real cloud, but you just take some VMs. It was before there, yeah, there was a Datomic cloud that we migrated.
00:33:57
Speaker
And we would have also had to switch to AWS. Yeah. So how was the transition?
Comparing Clojure and Erlang Backends
00:34:05
Speaker
Like, how do you compare the backend in Elixir or Adlang with the backend enclosure? Yeah, I think it went fairly well. There's still a couple of things that I sometimes miss from
00:34:22
Speaker
from the Beam VM. Like I feel like, yeah, CoreAsync or yeah, concurrency, definitely like this is just a bit safer in Erlang Elixir. Like, yeah, you can shoot yourself in the foot more easily with CoreAsync. On the other hand, yeah, this, like this Interop enclosure script, it's,
00:34:47
Speaker
or enclosure, it's really nice being able to know. We don't use it at this point yet, but being able to do really fast number crunching when you want to parse or archive parts of data to know that there's all these libraries for the JVM, which wouldn't exist for the Beam is really nice.
00:35:16
Speaker
And performance-wise, did you see? Because usually you hear, or at least on the internet at least, people saying, OK, you can run crazy amount of users per machine, whatever, on airline. So how, in your experience, how is closure scaling compared to airline?
00:35:36
Speaker
Yeah, we don't have this problem yet. Unless you move your sheep's head in the closure. Right, but we're not going to touch that anymore. Don't touch the working system.
00:35:55
Speaker
No, so I mean, it's definitely like, yeah, I think it's hard to write like something that scales as well as like an Erlang web server or like if you want to do this like web socket communication and things, it's really a perfect match there and the way VM uses all the cores effectively and like it doesn't care what you do in one process, it's never gonna
00:36:24
Speaker
like the whole system. Yeah, I think that's going to be hard to, or it's harder to do this enclosure than it is in Erlang. Is the Beam VM also a GC-driven thing, like JVM? Yes, but it's pretty simple to GC because every processes don't share any state.
00:36:53
Speaker
So a message that's sent to another process is just copied. And it's just binaries that are garbage collected on it. Yeah, but simple terms are just copied, basically.
00:37:10
Speaker
Yeah, yeah. So that's why it's a bit more, not a bit more, that's why it's more efficient. One of the things that I know, I mean, whether, I don't know if Rich would say this himself, but like one of the criticisms that he seems to have of Erlang is the Akta model.
00:37:28
Speaker
you know, because it's not as you have to send messages, it's not as expressive as, as, as like transit or even is in terms of message passing or, you know, function passing. Is that something that you've found as well? Or is it something that you haven't, doesn't concern you? Well, I definitely think we are we are writing
00:37:51
Speaker
code a bit differently now in Closure. When you have these processes, there is quite a bit of ceremony also setting up, getting them to talk to each other. In Closure, you realize maybe you need your state only at the very edge of the system and do a lot more with pure functions.
00:38:17
Speaker
Um, yeah, so yeah, I'm kind of with with rich by now on this, I think. Yeah, definitely. Um, yeah, on a lot of things. Um, what, what would be nice if I think, yeah, the, we had sort of this, this trade off that, um, Erlang and Elixir, um,
00:38:46
Speaker
makes for building stable systems. We are running at a bit less top speed, basically, but therefore we don't have to worry about locking.
00:39:05
Speaker
So the Beam VM, when a process does something, it counts like how many operations it does. And then after a while, it acts as a, like it stops this process and lets the thread do something for some other process, right?
00:39:20
Speaker
It's like a yield. It's like a, what do they call it? Like a yield, like a cycle. Continuation. Yeah, continuation process. Yes. Yes. Yeah. So that's a bit slower in top speed, basically. But yeah, you don't get these things. Like when you do a blocking read or something, it can't hold your whole thread.
00:39:48
Speaker
So what are the things that you, other than this one, so what other things do you miss from AdLang when you're writing things in Clojure? Well, I think like with Phoenix, the getting started experience, like, yeah, it's this frameworks versus libraries debate, right? There's definitely two sides to that.
00:40:10
Speaker
Like I remember like getting started, like diving into the closure ecosystem was very overwhelming for me. And like, you have to make like, yeah, you think you settle on this language, but, um, yeah, that's when all the choices basically start. Um, and yeah, it goes down to so many levels. Um, and yeah, it's, it's also like, I guess coming from this or.
00:40:38
Speaker
Yeah, these choices aren't as consequential as they are doing at Pigrails or Phoenix or whatever. But yeah, I was coming from this place where I thought they would be as consequential, and then it's quite stressful to make them all. Yeah. So at every point, you have to make a decision which library you're going to use and how to do something. Yeah. Am I right or wrong? Basically.
00:41:07
Speaker
Exactly, yeah, yeah, yeah. But that is, I think, one of the biggest.
00:41:14
Speaker
concerns or I would say even complaints. Every time I see on Reddit or something, people asking for, okay, how do I start building web application? Because there is duct, there is yada, there is composure, there is, you know, okay, if you go to a level below, there is ring. And of course, Luminous is helping you in setting up a template. But then it also gives you a lot of choices, like which one you want to use front-end and the back-end and every other library. I think Rails did that pretty well, I think.
00:41:42
Speaker
No, I definitely like this getting started. I mean, when we made the choice to move to pedestal, I think we didn't get anything. It took us like two weeks to get the very basics.
00:41:58
Speaker
like who's swallowing my logging output and stuff like that. This is definitely a big issue, especially with the getting started experience. I think you get that time back later in the process.
00:42:19
Speaker
I think you can also maybe have both sides there really, but yeah. Totally. I think that's the thing. I think when you start and then you don't have this getting started experience and then you build something and you are already past the advanced beginner stage, so you don't look back anymore because you already know.
00:42:42
Speaker
And once you cross that boundary, then people don't think about how it is like to be at that level. So it is always tricky. But these days, I think, at least for the front end and other stuff, things have become a bit more reasonable. I think better still times were different. Yeah. No, I definitely think, yeah. Also, like, yeah, we had our fair struggle with tooling. Oh, yeah.
00:43:11
Speaker
Yeah, like we started with Lightning End and moved to boot and back to Lightning End, and then finally took two steps and figured it in. Yeah. And yeah, it was also like, yeah, got some good help there from Arna, yeah, who's running Lambda Island and Heart of Fire. Yeah, like without that, like I think we would have been pretty lost there as well.
00:43:41
Speaker
It's true. I think it's a bit of a shame, right? Because especially when the language and everything else is so stable and so attractive to build systems, then the whole beginner experience is kind of tricky for people.
Frameworks vs. Clojure's Flexibility
00:43:56
Speaker
You've been writing software for decades and then you come to expect some things when you start with the language.
00:44:04
Speaker
And a simple thing like Buildool has like two different communities and three different communities now. Yeah, but maybe it's also these two sides of the coin, like how powerful and productive you can be, right? Yeah, that's true. Yeah. Personally, I think, by the way, that I think this notion that a framework is like the greatest thing. For me, it's a bit of a problem because
00:44:33
Speaker
I think, like you said, Martin, everything is consequential. If you've got this big decision to make at the beginning of your project, then you're very reluctant to change from rails to something else. Whereas with closure, you start with pedestal and then you move to something else and you talk about the route and everything is more composable. So
00:44:59
Speaker
So I think, I think it feels a bit daunting at the beginning, this like beginner experience, because you haven't got all your choices defined in one, like, like the package. But on the other hand, I think the fact that you, you can make your own framework essentially out of parts is, is quite attractive, you know.
00:45:19
Speaker
You don't have to commit to a very big ecosystem. Yeah, definitely. But yeah, you also don't. Yeah, I learned this over time. Yeah.
00:45:30
Speaker
if I was to tell my three-year-ago self to not stress out about this so much. Yeah. Yeah. So speaking of tooling, so Emacs or some other shit. So what do you guys use at Next Channel? Yeah, Space Max these days. And there are finally some reasonable choice. I think it's also, yeah, this is also
00:45:56
Speaker
This was very painful for me in the beginning because of questions like this. You think, like, yeah, I can't, there's no other way for me to do closure than the Emacs. Vijay still believes that. Yeah.
00:46:11
Speaker
But by the way, I mean, you come from Adelang and I know I think late Joe Armstrong, he uses, he used Emacs all the time, right? He was using Emacs for Adelang and he's another. So I think it's, as Ray was, I think he's very rarely right. And he said that, you know, any software that starts with E.
00:46:37
Speaker
So that is something that I think he was right, like using the max is the only way. But how, so your team uses, everybody uses emacs for the, for the Closure Script and Closure as well, or? We have, yeah, so we have a shared space max config that's used by most of us. We have one IntelliJ person as well. One that's using VI.
00:47:08
Speaker
It's usually other way around. I think there'll be, I'm usually the emacs guy and then everybody's using intelligence or even VS code and whatnot these days. And also, yeah, I don't have to touch my, my config. I'm just telling my colleague, Dieter to have this problem. Can you please figure this out? Just out of interest. I mean, I mean, we kind of often joke about these, uh, editors, but, um,
00:47:32
Speaker
You know, we, none of us at my work, none of us use Emacs, we all use IntelliJ. But I don't, personally, I don't feel like it really affects our workflow too much, but I get the impression from Emacs people that it kind of does, I don't know. What's your feeling? I think... Maybe it's Martin, can I answer this one? Yeah, of course, yeah. He's the guest.
00:48:04
Speaker
I can write on this shit forever. Although I actually missed one. We have one person using Adam with par unfair. And yeah, he's very happy with this. And yeah, that's what I would definitely recommend for, for getting started as well. Like, do you use par unfair? I mean, you know,
00:48:34
Speaker
I'm aware of it, but I'm too used to the bath and slurping and all this kind of like code movement around. And yeah, I mean, the problem is, I mean, I think, again, I think it's one of these things where if you, if you're starting today with, with, with, with cursor or something, you probably would go with power infer.
00:48:57
Speaker
Because why wouldn't you? But when I first started, it wasn't available. So I had to learn these other commands. And now it sort of feels like I'm kind of used to it. I think it's like the Emacs people who are kind of used to their way of doing things. But I've used PowerInfo for, like I made a web repl, and I use PowerInfo there. And it works quite well. And I like it a lot.
00:49:27
Speaker
but I just don't use it every day. Do you, does Parinfo work on, I don't think it works on Emacs, is it, Parinfo? No, no. I think there was one port at some point, but obviously, Parinfo is the way to edit the code anyway. So it's been there for 25 years. I think Emacs doesn't have enough editor affordances, does it, for Parinfo? I'm not joking about that. I think it's because it reads from your apple. It doesn't know what's going on, you know? Yeah.
00:49:57
Speaker
No, I mean, parenfera is just the editor side of it, right? You just indents the stuff based on... No, but it gets information from like where the cursor is and where the previous and next steps are. Oh, yeah, yeah, yeah, yeah, yeah, yeah. I think the Emacs editor model is a bit different compared to the other ones. Yeah, that's true. But as you're saying, I think it makes sense that regardless of which editor we use, I think
00:50:20
Speaker
In Closure World, it's pretty much like whether you can send a snippet to Repo and then get the feedback. Once that is set, it doesn't matter. Every editor, the editing experience is basically what you're used to. But the whole building the program experience is pretty much the same, I think. Well, I wonder about the pairing part of things. Because when we pair, we always pair. We pair with IntelliJ. And I kind of know what's going on. I can see what's happening.
00:50:50
Speaker
I don't know if it affects your pairing experience, Martin, when you're doing, like, someone's doing Emacs and someone's doing VI or, you know, when you've got like a heterogeneous environment of editors, you know. Yeah, that's problematic when I'm, yeah. So we usually let the person drive that's using his setup, but then, yeah, it's like reaching over.
00:51:18
Speaker
Then I can't do anything there. That's the problem. Yeah. I think this is something I'm facing a lot, because I'm working with the folks who are using VS Code and whatnot, and on Windows, and all the things that are burnt into my fingers because of Magit and all that stuff. And I need to go there and then type some weird shift to get to just commit some things on stage. And it's just really painful.
00:51:46
Speaker
I mean, once you, especially if you use Space Max and then it's like three keys that I need to type to get to the git status and everything. And then there I need to figure out... Yeah, anyway, Vijay, we're down in the weeds of your editor experience here, which is fucking fascinating, mate, I'll tell you. But maybe as we get back to next journal... So next journal, does it support Emacs key bindings?
00:52:12
Speaker
But if not, why not? Exactly. We demand it now. Honestly, that's what I think I'm most interested in, to learn from Emacs as this system that does everything, which is an external aspiration as well. All we need is next general mode for Emacs.
00:52:42
Speaker
years after working on Next Journal. Well, did you hear about org mode, which does everything you've been? I think that's pretty much everybody's answer in Emacs community. Just go there and then tell them something. Oh, what you did. Did you hear about this dot EL?
00:53:01
Speaker
Okay. Anyway, so in terms of building the front end, though, for the next general, so what type of challenges did you face? And maybe similar, maybe like an additional question for that, like now you have closure everywhere, how much is the code sharing between these two? So in terms of challenges,
00:53:26
Speaker
Yeah, I think it was less on the closure script side. Like we've been really struggling a lot with this content editable stuff. And this has almost been like switching out libraries there has been also like a rewrite of its own, of the front end. What does content editable mean, Martin? I don't quite understand that.
00:53:55
Speaker
when this is this browser stuff where you can type HTML in the browser. So whenever you're editing text and you want the native browser selection to work, you have to deal with this content editable stuff. Like the original browsers. That's why here it's the worst API of the browsers.
00:54:23
Speaker
And like, yeah, every browser does this differently, like what happens when you copy and paste and stuff like that. And yeah, we started out like in our original model, like each node that we had, like each paragraph would be like represented by closure script object and
00:54:47
Speaker
So and we'd only use the editor like for the small paragraph would be each its individual editor. And but then yeah, you can't select across these boundaries. And so in terms of the like notion does this as well. Like if you've used that, like you notice when you select more than one paragraph, it draws this their own custom selection.
00:55:14
Speaker
Yeah, yeah. The notion is the note editor, right? Note taking app. Yeah, yeah, yeah. I used it for like a trial or something or something, yeah. And yeah, so we like earlier this year decided to
00:55:30
Speaker
Yeah, to move to a proper content editable document. So yeah, you get your selections working nicely. And yeah, this was another thing. I think we estimated this to take six weeks, and then it was almost half a year. Whoa. Wow.
00:55:50
Speaker
But it is not really a closed script. No, this was, yeah. So, like, it's mostly more dealing with, yeah, these, yeah, JavaScript, these things that are just hard in the browser. Other than that, like, yeah, tooling definitely was a problem. But this has really gotten, like, yeah, with tool steps and Fig wheel main,
00:56:20
Speaker
Yeah, we've like especially doing like we we have like I think six JavaScript targets. And yeah, frequent main has this extra mains thing now where you can use the same closure script process or closure process to compile all those. This yeah, basically made our compile times much saner.
00:56:51
Speaker
And what else did we struggle with? I mean, the server-side rendering was, yeah, that's, I guess, a bit of a pain sometimes to deal with the differences. Like, but yeah, now we, like most of our code, like all the view code and stuff is now CLJC. And that now works pretty well. No, I think, yeah, we're really,
00:57:20
Speaker
happy with how things are going on the Closure Script site. Okay, so how much code is shared between the server and the client? More and more. The view code is shared. The routing, we're still on our to-do list. Currently, we still have separate routers, want to move to Reddit on the server as well. Yeah.
00:57:50
Speaker
And yeah, currently the thing that's closure only is like this code execution stuff and the pedestal handlers and stuff. So are you using a kind of classic REST API on the back end there, Martin? No, I wouldn't call it classic REST. So we're basically like when a client requests a page,
00:58:18
Speaker
gets served just the data it needs to render this template as transit. And when we're like passing, like we have this, like where you normally on the closure side would render this and pass in your view data to this function, but we're doing the same call on the client and passing it the same data.
00:58:43
Speaker
So like the first time you first visit a site, you get the server side rendered thing and like ClosureScript kicks in and gets the same data, renders the same page basically, and then the client side routing takes over.
00:59:03
Speaker
Okay. And how does this, because you said there can be multiple language cells in this thing. So if I open next general notebook and then start typing R or Python, so those processes are started on the server side and then still being routed, the data is routed to them. And then how is that working?
00:59:29
Speaker
In the back end, OK, so this we're also using pedestals, interceptors for this for the code execution. Your your say you have some Python cell and yeah type 123 in it press run. Yeah, then we get this request. Like then we figure out like we don't have an instance to run this on. Then yeah, we assign you an instance. Boot this up and.
00:59:59
Speaker
Then we take this run request and then figure out, we use these pedestal interceptors as well to figure out what's the current state of this. We call this runner and what needs to happen to serve this request. First, I need to download this Docker image. I need to boot this runtime process and then I can start executing
01:00:29
Speaker
this cell and that makes sense up to here. Is this a kind of, what should we say, a kind of imperative protocol or a kind of declarative protocol or how do you do this? Because I imagine that this is quite gnarly to work out what the various code paths are or what the various, you know, what the various permutations of this
01:00:58
Speaker
We have a first interceptor that's similar to what would be a router. In a web app, it's figuring out, I have this request and what kind of work needs to happen as a consequence of it. And then it enqueues other interceptors that do the actual work. Download this Docker image.
01:01:28
Speaker
and boot up this runtime and then execute the cell. So essentially, you capture the S2D error and S2D out from that process, or how do you get the output back? Right. Yeah, this is what the runtime protocol then does. Then you have, like, it can either be P REPL or Jupyter or our own thing.
01:01:52
Speaker
Before we move on, maybe it's just because I'm interested in how this web protocol works because I still feel like I haven't quite got it in my head yet. You get essentially a request coming in and it goes on one part of the router.
01:02:12
Speaker
the router, whatever you call it. And then that triggers some sort of payload or something that triggers what is the next, it must be some sort, is it a dynamic suite of like interceptors or is it like more declarative in the other side? Yeah, so yes, this is one interceptor that does this planning work, right? And then it uses other interceptors to perform the side effects.
01:02:40
Speaker
The plans are predefined, essentially. Yes, predefined. Yes. You know what the possible plans are that will run, or is that something which you kind of like spirit up at the time of the request?
01:03:00
Speaker
Well, yeah, you look at the server's state, like, yeah, if this runtime is already running, right, I don't need to boot it, for example. And we just have this separate planning interceptor that, like, we can test this declared, like, what this interceptor returns is just this new context map with our interceptors enqueued, right? We can test this.
01:03:29
Speaker
without performing any effects. So that's declarative in a sense. Who is the target audience right now? Is it only people who are writing scientific papers or is it actually targeting like Jupiter and data scientists and all this stuff as well?
01:03:46
Speaker
Well, good question. We're going a bit back and forth on this. I think it's heavy on academics currently. But yeah, there's definitely some people that want to do traditional data science.
Future Integration with GitHub
01:04:14
Speaker
The idea would be to have a GitHub kind of model. So yeah, people using it in public, using the product for free, using it in the open. And yeah, people that want to use it in private should pay for it. But yeah, our pricing, we're making some changes to the pricing. Currently, you can't even try it out in private. To get secrets management, for example, you have to sign up for a paid plan.
01:04:44
Speaker
And yeah, making it easier for people to try it out, like with five private notebooks. And then hopefully we'll also get more people using it for traditional data science stuff. So what's next for the next journal? So what is on the roadmap? What's next? Like, I think we... What do we want to do?
01:05:14
Speaker
the import, or I think we want to do a lot better on interop with GitHub. So we're definitely noticing that there's quite a few people that would like an external
01:05:35
Speaker
like would like their notebooks to be runnable by Next Journal, but they don't necessarily want to use the UI to create these notebooks. I mean, there's tons of notebooks already out there, like the Jupyter or some markdown format and like getting these to easily import and run and only updating them on GitHub and then syncing them to Next Journal.
01:06:03
Speaker
That's one of the main areas where we want to focus. I'm trying to remember there was another service that starts with a B, I think, that does similar kind of things like on-demand free notebook. Maybe it was, I think, a notebook.
01:06:27
Speaker
I'm not sure. It's a pretty obscure name. I only remember the first letter was B, but probably I'll remember at some point because they had the similar kind of idea like the GitHub for notebooks pretty much, but not like you like extensive reproducibility or something. It's just like a
01:06:44
Speaker
It might be in my browser history at some point. I need to look it up. Essentially, if I have a repository with notebooks in GitHub, I can just pull them into Next Journal and then I can work with them. That's the idea. I think the notebook format would be nice for tutorials.
01:07:13
Speaker
Um, yeah, I guess this, this whole interop story, like making, making it as easy as possible for people to get their content in and keeping it up to date. Um, yeah, that's where we can spend a lot of time still. You mean like, uh, you mean having some sort of hooks into the GitHub API to do that at the backend? Yeah.
01:07:35
Speaker
Nice. So, I think we have just finished almost one hour. Any final thoughts, questions? Questions? Have you got any questions, Martin? Come on, mate. Ask Vijay about his Emacs config. We want another hour of content after all.
01:08:04
Speaker
Of course. We're going to spend more time on that. Are you also not getting any work done because of Advent of Code? Oh, this year I started doing it in Rust. That basically means I'm just stuck on everything, which is amazing.
01:08:27
Speaker
So I got until day three so far in Rust. It's been fun because I have no fucking clue how to do even loops or anything like that. And especially when I start using this closure way of thinking, like just having an iterator and then start to iterate over it using map or whatever and different things, then everything is fucked up because I have no idea what borrowing means. And it keeps giving me weird errors and shit.
01:08:53
Speaker
And it's been some time since I used color. So on Rust, what happens is if you call an iterator, if you split a string, then the type that you get back is a split.
01:09:07
Speaker
Let's not get into this though. Let's not get into it. We don't really need an hour of your Rust content. There's not a podcast for that. So I just spent like, I don't know, 200 hours so far trying to figure out that shit. I'm stuck at day three now. And day three, one, first party. I'm beating you then. I'm at day five with closure. But day five is tough. I mean, day five looks like a nightmare, but I don't know.
01:09:37
Speaker
Okay. I don't know. And you're doing it as well? No, I want to do sick pee before I start on app. So it will be a couple of years.
01:09:51
Speaker
But you're really going to do it perfect at that point, though. You're going to nail it. Absolutely. But it seems like fun, though. I think I'll catch up on closure maybe during the Christmas time, once I have some time, maybe in the weekend sometime. But every day, doing it is not going to be. And what I realized is that Advent of Code is not really an Advent of Code. It's Advent of Math, of maths.
01:10:16
Speaker
It's kind of bullshit that they call it code because it doesn't make much sense because it's more about thinking mathematically and then representing that in your language rather than solving a code problem, I think. Well, there's a joke that Bork dude made, which I think is Rich says, don't do solve problems, not puzzles, except for Christmas.
01:10:45
Speaker
We're allowed to solve some puzzles this month. It's a bad time. Also, I saw that I think it's probably Michiel or somebody retweeted, it's like a nerd sniping. If there is a problem, we want to solve it. They just put it somewhere in front of you and then you're attracted to solving that shit, regardless of what kind of problem we did. It's like the cat and the light. Exactly. Anyway. Nice.
01:11:12
Speaker
Okay, so just a quickie before we head off, I mean, just final point, actually, because I think this is something which is a bit, you know, would be interesting to get from you, Martin is like the, you know, you're in Berlin, and it's like the beating heart of closure and Europe is in Berlin. So how do you find like the closure community there? Are you involved in it quite, quite a lot? Or is it something which you just sort of know? Yeah, I was really surprised, like, yeah, getting like how
01:11:42
Speaker
like yeah how big and nice every hour like yeah how big the community is how nice everyone there is and like that it's almost like yeah i found there to be like an inverse hiring problem like enclosure in there's there's really good people um that
01:12:03
Speaker
that used Closure and now it's getting more and more that are doing it full-time as well. But yeah, it's a great community there for sure. And yeah, was quite surprised that Berlin really seems to be special in that regard as how big it is and how much is happening.
01:12:26
Speaker
And speaking of Berlin, I think closure D is announced, and then it's going to happen on February something 23rd somewhere. Let me get that right. And the call for papers is still open, I think, or get your tickets. Yeah, 29th February in Berlin. So I think that's happening. Yeah, I'll be there also at Bob Conf the day before.
Announcement of ClosureD Conference
01:12:51
Speaker
This also sounds nice. Yeah.
01:12:56
Speaker
So it's a nice gathering of all the Berliner closure people. And there's a data science meetup happening around ClosureD as well. Yeah. Is it the day before, on Friday? Or is it? I'm not sure if it's been, I think BobConf is the day before, so they were thinking of doing it Thursday or Sunday, I think. Yeah. Oh, OK. OK.
01:13:21
Speaker
It'd be great to have that one as well. I think the one that you're talking about is the Daniel started. I think he was there at closure, closure tray as well. Yeah, I wasn't there. Yeah. Yeah. And this might happen at our office, I think, depending on how many people sign up. So yeah, if too many sign up, then we move to another place. Then it will move to some other place. Okay. That's nice. Fantastic.
01:13:51
Speaker
It's been really nice talking to you, Martin. I think that the whole journey of going from
01:13:58
Speaker
one language to the other and then how you discovered the coziness of parenthesis.
Next Journal's Impact on Scientific Research
01:14:08
Speaker
And then finally you realize that Emacs is the only tool. I think that's a really nice discovery. You have peaked with Emacs already. Are we going to stop this? Come on. No, I think we need to talk about Emacs more. Wind it up, mate. Come on, wind it up.
01:14:28
Speaker
It's a fascinating project and I'm pretty sure that, you know, I only use Jupyter these days, but I'm curious how far you take Next Journal and especially I think these days of people producing papers without any, this is one of the biggest problems that I see in the scientific community that the whole reproducibility is not there at all. So people talking bullshit and then you end up with no data available, no reproduction path available. No one can verify their claims.
01:14:55
Speaker
I think, in a way, this is not just technically, but also, I think, socially, an important tool that you're building, I think. So it's really awesome. And it's really awesome that you're building it in Closure and Closure Script and Emacs.
01:15:13
Speaker
So congratulations on the continued success of Sheephead. And hopefully, you guys will do some amazing work with Next Channel as well. And so people can go to nextchannel.com and then sign up. Yeah. Play with it for free, pretty much. We're going to make those pricing changes soon. So you can also use all the secrets management and stuff
01:15:43
Speaker
And yeah, you can run stuff on GPUs and typically a pain to set up. Yes. So you're solving a big problem for people to start up with the data science and this kind of things as well. And for all closures, like right now where you're thinking, yeah, if we're going to have to integrate par infer or par edit or
01:16:11
Speaker
Yeah, and which to make vault, so yeah, anybody who has an opinion on that should please reach out. Yes, so tweet at next channel or Martin's tweet that we'll put it in the in the description. So that's it from us for today, I think. And a big thanks for our patrons, Patreons. We have a couple of new folks helping us as well.
01:16:38
Speaker
And we have another episode scheduled pretty soon that we're going to record next week. So hopefully this episode will be out in one or two weeks once our engineer comes back from vacation. Yes, we might get more in for Santa Claus, you never know. Yes, exactly. So that's episode number 55. See you guys and girls and folks, everybody again in the next episode. Thank you. Bye bye.
01:17:32
Speaker
you