Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#64 - James Henderson - Crux
66 Plays5 years ago
#64 - James Henderson - Crux by defn
Recommended
Transcript
00:00:14
Speaker
Welcome
Pandemic Situations in Europe
00:00:15
Speaker
to Defend. This is episode number 64. Still, I'm in Holland, where the pandemic seems to be the second wave. And we are not allowed into Belgium and Germany. So I guess that's okay, I guess. Nobody goes to Germany, sorry, Belgium, anywhere. Who the fuck goes to Belgium? It's Belgium. You're not allowed to come to Belgium, especially you, you know.
00:00:42
Speaker
like the Hichagr guy doing like the worst word in the game universe in the galaxy. I think we're all getting fucked by Corona. So yes, in Belgium, it's because of these bloody idiots who want to send us kids back to school. And I'm one of them, unfortunately, because we have got a teenage son and he's back in school.
COVID-19 Impacts in the UK
00:01:02
Speaker
They've already had a case at school. God knows.
00:01:06
Speaker
if they're going to change their mind. It seems like a weird concept to have all this social distancing, small bubbles, and then say, oh, you know what? All these people that are known to be super gregarious and lack any discipline, let's just get them all together. Anyway, so what about the UK, Mr. James?
00:01:30
Speaker
Welcome, welcome to DeafN. Thank you very much. Yes, UK, they're starting to talk about second waves again here. I mean, it's just about starting to make the news again, in terms of people talking about it and talking about potential lockdowns and that kind of thing. But I think we're in a similar situation, schools went back for a crack, it must be a couple of weeks ago now.
00:01:51
Speaker
Yeah, they've just finished the eat out to help out. So we've all we've all been sort of dining out for half price on the government. Just rather nice. Thank you. Thank you Chancellor for that one. Okay. What is it called sunny? Rishi Sunak. Rishi Sunak. Yeah. But yeah, and then everyone back in the office as well apart from us obviously as techies.
00:02:21
Speaker
And yeah, and they're starting to wonder why cases are going up again. Yeah. Anyway, let's start. Yes. Agenda item number one, the running order. Exactly. Yeah.
00:02:35
Speaker
Check everyone's, well, have you, I mean, funnily enough, well, we can just do this little quick check, you know, the COVID related kind of stories. One of my colleagues at work, we all work remote, so it's not like a breeding colleague. Well, he is breeding, but he's not of a mate. Not breeding in your case.
Personal COVID-19 Experiences
00:02:54
Speaker
I've seen him breeding on the Zoom or whatever it is that we use, you know.
00:03:03
Speaker
He did get yeah exactly he did get a he did get covered and he's a young lad very fit i mean that young lad he's about thirty five but you know it's not like a teenager but it's for young man was kinda like mma guy mixed martial arts and i know super fit block.
00:03:20
Speaker
And he had it and he's like, well, I think it's like two or three months on now and he's still kind of like feeling it, you know, in his chest when he starts to do stuff. So it's pretty bad. Do you guys have any experience of colleagues or family members that have got it? Unfortunately, not for me. Nextdoor neighbor had it, I think, quite early on and he said it was really quite rough. Yeah.
00:03:45
Speaker
I think next door neighbors count, you know, I mean, it's right next to you, you know, like half a mile, you know, 300 miles away, but the next door neighbor, yeah, he's out here, but you know, we're still kissing. We're still going to come around for dinner every Friday, you know.
COVID-19 in New York
00:04:08
Speaker
And one of my colleagues, he got a COVID test and he said, it's horrible. The test itself is like really, really bad because they put the swab up your nose, almost touching your brain. So that's not a good experience, but luckily it came out negative for him. But there are folks in New York, in one of the branches of the company that I work for.
00:04:31
Speaker
Um, they mean, uh, their mother or somebody passed away into COVID, uh, in the first, uh, really, really bad. Uh, it was really bad in New York, wasn't it? Yeah. Yeah. Yeah. Yeah. It's, um, yeah, I mean, it's kind of a strange thing. It doesn't seem to have an end. And now they're talking about maybe they'll put some sort of a curfew. If goes, if it goes really, really bad in Amsterdam, for example, uh, in the Netherlands, actually we have these big cities and then the three, four big cities together, that is the.
Rising Cases in India and Netherlands
00:04:59
Speaker
Like the commercial center of the Netherlands we call Randstad, like the Hague, Amsterdam, Rotterdam, Rotrecht. So that is the place where the most cases are raising these days. So they're trying to see if localized restrictions can be put in there to reduce. And the weather is not helping because the fucking summer keeps going on forever here. It's super annoying for me coming from India.
00:05:23
Speaker
You know, running away from summer. India is having like three million cases a day, isn't it? So it doesn't look like the heat is not good for this thing. We have like billions of people, so statistically, they're going to have a lot of stuff. That doesn't make news over here. They probably should. Yeah, it's mostly local. And also, I think the death rate is a bit lower in India, though.
00:05:52
Speaker
But the death rate is not at the same level as we had in Italy and Spain, which is surprising. So it's surprising. Yeah. Maybe it's because India is more, quote unquote, younger population because we did have that
00:06:08
Speaker
I think this
Desire for Lighter Topics
00:06:09
Speaker
generation is there is a lot of folks who are very young in India, compared to Western Europe, where the population is skewed towards older generation. Yeah, that could be one thing. But anyway, obviously, as I'm a epidemiologist,
00:06:23
Speaker
export obviously, so I think my opinion is most important. I'm just gonna say whatever the fuck that I think. Okay, so... But dude, why vary from the rest of the fucking podcast?
00:06:42
Speaker
I think we should just keep talking about COVID. And then James will be like, OK, I'm going to check. I'm like, fuck this thing. I mean, people are hearing about this stuff already, like 24 by 7. I know, I know. It's a little. Yeah, yeah, yeah, it is. I mean, we shouldn't avoid it because it's pretty serious, I understand it. But we know we need our little, little respites away from.
00:07:02
Speaker
Yeah. To breathe and then, you know, OK, let's let's try to focus on something. Anyway, so on that. Just on a happy note, actually. No, I am interested actually as to because I know some people I mean, for me, Covid hasn't been too bad and I'm very I'm very privileged to have that situation where I've worked remotely before Covid and
00:07:28
Speaker
We're in a relatively rural area, so we don't have a lot of people. I'm very lucky that it has been relatively strict and I've kept working, so the economics haven't really affected me too much. It's incredibly fucking lucky, basically. For me, I got a dog after the lockdown.
00:07:52
Speaker
Yeah, but honestly, it's made me into the neighborhood. So many people in the neighborhood know me now, and it's like getting invites to come and, I'd say anti-COVID actually in many ways, to go around to people's houses because they're home.
00:08:11
Speaker
But it's, but it's a, it's a remarkable, I mean, you know, I got the, I didn't get the dog just cause of the lockdown, but it's, it's definitely helped with this isolation, I think. Um, and, uh, and it's definitely helped to integrate me with the more neighbors.
Positive Social Interactions in Pandemic
00:08:25
Speaker
And we have, I've lived here for 12 years. I've never spoken to the neighbors as much as in the last two months. So, so it's, it's a bit of a nice story to me coming out of COVID, but I don't know what you guys, if you've got any kind of nice reflections on things that have happened that are good rather than.
00:08:41
Speaker
We saw something quite similar here. We've had a doctorate a few years now. As soon as they came out and said, everyone's allowed one walk a day, you're allowed out for an hour. We found a whole load more people. We've got a green area over the back there that we take every day.
00:09:02
Speaker
But we found a whole load more people, sort of families out, you know, you name it, sort of people looking quite lost, like where I never realised we had this green folks around. But then, but then all of a sudden, the all the restrictions relaxed. And when everyone was allowed out, all of a sudden, it's got really quiet again. It's like you're allowed there now. Yeah. So so you're finished with all the spaces back. Yeah, it's just the dog walkers again.
00:09:33
Speaker
I got the same thing. I got a really small puppy, so he needs to go out a lot in an apartment, pretty close to the dunes and the beach, like five minutes outside. So I started going out like every two hours, like a clockwork. It took almost two weeks off just to stay with the dog.
00:09:51
Speaker
We stay with the puppy and then just walking him. So I'm showing up at 5.30, 7.30, 9.30. Every two hours I'm just walking the puppy outside. In the night as well. First week I did in the night as well, but I think the nice part is that then the house broke like in one week.
00:10:13
Speaker
So he stopped doing his shit in home in second day already, which is nice. But now he wants to go out like every three hours. And the funny thing is, just like Ray, I've been living here for 13 years in the same apartment. And obviously I don't speak Dutch, so that's a bit of a kind of a gap.
00:10:35
Speaker
Usually Dutch people are much more independent. So they don't get into your noses, you know, like, well, Indians usually. But now since I got a dog, now, you know, 13 dogs names now already. No, I have no idea the owner's name, what they are, but I know all the dog's names now. You just refer to them like, Oh yeah, your Toby's dad. Exactly. They're like, Hey, this is pepper. This is Sam. This is joy.
00:11:05
Speaker
What's the weirdest dog's name? Uh, I don't know. I think, um, I heard if, uh, I don't know what that means. Uh, there are Dutch names that are, that I think it goes. Okay. Yeah. Okay. Um, it's normal, I think. And then there are some salmon joy. Those are pretty okay. And Zian.
Dog Ownership During Lockdown
00:11:23
Speaker
Wow. This is a dog naming it. Yeah.
00:11:27
Speaker
But yeah, think, James, what's your dog's name? My's Ruby. Ruby, nice. She's a rescue dog. Oh, nice. I think same as Ray, then. Yeah, we have a rescue dog as well. My dog is called Eula, as in end user licensing agreement. Because, you know, it's complicated, but you just have to accept him, you know?
00:11:52
Speaker
I'm pretty sure I've told this joke on the podcast before, but you know. Nobody listens to this shit anyway, so, you know. Which is why we can afford 20 minutes of bullshit to begin with. Exactly. I think I named my dog Baurik Wowbagger from Hitchhiker Guide to the Galaxy, and then that was too long for people, so.
00:12:13
Speaker
I just call him Bowie and then, oh, it's like David Bowie. Like, I guess, because I've never heard his music. Oh, man. I thought it was called Bowie for that reason. Yeah. No, no, no. He's he's called he's Boric Wahlberger. So short Bowie or Bowie. All right. OK. You've got to choose it for something you don't mind shouting over a field, though. Wahlberger.
00:12:36
Speaker
And the weird part is every time I introduce the dog to somebody, and especially some older generation people, they're like, oh, let's dance and something. I'm like, what are the different? I have no idea. Yeah, that's a song by Power Guys. That's what I figured.
00:12:53
Speaker
Apparently he's big. I don't know. Okay. Yeah. Or he was. I don't know. Anyway, so, um, I guess we covered all the topics already, you know, uh, we talked about pandemic, we talked about dogs. I think most important stuff. Let's get to the less important things. So James,
James's Career Path
00:13:09
Speaker
um, please, uh, please let us know your, uh, about your closure journey. How did you end up in closure? Wow. Pleasure journey. Okay.
00:13:17
Speaker
So mine, I started off working with it about, I think it's about eight years ago now, as a recent graduate at the time. I was looking back on it actually, I was really fortunate to be in the right place at the right time with this one, because I was working for an investment bank and looking for my next graduate rotation at the time on the grad programme.
00:13:39
Speaker
Um, and I bumped into, bumped into Malcolm, uh, Malcolm's box of, uh, some internal graduate friend to the shore. Yeah. So I bumped into him at some network event. Um, and he had an opening for a graduate who was interested in functional programming. Um, and I've done a little bit of Haskell at uni, um, but, and despite me not even having heard of closure, um, he decided to take a chance on me. Um, and I guess, I guess, I guess the rest is history really.
00:14:08
Speaker
So we both left the bank about a year after that, I think. He went to form Juxt with John. And then I went to a small little closure startup. And I've been writing closure ever since, really. For those last eight years.
00:14:24
Speaker
And then I moved back to, I moved back to jokes, I say back. I didn't work for jokes originally. But yeah, social circle, yeah. Yeah, yeah, yeah. Nice. Okay. So when you were at college, then between college and this job, you know, was it like, was that starting with like, Java or something? Or what were you doing something there beforehand? I'm
00:14:51
Speaker
Or was it always a functional programming world that you were getting involved in? No, funny you should ask, actually. So I went straight from uni into this graduate program. But for the first rotation, I was actually put into a business analyst role because they'd bug it up on the numbers, essentially. And all the people, all the software developers that hired off of CS courses, they had too many
00:15:18
Speaker
too many offers out for the amount of placements they had. And so they essentially picked a few out of a hat and said, right, you're going to business analysis. Okay, just quite quite an interesting first few weeks, not really what you not what you expect when you come into the job. But actually, I say this, but sort of recent graduate me absolutely hated that role. It was it was pretty much sort of straight out of, you know, I want to get into some sort of serious industry code now. Yeah, you are. And what I thought at the time was essentially sort of
00:15:47
Speaker
you've put me into this role, which is all about sort of writing specs and talking to people. What's this about? Don't get me wrong, I really like the people I work with, but the job really wasn't for me. But I tell you, looking back on it, I'm really sort of thankful for that, because sort of seeing that other side of it, seeing that other side and sort of really getting to chat to business users, I think was a good start there.
00:16:14
Speaker
And then when I did get a choice in the matter, when I did manage to get back to coding. Yeah. Well, it's nice that you have that experience early on. I remember when I was starting, one of my mentors, he was telling me that, okay, first you should work in customer service. And then as a QA, it used to be a role back in the day, then you become a developer.
00:16:38
Speaker
that then you would understand, you know, like what the hell we're doing. And then then there is a nice link between the actual product that you're building. Otherwise, I mean, you know, I've seen some people who have been working in only as a only as a developer, working as a developer, and then they're so, you know, detached from the business reality of the situation. They always want to try out new shit or whatever. And then, OK, look, for the business, I want to have boring shit that can run the stuff.
00:17:07
Speaker
I know not some point, not one, at least somewhere. So that's a bit of a, yeah, I think it's a nice experience to have that, right, in the beginning of the career already. So I've seen the other side of it. Yeah, I'm certainly glad I'm in hindsight. Yeah, yeah. So you started with Haskell, or you learned Haskell at the university in a little bit. So really, I mean, it was the, yeah, the first term of programming was Haskell.
00:17:35
Speaker
And they introduced that, I think, as a little bit of a leveller, because there were probably a few people in the course that had done various bits of programming before, but they could now guarantee that no one had done Haskell. So, it really did level the playing field. Again, it's that mindset, right? It's that functional mindset, I think, that we're really looking to really teach there.
00:17:54
Speaker
Yeah, they did then cover Java afterwards. But you could tell all the lecturers were sort of treating it as a kind of a second class language to their preferred Haskell. But yeah, it's, I mean, Haskell is great. I mean, I know, I know you said that sort of, I know we're on Closure podcast and all, but it's one of those languages I really feel that changes the way you think about programming, like Closure.
00:18:16
Speaker
Yeah, yeah, yeah, yeah, certainly. I think that, um, no, I don't, I don't think anyone, sorry, go on. Yeah, you can say that, you know, closure took a lot of ideas from Haskell, right? And then just remove the types from it. Cause if you see the closure code, uh, core functions, they're pretty much similar to Haskell prelude. Um, I think there are a lot of ideas.
00:18:35
Speaker
Haskell has been, I think, as maybe I'm misquoting, but who cares. I think the Simon Paton Jones was like, you know, Haskell is like the idea pool, so you can pick good ideas from there, similar to Lisp sort of way. I would say, yeah. I was going to say that, yeah, I think for me, the
00:18:55
Speaker
I think things like Scala hired types and Idris and stuff like that. They're very mind-bending experiments. I remember when I was getting into Scala, I even went as far as buying the Godel Escher Bach book.
00:19:14
Speaker
I didn't read it, but I bought it. I was excited. I was like, oh, yeah. I think I've given it away to the charity shop now, but it was just too much. For me, I think maybe it was just too big a reach to go to those type systems that are so complicated. I'm sorry to say, but they definitely bring benefits. I can see that.
00:19:41
Speaker
But it's just like actually working with it and designing stuff with it was beyond me. I'm pretty sure once you get over the hill, then you can feel it and it probably feels a bit more plastic than it does on the surface when you look at it from outside. I mean, how did you find that? Once you grok that you could throw things together pretty quickly?
00:20:09
Speaker
I don't pretend to have dropped it for a moment. We did probably one, maybe two terms and then a few exercises after that. We are all Haskell experts here then. Let's talk about Haskell. What is wrong with Haskell? We know everything now. I think with Haskell, there's a good few developers I really look up to that really appreciate it. There's definitely something there.
Bridge Project Exploration
00:20:39
Speaker
Yeah. But talking of Haskell and Closure, you're doing this... Well, you tell me what the status of this project is, this bridge project. Yes, bridge. Bridge. So bridges, it's a compiler that I'm working on. I'd say quite a toy compiler, really. Because I was looking into various things. It's mostly for my own learning, I would say.
00:21:05
Speaker
It's it's a kind of a halfway house between probably closure Haskell and any other than any other number of other languages. It's based on growl VM and truffle. So I think that's got a good few buzzwords out the way already. Oh, yeah, yeah. This episode is like so high. If they were making a drinking game for death, then it would be like everyone would be absolutely pissed at this point. Yeah.
00:21:35
Speaker
Yeah, there we go, there we go. But you know, I want to do a non toy project. It's actually written in Kotlin.
00:21:40
Speaker
That's what I just noticed. It's pretty interesting because your main project or your eight years of experience is enclosure in, well, dynamic typing, dynamic but strongly typed. And then you switch to Kotlin, which is a bit more strongly typed than I think static typed, you can say Kotlin. And then you're building a statically typed Lisp in Kotlin.
00:22:08
Speaker
So, why? You just said it was for fun. Don't ask why. It's a side project, you don't really need to have a justification for side project. So, I mean, partly it was to learn Kotlin because I've not done Kotlin before. And I was a little bit conscious that I have really, in industry, I have really only seriously worked in closure projects.
00:22:32
Speaker
Yeah. So it was just a little bit of an insurance policy there, if you like. So if you want to become an Android developer, you could do. Yeah, well, yeah. But so I wanted to learn how compilers work, how they're implemented, that kind of thing. I wanted to learn how type systems work.
00:22:54
Speaker
Yeah, actually probably in truth I've spent more time sort of way more time reading and researching than I have writing bridge in the end. I've been looking at a lot of different languages and how they approach various things, why they make the decisions they make. Isn't that the classic hammock time?
00:23:12
Speaker
you know so so you're doing you're doing the right thing you know yeah i hope so i mean if nothing else it's been it's been quite fruitful for that um and you and you really do get an appreciation for the level of sort of thought and consideration that needs to go into writing a language
00:23:26
Speaker
And closure, especially, I think makes a lot of good decisions, right? I mean, that's why we're here, right? In closure. I guess I'm also interested in a kind of spectrum of solutions, right? So we've already got a little bit political, but without getting too political, debate seems quite polarized at the moment. You've got static versus dynamic. You've got OO versus FP, right?
00:23:49
Speaker
I don't know, Trump versus Biden. No, I'm not going to go there. The world doesn't need any more foreign election interference. I'm not going there. I'm not sure. I'm not sure. What is the UK equivalent of it? Whatever that is. I think we're just joining for the next few years, aren't we? Anyone else? Yeah, we're deciding. So was it 2024 or whatever until the next one?
00:24:12
Speaker
We've got a while. But yeah, joking aside, as with most things, there's a lot of middle ground to explore, hence Bridge, hence the name Bridge. Yeah, that's a really interesting thing because I haven't seen, well, I dabbled a bit in Haskell and also I wrote
00:24:34
Speaker
reasonable amount of Scala got paid for it. Um, so I was looking for, you know, I want this metaprogramming closure level, lispy type thing, but possibly a bit more, uh, static typing as well, because I've tasted that as well. Like, you know, what compiler can do in Haskell and Scala. I don't want the whole shebang that they bring in, but you know, then there are like,
00:24:59
Speaker
certain class of things that become, they become very easy when I'm, I feel more comfortable, you know, writing that code in Haskell or Scala. So I did look for, is there any statically typed list? You know, of course there is a typed closure, that one, and then Racket had a date on version of typed Racket to introduce types gradually, gradual typing. And then there was a small language done by, I think somebody from Latin America, probably Argentina, not sure, or Venezuela, called Lux.
00:25:29
Speaker
I think he also gave a talk during one of the closure cons or something. So that that's ML, but lisp, you know, essentially that kind of thing. So it was all the union types and everything.
00:25:42
Speaker
ADTs, but somehow the syntax didn't click with me, you know? Introducing these types into Lisp, they always look a bit strange because maybe we are used to Clojure too much. Yes, yeah. And you feel sort of Clojure is very simple in that respect, isn't it? They've got very, very little in the way of syntax. And when you bring types on top, you naturally have to introduce quite a lot of these kind of mechanics. Yes, yes, they did additional syntax.
00:26:09
Speaker
What is your approach in Bridge then? I mean, how are you thinking to introduce the types? Like what type of type system it is going to have or it has already? So, I mean, I'm kind of looking between the two, right? Because I quite like the, I'm really interested in sort of the flexibility and ease of development of closure. Yeah. Again, I suspect it's one of the reasons we're here doing it.
00:26:30
Speaker
But I'm also interested in what static guarantees are out there, as long as they can be a help rather than a hindrance. One of the big things you hear about from dynamic typing people about static typing languages is that it gets in your way, Java. You don't want to write all that out and all the rest of that. But there's a world of difference between those static type systems and what's available, what can be implemented.
00:27:00
Speaker
What am I looking for? It's got probably the closest thing I could find to the way closures maps work. Polymorphic records and variants. It provides you the flexibility of dynamic maps with the additional removal of keys. Quite like we used to in closure really. It's not too dissimilar to spec in that respect.
00:27:29
Speaker
in terms of saying, I'm expecting this parameter to have these keys and that kind of thing. And you can infer a lot of that now. There was a PhD thesis came out probably a few years back now called algebraic subtyping, which dealt quite a lot with those kinds of things, how to do type inferencing when you look up there.
00:27:53
Speaker
not got as much to go on from the user when the type system's got to infer a lot more. And then you've got variants, which are the dual to that. So they're your sum types, so your union types. So they're ones that are useful for checking that you've dealt with all of the cases. Looking into that, certainly is a bit of an area of research.
00:28:18
Speaker
So the types, they're not like gradual types or something. They're carried along with the object themselves in Kotlin. It's fully statically typed. So I've not gone for gradual typing. It's not really an area I've looked into, to be honest. But I felt that it gets a lot of benefits from being 100% typed. Gradual typing introduces further complexity at the borders.
00:28:44
Speaker
Isn't that also why you are able to target GrowLVM?
00:28:50
Speaker
Because
Bridge's Technical Aspects
00:28:51
Speaker
then you can, you know, if you, if you, one of the, one of the issues, like, why, why did Michael, Mikael, Borkent write the small closure interpreter? It was because, well, you know, we just can't do eval. You know, we can't, we have to remove a whole class of these things. So I think, I think like, and it's an interesting question to you, whether you took any inspiration from the small closure interpreter.
00:29:14
Speaker
that runs on GraalVM versus what you're trying to achieve. Because it feels like that approach that Mikhail has taken is sort of, okay, it's still closure, essentially, but it could lead the way to if you were doing your kind of thing to static typing as well. So I was wondering if you felt there was any crossover there between SCI and what you were doing.
00:29:40
Speaker
And so I would say with Graal, they've got implementations for JS and Ruby as well. So it doesn't necessarily need to be statically typed to go on Graal. And Graal does quite a lot really with dealing with dynamic languages and dealing with them when they can be unpredictable and change types of various variables and that kind of thing. And it does a lot of optimizations under the hood for how to deal with that efficiently.
00:30:07
Speaker
And with SCI, I'm not looking a huge amount into the details of how it works on graph, but you certainly get a fair bit. And you certainly get a fair bit for free with graph, which is one of the reasons I was looking at it. And so you talk about the native image side there, which is what makes SCI really quick. Now to start up, I mean, they've also got quite a bit of tooling as well. So you can implement a fair bit and it gets you a debugger. It gets you profiling that kind of thing.
00:30:35
Speaker
Yeah, and do you think about the metaprogramming with the types, I mean, closure type metaprogramming in Bridge? How is it going to work? Sorry, not really, no. Not really, no.
00:30:53
Speaker
We always have the hard-hitting questions. I've been looking into macros and that kind of thing. So the macros you can't quite do in the same way, because at least the way I've done it in Bridge. So Bridge doesn't have heterogeneous collections in the same way that Closure does.
00:31:20
Speaker
Yeah. See what he said there. He's like, Oh, okay. But, but that's it. But that's the thing, right? Because in, in, in, I'm not sure in Haskell, in, in Scala, there is a library called H-list, you know, like you can have a heterogeneous list, still typed, you know, that, but that is the, because this is something that, that fascinates me, you know, like bringing, bringing this kind of ideas into, into lists with types is, is, is challenging because, you know, fundamentally they're, they're, they're
00:31:48
Speaker
I mean, in my mind, at least they're incompatible.
00:31:51
Speaker
But isn't it just a container that you just have a sum of? The way that I've done that is a container that you just have a sum of. So yeah, the forms as a data structure is a sum type. And then the type checking doesn't happen until, I mean, it type checks that you return it a form, right? Your macro is like a function that returns a form. It takes a function that returns a form.
00:32:18
Speaker
Yeah, yeah. But then it type checks it after that. So after you've done the macro expansion, that kind of thing, is then when the main type checker kicks in and makes sure that what you've macro expanded to is about in time. Yeah. That's perfect. Yeah. So it is still in experimental phase, or is it like a 0.0.1? That means we can use it already. Oh, crikey, no. No, it's very much experimental.
00:32:47
Speaker
To me, it's still got a fair bit that I'd like to do with it. There's loads to think about. I mean, I've not really touched on because I've been looking to things like sort of side effect tracking as well. Structured concurrency and that kind of thing. Again, more passwords for you. As a small thing, before we get into monads, please.
00:33:16
Speaker
The sound effect tracking, funnily enough, and we're going to come on to Crocs in a minute, but I was doing some, and this is like as a user of Crocs actually, part of my usage of Crocs. In our work, we wanted to use Crocs to do some things, and one of the things that we have in our blockchain
00:33:42
Speaker
For testing at least, it's not always true for the real thing. But in the testing side, we have some atoms. And so in the real world, we have some atoms. Yeah, we do that. The atom's a thing. But in our testing world, I've used a watcher.
00:33:59
Speaker
on that atom. So it's a really nice way actually of kind of looking at the effect of the updates of the atoms and the previous and the next kind of like before and after type things. And this, I don't know.
00:34:15
Speaker
I know about this stuff and I've played around with it, but this is the first time in the last few weeks that I've really used it in production code, in test code, but it is code that we're going to use. Everyone's like, whoa, shit, that's really nice. That's a nice use of that tech because it's not seen very often. I wonder if that's related to the thing you're talking about in terms of side effects or tracking these things.
00:34:39
Speaker
There's quite a few of those enclosures, isn't there really? I sort of remember earlier on when closure came out, everyone's talking about sort of atoms, refs, agents, and all that. Yeah, yeah. And the whole SDM world. He's very little of that now. Yeah, very little. The patent that's gone away, and everyone sort of sticks their atoms and...
00:34:57
Speaker
Yeah, I think agents, the agents and then refs and then atoms. And then now pretty much everybody just use only atoms. I haven't seen any code with refs or agents at all. I mean, agents used to be something similar to actors, but not really. So you could get something done. The problem with those things is that you end up hitting all kind of edge cases and corner cases that just don't work very well, basically.
00:35:26
Speaker
You end up getting locked out, basically. Shit, we don't want that. With Adams, you can't be fucked.
00:35:36
Speaker
Just just to get the first fucked in there. It's not the first one. Oh, sorry. We had seven already. I mean, we're so, we're so how do you call that? Like, we're so blind. Yeah, exactly immune to it. We don't even notice that anymore.
00:36:00
Speaker
Anyway, but coming back to, I think we talked about Crux and so maybe it's a good idea to introduce to the people what Crux is, you know, like me. We
Introduction to Crux Database
00:36:13
Speaker
didn't actually say actually that you're one of the core developers, aren't you James, on Crux? Yes. Yeah. I look after the day-to-day development at Crux. Yeah.
00:36:25
Speaker
So you can give us the crux of crux. Funny enough, yeah. So crux, it's an open source, immutable graph database. In short, it's the tagline. But it's by temple. So it allows for all of those sort of time travel updates and updates and time travel queries. And you can now query it with both data log and SQL.
00:36:49
Speaker
I've recently added SQL support to that. Essentially, it's an unbundled database. A database inside out, if you like. It's based on all the industry titans like Kafka, JDBC, Roxy, the underlying storage. We very much benefit from their scalability and their availability, the decades of work put into those projects. We're very much basing ourselves on them.
00:37:18
Speaker
Yeah. So, yeah, maybe you can, can you explain what do you mean by temporal? Like, what does it mean? It means he buys temporal. You know, it's like the consumer, the consumer society. Yeah, exactly. Yeah, by temporal, so literally two times.
00:37:43
Speaker
And the two times we have are what we call valid time or business time and transaction time or system time is often called. They've got so many different names. That's four so far. Two names, three to two concepts, yeah. So what are the two concepts? So go on, go on. So we call them valid time and transaction time. But every paper you read on it will have a different two names. Every database that you come across will call themselves something different.
00:38:12
Speaker
I guess bi-temporal sounds better than two-timing. Two-timing sounds sketchy, doesn't it? Yeah, exactly. I've got a two-timing database. It's like, oh, OK. If there is a paper about all this stuff, it's like, oh, it's a two-timer database. Like, OK, that's sketchy. But the main thing this adds to you, you've got the transaction time, which is the one that's always moving forward.
00:38:40
Speaker
So this is the one that gets automatically added to any transaction you put in. And you know that the transaction time is always increasing. That's the wall clock time, isn't it? The wall clock for the system. But the valid time is one that the user chooses. So it's at that point I can explicitly make corrections in the past or even put code input data into the future as well.
00:39:08
Speaker
So if I want to schedule data to become true at a certain point in the future, I can do that using valid time. And I kind of think of the combination very much as thinking about corrections. So whether you think about corrections or not. Because if I'm looking for audit, for example, if I want to go back in time and say, what did we know at this particular date? I can go back in valid time.
00:39:35
Speaker
And what that does is that then says, what do we now know to be true about that time? So I do see corrections in that case. If I go back in transaction time, that's without corrections. So what did we know then? Why did we make that trade? Why did we accept that contract? That kind of thing. Exactly what data did we know at the time, regardless of what was corrected later.
00:39:59
Speaker
Yeah. So this applies to the entire graph or because you said it's a graph database. So does that mean it's a graph database in the traditional sense? Like you have nodes and edges and then the whole thing has these two times or is it individual nodes?
00:40:20
Speaker
So you can think about it as a graph kind of evolving through time. So at each different point in time, you take a snapshot at that point. And I guess you can kind of think of it as a, sorry, steal a common phrase in Clojure. It's like a graph as a value. At that point, take a snapshot of the graph. And then that's what we show at that point in time.
00:40:50
Speaker
Incorrects its documents rather than nodes and edges specifically. But it behaves in the same way. So your document is, your document kind of parallel to your node at that point.
00:41:02
Speaker
And then the edges are done by values within that document. So if I have a key value in that document, and the value happens to be an ID of another document, that behaves as an implicit edge for us. Oh, so that's an automatic one. It's not like you need to, yeah. Okay. Because a long time ago, I was using MongoDB.
00:41:24
Speaker
before MongoDB became a reasonable database. We had to build a similar kind of thing, like kind of a graph on top of MongoDB, and then we had our own keys pointing to other document IDs. I was wondering how this works in Crux. Any value pointing to the ID of another thing automatically becomes
00:41:49
Speaker
edge for that node. Is that? Ah, okay. And how did you implement SQL on top of this? Because I'm familiar with at least two graph databases, Neo4j and SAP HANA. That's enterprise crap. So they have this graph in memory.
00:42:12
Speaker
columns store everything together, but they don't provide SQL as a normal SQL anymore because you can use either Cypher queries or GraphScript. That's the thing that they have. So how does SQL work on this kind of model?
00:42:29
Speaker
And so in terms of, in terms of the modeling, um, with, with the crux SQL module, um, we get the user to specify, um, their relations. So the relation is a query over the graph. Um, and that, that then ends up with, um, that then ends up with a relation or a table, um, as you'd have in SQL. Um, and then when you pass SQL to it, um, you pass, um, the SQL you write is in terms of those relations that you defined.
00:42:56
Speaker
So I might define, I might define a data query that says, here's all the users in the system. Find user where user has a user ID. That kind of thing. That then gives us something akin to a table. And I might then write things like forum software, let's say comments. Here's a table of all the comments in the system. And here's a query for how to get them.
00:43:24
Speaker
The SQL was then interpreted by a library Apache calcite, which gives us a hell of a lot of SQL parsing for free. We're certainly not in the business of writing a full SQL parse, but then what then happens is that then breaks it down into an AST for us. We can then combine the queries for each of the relations, in fact, if they're joined.
00:43:54
Speaker
with the query IST. And that then essentially translates into one big data code query. That we then saw. We are translating it back to data log and then applying that onto. That's nice. Isn't this the nice thing about like, you know, like the fundamental architecture of jokes, essentially, is that it's, that it's scheme-less so that Crocs, did I say jokes?
SQL Integration in Crux
00:44:15
Speaker
Yes. Oh, we do it all the time.
00:44:19
Speaker
Yeah, well, you know, I've had two glasses in, so... Yeah, so the architecture of Crux, which is essentially schema-less, because that gives you the power to essentially define not only schema on read, but actually type of schema that you want to produce as well, that you want to support.
00:44:44
Speaker
Yeah, absolutely. And so we can see that being extended to other query languages as well. So we're looking at how to support that for a similar type of pattern for defining GraphQL schemas. And then exposing a GraphQL API in a similar kind of way. Okay. I mean, this is really nice, I think. Go on, sorry. Just another question about the SQL stuff, though. So... You've picked them online for this, by the way. You want John back for that.
00:45:12
Speaker
We'll get everybody out. Bring him in. Get him online. I'll give him a quick look. The databases are fairly interesting for me. I keep following some interesting tech that is happening in the outside closure as well.
00:45:31
Speaker
that the SQL interface that you have, that means it also exposes these two times the bitemporalness to the SQL level as well, that I can query for where valid time is this or something like that. So unfortunately, we don't fully expose the bitemporality to SQL, at least not yet. So I mean, there was quite a lot introduced in SQL, whichever standard that was, 2011.
00:45:56
Speaker
I think it was 2011, wasn't it? With all the bitemporal functionality, which I don't think calcite yet supports. So we've not gone for that as yet. There aren't many major databases that do it. I think DB2 supports it, and I think maybe it's Oracle with some extensions or some shit. But it's not that common. I don't think SQL Server supports it, for example.
00:46:21
Speaker
So even the big guys, and I don't know, I mean, I don't think that Maria SQL supports it, for example. Yeah, probably not. Because there is a, I'm sure you might have heard this one, there is a database thingy called materialized. It's based on similar kind of ideas, but
00:46:41
Speaker
The thing is it has SQL queries, but the SQL query can stream the latest ones because the underlying storage is changing continuously. In the case of Crux, the graph is evolving, but I make a query and then I'm getting a snapshot of the data.
00:46:56
Speaker
But if I have to get a new information, then I have to query again. But the materialized guys, they push the data or not push, but actually they're continuously updating the back, the storage table or view, essentially materialized view.
00:47:12
Speaker
So this is a pretty interesting thing for analytics, for example. Now you can get like streamed results. So I was curious, what plans do you have in that direction to take SQL being one of the primary ones? So, I mean, we've obviously got Datalog as kind of the primary language. That's the language Aquarian is written in. We have done a little bit of unbundling in that area as well recently.
00:47:42
Speaker
we could start to look at things like having multiple languages based on top of it, essentially just as guests of Crux, of which SQL might just be one, data log might just be another, GraphQL might be another, that kind of thing. In terms of by-temporarily we're exposing, so Crux at the moment really only deals with point-in-time queries. So when you take out that snapshot, you take out that snapshot at a valid time and at a transaction time. So that's all we expose, even to the data log at the moment,
00:48:13
Speaker
But one of the things we are looking to do, and it's part of our ongoing research, is looking to academia to find out sort of what by-temporality does make possible. And things like advances in indexing and querying by-temporal data as well. The indexes we've got at the moment are very much suited towards this one-point-in-time query. We've done quite a lot of work in optimizing them, so to answer those queries.
00:48:37
Speaker
Yeah, I think we probably need a bit more on the way of indices in order to answer that. And I guess the kind of things you might be looking for there, like time range queries, which we do for one entity. So you can say, give me the value of this entity over time. But that's obviously not as powerful as being able to do a full day slot query over different times.
00:49:03
Speaker
Yeah. But, but if you, if you get an entity and it has a reference to reference, quote unquote, reference to, or an edge to, to some other entity, um, when, when you get the, when, when you query it at a point in time, um, that might not exist, right? The other one, the, the, the one it is pointing to. So you, you, you can't actually traverse in that way or is it possible? So if we don't provide any,
00:49:33
Speaker
integrity constraints about sort of foreign key constraints or anything like that. If you've got the data in a document, it happens to match, we'll join it for you. But we don't, it's merely a case of you throw a document in, throw whatever documents you've got in and then query the results. Just a quickie, don't you also have this ability to respond to index building?
00:50:04
Speaker
So you can have clients that build the indexes and they can watch the logs. It's not quite like the constantly evolving materialized view, but there is some sort of event system inside of Crux, isn't there? Yes. A lot of the events are largely internal, although we do expose one, which is the transaction event. So you could listen to all the transactions that are coming through the system.
00:50:34
Speaker
personal, put up your own indices or whatever, whatever may be, um, to suit those kinds of queries. I've not seen anyone use it like that as yet. Um, but I'm certainly interested in it. No, but I think it's interesting to, you know, when like, if you're doing like event streaming, then you do want to do that. You know, you want to be able to listen to all the events in Kafka. Yeah.
00:51:00
Speaker
In jokes, crux, Kafka, and then we have a use case, for example, at the moment in our organization where we want to listen to database changes. And then essentially for certain types of changes, we want to be able to send them out to something like AppSync or Firebase to kind of like synchronize with a whole bunch of mobile clients.
00:51:29
Speaker
which is not exactly the same kind of thing that Vijay is talking about, but this idea of event streaming rather than query streaming is still supported by your tech, I think. At least it better be. We'll talk afterwards if it isn't.
00:51:54
Speaker
Otherwise, you're going to open an issue for that one on GitHub. Fix this thing for me because I thought it is there already. Make it happen.
00:52:03
Speaker
I've sold it already. Exactly. James, you're talking about it's like database unbundled, right? So there is the storage engine, there is obviously query engine, and there is probably the transaction manager, something that is supposed to write all this stuff. So where do you store all this data? Where does it end up, actually?
00:52:25
Speaker
Yeah. So we split it into three main parts. So we call it the, let's say we don't have it into an implementation details or crux here. We split into three main parts. So the transaction log, what we call the transaction log, and the document store, which are our two golden stores. So they're the ones that really need to get backed up and kept.
00:52:48
Speaker
And then finally, on each node, we keep a set of query indices. And those in theory are throwaway. That's not in theory. In practice as well, they're throwaway. So they can be rebuilt from the transaction log and from the document store. So those query indices are really stored in the format that the query engine needs. So they're very much of the documents broken down, shredded into the indices that then mean that we can run the queries faster.
00:53:18
Speaker
The reason for the split between the transaction log and the document store is mainly for eviction. So we allow to evict documents irretrievably from CRUX. It's one of the things that particularly came in after the GDPR regs a few years ago. California's equivalent as well. That's the one.
00:53:42
Speaker
That's the one. So what we do is we put the events themselves, what we call the transaction events going to the transaction log, but they don't contain any data. They don't contain any data of the document itself. So that can remain completely immutable. And even if an eviction comes through, it's just another entry on that transaction log. And then it's the document store, which is essentially, it's pretty much sort of a content addressable store.
00:54:10
Speaker
So we'll hash each of the documents as they come in and then store them in there.
00:54:16
Speaker
Okay. But if I have an eviction, let's say T1 and then imagine that T3 is in the future, like T1, T2, T3, et cetera. So at T1, I store a document. And then at T3, I evict the document. I delete it. So the transaction log will still have capital T1, capital T2, capital T3 for transactions at time T1, T2, T3. But the document
00:54:41
Speaker
will be completely removed. So if I ask for the document at T1, I'll get a quote unquote null, like nothing. Yeah, it seems that we call it. Yeah. Oh, okay. So you got that. Okay. Okay. Yeah. Yeah. Yeah. So it's, um, so where is, by the way, so it's okay. But, but there are remains of stuff in under tombstone. So with that stuff, yes.
00:55:09
Speaker
But where is Kafka coming into the picture here? How do you use Kafka in the design?
Crux's Storage Options
00:55:15
Speaker
Kafka is one of the options that we have for the transaction log. In the transaction log, it can be Kafka, it can be JDBC. We're also looking into certain other transaction logs as well. Anything that can provide us with a sequential list of transactions, we can use as a transaction log.
00:55:37
Speaker
Okay. And for the document storage, is it your own format storing to the disk? So we actually, we store the documents in NIPI format at the moment. Oh, okay. As I just, yeah, like a compression, free of serialization. But that they, they can go in a JDBC table as well. We can put those in S3, you can use S3. Yeah, yeah. And we've even, we've actually even had a community contribution recently. So thanks, thanks, Henrik for this one.
00:56:05
Speaker
for Crux's, sorry, Azure's blob storage. So yeah, it's starting to expand out there as well. Nice. So essentially you can, well, as you said, it's completely unbundled, so to speak. So the storage engine can be anything, either S3 or Azure blob storage. And the transaction lock can be Kafka or something else, or a database, or a SQL database. Okay, that's interesting.
00:56:34
Speaker
The other thing that I thought was, sorry, I was going to say the thing that got me sort of, I mean, I'm interested in Crooks because it's like a, I mean, I know it's not an open source version of the Atomic because that would be crazy, but it's kind of like, it's inspired by the Atomic, isn't it? So in the sense that, you know, you kind of like,
00:56:57
Speaker
It's a database enclosure, so you're going to have some kind of relationship there in terms of inspirations. I was going to say, one of the things that was really nice recently that you guys did, that I missed was this whole idea of
00:57:16
Speaker
this notion of putting transaction data that you could speculate about rather than just kind of like data that was in the past or this by temporality in the past. You actually had a third time in some respects being the in-memory data that you have making it sort of a what-if type of database.
00:57:36
Speaker
Yes, yes, speculative transactions. I think it's quite similar to your, I think the atomic calls it with as well for that. But yeah, I think, I mean, just to talk briefly about the atomic, I think we'd be foolish not to look at it. And especially given so many people with enclosure are so familiar with how it works to, yeah, I think we'd be foolish not to sort of make the,
00:58:05
Speaker
make it seem familiar, if you like. But at the same time, we've obviously sort of the open source side, the unbundled and the... Oh, I can tell you, it's obviously very different, yeah. Yeah. Yeah, yeah. So what is this speculation transaction stuff?
00:58:24
Speaker
So that allows you to put transactions within the database without persisting them. So it allows you to do things like what if queries. So what if our sales data over the next couple of months looks like this? What does that then look like for our, I don't know, I'm making up a use case. What does that look like for our annual numbers or that kind of thing?
00:58:46
Speaker
But the idea is that you then get your own personal copy of all of the data. I guess you can kind of think of it as forking at that point. It forks the database at that point. Basically, you can do all the transactions in your own memory and never commit them to that. It's good for testing, isn't it, as well? You can sort of say, okay, this is the real database. It could be zero. And now I'm going to put all these transactions in memory
00:59:14
Speaker
and then I'm going to just throw them all away. But it feels, it's all committed, so it's all kind of visible in terms of its own, in terms of all the queries that are made on different threads inside of this own memory space. It all works. But then it's a node specific then, right? It's not going to flow across the node then. Yeah, yeah, that makes sense. So in the normal SQL world, then I will create a temp table and join them and then figure out how it's going to be, I guess. It's probably closer to a transaction you're not going to commit.
00:59:44
Speaker
I guess and see. Yeah, yeah, yeah, yeah. Yeah, you can say commit off and then, and then try it and then don't, don't commit this stuff. So you can set, but that is session level thing. I think in, in JDBC normal. So you're really thinking in relational database worlds, these days VJ. I mean, you know,
01:00:04
Speaker
No, I mean, I always think in relational database, but the main thing is that because there are so many people who know relational database so much, SQL so much, it's kind of stupid not to tap or give tools to them, you know? Like, like data log is probably known before people, I guess, you know, pretty much, you know, relatively speaking. And the amount of people who know SQL is massive. And it's like reach, you know, it's like, you know, JavaScript reaches sort of thing.
01:00:34
Speaker
So SQL has that reach and it's not going to go away. And it's the tooling as well, right? Yeah, exactly, tooling everything around it. So that's why you see SQL being applied to everything like streaming SQL, KSQL, and you know, Spark has one, like Spark supposed to be like, you know, big data querying engine or a processing engine.
01:00:53
Speaker
And most of the projects that I've seen so far is just your Spark SQL. That's it. You know, why bother? So they spent so much time in, in improving SQL performance rather than working on, you know, inner workings of SQL query planner and all that stuff, because that makes Spark accessible to everybody immediately. Not people who know, you know, Scala or, you know, RDDs and data frames and shit and whatnot.
01:01:19
Speaker
So I think it makes sense that you provide a SQL interface. It makes it super easy to introduce to people who don't need to learn a new language again. I'm interested in this about whether or not it invites the question about why bother.
01:01:44
Speaker
Why bother not just using the SQL Server or Oracle or like Aurora DB or whatever. If you're into SQL, I take your point that we're just having a little bit of a late moment here.
01:02:00
Speaker
Yeah, I like the idea of putting SQL on top of things. But I'm just playing devil's advocate a little bit. Once you start doing that, you're kind of inviting that question, which is, well, I might as well just use a proper SQL database. Yeah, that's true. But the thing is the underlying data and then the type of the usage of the data has changed a lot, right? Like, for example, the idea of what-if scenario. I still want to do that, but I don't want to learn a new language for this shit.
01:02:30
Speaker
Like, you know, I want to do this by using SQL. And then the data is coming every second for me. So that is streaming stuff invented. But can I use the same interface to tap into that thing? I think that would be my opinion. I don't know. What do you think, James? I mean... No, I'd agree with that. I mean, I think we obviously are sort of presenting quite a difference in terms of the data that you put in. We're just saying not supporting sort of insets or updates as yet in SQL and
01:03:00
Speaker
But yeah, I mean, it's familiarity, right?
01:03:06
Speaker
if it's one fewer thing I have to learn about a database to get started. I mean, we may also be talking about some different groups of people as well. So you may have certain people within a company, within a team that are way more happy to get their hands dirty with things like DataLog or GraphQL or whatever have you. But then you may also get people within the company that's
01:03:32
Speaker
Let's get data scientists, data analysts, that kind of thing. You're so familiar with that, but just want to stick with it, and that's fine. Yeah, sure. I think the idea of using SQL as putting emphasis on the queue makes a lot of sense. Yeah, exactly. I think, Gray, I understand your point because I think that is more developer point of view. If I'm a developer and I'm building an application,
01:03:58
Speaker
because I can learn SQL or I can write data log, that's fine by me. I always like to say to the developer, Viewpoint VJ against you, one of the management elites. Obviously, my opinion is highly valued because...
01:04:14
Speaker
because I'm in the management now. I think that the other side of it is that exposing the data to people directly, you know, rather than having this is, okay, I'm going to build a credit app on top of the ship. And then that makes a, that is a kind of a different, uh, in, in that case, I completely agree with you that there is no point. You're going to stick to SQL, whatever that is, that is implementation detail. But once we start exposing the data to, to, to quote unquote, you know,
01:04:39
Speaker
non-developers, then they have less amount of interest in learning data. No, it increases the reach of Crooks or any other sort of like esoteric database. Yeah, totally. And also it can plug into everything, right? I can now plug into Tableau, I can plug into whatever the tooling, speak SQL, you're done.
01:04:59
Speaker
I'll Power BI or whatever. That makes it awesome, actually. Anyway, so it's coming back to Crux again. So it's not a normal client server sort of JDBC thing, right? Because if it is similar to Datomic, that means that every application is joining. How does that work then, if I'm building an application using Crux?
01:05:27
Speaker
With Crux, in a similar kind of way, we actually bring down all of the data on each node. The node keeps its copy of the query indices. While they do share the JDBC service for the Golden Store side of things, the node then is responsible for pulling those down and indexing those in a way that they can then service queries. The node can be embedded.
01:05:55
Speaker
So the note can be embedded within an application. It's just a JVM application that way and in the same way. I mean, we do have like an HTTP server that if you do want to deploy crux separately from the application, scale it differently, that kind of thing. Can use the HTTP server there. But yeah, fundamentally for a query, it goes to its own local indices in order to satisfy those queries.
01:06:23
Speaker
So for that we've got, we mainly use KB stores, like fast KB stores for that. So Facebook, RocksDB, or LMB are the main two. And one of the reasons that we're currently have it local is that the query algorithm we use is quite chatty, it's quite back and forth.
01:06:49
Speaker
to service these graph queries. So looking at adding network hops to each of those conversations does add a fair bit. I mean, that said, we do have a reddish spike in the pipeline as well at the moment. So I'm interested to see how that goes. I'm interested to see what the performance numbers look like on that one. Nice.
Crux's Transaction Coordination
01:07:10
Speaker
And how does the writing work then? Because how is that coordinated?
01:07:14
Speaker
So the writing is coordinated through the transaction log, so your Kafka. And we do that via a single partition topic on Kafka. So each of the nodes, when it gets a transaction submitted, submits it to Kafka. And then essentially, if you want to read that right, you then wait for that transaction to come all the way around and get indexed into your local products node.
01:07:37
Speaker
Okay. But if you single, I mean, I can understand, I'm guessing you use single partition because you want to have the ordering guarantee. But isn't that then against idiomatic Kafka, because you really want to partition it so you can scale it too well, right? And the Kafka itself, because Kafka scales by partitioning. Yes. Yeah. I mean, even with single partition, Kafka still gets you a fair way. So that's going to serve for a decent amount of use, decent amount of use cases.
01:08:08
Speaker
But we'd have to look, in Crux, we've certainly considered how we could use multiple partitions within Kafka. And I think one of the ways would be to look at introducing a little bit more schema on top of what we currently have. And relying then on the end user to say, right, I can guarantee that these two transactions don't overlap. So here's you can put them on different, we'll hash them, we'll give you like an event hash or whatever it may be.
01:08:35
Speaker
which then goes onto different partitions. Yeah, so you can have hash partitions because I think that scaling up Kafka without partitions is going to be painful, right? And also you need to have... Yeah, because if every node... Sorry. So every node connects to the partition to read as well. The read happens via Kafka as well. So we've got an indexing process which happens on your local crux node.
01:09:05
Speaker
And that's the thing that reads down from Catherine and indexes it into our query indices. And then the queries themselves only look at those local query indices in the main, only look at those local query indices and the document store.
01:09:21
Speaker
Okay. And how is the performance, by the way? I mean, what kind of benchmarks or something, because I know Datomic famously doesn't allow people to benchmark stuff. Yeah, so obviously I can't give you a comparison. Yeah, that's completely understood. But we did have some public benchmarks, or at least a public benchmark suite, which we tested a few different datasets.
01:09:48
Speaker
So we do have those running every night. And what I will say is that it holds up. It holds up. We run a what-if, which is an RDF benchmark, which we then translate into crux documents and data log queries. And we've also started recently benchmarking TPCH, which is a standard SQL benchmark, as well as a few other smaller benchmarks.
01:10:18
Speaker
Yeah, we do we do have those running on a regular basis. Okay. So is your goal there to make sure that you don't like go off the charts with the next release on one of these benchmarks? Yeah, it's upwards. It's okay. Yeah, I was about to say on the optimistic side of that it's improving so that we can in the release notes we can say yeah, it's 40% faster.
01:10:44
Speaker
this time around, and we've got some numbers to back that up. So that's really useful. But yeah, regressions as well. Yeah, yeah, yeah. So Crux is completely written in Clojure, right? That's what I'm guessing. Yes, yeah. I mean, we've got some Java on the more sort of performance-heavy areas. And once you get down into the sort of depths of the query engine, it gets quite gnarly.
01:11:13
Speaker
Particularly for memory management and that kind of thing. There are a few transients and things like that around. Yes, yeah, fair few of those. Fair few of those.
Motivations Behind Bridge Project
01:11:24
Speaker
But yeah, especially once we get down into the query engine, we do have to do quite a lot of memory management. And we've got quite a lot of, the same way Closure does generally, we've got quite a lot of short-lived objects. So we need to try and cut down on those as much as we can.
01:11:40
Speaker
Okay, so is this the project that pushed you to build Bridge? No, I think I mean, Bridge Bridge, I've been working on and off for must be a few years now. No, it's been a hobby project for a long time. But yeah, so actually, before I before I joined Crux, I did
01:12:06
Speaker
I did start to investigate what's it like to implement something like the Atomic, and then drugs came along and was open source. I was like, okay, now I'm going to have a look. I think there were one or two blogs written about the implementation of the Atomic, which had been around so a while back.
01:12:25
Speaker
Speculated implementations at the time. Yeah. Yeah. And also data script, for example. Yeah. Yeah. Yeah. Yeah. Hmm. Well, it's a very, I think, to me, it seems like a very fertile space, you know, because people do want, I mean, people do, I think Rich was right, actually, that, you know, this whole idea of like having all this nice stuff, enclosure and Eden and all these things, and then having the sort of,
01:12:51
Speaker
talk to some messy back end that didn't know what was going on and wasn't immutable, then it's kind of tedious at the best and kind of disappointing at the worst. Maybe it's not the worst to be disappointed, but definitely it's like, why can't we have nice things?
01:13:13
Speaker
whereas now it seems like there's a conicopia of nice things, which is really good, you know, because this space is kind of like blossoming a little bit, I feel, you know? Yes, yeah, it certainly feels that way, doesn't it?
01:13:25
Speaker
Yeah. And I think what you guys have done as well is really interesting because you've made it on bundles, you've made it open. And like you say, like the sequel stuff, you didn't, I mean, I know you stitched it together in the end, but someone else puts those PRs together. And I'm pretty sure there are other community contributions around these things that you've benefited from, you know?
01:13:47
Speaker
What's the story on that actually? Are you getting some, is there a fair number of community pickup and community involvement in the code or in the add-ons? Yeah, community is certainly growing on that front. We've seen a little bit of a spike recently, which is quite nice. The number of questions are getting through and inevitably the number of issues that are getting raised. That's going to happen as well.
01:14:17
Speaker
So we're getting quite a little bit of that. We've not seen an awful lot into the core, which I kind of understand. There's quite a lot to get sort of up to speed on there, but especially around the edges. So we're getting a few modules coming in now. It's particularly sort of data storage. I like Crux, but I'd really like it to store its data here.
01:14:41
Speaker
Yeah. I mean, the protocols on the edge are quite simple, really, for what they do. I mean, document store, for example, the protocol for that one is submit docs and fetch docs. So it's plenty accessible to be able to do that. Transaction log has to be a little bit more complicated. But yeah, it's definitely feasible.
01:15:09
Speaker
And wasn't Malcolm talking about something they do with timelines recently as well? Do they do with schemas? Yes. So he's talking about the separation of time code data. I've got those in the wrong order. Time, data, form, and code. Right. So apologies, Malcolm, if he's listening to this. I'm probably going to do a terrible explanation of this.
01:15:37
Speaker
But the separation doesn't depend on this, James. We talk in the closure industry constantly about de-complexing, a word that I've never heard before. I'm not even sure it existed before closure.
01:15:59
Speaker
But we talk constantly about de-complexing things. And the theory there is that if we can split those four apart, that will lead to being able to work with simpler systems for longer. So keep systems simpler for longer. And Crocs very much, I think, is the embodiment of that, the splitting of those four.
01:16:26
Speaker
So what are the four again? Time, code? Time, data, form, and code. OK, what is form? So form is structure, essentially. We've got to think it jutched for four-letter names. So it had to be a four-letter name. Yeah. Structure law. I think the fact that it's just a core, exactly.
01:16:51
Speaker
It makes it particularly appealing to us. The separation of form and code and form and data is really to say that we don't know form and structure up front. There's quite a lot of solutions out there which rely on you knowing how your data is structured up front. You think of it like a relational table or whatever, sure you can make migrations.
01:17:17
Speaker
Fundamentally, we don't know the structure until we've tried to use it. And until we've got some new styles of query coming, I never realized that I wanted to use this data in this way. And so if we can decouple form from the underlying data, we can then make systems that are more flexible. Well, I think my question, and I think that we're going back to the original point, was about
01:17:42
Speaker
having like this notion of schema itself being on the timeline, you know, so you can, so, you know, um, I think one of the, like, it's interesting. I mean, I work in a kind of like in this blockchain kind of like domain. And for us, like things like the contract evolution or the sort of evolution of the data form, those kinds of things are, you know, they're definitely interesting. And obviously this whole thing with rich, with, um, his like, um,
01:18:10
Speaker
is dependencies mapping and all these kinds of things that things change over time and they should grow. And, you know, if they grow, it's okay. Yeah. If they grow, it's okay. But if it's, if they, if they don't grow, if they change what, what, what they return or what they expect, then it's a break. So.
01:18:28
Speaker
So, you know, I think with schemas, you want to also have that kind of, um, you know, schemas essentially, we define a contract on there. So you want to see where that contract is, like how that contract evolves over time, essentially.
Schema Evolution in Crux
01:18:44
Speaker
But isn't this a similar to maybe I'm not understanding it properly. Um, isn't it similar to something like Avro because you have this, this data and then schema and then that can evolve over the period. Uh,
01:18:58
Speaker
Yeah, but I think it's not a first-class citizen in Avro. I think the problem with Avro is that Avro is a data structure. It's nice in the sense that these schemas travel with the data. So the version that this data is of is version two or version three, so the interpreter on the other end can understand that.
01:19:17
Speaker
But Avro itself doesn't lend itself to a global view of what that schema looks like. Maybe as you can, I'm pretty sure the Confluent guys can build one for you. But it doesn't come out of the box with Avro. Whereas I think the idea with Crooks is that you can see the evolution of the schema on your database in a separate timeline to the evolution of the data. But aren't they codependent? I mean, the schema cannot live
01:19:47
Speaker
Well, schema is meaningless without the data, right? Well, obviously not. Come on, mate. Keep up. Yeah, so the schemers we have it very much lives on top of the data. Data in itself is in as raw a format as you can possibly get. Yeah. And then applying schema to that, applying structure to that when you need it.
01:20:12
Speaker
So actually with them, you can put data into a database which doesn't conform to any existing schema. It may conform to a future schema or it may not. So I think that separation is kind of like confusing to the layman like you at the beginning. It's obviously not for me because I'm at SQL.
01:20:43
Speaker
I think we've really fully incorporated form into into crux as yet. I don't think we've really got that. I'm not sure I'm sure there's some there's some iteration to come on that one. Yeah, yeah. I mean, it takes it takes one follows function. No, I think that's the way to think about it. Okay, there is function now. Okay. No, but that's just that's just in the code. That's a cultural reference feature. Yeah, yeah. Form follows function.
01:21:10
Speaker
Because it already took significant amount of brain power that I've used completely now to understand the change from relational database to streaming shit for me. I still don't understand streaming. So that's a good thing. So I can just live in my cozy SQL world. I always think of it as the equivalent of having a drink of water versus pissing. I'm intrigued to see how you get out of this one. So what is streaming? Is pissing?
01:21:40
Speaker
Well, ingestion is the drinking and streaming is the pissing. Yeah. But you do stream water into your mouth. You don't just, you know. That's ingestion. Come on. No, but you do stream that. I mean, there is a pipe there. I guess you could stream the piss. I don't know what you guys do in Belgium, man. I have no idea.
01:22:03
Speaker
You'll forgive me for not using this analogy in that documentation. But anyway, I mean, the thing is that for me, it feels like the complexity is raising by decoupling things too much. At some point, there are too many moving parts. And when we introduce these abstract concepts that are being decoupled,
The Role of Schema in Databases
01:22:30
Speaker
That feels even weirder. Okay, I have data, if I understand you correctly Jim, so I have the data that is in the data storage, but I don't have the schema for it yet. So because my interaction with the data is via schema, at least that's what I would think. So for me, that kind of contract breaks there because I cannot retrieve the data because I don't know the schema.
01:22:52
Speaker
But it's a schema of this database. It's a schema of this database. I mean, you have to... Yeah, that's bullshit. What the fuck is schema of this database? That doesn't make any sense. Well, it is because in a sense that you can... MongoDB, mate, you know, remember that good old days where you could... It does have schema. That is okay. Just because it's no SQL doesn't mean it's not... No, but the schema is only the document that you insert. So, you know, in that sense, JSON acts as the schema. Yeah, exactly. So there is a schema. Yeah.
01:23:22
Speaker
Yeah, but that's, but that's a bullshit thing because like Jason and Eden don't change. Whereas your expectations of the data, day to day, they do change, you know? So, you know, you want fields of, you know, you want a certain number of fields in this document. You want a certain, you know, a certain type of these fields in a document on day one, where you want something
SQL Layering and Complexity
01:23:44
Speaker
else on day two. That's the kind of classic scheme, isn't it?
01:23:46
Speaker
Yeah. Yeah. Yeah. So that's, that's where, I mean, you know, they have, they have like Eden as a schema, but that's true everywhere. That's a base, but then they want, if you want to like enforce a spec type schema on top of that, or a data schema, which is, I think a data log is always schema on read, isn't it? So, you know, so that's, that's why if you, if we come back to the, um, the SQL that we talked about earlier, um, where you define, you define the relation separately.
01:24:11
Speaker
So you've already got all of your documents in Crux. We then define the schema as a query on top of that. So you're going to have access to that relation when you write your SQL. So your SQL then is based on top of that structure.
01:24:27
Speaker
Yeah. So it's more like moving the constraints. But that was not guaranteed on insert. That's the important point. At least at that point in time.
Managing Tech Complexity
01:24:38
Speaker
So the idea about this idea of evil eating schema, according to what I understand from Malcolm's talk anyway, is that you can kind of say, okay, well, what can I expect from these queries at a particular point in time?
01:24:53
Speaker
Okay. It's like looking into the future. Well, actually the past isn't it? Either. No, no, it's like, either. Okay. But mostly the past because, because schema one existed then schema two and schema three and schema four. So you want to see the difference and you know, track that timeline. Okay.
01:25:12
Speaker
Well, I mean, that went right through above my head. So obviously, I guess, you know, I think my schema is not that separated, I guess. So anyway, so is my schema your next database? Well, I mean, I'm planning to I'm planning to retire in 2025. So, you know, I don't care after that. What happens in computing? You know, it's like, fuck it. We have we have enough complexity already. My brain cannot take it anymore.
01:25:41
Speaker
So I'm giving it up like, okay, fuck it. I'm done. Because the world is moving towards all sorts of weird directions that I can't keep in my mind, or at least my brain cannot tolerate. Yeah. That is one of the things about tech, isn't it?
01:25:57
Speaker
Yeah, exactly. I mean, every day there is something new that we need to catch up.
Clojure's Stability
01:26:02
Speaker
But at least Closure has been one of those reasonably rock solid foundation stable thing for long enough for me to understand. So it took almost 10 years. But yeah, I think I finally have some idea of what Closure is, at least. Well, one of these days, if you could explain Spec 2 to me in the next podcast, I'll be very interested.
Deployment Challenges in Crux
01:26:25
Speaker
Why are you asking me? You're the schema expert. So you should explain that shit. I just have one last question about Crux, by the way, because it has so many pluggable components, right? I mean, because these things can be pluggable. So isn't the deployment, you know, gets a bit scary?
01:26:46
Speaker
when you're configuring and then maintaining and all this stuff. Yeah, so I mean, I guess that's one of the downsides, if you like, of the unbundling is that it can seem as if there's so many choices to make. Yeah. And so I think one of the things we're working on is both sort of making the boundaries clearer. And one of the things we've been working on a lot in the last six months or so is making those boundaries clearer, making sure that the sort of the components that we have are clearly defined, the responsibilities are clearly defined.
01:27:13
Speaker
Yeah, I think a lot of it as well as about sort of writing up guidance about what to consider when making those decisions. Yeah, so yeah, yeah, very much. I mean, it does. I guess you could say it's a similar way to closure in a way.
01:27:26
Speaker
and Closure's kind of got a little bit of a reputation for a lot of wiring up. We got a question on Crux the other day about what would you now recommend for how to make a web app in Closure?
Introduction of Crux Builder
01:27:40
Speaker
I think if you got 10 different Closure devs on that one, you get 10 different answers. So I think there is a little bit, yeah, you're right, there is a sort of a paralysis of choice there if you like.
01:27:57
Speaker
that we need to help people through. And that said, we are working quite a lot on some deployment and operations as well. So things like we released what we call Crux Builder a couple of months back. And that's the thing for you to build your own Crux artifacts. So if you do want sort of catheteron one S3 on the other and then rocks as indices, you can then go and make yourself a Docker image for that or an Uber jar for that, whatever you need.
01:28:26
Speaker
So we are writing up a lot more about that.
01:28:29
Speaker
Yeah, yeah. Because the tooling can help you a lot, right? Especially if you want to put this into the deployment, whatever that people use, like Ansible driven things or whatever, you know, it's much easier to have some sort of a tooling available.
Crux's Flexibility in Queuing Libraries
01:28:43
Speaker
But the thing is, I mean, you're going to have people with different constraints, right? I mean, you're going to go into some business and go, right, this is, we're based on Kafka and that's an instant no, or we can only provision sort of JDBC databases easily.
01:28:56
Speaker
or we're working with a certain type of blockchain or whatever it may be. You're gonna get these constraints. So yeah, we've chosen best to be open there. Yeah. But are you like Kafka is more kind of necessary thing for Crux right now, or is it something like, are you looking into implementing MQP related stuff so you can use WMQ or something else?
MQP's Role in Crux Operations
01:29:24
Speaker
And so we've also got, so we can also use JDBC tables for that transaction log side of things. So you can use JDBC across the board for the transaction log and doc store. I'd say we could add support for other queuing libraries as well, other queuing services as well.
01:29:43
Speaker
Or we could get some community contributions. Yeah, exactly. Yeah. Because MQP is more or less like a protocol level thing, right? So once if you enable that one, then probably you can talk to RabbitMQ and maybe zero as well, I guess. Does MQP guarantee ordering?
01:30:00
Speaker
That's the only thing that you need, isn't it, James? That's the big thing. Yeah, yeah, yeah. I think it does, because RabbitMQ... That must be confused by that. Because I only know RabbitMQ side of it, so maybe. Obviously, people who are listening to this one are way smarter than at least two of us.
Community Contributions to Crux
01:30:19
Speaker
So please, please let us know if you can build stuff with the MQP and then that could be a nice community contribution.
01:30:29
Speaker
then suddenly it opens up more possibilities for Crux. Nice. So just one last question. I said that last question, but it's not about Crux anymore. So I still have one more question though. So I've been writing Closure for eight years, eight, nine years almost.
Development Tools Preferences
01:30:47
Speaker
So is it Emacs or some other shit?
01:30:50
Speaker
I've got to be careful with this or not. For me, it's Space Max. Yeah, there you go. See, I think you're a quality person, so I can imagine you're using Space Max. There you go. I mean, even smarter people use Space Max, obviously. Why not? Why not? You don't want to talk into this guy. You find stuff out on the shore that you never didn't know.
01:31:19
Speaker
So this is like one of those memes, you know, like James is not my friend anymore. At the beginning of the show, he was our friend, but not anymore. That's nice. So you build entire crux using Space Max. Oh, I can't say I've built an entire sea of crux, that's for sure.
01:31:39
Speaker
But you're developing using Space Max. I mean, you know how marketing works, right? I mean, there is a small star somewhere and then there is condition supply. Yeah, right down at the bottom there. Exactly. It may or may not be true. Exactly.
01:31:57
Speaker
Not actual not actual gameplay. Yeah, yeah, exactly. Sequences may be shortened. transactions may be lost.
01:32:13
Speaker
So now you know that, you know, um, entire crux is built using space max. So yeah, that's, that's on that, on that bombshell.
Open Source and Commercial Support
01:32:24
Speaker
So, um, on that death planet. Yes. On the death star of, uh, space computing. Yes. Um, so yeah, I think that, I think we covered a lot of ground. Is there any other thing that we, that we need to mention?
01:32:40
Speaker
Obviously, Crux is open source so it is available and people can download, install, use it for whatever the thing.
01:32:52
Speaker
If you are into code, then you can support by writing code, writing extensions, writing the add-ons, whatever the different things that Jim's discussed about. So please go and check out. And obviously, you guys offer commercial support, right? Yes. For Crux. Yeah. Yeah. So just write commercial support.
01:33:10
Speaker
Yeah, so there is a there is a solid closure company behind Crux. So that means, you know, you know, the ins and outs are also fit, obviously. And you've been doing closure for long enough. I think I think within Jext, it's like, how is it like 50 years of closure experience? Oh, more. Probably more. Yeah, yeah, it's got to be it's got to be more. Yeah, yeah. Let's say with a start at the bottom.
01:33:35
Speaker
may not be true. That should be this podcast that you were talking about doing some 2020 merch. We should say some like deaf and closure podcast star may or may not be true.
01:33:51
Speaker
No, definitely the best Closure Podcast. Oh, yeah, yeah, yeah, yeah, yeah. May or may not be true, you know. Music also has the best vegetarian Closure Podcast, you know. I think that's probably still true. Yeah, it's probably still true. But yeah, that's really good. Yeah. Nice. And James, I think it's been a
01:34:12
Speaker
It's been a pleasure to talk to you and then getting deeper into into crux. So I think people who are listening to it, you can go and check crux and github.com slash.
Crux Community Dynamics
01:34:23
Speaker
slash jux slash crux slash crux. Yeah. And, you know, there is crux channel in I think all the usual places where you suspect closure folks are available, like Closureians. Yeah, so we got a crux channel on Closureians. We've also now been invited on to the GitHub discussions beta. That's a good thing to try out as well. Okay. And our official support channel is Azulip channel, which is also
01:34:50
Speaker
available there. We can put links to these in. Yes, we do. I'll put the links to all the stuff. I think that's it from me. Any last words, right? Or last instance to your co-host? No, to be totally honest, I've used Crooks in the past few months and I find it to be a real pleasure.
01:35:19
Speaker
And I think James and his mates are doing a great job there. So I think this whole space is very fertile ground. And so I think it's just excellent that people like James, especially without being too patronizing, a young man like you, James. Young man. Oh, yes. Young man like you. Going up in the world.
Acknowledgment of Crux Team
01:35:45
Speaker
No, it's really nice to be delivering such great
01:35:49
Speaker
technology into, you know, into the sort of into the community. You know, and because the reason I mentioned in terms of last words is because I think, you know, Rich, I think, put his hand up at a conference in a year or two ago, maybe a few years ago, I don't know, saying, I know, who's like, who's used like closure, who's been to a closer conference before? It's all the old crusties, basically. And his idea was, you know, well, maybe it's not too bad if it's all old crusties.
01:36:14
Speaker
But actually, it's quite nice to see, you know, I know that Juxt is quite a young company, actually. I know that that Malcolm and John are like more like senior, a bit older, a bit more gray hair or whatever. But but obviously, they're, you know, they're they're young at heart. That's for sure. But but clearly the, you know, the company that you've got going is, you know, it's all kind of like recent graduates in their 20s and 30s. And you know, it's really nice to see
01:36:41
Speaker
people doing this groundbreaking work, young people in this community. And I think I'm really lucky to be a consumer of this stuff and to be talking with you guys and hopefully kind of like helping to promote it a little bit with this podcast with Vijay. But James, I mean, a real gent and it's been a real pleasure to talk to you and congratulations on the work you've been doing so far. And it's a great start to your career, I think.
Working with Clojure
01:37:09
Speaker
Thank you for having me on as well.
01:37:11
Speaker
Totally, yeah. It's been a pleasure, yeah. Awesome. So, 100 years of closure experience building crux in Space Max.
01:37:22
Speaker
That's the new tagline. There we go. Yeah, exactly. May or may not be true. So that's it from us for episode number 64.
Podcast Wrap-up and Engagement
01:37:35
Speaker
And wherever you are, stay safe. And I think we'll be back with another episode soon-ish again on our regular or irregular schedule that we are on. So thank you. Thanks, James. Cheers.
01:37:50
Speaker
Thank you for listening to this episode of DeafN and the awesome vegetarian music on the track is Melon Hamburger by Pizzeri and the show's audio is mixed by Wouter Dullert. I'm pretty sure I butchered his name. Um, maybe you should insert your own name here, Dullert.
01:38:07
Speaker
if you'd like to support us please do check out our patreon page and you can show your appreciation to all the hard work or the lack of hard work that we're doing and you can also catch up with either Ray with me for some unexplainable reason you won't interact with us then do check us out on slack closure in slack or closure verse or on zulep or just at us at deafened podcast on twitter
01:38:36
Speaker
Enjoy your day and see you in the next episode.
01:39:14
Speaker
That's why people stopped, you know, coming to our show. Like, this is a shitty show. I don't want to be associated with it. Dude, people didn't stop coming. Don't tell him now. Okay.