Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#28 - Max Penet aka @mpenet
38 Plays8 years ago
An entertaining episode with the Max Penet who has made his own keyboards as well as some popular Clojure libraries for noSQL databases.
[ We know the track is slightly crackly but we think the content is good so we hope that it's good enough for the non-audiopholes out there ]
Recommended
Transcript
Introduction and Holiday Humor
00:00:16
Speaker
Welcome to Death and Episode number 28. This is Vijay from the Netherlands. And it's Ray from Belgium. And it's Max Beneth from Switzerland, but in Sweden. Hello, Max. I thought we were going to introduce you, but anyway. Oh, OK. You just get there. You just popped in. That's awesome. That's pretty nice. Anyway, it's Christmas time, so everything is OK. Either Christmas or holidays.
00:00:45
Speaker
I didn't get it. We say anything that will upset Donald. Yeah. So happy holidays. So fuck Christmas. That is going to offend pretty much everybody now.
Advent of Code and Clojure Community
00:01:04
Speaker
But just before we started recording, Ray, you're talking about Advent of Code. That is holiday thing, right?
00:01:13
Speaker
Yeah. I mean, you know, we can dispense all the commercial bullshit, but yeah, we can do the Christmas of court. Yeah. The advent of court. That's good. Yeah. Okay. Yeah. So advent of court. Yeah. A few of the closure guys are doing it. And this guy, I think you know him. Is it Michael Bork? Yeah. Michael Borkant. I think you're, you're Anglicizing his name.
00:01:37
Speaker
borkdude at borkdude on Twitter. Yeah. That's his problem. Plenty of plenty of people doing this one. Yeah. Well, he's made a repository with about, you know, I don't know, 20, 30 people are doing it like, um, and there are various people. Um, yeah. And Bruce Hamman and friend of the show, Mike Fikes is doing it as well. Yes. And he's doing it in, uh, he's doing it in bootstrapped closure script. So I think there are people doing it in different kinds of forms and different ways, you know,
Max's Move to Sweden and Lifestyle Changes
00:02:07
Speaker
Awesome. But I was looking at it and I was thinking it was very interesting because if nothing else, you can learn some tidbits about closure as you go along and compare the styles of people. As one example, there was one interest, two examples, okay, two examples. One quick example that I know that Michael and Mike were talking about was the difference between enthrest and drop.
00:02:32
Speaker
it was on Twitters and apparently it said the drop is lazy and nthrest is not. Yeah, it's holding onto the header, apparently. Yeah. Another thing that I saw in Mekel's code actually was plus prime, you know, where you say reduce plus prime.
00:02:51
Speaker
I've seen it in the code before, but I'd sort of ignored it and I thought, what the hell is this? And then I finally thought, okay, screw it, I'll look it up. And it's this one where it goes back to pre-collision 1.3 behavior, where it promotes the numbers so that you can have arbitrary precision. Yes, to big decimals or big integers.
00:03:17
Speaker
But these are the things that you generally don't get to unless you need a specific use case or something. Yeah, but it's interesting as to why do these guys put this stuff in there. To me, it was kind of interesting to see the code that they're writing. I guess you can have a little discussion about that. I don't know if you've done anything like that yourself, Max.
00:03:42
Speaker
I've never heard about it, actually. I was wondering what it is.
Max's Programming Journey and Clojure Discovery
00:03:46
Speaker
Well, in one of the libraries I thought there is some unchecked mass and stuff, but I mean, it's just to iterate a bit faster over some data structure. In general, no, I don't use these kind of low-level things, to be honest. Yeah, I think anything that just a sort of distraction, isn't it, from the cord? You have to have a very good reason to use it.
00:04:11
Speaker
Yeah, usually I start simple and then if I really have to, I go down to these the gory functions that you never use. Don't check the stuff and I really sometimes if I really have to down to Java, but kind of down to Java. Yeah, it's like assembly at this point Java. This is the first time I'm hearing somebody saying down to Java. It's like using Java as an assembly language or something.
00:04:39
Speaker
Yeah, but I mean, that's not uncommon. I've heard that a couple of times in the past from other people doing closure. I can't remember who was saying that. But yeah, I mean, you get the best performance you could get on the GVM more or less using Java. And I mean, from closure, it's kind of rare that you have to go to that low level. I mean, that gentleman, for instance, has done that a couple of times for primitive math and
00:05:03
Speaker
a couple of things dealing with byte streams and so on. But I mean, yeah, the normal closure user, let's say, if we can say it like this, doesn't really have to go to solo level. So I think this is a nice time for you to introduce yourself, Max. Yes. And then talk about how did you get into closure.
00:05:25
Speaker
Yeah, so what should I start from? From when I started using closure or... Oh wait, so I'm Max. I live in Sweden. So I used to... Well, I'm from Geneva originally, so I moved here like three years ago.
00:05:40
Speaker
And I work for a company that's in Switzerland still, so I work remotely, mainly. It's not really a big upgrade, right? I mean, going from Switzerland to Sweden. For me, it's kind of a downgrade from many things. I mean, the weather is not as good, let's say, as the one in Geneva. Exactly. Not that the one in Geneva is tropical and nice, but here it's a bit worse.
00:06:09
Speaker
That's the light. Me, I'm not super happy about Sweden in general. Let's say I'm living here. The skiing is a bit easier there, isn't it?
Affinity.ai's Evolution and Clojure's Role
00:06:18
Speaker
The skiing, yeah, it's a bit flat. I used to bike a lot, for instance. I was big on going climb some mountain or some hill in the weekend with my bike, but now I can not really do that. The best you're going to get is the wind hitting your face really hard on a flat section.
00:06:36
Speaker
No, other than that, there's a place to be for sure. It's nice for many things. I cannot complain too much. Why did you move there then? Come on. Because the work of my wife, she works with particular accelerators and there are not many around. There are building one around here, so we're here for as long as it lasts.
00:07:01
Speaker
Is it the same big one that goes under the entire Europe and then you're going to the other part of that one or is it actually a different particle accelerator? I don't think there's one that big. There's one close to Geneva, I mean, well, in Geneva, which is CERN, so it goes under parts of France, parts of Switzerland, but I think it's like 30 kilometers in diameter if I'm not wrong or a bit less. But that one is a linear one, so it's straight and so it's a bit different.
00:07:28
Speaker
But they are building it from scratch so far. It's super interesting to be here. I mean, she does research on this stuff, so she gets to put her on the process from scratch. So we are here for that. I think we should get her on the podcast at some point. Yes.
Ergonomic Setup: Emacs and Mechanical Keyboards
00:07:45
Speaker
I bet she programs in Python though or something. No, she uses awful stuff. Mostly old code from the Fortran stuff. Oh, man. Best programming languages that
00:07:57
Speaker
came out of an awful stuff. I did use some closure in the past. I helped with Interop through GNI with closure. I mean, yeah. But... That's true love, that is. Yeah, exactly. So that's the best I can do to help her. Then it goes to crazy math. I don't even understand our physics stuff. That's like, yeah, way over my head.
00:08:25
Speaker
Nice. So that's why you ended up in Sweden. But before getting into the closure stuff, what were you busy with? What am I busy with?
00:08:35
Speaker
Yeah, I mean, which languages that you use? At work, we use Closure. I mean, mostly Closure. We used a bit of Erlang. We did use a bit of Erlang in the past. Well, it's still in use, actually, but it's more like running and hasn't been touched in a long time and it's doing just fine by itself now. But it's mostly Closure. What about in the past, though, Max, when you first started programming?
00:09:01
Speaker
Oh, well, it's been a long time, so I've used a bit of everything to be honest. Well, I started very early with, well, Amiga stuff, assembly and all this stuff, and then through university. Actually, the first language I was introduced to at university was Java.
00:09:18
Speaker
Java and Emacs together are well done. It was in the 1990s, so Java was not so well developed, let's say. It was a bit of a pain to use. There was no real idea like there is today. And then to work a bit of everything, it went for a couple of variants of VB, a lot of Python.
00:09:44
Speaker
And then there were the startup days where I was like hopping from one to the other and it was mostly Python before Clojure, I would say. So what made you come to Clojure, Max? What made you fall in love with the parentheses?
00:09:59
Speaker
I was a bit playing with a couple of lisps. One of them was called, I think it was called Wasp. And it was something related on security. It was basically allowing you to build distributed system. Well, we can call
Data Processing at Affinity.ai with Clojure
00:10:16
Speaker
that like this, but it was a bit more around security in general. And it was like a very experimental thing. And I got hooked.
00:10:24
Speaker
I just realized that the parenthesis thing actually, yeah, the structure of the language was really nice. And from there I tried Common Lisp and then Closure came out and I was, I think it was on Reddit, I discovered about it and then tried it in the early days, which was, I don't know, I think it was around version 1.1, something like that. So yeah, 2010 probably.
00:10:49
Speaker
And yeah, so I was using it basically on the evenings after work. And the startup I was working for at the time, basically closed down, was not doing so well, didn't get any, another round of investment. And I was working on the little toy project on the side enclosure and I showed it to one of the, well, the owners of the company back then. And we use that as a buoy, I would say.
00:11:16
Speaker
It was like the way to, I mean, yeah, we sold that to clients and from there we created another company and that company became the one I work for now, let's say after five or six years. Then we merged with another company and so on, but that was how it started. Okay. Well, I hope they give you some shares, Max.
00:11:39
Speaker
Yeah, well, we are doing okay, but we're not Microsoft big or Google big.
00:11:47
Speaker
But which company is this? It was called Shoreware in the past, but now it's called Affinity.ai. But it's the company that has a couple of projects, some of which are a bit more known than others. One was we merged with a company that used to build a paper.li, which was maybe you know about this service, but it can basically set up an environment where you get your own newspaper published every day.
00:12:17
Speaker
It's a bit like what Prismatic was back in the day. Kind of a soft hire. It's similar in some way, but maybe less focused on one thing like Prismatic was. It's a bit more of a disco. And it's just one project because our side of the company was working on other things.
00:12:38
Speaker
We were working on some knowledge center kind of project where you ingest tons of data from whatever source you get and you give the tools and the UI to the people in charge to basically drill down these data and figure out what's what.
00:12:55
Speaker
So some people use that for other things, some people use that for other things like conferences to show fancy dashboards with tons of graphs and things moving in real time, these kind of things. So it was more about the icon, this project specifically.
00:13:14
Speaker
When we merged with the development team that's behind Pepperley, we basically created a couple of new things.
Clojure in Data Science and Machine Learning
00:13:25
Speaker
There's a lot of data science now behind the scene. There's a lot of data aggregation. I don't know if I have a couple of numbers. I think we get half a billion posts per day.
00:13:39
Speaker
And it's like 50,000 articles that are analyzed per day. No, millions, not thousands. And yeah, so when I say analyzed, there's a lot of, I don't know, entities detection, categorization, sentiment analysis, and all these kind of things. And all of it is written in Clojure. So you're a kind of a Clojure version of Google News.
00:14:02
Speaker
Yeah, it's a bit similar in some ways. But I would say we focus more on allowing the users to have very precise querying capabilities. We have our own little programming language that they can use and they can express very complicated queries and do custom waiting on a lot of things through sub-queries. The users that we have, the private companies that use the thing,
00:14:31
Speaker
Typically, there are people who want to know what their competitors are doing, or when there are early funding happening in their domain. But it's very often extremely precise, okay, I'm in Mettech, but I'm interested in all these specific fields.
00:14:47
Speaker
But before we get, I mean, I would love to discuss more about the stack that you're using, but I'm curious about how do you type all this shit and in which editor? Yeah, it's a bit of a surprise, but I'm a new Max user. See, I knew it. I mean, it's really difficult job, so you need a proper ID. Exactly. No, I wouldn't be possible without you Max. I wouldn't be possible without you Max.
00:15:13
Speaker
No, yeah, it's been a long time. There was a little yet to use, well, I didn't use Emacs for a couple of years, but then I quickly went back to sanity and just, yeah.
00:15:26
Speaker
No, university were introduced to it. For two years I was insane and then I came back and everything is fine again and then I started using IMAX. It really just feels like a religion to me. I think you get used to so many things that it's hard to go back to something else. Yeah. Unless you really have to for some reason. I mean, I was playing with Kotlin, for instance, a couple of months back.
00:15:55
Speaker
I mean, the idea is so well done that the Max integration is so poor that, I mean, in that case, I understand it doesn't make sense to use the Max. I mean, at least for now, then just wait a couple of years and who knows. But yeah, at the beginning, I hated it. Well, Kotlin is a little bit kind of, it's got an unfair advantage, hasn't it? If it came from IntelliJ, so they're kind of like definitely privileging their IDE.
00:16:23
Speaker
Yes, that's exactly that. I mean, it's extremely powerful. I mean, you can basically convert, I mean, normal Java code to Kotlin in one click. I mean, you can do crazy stuff. But then you have to leave with the rest of the ID, which is not so nice, in my opinion.
00:16:45
Speaker
So you also use your own keyboard. Yes, well, it's not my own. It's the one that was designed by Phil Hagerberg, actually, the guy behind Lightning Eye. Oh, yeah, yeah, yeah. But yeah, I've been using for quite some time also mechanical keyboards in general. And with time, as time passes, it seems like they shrink down a bit more.
00:17:06
Speaker
Started with a full size keyboard and 10 kilos, 10 to 60%. And now I'm down to the Atreus, which is the one that field design. And yeah, it's an open source design. So it's really nice for that. Yeah, because I've been lately dabbling or kind of thinking about getting one of those keywords because I saw Arne on a browser.
Performance Optimization with Cassandra and Elasticsearch
00:17:28
Speaker
He got this Ergodox DX or something. Yes.
00:17:32
Speaker
Then I've been itching to get one of these keyboards. And one of my colleagues, I work at Workspot, and one of my colleagues, our software architect, Marain, he 3D prints his own shit. So he's just 3D printing his keyboard. And it's like super, I mean, very ergonomic, actually. It's a butterfly design, so it's a split things.
00:17:53
Speaker
Yes, exactly. But I think when Phil designed that keyboard, the main goal for him was, well, he had to be very ergonomical. Because I don't know if you've seen, but Phil is the guy who used to wear pants with a keyboard split on each side of the pants, on one leg on one side and on the other. So he could basically work standing with a split keyboard sticking on his pants. Anyway, so it's very comfortable. I mean, that keyboard, the Atreus, is really nice for that.
00:18:22
Speaker
I mean, it's extremely small, so you might think, okay, but how do you use such a small keyboard? You're missing tons of keys. But the way the layouts are designed and the well, so your fingers are positioned, makes it extremely comfortable and you actually, it's less stressful in general than a normal keyboard, in my opinion.
00:18:44
Speaker
How do you use them with your laptop then? You're basically bound to your keyboard. That's the pain. Before the podcast, I had to take my laptop without the keyboard and go on the sofa downstairs. Typically, it was one of these moments and I have to type on a normal keyboard again. For a couple of seconds, it takes a bit of an effort to not use your thumbs for a lot of stuff. It's strange to go back and that's one of the downsides for sure.
00:19:14
Speaker
The other thing I got to feel was to make it very small, very portable, because apparently it was going on working from coffee shops a lot while remote presses, let's say. I could have taken it on my lap right now with me, but I didn't.
00:19:33
Speaker
I've seen these other guys who use these blank keyboards where they don't have anything on their keycaps. It's like this DAS, a company called DAS, make mechanical keyboards. Yeah, that was one of the recursors of the way of doing things. Well, mine doesn't have any either, but it's not because- It's a security feature, is it?
00:19:57
Speaker
No, but if you, I mean, yeah, your wife cannot use it so far that, no. But, no, I mean, anyway, we have so, I mean, the layout is so weird anyway that, yeah, I mean, nobody's gonna really able, I mean, my mom is not gonna be able to use it just by reading where the letters are, mostly.
00:20:19
Speaker
Especially the layout I have, because the return key is not even on the main layout. I mean, you don't have a return key. You use control and M basically to do return. So it's a bit odd. But a lot of shortcuts like this are inherited from all the unique stuff. So you get used to these shortcuts before having an address. So maybe the transition was a bit easier.
00:20:40
Speaker
But I don't have tab, I don't have return, I don't have arrow keys. No backspace or control H. Yeah, no backspace. Yeah, exactly. Good times. But in the end, seriously, your hands are always on the home row, more or less. Not always on the home row, but they don't move laterally. Almost never.
00:21:02
Speaker
And vertically they never move, it's just a little finger movement. So it's really comfortable and you can get to very fast typing speeds. And yeah, I like it that way.
00:21:13
Speaker
But I've been thinking, what would be the next way of writing code? Because we've been discussing this a lot at work as well. You have to optimize it a lot, of course. You switch to one of these fancy keyboard layouts and everything, and for two days, your speed goes down. Because you really want to communicate your thoughts faster to the screen. That's the whole idea.
00:21:32
Speaker
Yes. I don't know how we are going to solve this because at some point that the keying needs to stop. Maybe we should get this one of those brain implants shit or something. I don't know. Just keep looking at the screen.
00:21:45
Speaker
I mean, it's muscle memory, in my opinion. I mean, to be honest, sometimes when some people ask me, okay, how do you do these cinematics with what keys? I don't even know. I just do it with my fingers and I couldn't tell you which keys are doing what just happened. So, I don't know. It's probably good enough that way, I would say. But then it's a matter of
00:22:05
Speaker
Yeah, you wanted to say something? Yeah, I was going to say I used to have one of those type matrix keyboards. Yes. That was where you have the enter bar in the middle, the enter key in the middle. And it's kind of laid out a bit like you can lay it out a bit like you can have it as dirty or quirky or whatever you want. But again, they're kind of very compact keyboards. Yes.
00:22:30
Speaker
So how did you get into this whole keyboard thing then? Everyone just gets a QWERTY or a ZERTY keyboard. The Europeans use a ZERTY quite a lot. But these QWERTY things are a piece of shit, basically. Everyone kind of knows that.
00:22:47
Speaker
No, I still use regular layouts myself. I still use QWERTY, but just the keyboard that's different. I don't use cold mark. I mean, this kind of layout, no. But yeah, it's heavily modified from how I like to use it, but it's not as extreme as... I don't know, I tried cold mark a bit, but I didn't... I mean, the learning curve was kind of high. I mean, for basically a couple of weeks in the days, you're basically a bit useless with the keyboard, and then you get to it slowly.
00:23:16
Speaker
It's already difficult enough with something with the attributes. At the beginning, for a couple of days, you're really lost. I mean, chatting is super easy because it's like the layout you're used to. There are just two keys, basically, that change. But all the modifier animacs, they change positions. I mean, you have to get used to a lot of different ways of doing things, especially animacs. And yeah, it gets worse before it gets better.
00:23:42
Speaker
And that's for sure. But then it's a lot better, in my opinion. And I mean, it's fun also, the whole electronics part, you get to build it yourself. If there's an issue, you can basically fix it. You can replace the single key that fails if you have to. How come this is a DIY pack?
00:24:00
Speaker
Yeah, yeah, that's how it works. So you can order from field like packages with, well, you can have it pre-built for let's say a premium price, but otherwise you can buy for 100 something dollars, I mean the whole set and build it yourself.
00:24:16
Speaker
And it's well made and serviceable and everything is open source from the CAD files for the plates and all this stuff in the case and to all the electronics and the PCB and everything.
00:24:32
Speaker
Nice. I should get one of these things these days because I need to get my Mac for repair because the N key got stuck on my keyboard. It's so fucking annoying. I take it to them and then they say, oh, we need to replace the entire keyboard.
00:24:47
Speaker
What, for one fucking key? Yeah. Okay. And then give it, give it to us for three to five days. And this is the new keys, you know, the butterfly shit, whatever. That's, that's just irritating. So I was thinking I should get a different keyboard. But the main problem is, you know, if you're in a train, if you're traveling, then you get back to this keyboard again. So.
00:25:07
Speaker
That's why I chose that keyboard in particular because it's really small. It's less wide than my laptop keyboard. It's really compact. It's really easy to carry around. If anything breaks for whatever reason,
00:25:25
Speaker
If security on the airport decides that it might be a bomb, it's not too much of an issue like it would be with others. But, yeah, I mean, there are Godox, for instance, that you were mentioning before. You can get a pre-built one. Godox is easy, I think it's called, but it's going to cost you, I think, $250 or $300 coming close to that. And probably even more if you're in Europe. So,
Deployment Processes Using Ansible and Palette
00:25:48
Speaker
I mean, it's really nice keyboard and everything, but I still prefer to have a one-source hardware for this kind of things.
00:25:56
Speaker
Okay, so this is the keyboard that you use to type all this closure code then. Yes. In Emacs. Yes. Yeah, exactly. It's the Triumvirate, the open source keyboard, the Emacs and the closure. You're living the dream here, Max. Well, as far as VJ is concerned. I'm salivating, drooling.
00:26:18
Speaker
Yeah, so we should move on to some of the libraries. Yes, exactly. So you use Emacs and then keyboard, so we talked about how, and then now we're going to talk about what. What exactly are you doing these days and the library that you're working on? Well, mostly all the libraries that I've written, almost all of them at least, that are on GitHub, they're libraries that I need at some point to do something at work.
00:26:43
Speaker
There were a lot of libraries around Cassandra, for instance. I wrote, I think, three different drivers for Cassandra with the years. I started six years ago. There were libraries for Java, but they were not very nice to use, full of annotations and stuff like that. There was a way to learn how Cassandra worked also for me back then.
00:27:11
Speaker
to implement one from scratch. I think it could be a nice idea to give a quick overview of what's good about Cassandra and why you started using that one. Cassandra, I said distributed database, but it's very much oriented. It doesn't have a single point of failure in its design. Basically, every node is equal to another. There's no master or slave or anything like that.
00:27:40
Speaker
Yeah, so it's very... Well, they say that it's very low maintenance, but I mean, it's partially true. I mean, there's still some maintenance once you grow the cluster past the size, but it's very low operation. I mean, at least when it came out, the big competitor was HBase, and HBase was a monster in terms of installation and maintenance. So Cassandra is very low friction for that.
00:28:04
Speaker
And you use Cassandra if you need to store a large amount of data or if you need a high amount of writes, a lot of writes for whatever you're doing. So it's very good for analytics in general. People use it to store time series and counters about a lot of stuff.
00:28:25
Speaker
Well, to talk about the thing is not really good at, so it's one of the databases in the class. It's AP, I think. So there's no, the consistency is not, you cannot have transactions or anything like that. There's no guarantee that you're going to get through a read what you just wrote, basically. But other than that, it's really resilient. It's never, I mean, it's very hard to
00:28:52
Speaker
to get into an unusable state with a cluster, even if machines are down or if you have a net split or stuff like that. Sweet brains, I would say. Net split. Where did you use this thing then? Cassandra is used by your company right now, Infinity.
00:29:13
Speaker
Yes, Cassandra is used in different ways. The company, I think, has a 14-node cluster right now. And there's another one a bit smaller. But overall, there are 19 machines that use Cassandra in one way or another, I think. I don't have the exact number. Do you have this cross-data center replication as well in Cassandra?
00:29:38
Speaker
Yeah, you can have this. Do you guys use a cross data center? No, we don't have cross data center. Everything, all the main clusters are on the bare metal on OVH. It's actually a problem because we had a nothing recently and that was a discussion about supporting cross data center application. But yeah, then the cost changes.
00:30:05
Speaker
And we can also use it at a much smaller scale. We use it in other projects at a very small scale, let's say, and it works just fine. And it was originally by design, so that when the setup of a client would grow past a size, you could just easily make it grow.
00:30:25
Speaker
Because Cassandra has a very good thing is that it's very predictable in when you need to scale. If you need like twice the throughput, I mean, you can just add basically double the nodes and you're going to get that more or less. So that's one of the really good thing about it. So how come closure of them? Closure plus Cassandra.
00:30:48
Speaker
How come? Well, closure was because I just fell in love with the language that I mentioned. But yeah, the drivers were so sold, let's say, for Java, and it was a good exercise for me also to learn Cassandra. I said, okay, I'm just going to write one of these drivers, and it was based on the thrift interface back then.
00:31:12
Speaker
So, and there was one only for Java, but it was using the regular drift interface and there was an existing interface using async drift back then, so the asynchronous one. And since it was the hot thing to be asynchronous and everything back then, I tried. I said, okay, I'm just going to use that one.
00:31:29
Speaker
And so I implemented part of the protocol for a couple of things, like collection types and maps and all this stuff, and all the funny stuff, the retry policies, the load balancing. I mean, it's a good experience, but I learned a couple of things doing that.
00:31:47
Speaker
And we used the driver for quite some time, for a couple of years, and he was doing really well. I mean, he was doing okay. Then Netflix came out with their own driver, which was not for closure, but for Java, which was really nice. I call it a Stenax.
00:32:04
Speaker
So we were thinking, okay, maybe we should transition to this. And then the company that's backing Cassandra commercially right now came out with their own, which became like the official Java driver. And it was also written in a way that was kind of nice and made it a good fit also for Closure.
00:32:26
Speaker
So that's when I decided, OK, I'm going to just write a simple wrapper for that one and try not to grow it too much. That was the original idea. And then I went out a bit of control. And it's becoming what it is now. It's the library called Alire that's on GitHub.
00:32:47
Speaker
That's what we use right now. At least on the clusters that are running Cassandra 3.0 and the same thing, that's what we're using because we have still all the installations. One of the things that you emphasize on the page, Max, is performance. Maybe you can tell us a little bit about why, for instance,
00:33:11
Speaker
other closure wrappers or closure in general might not give great performance by default. And what have you done to mitigate that?
00:33:20
Speaker
So there were a couple of things. There were a couple of drivers existing in the past for Cassandra for closure. But there was a lot of, I mean, they wanted to make a super beautiful closure API, which was, I mean, a nice goal to have. But then there was a lot of, I mean, there was a lot of abstraction and it made things very slow. A lot of intermediary sack everywhere. And when I like profile some of them, it was not really nice to see all the garbage created left and right.
00:33:49
Speaker
So one of my goals was to make the, so the Java driver was actually really lightweight and low level. So it was kind of easy to build just a very small like closure layer on top that basically was a couple of function calls away from the Java thing. And I was trying really not to abuse well, laziness or sex and all the things in closure that can make these things a bit slower. And so yeah, just try to keep it simple in short.
00:34:19
Speaker
Yeah, a lot of profiling also to make sure I was not going a bit too wild in some cases. And the parts that would be slower for one reason or another, like there's another like border library of the main driver, which is like a DSL, a bit like what exists in closure for working with normal SQL or Corona in the past. So we can basically use maps.
00:34:46
Speaker
to generate your queries, that stuff was all optional. So, okay, that was a bit slower. We do a lot of string manipulations on that one, but you don't have to pay the tax of performance at this level if you don't want to. And actually, we don't really use it that much. We use it just for the migrations and setting up tables and all these kind of things, but not really in the real queries. They all end up being like prepared queries when you care about performance anyway.
00:35:17
Speaker
And another thing was there's a lot of iteration, obviously, in this kind of thing. So you're going to iterate over rows in a table and then in other color facilities that made it really fast. So you can, for instance, iterate over rows using a reducible. So there's going to be no cost, no sec created at any point in one of the ways you can query the database.
00:35:46
Speaker
Also, you can do that in some fancy ways. You can, for instance, do a select star on the whole Cassandra table that can contain terabytes of data and get a reducible about that and just iterate over the content without holding the head on anything. It's a streaming protocol on the Cassandra side. So it's going to just come at you as you call the function, basically, and halt when you want it to halt.
00:36:13
Speaker
And there's another part of the API, which is a bit, well, totally lazy, which allows you really to, well, to basically have a lot of fine-grained control over iterations about this data, about your queries, about what they're written. And for the query language, you have your own version, right? You have your own DSL for this one? Yes.
00:36:37
Speaker
Yeah, that's what I was saying before. We have a library called hate because a comma that existed that was kind of nice. I mean, the idea at this was nice, but then the implementation was so-so for some reasons. It was a lot of macros and I don't think there was an intermediary representation also of the queries. There was no data basically that you can manipulate.
00:36:55
Speaker
So the idea was to take inspiration from what Karma was doing, at least at the surface layer, how the API works, but then have just data structures that represent your queries. So you have just maps. Yeah, basically just maps in short.
00:37:14
Speaker
And you can easily manipulate your queries that way. You just use the normal closure function, crunch, merge, and all these. And so yeah, you can generate prepared statements or normal statements with that. And also on that side, there was a lot of
00:37:30
Speaker
focus on performance even though, I mean, if you really care about performance, you will not generate tons of string, I mean, do a lot of string manipulations at runtime through that library. You're just going to use that to generate prepared statements and then use the prepared statements instead. So, Max, why do you hate Cassandra? Why so? I'm sorry. Why do you hate Cassandra?
00:38:00
Speaker
What is the name form, actually? I don't know. I know it's not spelled. It's spelled H-A-Y-T, isn't it?
00:38:08
Speaker
AY, no, I'm sorry, the library, the DSL library, you mean? Haight. Haight, because the main library, the Java driver is not a closure driver, it's called Alaya, which is a character in Dune in the book. And Haight is another character, basically, in that book. And they have like, well, let's say, links. And I don't know, it was just a name that I picked like that from the Wikipedia page of Dune.
00:38:35
Speaker
Yeah, I'm not very good with naming. Particularly, you hate Cassandra, so you decided to make it for you. No, not at all the intent. It's a subtle stab on Cassandra, I think. I hate you, but I'm going to hate you.
00:38:49
Speaker
Well, Cassandra writes a love-hate relationship that we have. That's true also. No, it can be really good and it can be annoying for some things. It's not the perfect solution. Sometimes you just wish you selected Postgres for something instead of Cassandra. You also do a library for Elasticsearch as well, don't you?
00:39:09
Speaker
Yeah, Spandex. Spandex, that's right. Yeah, exactly. Yeah, like you said, names, yeah. So, yeah, same. I'm very impressed that you're wearing Spandex for this podcast. That's quite impressive, actually. That might be. I think you didn't have to wear yellow, but I'm impressed that you did.
00:39:32
Speaker
It was the same situation a bit. I mean, no, it's not really true. But there are some libraries for Elasticsearch. But some of them, they're not maintained really anymore, or they're in life support. I mean, it feels like some of the maintainers just merge pull requests, but there's not really any work done on them anymore. And if you've worked with Elasticsearch in the past, probably know that it's a super fast-moving project. I mean, every version, they depreciate a lot of stuff.
00:39:59
Speaker
There's a lot of new features and the low-level bits change and it becomes really, I mean, even just for a regular user to follow the API sometimes as it evolves. For the drivers, it was a bit the same problem. Even on that side, the people running Elastic decided to basically ditch the... Because there were two things in the past. There was the native transport-based library.
00:40:23
Speaker
and the REST library. So you could choose, okay, I need like extreme performance and whatnot, and I'm going to use the native transport one. Or I can just give up a bit of network latency and use the HTTP layer based library. But I mean, now with the kind of, I mean, with HTTP, you can go very far and the difference is not so big anymore between the two.
00:40:45
Speaker
And they said officially not too long ago that the way to go would be to use the REST layer and forget about the native transport other than to basically build the Elasticsearch itself. It's funny, isn't it? Because CouchDB had a REST API from the beginning.
Clojure's Productivity and Access to JVM Libraries
00:41:03
Speaker
And all the Elastic people were, oh, the REST bullshit. We want the real native APIs. They get really good performance. Yeah. They've eaten their words big time.
00:41:15
Speaker
I mean, the idea, yeah, in theory, it's nice to have your own native protocol and everything like Cassandra does. But, I mean, Elasticsearch is not in a domain where it's necessary to go to these extremes. And HTTP also is changing a bit with HTTP2, and you can get to, I mean, very close performance. You don't pay the round trip latency like before, I would think.
00:41:42
Speaker
What do you do about, because one of the things that they have in Elasticsearch is these aggregation pipelines? Yes. That's something we use a lot actually. Well, so just to continue a tiny bit on the Elasticsearch part, I mean, one thing that was extremely annoying for us is that every time you upgrade your Elasticsearch version,
00:42:01
Speaker
With the native transport, you have to use the exact same transport, basically a driver. It was a huge pain. If you have many different servers for many different clients like we had, sometimes running still 1.75 because you have to run sometimes 2.4 and sometimes 5. something, it becomes a nightmare. That's why we decided, okay, we need something more robust. We're going to build on the new HTTP client. Since they just release their own and they maintain it and
00:42:28
Speaker
I mean, they do the work with paid people for us. We're just going to do, again, a thin layer of that and use that. It's going to be just fine. So it started as a couple of hours library, and it grew out from there. And so, yeah, Cassandra is really good at a couple of things. So well, first of all, it's not a replacement for a normal database. It's not a database at all, I would say. It's just an indexing engine. No, you mean Elasticsearch. Yeah, Elasticsearch.
00:42:55
Speaker
Yeah. So you always have to have some kind of data store in the back that's going to store really what you're going to then index on L6 search, in my opinion. Then it's extremely good at searching stuff, obviously. And you can express really complicated queries and do like a waiting, custom waiting on terms or on value of fields and
00:43:19
Speaker
do like, well, really a lot of crazy stuff just for the search part, but there's also the aggregations. So aggregation, you can do well, very complex analysis stuff that you used to do by the normalizing data running batch processes in the past. But you can do it live more or less with a basic search. So, okay, yeah, it's gonna cost you a bit more resources, obviously, but you can go very far with what it gives you.
00:43:45
Speaker
What do you think about this? Because I know Datastax, the Cassandra people that you mentioned earlier, they also ship. They have their enterprise edition of Datastax, and that comes with Solar rather than Elasticsearch.
00:44:05
Speaker
both based on Lucene in the end, of course. So it's the same fundamental indexing. But one of the advantages they claim, and I think it's a decent advantage, is that you don't need to move the data around. You can do the searches in place on the data that you've got.
00:44:24
Speaker
Yes, exactly. Well, they integrate directly with what's starting Cassandra and gives you like knobs to choose how data, basically how long it lives and so on. But why did you start in Sadavera 6 search? To be honest, I don't really know. And we didn't choose their solution for a couple of reasons. One of them, because it's what you have to pay to use it. It's a DSC, so Data Tax Enterprise.
00:44:48
Speaker
And yeah, it's a bit, I don't know, we use that sort of C-search in the past, so we just knew how to use it and it was the obvious choice. And we do a lot of, very often our clients, they have their own platform with their own Cassandra, their own C-search and their own everything. So if every time we have to pay a license, it would be a big problem for us. Same for the, I mean, Datamy, Datamy, we even thought about it at some point.
00:45:17
Speaker
Not too long ago, we really evaluated, I mean, in terms of feature, it would be a perfect fit for one of our products. But then, yeah, it costs too much. And yeah, that's one thing.
00:45:31
Speaker
So I'm curious about your stack. So use Cassandra and Elasticsearch. So obviously Cassandra for storing and Elasticsearch for querying and your own libraries for both using Closure. So where is the business logic running? What is the Closure stack?
00:45:50
Speaker
Well, the closure runs everywhere. That's what we call Finiti data, which is like the data platform, the one that does all the streaming aggregation and enrichment. That's basically the huge data provider for the rest of the clients, let's say peers, as we could call them. It's used everywhere. It's used for normalization of data. It's used for the machine learning stuff. It's used for the enrichment.
00:46:15
Speaker
But which libraries? Do you use any Spark or something? Not really. It's mostly Cassandra, Kafka, Elasticsearch. No, there's not much of... The computation, for example. Do you use Onyx or have you looked into Onyx? No. Yeah, we did. For a project that we had recently, we evaluated it.
00:46:37
Speaker
I was not working on that one, but then they went with BigQuery just because it was not exactly the same, of course. It's not totally different, but it included all the operational stuff was uploaded to Google, basically. And on top of that, you had the data store that fit the need. But yeah, it was something that I really liked and that I suggested to the people that were working on that when that came on the table.
00:47:04
Speaker
And that's a project I would use myself if it's a good fit for whatever we do. And the front-end is built in ClosureScapes as well. Yes, no, not ClosureScapes. It was at some point very early on. But I mean, paper that I like, the other service is still in Ruby, I think, the front-end.
00:47:24
Speaker
But the other products, they are all in Closure, I think. I mean, the back-end that serves the HTTP requests from the users or the clients. Then there's JavaScript, there's a lot of React, and other kind of frameworks like this, but it's not Closure. Simply due to the fact that the people who are working on it,
00:47:46
Speaker
They are like JavaScript developers. I mean, they're good JavaScript developers. They are very familiar with React and so on, but Closure Script is not something they would be comfortable using, I think, if we ask them to...
00:47:59
Speaker
In this stack, using closure, what was the biggest challenge for you using closure? Because sometimes you can pick these days. You have lots of options. Of course, you're also familiar with Adlang, for example. So picking closure, what was the biggest challenge? The biggest challenge, that's a very good question.
00:48:20
Speaker
Do you mean challenge or motivation? Yeah. Kind of the same thing, really. What did Closure offer that these other languages made a bit more tricky? If you have to tell your manager today, can you still defend Closure as a good choice?
00:48:39
Speaker
It was the access to a lot of libraries, first of all. Access to the GVM, but without having to write 10 times more code with Java at the cost of a tiny bit
Machine Learning and Natural Language Processing
00:48:49
Speaker
of performance. Not a tiny bit, but at the cost of performance probably. But we all got extremely productive, I think, using Closure.
00:48:57
Speaker
And right now, when I try other languages, even easier, let's say versions of Java, like Kotlin, where a lot is done upfront by the IDE, and the language is a lot simpler, but still, Azure is unbeatable for productivity, I think. I mean, working from a ripple is a huge win over the previous workflow I had, especially UVI Max. Of course.
00:49:29
Speaker
No, but productivity was huge. You get access to crazy good quality libraries for everything and anything. Erlang is awesome for a lot of stuff, but you're going to get constrained to a couple of domains.
00:49:44
Speaker
And you're never going to do a machine. I mean, not now, at least machine learning with a long. And there are a lot of stuff you have to write it from scratch. So it's good for some things, but not all. And we just use whatever is good for whatever we need to do at a point in time, I guess. So what kind of machine learning things that you guys use there?
00:50:06
Speaker
There's a lot of actually custom code. We have a guy who's really good at all this stuff. Well, we can call him like the data scientist of the company. So there's a lot of custom written code. And let me think for a bit. On top of my head, I couldn't give you names of libraries or specific frameworks. But yeah, we use a lot of open NLP, for instance, for some things that I wrote. Because you deal with lots of text then. So mostly text related.
00:50:35
Speaker
Yeah, so there's a lot of enrichment, gender detection, all these kinds of things, categorization. Also trying to deduplicate content. Since we deal with articles, one of his tasks, for instance, was to detect articles that are extremely similar and that were basically copies of each other. So you get rid of the content forms.
00:51:00
Speaker
Yeah, exactly. And there are a lot of challenges like this that are not really visible on the surface of what you receive through your data when you're a client. But there are a lot of little tasks like this that require a lot of filling with machine learning or statistics or these kind of things. So it's like they say then, you basically hire a PhD guy and then you have him spend him spending 80% of his time
00:51:28
Speaker
running Perl scripts to clean up the data or the equivalent thereof, you know, enclosure. Yeah, that's a bit nice. It doesn't use Perl. But other than that, yeah, it's a bit like this. But we all, I mean, I wrote a lot of the enrichment stuff also before the merger of the two companies. So some of it was my responsibility and they use some of what I wrote in the past,
00:51:54
Speaker
but it's mostly his work and that's used for all the rest. That's the big part of the machine learning tasks we have now.
00:52:06
Speaker
And a lot of it, actually, we can, I mean, it's not really machine learning, but a lot of very advanced stuff can be done also through Elasticsearch directly. I mean, it's not machine learning at all, but you can get really fine-grained precision of what you want from the data through scoring, custom scoring, and stuff like that. I mean, it sounds dumb, but you can create custom queries, like, subqueries that will score your original filtered queries in some ways, and you can go really far with just that.
00:52:33
Speaker
I think Elasticsearch focuses on what they call relevance, doesn't it? Yes, exactly. There's still magic to everybody at this point. There's a lot of maths behind that, isn't there?
00:52:49
Speaker
Yes, sometimes you have to dig a bit into how things work to understand what are going to be the implications in terms of performance or storage or about some features. But yeah, it's really magic in many ways. But yeah, it's at the cost of some control. When the person in charge of the data science part really works on this stuff, you're going to have a lot of guarantees and you can really tailor-made the solution to what you want to do.
00:53:19
Speaker
Yeah.
00:53:21
Speaker
So your system is only processing English language content? No, that's one of the big challenges. We process more or less anything. So we have clients that are in Spain, in a lot of central Europe, a lot of French, a bit of everything. We had some national French TVs here and there. So even in Switzerland at some point, Switzerland is like three languages.
00:53:49
Speaker
So from the start, it's a challenge because you find a lot of models that are in foreign English, I mean pre-trained models for whatever. But as soon as you need like Chinese or I don't know. Even German, I mean stemming and everything and it changes the whole
00:54:07
Speaker
So yeah, you have to do it yourself, I guess. So you find ways to do it for categorization. The very early versions were basically calling news websites, even though maybe I shouldn't say it. But anyway, we are doing that through the different categories on the sites to get content that were matching a certain domain, like sport, finance, or whatever.
00:54:27
Speaker
and you train your models on that. But then you use Wikipedia a lot when you evolve a bit on this kind of websites that gives you a bit more precision and where there's a lot of freely accessible content. But yeah, it's one of the big challenges, finding the data.
00:54:44
Speaker
So just to maybe just to move on a little bit, Max is the one of the very, very small library, but I think he's done a lot of thinking about it, which is the stuff around spec, because we were talking about data quality. Presumably, this is something which, in theory, at least, once you can start to describe your data a bit better,
00:55:05
Speaker
you should have a bit more power of its entry into your system. So, tell us a little bit about how you're thinking about that in terms of its usage in your domain.
Scale of Operations and Development Dynamics
00:55:16
Speaker
Yeah, that's good to mention it. We use spec a lot, actually. And I was the one pushing it. But because I liked the principle at the beginning, who was using schema before,
00:55:26
Speaker
And I was very skeptical about spec at first, I have to say. I don't know. It felt the way to write the spec was not as natural as the way you would write schemas in the past. Schemas, basically, when you were speaking while doing a schema for a map, it was a map. And when it was for a set, it was a set. And it was really easy to read. And the spec was a bit, it was different that way at the beginning.
00:55:53
Speaker
For me, I was very skeptical, but then I started using it, and I slowly fell in love a bit with the way there's the split between the keys and the maps and so on. So yeah, we use it a lot. We have thousands of lines of specs probably in our code base right now, and it's put up some very pernicious bugs, and we are kind of amazed by... I mean, we had very specific specs, for instance, for geographical coordinates.
00:56:21
Speaker
But we're saying, okay, a latitude should be within these bounds, a latitude within these bounds. And before, we were not caring about that stuff. We were just saying, basically, okay, it should be like a number. It was a bit more precise, but it was more like that. And it was finding bugs like we had coordinates that were not right, sometimes coming from some data sources. And then, I mean, some really weird bugs that we would never have detected, were detected by spec.
00:56:44
Speaker
But we are not, I would say, poor users. We don't use everything from spec. I'm not using instrumentation a lot. I use it a bit in development, but very often I'm getting bit by some of the shortcomings of instrumentation, I think.
00:57:02
Speaker
Can you talk a little bit about that? Because I know that, I mean, I like instrumentation when you're doing tests and stuff like that, because obviously it makes it very easy. But obviously there's a runtime overhead. So you're seeing some other things as well.
00:57:16
Speaker
Now what I do usually is basically I split, I create a separate nip space just for the spec and I load it as I write some code and I use an instrument there, but not for everything. But sometimes I have just to put assets in seven of an instrument, of using instrumentation. Typically when I have a higher order function passed to another function,
00:57:37
Speaker
you get into some weird situation where generative testing is going to be called on that function. And then depending on what the higher order function does, the anonymous function that your function does, it's just not going to work. When you have to deal with interrupt, for instance, generative testing is
00:57:57
Speaker
are going to be strange because you have to create a lot of code just to have the generative part works, even if you don't really care about it. For this, I would sometimes wish it would be a bit simpler, like you would just wrap the function that you pass as an argument with a search, basically, and do validation at runtime just as simple as that like this.
00:58:19
Speaker
But that's one of the things. But other than that, I use a cert like everywhere on the fact that you can enable disable it at will. And I mean, either at compile time or either at runtime. This is kind of awesome. And sometimes you just want to enable it in even not in production, but in the exchanging environment.
00:58:40
Speaker
see where it fails at some point, it's super useful. And yeah, I had a taste of that with other languages I used in the past. And yeah, now it's probably something I couldn't do without. But I'm really looking forward to, I mean, I know that there's going to be in the next iteration of spec coming out in the near future. I've heard about Critch working on that.
00:59:03
Speaker
And I'm really wondering what's going to come next and if there are changes to some of the existing things that I think are a bit, that could be improved. And yeah, on my other big, not disappointment, but the thing I don't really like is the way you compose specs right now.
00:59:22
Speaker
Sometimes you have to get to, I mean, overall it's really good at composition, but sometimes you are in cases where you need something a bit advanced and you need to compose a lot of specs in weird ways and you have to use eval, because it's all macros and it's kind of ugly to have to start to eval to do this kind of stuff.
00:59:41
Speaker
And you just wish you had more data. I mean, it was a resident more in data than just like forms. And I know the data tickets about that in JIRA. Like they're talking about the possibility to confirm a spec and then basically modify it because then it would be like a normal project data and then unform it and you get another spec. So that would be a way to do it. But it sounds still a bit convoluted, the whole process.
01:00:10
Speaker
I don't know, but I'm waiting to see how it is. I'm a bit quick to judge sometimes, so we'll see how it is when it comes out. Did you use any statically-typed languages to have a comparison?
01:00:28
Speaker
Yeah, I've used some OCaml, and I've used that scale as well. But OCaml, we actually have a couple of small things written in it. So yeah, it's very comfortable to have a type system, but you're still going to generate bugs no matter what. And most of the time... That's kind of a stupid argument. People make that you're not going to have any bugs.
01:00:51
Speaker
No, I mean it's gonna catch the stupid stuff that yes, but I mean most of the bugs that we have I mean if I look at all issue trackers is not because you made a typo because it's the wrong type of stuff passing here or there is because of some weird domain thing that makes it that you missed a case or something like that, but yeah, it helps and
01:01:13
Speaker
I don't know, I think there's a sweet spot between Haskell or OCaml-style strategy type system and Closure. And maybe Erlang is a good middle ground, actually. Erlang has no nil. The data types are fairly simple and use tuples for all that stuff.
01:01:31
Speaker
And you have a static type checker that's good, but that's not crazy like what you would have in a scale or a camel. And basically, you are almost guaranteed if you spec everything you write, while almost everything that your code is going to be bug free and is going to run just fine for years. I mean, it's kind of crazy the reliability of Erlang code compared to even closure. But we might just get there with spec because it's not far from what exists in Erlang.
01:02:03
Speaker
So it's very impressive. I see the numbers that you're talking about with the affinity thing. So many millions of articles analyzed and different amount of data that you're analyzing. So I have two questions about surrounding the closure. Look, do you have any estimate on how many lines of code that you have in closure? And then what is the team size?
01:02:23
Speaker
I have no idea about the lines of code. It's a good question. I see absolutely no idea. No, but if you take out all the libraries that you wrote and then released. There's a lot of them. I would probably in like 50 libraries probably. I don't know. We also use Git for like configuration. So there's like hundreds of them, but that's probably in the 50 of real libraries. I don't say.
01:02:46
Speaker
And then the number of people before the merge, that was just me working on Closure stuff. Then when we merged the team on the other side, there are two people, well, almost three I would say that work with Closure, but really two full-time.
Clojure's Impact on Rapid Development
01:03:03
Speaker
Yeah.
01:03:04
Speaker
And they inherited from code that was written from other Closure developers, so the team changed a lot on the other side of the company. And it's all people from Switzerland that are still working with Closure for most of them. And so I've heard about one of them had one of his projects presented at a huge AWS conference not too long ago, showing off another Closure project he's working on now, which is kind of insane. That was Olivier.
01:03:34
Speaker
Yeah. And I don't know exactly what they are doing, but it's probably very impressive. And the other team is the guys behind Exoscale, which is a hosting provider in Switzerland, which are doing also, I mean, a lot of good stuff.
01:03:50
Speaker
It's a bit similar to digital ocean in the way the service works, but with other selling points and it's very impressive what they're doing. And that's mostly, well, I know Pierre Richard that works there, and I don't know about the others. I know a couple of them, but from Python, they're mostly from the Python world, from what I know.
01:04:13
Speaker
We'll talk to him in the future because I worked there for a while. We'll leave that one for another episode. Exactly, yes. It is pretty impressive with the stack that you have and Closure is running behind this big organization. You don't get to hear these things much. That is the interesting part. When things are running smoothly, nobody complains. Nobody knows what is happening.
01:04:39
Speaker
Yeah, it's true. I mean, it's not a lot of people for a lot of work when thinking about it. It is true. But closure makes it easy to iterate very fast. I mean, we did a lot of stuff, I mean, small projects that failed also in the past on both sides of the company. And it allows you to do that without losing too much, I would say.
01:05:02
Speaker
And how do you deploy that? I mean, let's talk about the final stage. So you write the code, and then you spec it, and then you make sure that it's bug-free, and then now you need to deploy the code. There are two ways of working. I mean, on what I call our side of the company, even though it means one company, one team now, but on what was our company at the beginning, we use Ansible, but there's a wall, and you pass by CI, and then Jenkins build the stuff, and then
01:05:30
Speaker
We have our own internal tools to do deployments and to follow the lifecycle of every deployment for the clients. So tools written actually in Closure and Closure Script. That's the exception for Closure Script. So yeah, and on the other side, they use Palette, which also is written in Closure.
01:05:46
Speaker
which is an orchestration library, which was a bit inspired from probably a pallet chef, all this stuff. But that runs without an agent, also, which is a bit more like a hands-able that way. So, yeah, that's how it works. And the infrastructure, I think, is like 90 servers on one side and around 50 on the other. And 90 bare metal and 50 that are split between exhaust kiln and digital ocean, so it's VMs for the other side.
01:06:16
Speaker
But there's a bit of everything. It's fairly automated. Impressive. Yeah. Well, you do your own hosting then. You mostly build your own racks and do your own things. No, it's on OVH. When I say bare metal, it's a lot of games. That's what I mean. So it's basically cheaper infrastructure and a bit better performance depending on what you do. All the important care centers are all on bare metal, for instance.
01:06:45
Speaker
All right, right, right. Okay. So it's a rack somewhere in a cave. Exactly, yes.
01:06:55
Speaker
Okay, so that's a pretty impressive explanation of... I mean, I've never heard that this is... Paperly was built in Closure. I mean, this is the first time I'm hearing it. So that's interesting for me. And of course, now I see the bigger picture. It makes more sense. It is impressive to hear that Closure is being used at this scale. That is pretty awesome.
01:07:20
Speaker
It's also quite one of those things where you hear people talk about the sweet spot for closure is big data. But then everyone's saying, yeah, but where are the examples? So it's quite nice to hear a good story, a good example. Like you say, there are very few developers moving a lot of ground, pulling up a lot of trees. So that's quite impressive.
01:07:44
Speaker
There are a lot of projects, I think, that are probably low profile and that we don't hear about that are running closure behind the scenes. Even some of the developers on our teams, there are not people that are closure famous, let's say. There are not writing terms of closure libraries and they are not on Slack all day. But they are still extremely, they are really good programmers and you never hear about them, that's all. And it's probably not the exception. It's probably the rule, actually.
01:08:13
Speaker
It's to add that some projects that don't come out more in strength are talking about what they are doing publicly. That's true. But I don't know. Maybe something that should be improved via conferences. Well, podcast is a good way to do that also.
01:08:29
Speaker
Yeah, yeah, we're trying our bit. Yeah, exactly. But this is why we want to talk, you know, people who are, who are, you know, so metaphorically speaking, neck deep into closure and then, you know, that they're using closure to build these kind of,
01:08:45
Speaker
awesome things. And of course, I knew about you when Ray was mentioning about Cassandra Library, but I didn't know that you are part of this, you know, there is a huge iceberg. And the two libraries were basically like, you know, tip of the iceberg, so to speak. So it's very nice to hear. And then it's very, you know, motivating for people who are listening as well. They're like, Oh, sure. I mean, we can do all this shit enclosure. You know, that's that's.
01:09:09
Speaker
When you think about how it started, I was just doing with closure on a small project on my side, which became a product, which became a company, which merged with another one, and so on and so forth. In the end, it doesn't need much, and small things can become bigger.
01:09:28
Speaker
Fantastic. So I think we are almost one hour into the recording. But very insightful discussion for me at least. Now I can talk about, hey, you can use Closure for big data stuff. It can't get bigger than this, right? Bigger as in, of course, you can keep adding more and more websites and everything. But almost 50 million websites and blogs
01:09:54
Speaker
It's a big beast, I would say. So any conclusive thoughts? Good question. Did we miss any of the kind of areas that you wanted to chat about, Max? Let's see. Excellent, excellent. Otherwise, you need to come back, and then we need to do another episode.
01:10:23
Speaker
Yeah, yeah, exactly. No, I'm curious about what would come out in closure. It was a bit like the Steve Jobs effect at every closure conference and we have a bit less that and I think there was some talk about closure or popularity slowing down. I don't think it's really true in the end.
01:10:42
Speaker
Yeah, I think it's good that you're doing this podcast and I hope more people like push the envelope on the show and what they're doing and so that popularity keeps on going. Yeah, well, we're... I was going to say it would be very welcome for people to talk to us on Slack or on Twitter, you know, to, you know, because we have a kind of process for getting people onto the show and often it's just
01:11:09
Speaker
me or Vijay liking the look of people or liking the look of repositories and hunting around for people. And that's, you know, that's fine. It's all working out quite well. But definitely shout out to the gang out there, you know, if you want to talk about a project on, what's this called again? It's called Deafen here. If you want to talk to somebody about Deafen, you know, we're quite welcome. Yeah.
01:11:33
Speaker
Yeah, of course. I mean, this is how we want this podcast to be. Not only just people who are already in the conference of Qt and doing their bit to encourage closure usage, but also we would like to talk to people who are every day getting things done with closure and then what kind of challenges that they're facing and what kind of advantages that they have compared to the other.
01:11:59
Speaker
other development teams and how happy they are. That's the whole idea, right? Yeah. And of course, as Rich was mentioning, I think it was like, it was for the boring people, you know, closure is for boring people who want to get shit done. So that makes more sense. Yeah, that's fine. That's exactly it.
01:12:22
Speaker
Okay, so I think we can conclude now. Just before we close off, we'd like to... First of all, Max, thanks a lot for joining us. You're welcome. Thank you. It was a pleasure talking to you and we hope that you're going to use Clojure more and more and then grab more and more. Keep adding zeros to your numbers, both to your posts and also hopefully your company financials. Thanks.
Conclusion and Thank You
01:12:47
Speaker
It was a pleasure being with you. Thanks.
01:12:50
Speaker
Yeah, it was really, really fun. And obviously, we just want to thank the people who are already supporting us on Patreon. There are already six patrons for us, and we are extremely happy to have your support. And I've already ordered my Tesla with your money, and I'm going to send it to Mars right after Elon Musk sends it.
01:13:16
Speaker
No, we're going to order a Tesla and then send it to Mars, right? That's the idea. Oh, yeah, yeah, that's right. Yeah, yeah, yeah. That's what Elon Musk is paying now. So all the cars are good for. Exactly. But all the jokes aside, I mean, we very much appreciate your support, your patronage. And please keep checking your private feed because I'm sending some messages to you folks. And the people who are not on Patreon, this is a slight nudge to you that
01:13:45
Speaker
You might get more information if you become a patron for us. And of course, the usual credits. This is me from Holland and Ray would do the honor for our rest of the crew. Okay, from Belgium, yeah.
01:14:03
Speaker
Well, we've got the music from Pizzeri, and he's got the famous Melon hamburger coming at you right now, I imagine. And we have Wouter doing the sound editing for us. And just a small final word, by the way, we're all going to be doing a Daffan road trip to Closhed in Germany in February. Yeah.
01:14:28
Speaker
And so that's going to be really good. If anybody is going to go to ClosureDee, if you can hit us up between now and then, that would be really excellent. So, you know, myself, Vijay and Wouter, we're all going to go and we'll be there in force and hope to see you there.
01:14:46
Speaker
Yeah. And most probably not most probably, uh, okay. I mean, we'll, we'll, we'll probably try to record something there or, um, in the worst case, if we are going by car, which I don't, we will be, but, uh, then we'll start recording in the car and then annoying people raise already. Like there's no way there's no way I'm driving the Berlin.
01:15:07
Speaker
But of course, I mean, if you're popping by Close Your D in Germany, hit us up and we would love to talk to you. And we know that our biggest fan base is in Germany. Exactly. This is like we're going to be autographing everywhere on the streets and we're superstars in Germany apparently.
01:15:28
Speaker
Okay, that's it from us for today. And we will come back to you with another episode soon. And we wish you good luck with Advent of Code if you are participating in it. Keep writing closure. And again, thanks a lot to Max for joining us from Sweden. And good luck with your closure stuff. Bye bye. Bye.
01:16:24
Speaker
Sorry. I'm allergic to Linux.