Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#88 Matthew Davidson a.k.a kingmob
41 Plays1 year ago
Matthew Davidson, maintainer of Aleph/Manifold joins us to talk about his Clojure journey and https://www.speederino.com and zen :)
Recommended
Transcript
Recording Confusion
00:00:16
Speaker
But you're not, obviously, you're not originally from Indonesia, right? Not very much. Shall I start this bill now? It looks like it's recording. Oh, yeah. I mean, I already clicked record because we usually like, you know, just start and then. OK. Yeah, yeah. As we go.
From Virginia to Southeast Asia
00:00:33
Speaker
No, I'm an American. I grew up in Virginia and went to school there. And then when I was like 20 something, I moved to New York and I lived there for almost two decades.
00:00:46
Speaker
And then, yeah, my wife and I decided to seek adventure. Partly, I saved up a bunch of money and wanted to start a little software as a service. And so we were just trying to save a bit of money by moving to Southeast Asia. And COVID kind of threw a wrench in our plans. So we didn't originally plan to come here at all.
00:01:09
Speaker
My wife studies Japanese and she really wanted to pursue that. And so she went off to Japan in late December, first year of the pandemic, I think. And then the plan was I was going to pack up the apartment and I would follow up with her.
00:01:26
Speaker
later. And two days after she arrived from Japan, they closed the borders to everybody, not just random tourists or people whose spouses had visas, but also like permanent residents, like people who were married to Japanese citizens could not get back in.
Relocating to Bali Amidst COVID-19
00:01:44
Speaker
So we kind of played it by ear and every semester we'd reevaluate and I'd be like, well, it's three months later, can I get into Japan now? And the answer just kept being no, no, no.
00:01:56
Speaker
So we eventually decided to give up on Japan. We met in Portugal. We were there for three months. It was nice enough. And then we were looking at Southeast Asia. And we'd both been to Thailand many, many years ago. But Thailand, their tourist visa setup was, they had something called the Special Tourist Visa for the pandemic, but it was so incredibly convoluted, almost nobody took it.
00:02:23
Speaker
So we looked at where we could go. Special tourist visa for pandemic, as in you can just go and visit when there is pandemic. So if it all worked out, we'd have been able to spend nine months straight in Thailand. But that didn't work out. So we looked at where we could go. And Bali in Indonesia, it's a very tourism-heavy place. And they were desperate. So they gave us a business visa. So I showed up.
00:02:52
Speaker
and started working on my solo idea. And it was a terrible idea, so I had to kill it eventually, but now I'm working on a different one.
Starting a Business in Indonesia
00:03:03
Speaker
Hey, maybe I should do the introduction then now. The person you're listening to is Matthew. Welcome to Deaf and number episode 88. Anyway, but you were working already in the US and then now you moved your business there or what was the main reason then?
00:03:26
Speaker
So I'm basically trying to start a software as a service and I don't have any customers yet. And so living here is just a way to keep costs down while I build it. That's the primary reason.
00:03:41
Speaker
We like Indonesia a lot, but we also wanted to check out other places. The food in Thailand is clearly superior. So it's a little bit harder to stay there, stay here. So we're going to go back to Indonesia next week. And then hopefully we can get a special tech visa for Thailand and be able to stay here a little bit longer than what's allowed for tourists. It's all very convoluted and ever shifting. We'll see.
00:04:10
Speaker
A lot of the countries here have talked about digital nomad visas, but not too many of them have come through with that yet.
Visa Challenges in Southeast Asia
00:04:17
Speaker
But that being said, I was just going to say Bali is the easiest. There are essentially companies that exist to put you on the books and give you a business visa and claim you're an employee, even though you don't really do anything for them. So it makes it, and you can just kind of renew them.
00:04:39
Speaker
for a long time. So it's very easy to stay in Bali, not nearly as easy to stay in Thailand. And is that it just I mean, we're kind of off topic to start with whenever on topic, you know, we can say if we if we hit the topic, it will be a miracle. But anyway, I was going to ask you was, do you do you think all of these things are just still a hangover from the pandemic? Or is it a general kind of way that the countries are organized?
00:05:09
Speaker
Well, the hangover from the pandemic is them still kind of trying to get organized. So a lot of the tourism heavy places were really hit hard, like Bali in particular, the majority of the island is, its economy is dominated by tourism. They were desperate to get people back. I think at this point, you know, it's back to business as usual. And the digital nomad visas are things they've been planning for a while.
00:05:37
Speaker
Generally speaking, a lot of like the countries around here are the opinion that as long as you're not taking work away from a local, they're more or less okay with you coming here and spending your money as long as you don't cause any problems. So for example, Ballet is in the news because a bunch of Russian Ukrainians got into some motorcycle accidents. Now they're contemplating banning foreigners from renting bikes.
00:06:04
Speaker
which to be fair is probably safer.
Safety Concerns in Bali
00:06:11
Speaker
The Balinese are amazing bike riders and I can't hope to touch their ability. How about training rather than banning now? Because most of these things you have...
00:06:24
Speaker
in the UK or in Belgium, for example, you have to qualify to ride a bike or qualify to drive a car. Obviously, VJ is going to tell me that that's all bullshit.
00:06:38
Speaker
I think you can still do it as long as you get a Balinese driver's license. It's just that I don't think they're handing those out to tourists anymore. You have to be a long-term resident. I'm not too clear on the details. If Bali is pretty much like India, I would assume that the training and the test mean nothing once you get on the road.
00:06:59
Speaker
I was told when I got there that realistically, as long as you wear your helmet and don't get into an accident, you won't have any problems. And that, you know, if the police might pull you over, but the fine is minimal and it's possible they're actually just looking for a bribe.
00:07:16
Speaker
I never personally encountered any of that. So I don't know for sure. I don't want to disparage the Balinese authorities and have some government official who also dabbles enclosure on the side to deny me entry. Yeah. I mean, you should, you really shouldn't because the whole, you know,
00:07:33
Speaker
Bali the entire island basically listen to this podcast like every day Exactly. Yeah, that's that's the biggest demographic that we have. So please don't defend any any Bali people I should actually balance this out a little bit by saying that it's definitely not just India or Asia because remember when I first went on holiday to Ibiza
00:07:58
Speaker
which, you know, a young man, you go on holiday at the Ibiza. Anyway, I hired him on a bike, as you do, to explore the island. And I came off it within like two minutes, scraped my knees off, wasn't wearing a helmet, anything like that. I was like, Oh, carry on. It was fine, you know, someone else in the group, there was like six of us, I think, and someone else came off as well. But it wasn't just the scripts, really, you know, so didn't get involved in too many crashes.
00:08:28
Speaker
Yeah, zero, zero training, you know, basically start, you know, turn turn the throttle, hit the brake, that's all. Yeah, I mean, I did, I took a day's worth of training, but I was nowhere near prepared by the time I actually had my bike. Well, the state of in the state of Balinese, I mean, there's, there's really not much in the way of like road infrastructure.
00:08:52
Speaker
outside of the main city, Denpasar. It's like every road is backwards. It's dangerous to even walk. It's not laid out anything like a grid. It's very much
00:09:07
Speaker
So a lot of traffic gets concentrated on very, you know, key roads. And when you're trying to travel parallel to the beach in certain areas, you have to go through what are colloquially called shortcuts. And these are essentially like concrete platforms, like five feet above the ground passing through ice patties. And they're super narrow. And if you can't, there's no shoulder on the side. So what happened is you will simply fall five feet into a rice paddy and good luck getting whatever your vehicle is out of that. Oh my God.
00:09:38
Speaker
The worst way around. Just to get to the beach.
Environmental Issues in Southeast Asia
00:09:42
Speaker
Yeah. But on the guardrails are like tests in your software. Is that the way that you wrote a blog post saying, hey, how tests are going to mess up your design? They're like, no guardrails, you should know how to drive. You can go faster with no guardrails. Right. Right. It's the empire style of road design.
00:10:02
Speaker
They never have any guardrails in Star Wars and everybody's always falling off. Same philosophy. Leave this little kind of mind. No one will ever get it. They'll never work out how to hit this very specific little point that will blow up the entire star. Leave that open. They will never know about this.
00:10:28
Speaker
So yeah, we took a little bit of time to travel and then hit Thailand. And the food is better. It's a little more walkable. We'd like to stay here if possible, but we'll see. And whereabouts are you in the Thailand? The original digital nomad place, Chiang Mai. Unfortunately, our timing is terrible. We're here in what's colloquially called the burning season, where a lot of the agriculture here is slash and burn.
00:10:56
Speaker
So when the fields are done to get the nutrients back, they just set it on fire. Bali did the same, but apparently it's really bad up here and the pollution is just staggering. Yeah. I think we had the same thing in the north west of India, like near Delhi and that place. During the season, basically, they burned all the things. And then because Delhi is already crazy populated with pollution anyway,
00:11:26
Speaker
So there is a thick layer of pollution on that level that even the smoke can't penetrate that layer. So everything is just like a, yeah, it's like a local sealed dome of carbon dioxide and all these gases that people suffer crazy.
00:11:45
Speaker
It's something similar because of the mountains here, they concentrate the pollution. It's like a landscape. Yeah. Actually, I was in Barcelona recently and apparently the air quality in Barcelona is really poor because of the mountains around it that it sort of ends up trapping all the pollution that comes off the vehicles and it just can't escape. It's just so trapped by the huge mountains inside. So yeah, it was a bit weird.
00:12:14
Speaker
It's really anticlimactic, right? Because in India, the mountains basically stop and they're the ones causing the seasons and monsoons and everything in India, because the colder air is being stopped by the mountains and then it bounces back and then we have two monsoon seasons because of that one. They were very, I mean, that is the reason why we have two growing seasons, especially in the northern regions of India that is so fertile, because the mountains Himalayas are basically blocking the sea air coming from
00:12:42
Speaker
east and west from the sea and that's why we have the nice monsoon seasons and then growing seasons and now they're also the reason why all the pollution is being blocked and you know fucking up there everybody was living there which is yeah sad
00:13:03
Speaker
I don't know what to think about. I think those are bigger problems than not having tests in your softwares. I think it's... Yeah. I mean, I was listening to... I was listening to Zach's podcast when you interviewed Tillman,
Software Efficiency and Ethics
00:13:20
Speaker
part of his prep for this. I'm like, I should probably know what my predecessor said on this podcast. And he was mentioning Brett Victor and
00:13:28
Speaker
But Victor is partly an inspiration for what I'm doing now, or rather, not inspiration is the wrong word, but more like the impetus. So I like doing very abstract, high-level programming as part of what drew me to closure. But at the same time, the part of me that's concerned about the state of the environment and the world is also concerned about software efficiency. How many cycles are we just burning and wasting?
00:13:56
Speaker
Some of these things like I refuse to really participate in proof-of-work cryptocurrency because I consider it almost an ethical use of electricity at this point. But I've also been thinking about how can we cut down on the cycles we use when we stop wasting energy in the CPU. Stop all AI, basically.
00:14:20
Speaker
Yeah, yeah. Sorry, but yeah, I'm not sure about that leap. But it's a very crypto like waste of energy in many ways, doesn't seem to have many good use cases. But okay, let's let's put that one till the end. I don't know what we're gonna do.
00:14:42
Speaker
Please go ahead. I'm fine if you want to talk about AI. That's good. I think that's the burning topic these days. Literally burning the planet top.
00:14:52
Speaker
Yeah. I was playing around with some of the stable diffusion stuff recently. It's definitely wild, which you can get it to do. Have you guys messed around with it yet? Yeah. All I do is just ask for, give me an image of my dog doing something, my dog's race doing something, and then it just gives me some weird looking wonky dog and I get bored and then I move on. Well, we're all prompt engineers on deaf and now. Fully certified prompt engineers.
00:15:23
Speaker
I was trying to get it to generate a business logo. And I had very mixed results with that, but it did generate something interesting. And in the interest of time, I just kept it rather than keep fooling around with it. But I have a whole bunch of publishing and art friends back in New York who were convinced the sky is falling. He's an anxious fellow, so he's convinced the sky is falling anyway, but it worked.
00:15:53
Speaker
But there is certain types of either images or text that can certainly be generated using this one. I did try a couple of things with the new chart GPT-4 because we are all technical people. I keep asking technical questions like, how do you
00:16:10
Speaker
deploy Helm Chart into Kubernetes and it keeps just giving an example of either nginx or WordPress, then you can literally see just read it from there. And I'm like, no, not WordPress. I just want generic, write just generic Helm Chart installation. Like, nope, this is how you're going to install nginx again. Like, no. So I think it's not for that kind of use cases probably. It's like a very sophisticated parrot at the moment. Yeah.
00:16:37
Speaker
The problem is the technology is just going to get better and better. It's just reading about universities being terrified that... Oh, yeah, yeah, yeah. It wasn't just they're terrified of plagiarism by the use of chat GPT. A bunch of researchers submitted an article on the ethics of AI and they had it written by chat GPT and it passed periodically. And it was accepted. Yes. Yeah.
00:17:02
Speaker
Yeah.
Plagiarism and Education Value
00:17:03
Speaker
I mean, I really miss not having like, you know, GPT when I was doing my MBA, because they asked like, write 2,200 words on something. And because I mean, I'm an engineer, and I'm, I prefer writing like bullet points, like four of them, and then that's it. I don't need to fill it up with words. And my God, damn it, 1800 mark, I have to reach 1800 marks somehow. And then you keep adding like, and furthermore, based on the previous paragraph, keep adding these filler words and everything.
00:17:31
Speaker
Jesus, I think all these kind of things will be super confusing for people who are Driving their education based on writing essays and writing these things Well, that's gonna be shitty You know, the thing is like pleaturism in higher education has already been like a really big problem for a while anyway, though There's before chat GPT. There's been like, you know essay mails where you can just pay to have somebody write an essay. Yeah, so I
00:18:02
Speaker
So real quick, I went to school for computer science, but I was also really interested in the mind. So after a few years at startups, I moved up to New York City and started volunteering at a lab that studied emotion and memory. And eventually, I became a research assistant, and then I became a grad student at Columbia doing cognitive neuroscience for consciousness. And as part of that, I had to TA a course every year.
00:18:28
Speaker
And I found that rather depressing because this is essentially an Ivy League institution. The students here are supposed to be some of the best in the country. And I can't count the number of plagiarism and cheating. I found it very depressing.
00:18:48
Speaker
Because I always came from a model that education is not just about attaining credentials, it's about learning something.
Education Costs and Global Perspectives
00:18:56
Speaker
And then the most egregious example I came across was somebody who copied a paragraph out of Wikipedia. And I had enough other stuff to do as a grad student and we didn't have sophisticated anti-phasurism detection tools at the time. I found out they copied a paragraph from Wikipedia because they didn't change the font or the background color.
00:19:17
Speaker
So in the middle of this essay of like black on, you know, white, is this gray paragraph with a different plot? And I'm like, well, this looks suspicious. So I, you know, googled the words involved. And the first thing it turned up was Wikipedia. I reported it to the professor, but, you know, the professor, he, he was an adjunct who enjoyed teaching. And I don't think he really wanted to like,
00:19:45
Speaker
get involved with the dean or helicopter parents or anything like that. So I don't know what came in. I think that the incentives are wrong, right? Maybe that is the reason, that the people who are getting into the education are not getting for the, as you mentioned, because they have interest in something or because they want to learn more about something.
00:20:07
Speaker
It's fairly, everything is rigged towards, oh, this is a carrier path. Everything is linked to a carrier path. That's super annoying. And then on the other side.
00:20:18
Speaker
I was going to say, even if you don't take the attitude, it's super expensive in the US to be educated. So if you don't pass, then there's a huge cost of failure. So I think that's another problem with incentives over there. Like over here in Belgium, it's like a thousand a year. Yeah, it used to be like that in the US like 50 years ago.
00:20:45
Speaker
I'll give you an example, right? Like I'm doing my psychology degree as a partial part-time thing from UK. So I finished half of it. And I thought because I'm a death citizen now, I'm going to apply to the University of Amsterdam because it's closer and I would like to do it in the Netherlands. And they tell me, okay, go through this exam to entry. And then I have to prove that I can study in English by taking an English test.
00:21:11
Speaker
And I have to prove that I said, look, I have a Bachelor of Electronics Engineering, basically. I know some basic calculus, and that's not a problem. They're like, no, you have to take an
Psychology and Academic Challenges
00:21:22
Speaker
exam to prove that you have the elementary mathematical understanding.
00:21:27
Speaker
and English. I'm like, dude, I just did my MBA, like two years of writing shit in English and then passing in the Netherlands, not even any other country, 50 kilometers from your university. They're like, no, that's not accepted. You have to go through some, I don't know, TOEFL or one of these exams and apply that one. Okay. You know what? I finished half of the psychology degree. Is that worth anything? Like, nah, no, no, really. So take the exam, take these two exams, then you apply. All right. Come on, man. This is,
00:21:57
Speaker
Okay, fine. I guess I'm going to go through this charade. It doesn't make any sense because they don't want to make any exceptions. Nothing. Everybody falls through the same thing and then you just do this. And I'm not trying to be a psychologist or a neuroscientist. I'm just studying because I'm interested in this shit. I can't make a career out of it being a therapist. I'd rather write shitty closure code than just stay in that domain. What in psychology were you interested in studying?
00:22:28
Speaker
Well, I mean, it was just like reading all the pop side books initially from, you know, all the, you know, the general New York Times sort of books. And then now I thought, okay, I want to get a bit more into biological psychology side. But then the first degree is always going to be holistic approach, right? So it's going to be more about both biosocial and psycho thing, the whole area.
00:22:52
Speaker
So I'm not sure what, I don't have any big goal. Like I don't want to become a therapist or anything, but I just picked up something like psychology and counseling and that's what I'm going to do. So I just finished like, I don't know, around half a year, half the course in the bachelor's degree. And then I thought, okay, I'm going to switch to something else because it's going to be closer and it's a nice university and it's going to be fun to do this.
00:23:18
Speaker
They're like, fuck, you need to do all these exams. Jeez, god damn it. That's another weird part. So many things wrong with academia. Obviously, I'm not still an academic, even though I was working on PhD, which actually leads into my closure origin story. Exactly. That's what I was about to ask. How did you come from? You started from this, and then, yeah.
From Academia to Programming
00:23:45
Speaker
So some of it was I was unhappy with academia in general. I felt that like what we were learning in terms of perceptual awareness and consciousness was very far away from answering things like the hard problem of consciousness, like what makes red, red, like what is the experience of red? Like I can say stuff like, ah, well, you know, there's associated activations in, you know, area V4, which is the low level color area and the visual system. And that's not really, you know, very satisfying.
00:24:16
Speaker
Some of it was I saw my advisor writing grants like mad trying to get tenure and it looked pretty unpleasant. Some of it was that my data wasn't doing so hot. And so I was like, well, do I want to just become a professor in some random college town somewhere in the middle of America? And it's like, I don't want to do that. I'm kind of tied to New York at the time.
00:24:40
Speaker
But what really, the moment that the straw that broke the camel's back, so to speak, I was with some collaborators in the Netherlands at Nijmegen, actually, I read about university. And it was like the perfect environment, you know, just like free scanner time, no teaching demands. It was the summer. It was, everything was perfect and I still wasn't really happy.
00:25:07
Speaker
Um, and for my then girlfriend's birthday, uh, I decided to mess around with, uh, this new language I'd been hearing about. So this would have been around 2010, I guess 2009, 2007, somewhere around then I'd have to like double check. Uh, I had been, you know, I'd come from like computers and I was still reading hacker news and I'd come across many of Paul Graham's essays and I took a look at arc and that wasn't really going anywhere. Um, and so the other.
00:25:37
Speaker
Closure was the big up and coming list that was available. And so I wrote her a program that would take any sentence you give it and display the words like a crossword puzzle, assuming that the word could be laid out like that. And I did that like, you know, one weekend for her birthday. And I had a lot of fun doing that. That was the moment where I was like, okay, I need to leave academia and probably return.
00:26:02
Speaker
So, oh my god. But it would actually take many more years before I would work in closure full time. I did a whole bunch of other stuff. But you were originally trained in programming and computer science? Yeah, yes. So I have a degree in computer science from UVA. I graduated in 99. And I went to work for a couple startups. They did not make me filthy rich.
00:26:29
Speaker
Um, and then I was sick of being in Virginia and a lot of my friends moved to New York. So I went there, uh, entered academia. So I leave academia and, uh, also my girlfriend broke up with me that same summer. Uh, and so all of a sudden I have like no ties after that closure program as well. I mean, you know, this is, that's a harsh review. Yeah. Um,
00:26:57
Speaker
So first thing I did was go on a long meditation retreat for a month and a half. And then I came back to New York and started working as a consultant and
00:27:11
Speaker
All my time in academia was spent with MATLAB, and I absolutely loathe MATLAB. But it is very popular, especially in neuroscience. So dominant open source packages are all written in MATLAB. And so I didn't want to do that. And I basically had to fall back on my earlier era of the web, which is essentially PHP. So I took a few jobs with that. And of course, I wasn't really happy doing it.
00:27:39
Speaker
And at a certain point between gigs, I was like, you know what? I'm just not going to take another job until it's in Clojure. And I kind of responded at random to the CEO of a startup called SwarmFI, which is a ClojureScript startup. They're still around. I eventually had to lead because the lead's thunder got a little weird and cut funding. So that kind of sucked.
00:28:08
Speaker
But in the middle of my time working at Swarmify, I kind of drunkenly responded to one of David Nolan's tweets on a Friday night. So he had a co-working group called Kitchen Table Coders and he inhabited some loft in like Brooklyn. And I looked it up and it was only like a 15 minute walk from where I was living in a different loft in Brooklyn.
00:28:30
Speaker
Um, and so I'm like, yeah, you know, I need a place because I can't work at home. I've been doing remote work for like forever and I just cannot get anything done
Career Transition and Consultancy
00:28:40
Speaker
in the house. It's not about having like a home office. It's just about like the temptation. Like I can just leave my home office and I could be playing the switch or talking to my wife, you know, any, any of these other things I could be doing. So for me, I always have to like go to a coffee shop or just get out. Um,
00:28:58
Speaker
And so I went to, I started working at Kitchen Table Coder and I got to know David. And as I was wrapping up, David was working on the Danish tax project, which I believe Vijay you might know about because you used to work at Flexiana. We had a bunch of Flexiana guys come through there. Yes.
00:29:23
Speaker
You have that look in your eyes that says, oh, yes, I've heard about this project. Yeah, I've heard about it. And because Cognitech, when it was essentially becoming new bank and they wanted to offload these projects to some other companies, usually, I think. So they wanted to interview me. And then I did the interview with you folks. And then I decided, well, this project is not for me. This is really bureaucratic.
00:29:51
Speaker
Danish development thingy. And if I look up the history of the project, it's, yeah, I'm like, no, I'm not going to do this. So I backed off and they're like, no, I'm not going to. And also because they said I need to be in Copenhagen pretty frequently or something. I thought I'm not going to do that. I mean, I'm not going to fly from Netherlands and then every week or two days, whatever.
00:30:13
Speaker
Um, so I decided I'm not going to do this. So, uh, that makes sense. It was a fun project. Yeah. It had a lot of turnover is a lightweight. Um, but yeah, David suggested I interview with cognitive tech. Uh, I didn't really technically interview with cognitive tech. They, they brought me on as a, I mean, I did have an interview, but I wasn't not to be an employee. Uh, I was essentially an outside consultant brought into, uh,
00:30:43
Speaker
essentially augment their staffing. And in fact, I mean, that project had to have had at least three or four different agencies because it just required so many people. So yeah, that was lucrative, if bureaucratic and frustrating in some ways, but the people were good. Yeah, yeah. I mean, it's usually never the people, right? I mean, people are always fun. I mean, if I look back,
00:31:12
Speaker
every good project or the bad project, it's usually the style of the requirements. It's usually the style of the management. Other than that, with the team, there was never an issue. The team, there is always fun, good people sitting together trying to solve something. So I think the intention in every project is good, but usually it's done by managed by something in a weird way or rewriting some shit for the sake of rewriting and all sorts of weird shit shows up. Yeah.
00:31:39
Speaker
Well, let's say I've never been on a project before that, that had switched databases. Like I always heard about that as a theoretical thing, but realistically, I always assumed, yeah, changing your database seems like such an incredible pain and risk that I'd never seen it happen before. And for that project that it actually happened three times.
00:32:00
Speaker
three times. Yeah, and also the, it's not switching from, you know, let's switch from post-customize equal, right? It's like a paradigm shifting database shift. Yeah, before me, it was a dutomic on Cassandra and then they found that's kind of slow. So then they ditched dutomic and then with Cassandra, they found reads were slow because Cassandra is designed for write ingestion. And then for a brief shining moment, we were just going to use boring Postgres.
00:32:30
Speaker
But then for legal reasons, it was determined that a lot of the case data had to be stored with this other vendor known to the dean called WorkZone or something like that. And so we essentially had a split database. We were storing half of it through Microsoft OData calls to this other database. And then half of it was in an event system stored in Postgres.
00:32:56
Speaker
Let's say I learn not to do that in future endeavors. Hey, there is always learning in every project. I remember when I was in this process of talking to them, I asked, how is the development enrollment like? And once that was explained to me, I was like, no. I can't deal with these many weird shit floating around, some postgres I need to connect to. I cannot have anything locally.
00:33:24
Speaker
Anyway. Zooming out a little bit though, I think what I find interesting about this is that it's like whenever we're doing closure stuff, that's the nice part about any project.
Programming Languages and Happiness
00:33:39
Speaker
It's all this other stuff around it where you discover the architecture is wrong, the database is wrong, or the connectivity is wrong, or the
00:33:47
Speaker
The thing that you're trying to build is just not actually needed by the business. These are the kind of things which kind of like make everyone miserable essentially.
00:34:02
Speaker
It was that your experience as well, essentially that it's like, because I think that's what you're sort of a little bit inferring VJ is like the building thing, like the making of the thing is usually because engineers like, like essentially get told what the requirements are, and then just do their best to make it happen. You know, and that's the fun part. It's it's all this kind of like, higher level stuff that is essentially a problem. Oh, yeah. You know, it's difficult because
00:34:29
Speaker
It's difficult for me to accept as somebody who thinks and feels very strongly about the language I use. But that's actually really a small part of what makes success in a project. I mean, people have been writing great stuff, low-level stuff in C for decades.
00:34:50
Speaker
Yeah, but I think it's a difference between, for me at least anyway, was writing stuff in C or Java or whatever and experiencing a much more expressive, like, yeah, I would say that a more joyful way of writing programs in closure.
00:35:09
Speaker
than you can in these other languages. I think that's the difference. In the end, it doesn't matter what the language is, you'll get it right or wrong. That's for sure. But as a participant in the process, I much prefer doing it in closure than in other languages. For sure.
00:35:28
Speaker
I forget which study it was. They were talking about programmer happiness and go and close your routinely rank very high on those metrics. I've been thinking about what actually unifies those two languages in terms of programmer happiness because they seem at the surface very different. It can't be the use of communicating sequential processes or anything like that.
00:35:51
Speaker
I think it has to do with the very tight feedback loops involved, because Go has a notoriously fast compiler to the point that you can kind of explore the compiler errors you get in an almost REPL-like manner. It can happen that fast. And then obviously, lists do have a REPL, and so you can do that kind of exploratory programming.
00:36:16
Speaker
And people talk about this, but I actually feel like that's kind of an undersold superpower of, you know, really high quality rebels like this have. And maybe Smalltalk had this as well. I've been curious that I've never played around in Smalltalk. And I think it's because there's just so much less to keep in mind when you have a very tight feedback loop. There's fewer things that can go wrong, and anything that does go wrong, it's much easier to identify what went wrong and fix it.
00:36:45
Speaker
essentially the longer between when you create an error and when you fix it, the more things you have to keep in memory, more things that could possibly be the cause of your problem. Yeah. You know, obviously, yeah, obviously something that makes it as a
00:37:03
Speaker
As a counterexample, or maybe just a point of an argument, don't you think it's the similar way if you have, quote unquote, sound programs in the languages like Haskell or some other way where you can tighten the program with types and you know that part is going to work. So you can focus on the other part. I'm talking about how much stuff that I need to keep in my mind when I'm working on a piece of code.
00:37:32
Speaker
with the closure, as you're mentioning, I can quickly bounce back between different things. But don't you think it's the same principle applies in other situations? I haven't really done any Haskell programming. I can't really comment on what it is like as an experience. The impression I get from
00:37:57
Speaker
Generally, closures are not going to find it too controversial to say that, obviously, types can't solve all of your problem. Every bug you've ever written has made it through the type system. It's an open question. Obviously, if you're getting the business logic wrong, you may or may not be able to encode that in types.
00:38:17
Speaker
I can't actually say about Haskell. I do think, though, that types add an extra burden on the working memory system. And I'm not actually sure if it's working memory that's even really the issue.
00:38:32
Speaker
Every so often, I'll try to marry my two major experiences, the cognitive neuroscience side and the grammar side. And there is shockingly little written about that. Very few people have truly explored the psychology of writing code. There has been some stuff written, and it's almost always for newcomers. What is the process by which newbies start to learn to write code?
00:39:00
Speaker
And some of it is, you know, they'll analyze like the corpus of code written in GitHub and try to make inferences from that. But in terms of actually being like, you know, what's going through your head when you're writing code, there's very little written about that. Do you think like programming needs to be added to DSM 8 or something that this is going to be one of the... I think a lot of us have something wrong with us, for sure.
00:39:26
Speaker
Something ain't right. I'm going to check everything. Yeah, exactly. Then it's a very specific way of thinking our process is happening in the brain to become a programmer. But don't you think it's an interesting question in the sense that
00:39:41
Speaker
When people are designing languages, they very rarely have this idea of joy in their mind. They're trying often to solve one thing. I mean, I know Matt's famous, he says that Ruby is for programming joy. But can he prove that? I don't know. I mean, obviously a lot of Rubyists think that it's kind of like
00:40:01
Speaker
That's a lived experience is true, but I've done Ruby and I don't think it's that joyful.
Joy in Programming: Myth or Reality?
00:40:07
Speaker
It was if you're coming from Java and then first time you build a Ruby on Rails application, you're like, holy crap, this is amazing. That's Ruby on Rails. No, but you're starting with the Rails way of building
00:40:24
Speaker
is enabled by the features in the language, like all the monkey patching that you can do, all the syntactical things that you could do, the whole active record stuff analysis is enabled by the language features. I understand that, but my point is that programming Ruby per se
00:40:38
Speaker
is not, it wasn't specifically joyful for me. I mean, I dropped out before Fairingham rails. So yeah, that's a different way of programming something. That's a really interesting point because then joy is a subjective experience, right? So I cannot design for your joy.
00:40:56
Speaker
So how would somebody design a programming language? It's easy to say. But my point is that he explicitly said that that was something that he was going for. But the interesting question to me, moreover, is like, can it even be true? Can the act of programming ever bring any joy?
00:41:14
Speaker
To your point, I think the answer is definitely yes. There's a long literature on, you know, what's called Flow by Michal Czejal, totally butchering his name. But he's, you know, studied the concept of flow like being in the zone, mostly in terms of things like music performance and athletics.
00:41:35
Speaker
And I don't know if anybody really looked at that encoding, but I definitely feel in the zone sometimes. What I find really interesting is why does it take so long to get into that?
00:41:46
Speaker
If I sit down at my laptop and I start coding, I've got to open this, and I've got to open these tabs, and I've got to review this documentation, and what was I doing here? Oh, yes, I need to look at the bookmarks I saved and double-check that. And it can take a good 20 to 30 minutes before I've really settled in. And then I can code with maximal performance, and I feel like I am in the zone. And then sometimes the hours can just fly by when you're doing that.
00:42:15
Speaker
And I would love to see what exactly affects that. For example, to the extent that... So it can't possibly be working memory that we're relying on, because working memory itself is super, super limited. There's a very famous site paper by George Miller, I think, called Magic Number 7, plus or minus 2, being the number of things that we can hold in memory.
00:42:37
Speaker
And that's actually the number of chunks that can be held in memory. And it turns out in later research shows it's even lower than that. It's more like 5 plus or minus 1. But we don't actually hold everything we're doing in working memory. So I think it's more about prepping the parts of the brain that are going to be called on, sort of like spreading activation by thinking about something. You are prepping it to be used as necessary.
00:43:05
Speaker
I'm not exactly clear how it all plays. A lot of it is below the surface. It's subconscious. But to the extent that I think working memory is taxed at all, I keep fantasizing that one day I will get around to writing an Emacs plugin that actually keeps track of the number of free variables at any given line.
00:43:26
Speaker
And it will highlight the line based on how many free variables there are. And it gets progressively redder and more alarming looking as you keep adding names that you have to keep track of. And I don't know where global variables would fit with this. That's a nice segue into my question. Do you use Emacs or some other shit? Apparently, it looks like it's only Emacs.
00:43:51
Speaker
No, I used to use Emacs. I stopped because... Because it was fucking awful. There's things I love about Emacs and things I love about Emacs. So it's a bit of a mixed bag for me.
00:44:10
Speaker
I stopped because I was doing this during the Swamify days and it's all 90% closure script and Cider had some sort of weird closure script issue for like a month. It was driving me crazy. And I just switched to IntelliJ and cursive.
00:44:25
Speaker
I just haven't bothered to switch off yet. And it's been helpful, especially for Java interop, which I've been doing more of lately. There's an awful lot of that in Alice. But coming back to your point about the free variables and stuff, are you talking about the number of let bindings that you have, or the number of vars, or?
00:44:50
Speaker
So I can like interpret it. No, no, you have the right interpretation. Left-binding would be obviously more salient because they're local, but technically any global arc would theoretically be relevant. But I think intuitively like our brain is somehow discarding most of those because they're just not relevant.
00:45:17
Speaker
So yeah, I think it would be a cool Emacs plugin. I have no idea if it would actually reveal any deep insights into how programmers code would be interesting. It does match some of our intuitive understanding that shoulder functions are easier to understand, stuff like that.
00:45:35
Speaker
Yeah, but I do think you're right that I mean, basically, if you're operating on inputs to a function, and you have some left bindings, and you're doing some actions, and then you return some values, and it's all kind of like right there on the page in like 10 lines or 20 lines, that is so much, so much more rockable than the side effecting systems. And I think maybe functional programming in general, if you if you kind of like stick to that pattern will be will be sort of
00:46:05
Speaker
Let's say more comfortable in the brain, you know, I don't know if you feel like that. Yeah. Pure functions are obviously, you know, much easier to reason about. Um, and I mean, going by that working memory example that you can say the answer is, well, you know, that once the function is returned, there's nothing else you need to keep in mind about that function. You know, it hasn't somehow screwed over another function and another thread somewhere else.
00:46:34
Speaker
Anyway, this is just my head theory. Yeah, but honestly, this is huge. When you do Java Interop, you realize that you're looking at some frameworks, Amazon frameworks or Spring or whatever, and the side effects fucking everywhere. And it's really difficult to grok what the hell is going on.
Functional Programming vs Java
00:46:53
Speaker
And I find that incredibly stressful, to be honest. It's one of the reasons why I don't like Java anymore or those kind of non-functional programming environments, because
00:47:04
Speaker
It just hurts my head too much. Going back to the joy aspect of things, that is misery. That is immiserating to me. Maybe we can argue what joy is, but I know what misery looks like. The joy is absence of that misery. Well, I'd like to think it's a bit more than that, but at least I know what pain is. Closure is the programming equivalent of stop banging your head against the wall.
00:47:33
Speaker
Yeah. This is like Sam Harris concept. Sam Harris is somewhat discredited in general. I think there's a neuroscientist. But he has this idea of that joy is the absence of pain or that our goal in life is to reduce the amount of pain in the world. I think this is all horseshit, frankly. But I would certainly say that the idea that
00:47:56
Speaker
Absence of pain. Absence of pain does not equal joy. That's for sure. So I think there's something more. But I definitely think that there are situations where you definitely experience misery and upset. And like I said, that's side-effecting things in general and free variables and all these things where you have to play a computer more.
00:48:17
Speaker
Those are things which I think, like, erode joy pretty quickly. Oh, for sure. And that's going back to Brett Victor, actually, isn't it, that you were talking about right at the beginning, you know, how Brett Victor is saying we want to use the computers to let them play a computer, let them be computer and let us reveal through systems, through, you know, work, what is important and interesting. Right. I wasn't thinking about
00:48:45
Speaker
I haven't seen that one in a long time. The Victor information for what I'm working on now is more about this whole website for exploring ideas for saving the environment. Oh, okay. Yes. That was the one in particular I was thinking of. And there's inventing from first principle. That's the interactivity one, right? The inventing, inventing on principle. Yeah, inventing on principle. Yeah. That's the one I'm referring to.
00:49:14
Speaker
Sam Harris said that joy is the absence of pain. No, no, he said that if you want to work out what should we say, what a non-Christian version of morality is, then it's about reducing pain in the world. I don't think that was the
00:49:43
Speaker
I don't know, maybe I'm talking from the purely Hindu version of it. I mean, it's more about accepting the pain rather than actually
00:49:52
Speaker
I don't know, trying to remove the pain from the world, because we all also talk like, oh, everything is my ass. So fuck it. So, you know, that's what would be that that pain is mandatory, but suffering is optional. Yeah. It's like the actual pain you experience. And then there's all the extra that you heap on in your head.
00:50:15
Speaker
Exactly. Yeah, I think the... Maybe that's the Eastern... I think it's like a lot of... it ends up being a sort of debate bro type argument where you put some like planks on the bottom like that and then you end up saying Islam is terrible, you know, and it's like, hang on, mate. You've carefully constructed this sort of these set of planks and then you then you're now criticizing
00:50:43
Speaker
every religion or a group or whatever. And I think that's the thing that I didn't like about his argumentation really, is that you kind of work on this like, difficult to argue principle that pain is bad.
00:51:01
Speaker
But you can't automatically leap from that to certain activities that in some subjective manner might cause some pain to some people is bad. Well, I think what we're trying to do is pain is bad and then switch to closure is better. Just leap from pain is bad to closure is better. Yeah, I think that's really what it is. Yeah, that's exactly what it is.
00:51:28
Speaker
There is a tagline to increase adoption. Closure beats pain. I like it. Less pain with closure. Yeah. Oh, it's like the anodyne, the anodyne of programming, or what do you call it in America, where you have this like, headache, tablets, paracetamol or something. Yeah. Oh, yeah. Yeah. The paracetamol of programming. It's not the panacea, but paracetamol. Right, right. You call it a sedimentary thing for some reason. I have no idea why.
00:52:00
Speaker
Ibuprofen maybe? Is that an international? I think the other generic names are standardized, but for some reason it's paracetamol in most of the world and acetaminophen in the United States. I can't explain that. Damn stupid. Or is it non-metric system, I guess. I like knowing how many furlongs were in a pint though. It is.
00:52:30
Speaker
Anyway, but so going back to your journey.
Entrepreneurial Ventures Post-Cognitect
00:52:35
Speaker
Yeah, exactly. So are you still working on the Danish project or that was part of the part of the journey?
00:52:44
Speaker
Yeah, Cognitec wrapped up their involvement shortly after the new bank acquisition was announced. They pretty much round down all of their consulting. And I stuck with them for another over half a year. I wasn't sure what I wanted to do next. But no, I left that right around the same time. So almost a year ago now, a little over a year ago, was right when I left Indonesia, I left that project.
00:53:14
Speaker
And so, um, what I had been working on all of last year is, uh, I mean, I don't know how much your listeners want to hear about, uh, starting up, uh, you know, software as a service versus. We definitely want to hear about this, of course. Well, it's, it's, you know, it's definitely a bit of a stretch. I've always enjoyed coding and, uh, it's always a little skeptical of those marketing and sales guys, but I find myself in the position of having to learn a lot of that stuff.
00:53:43
Speaker
So the first most painful lesson I learned last year is you need to find the problem before you write the solution. I think a lot of engineers who go solo have this exact problem because we fall in love with the way we think things should be and we have the technical chops to make that happen. So in my case, I have terrible hearing. I was born absolute crap for hearing and I've worn hearing aids for a long, long time and I pretty much watch everything with subtitles.
00:54:12
Speaker
And when I was in the middle of the Danish project, I started like a little side venture, not really venture, a side project to start to bring that technology to the browser. So, for example, YouTube has like a closed caption button that will, you can click it and it'll show the caption that will even machine learning generate them if they don't exist yet.
00:54:35
Speaker
Microsoft Teams has a button you can push and it'll try to caption everybody in your team. So the technology for that is there. They've been offered of cloud services for a few years now.
00:54:46
Speaker
And I thought, okay, well, I'll just take any audio source running through the browser, do a live transcription of this and stream it as subtitles to be displayed. And I was working on this on nights and weekends. I still had my day job and my wife and friends and all sorts of other things going on.
00:55:06
Speaker
And my old boss from swarmify came back one day and said, Matthew, I'm so sorry. I'm like, sorry about what? Did you see the announcement? So Google basically announced they were going to add that for free as part of Chrome, as part of their accessibility measures.
00:55:24
Speaker
But what I was building was actually going to be a little bit nicer because I was doing segmentation technology to figure out when and where to display it. So what Chrome does is just an endlessly scrolling wall of text. And what I was going to do would be more like traditional movie subtitles. So it would be less of a cognitive load.
00:55:44
Speaker
It would've only been like a few percent better and probably nobody would have wanted it. So I shelved that for a bit. And then last year I started thinking about another idea I'd had, which is a way to improve transcription accuracy by essentially looking at multiple sources and then looking at where they differ.
00:56:04
Speaker
So I figured it was the kind of thing that the really big companies like Microsoft and Amazon couldn't easily do because it would involve having multiple teams all write the same models and then do like majority vote. So if two out of three transcriptions agree on a word, that's the word you go with. And then the odd one just gets tossed. And this would actually work to improve accuracy as long as their errors weren't correlated, but because
00:56:31
Speaker
They're coming from different companies, which is the way I was building this. Essentially, we talked to Google, Azure, and AWS and compare their outputs to the extent that they weren't correlated. It would work out really well. Mathematically, I'd have been able to eliminate about 40% of the existing errors in transcription.
00:56:51
Speaker
The problem I ran into was, even though this would have worked, nobody cared. I started forcing myself to actually do the customer research part that many startups want to avoid.
00:57:10
Speaker
But I started talking to people who use machine learning transcription tech, or even human transcription tech, and they kind of fell into two camps. One camp were the people who had super high accuracy needs. We're talking fields like medical fields, legal fields.
00:57:30
Speaker
They've done transcription for a long, long time, and they have specialist human transcriptions who finish it, and they need very high accuracy because to get something wrong, somebody will overdose on their drugs or get sued. The other camp were people who were using it for whatever. Maybe they were transcribing their notes, maybe they were keeping minutes from a meeting, and the occasional...
00:57:56
Speaker
Do you offer transcriptions of this podcast? No, not really. I did try the Amazon recognition thing last year, two years ago. It was shit. I mean, it couldn't understand my accent, obviously. And then every time we say closure, it's closure. And God,
00:58:18
Speaker
And all the bleeping stuff that we say that it can't try, it can't put the thing out. But actually two days ago, Dieter, I think, on Mastodon, he mentioned he was using some service to do the transcription. That was pretty reasonable. And there is also an open source one called Mac Whisper or Mac Whisper. That's on my to-do list to try. But yeah, anyway, that's as an aside, yeah.
00:58:47
Speaker
I mean, if you're looking for one, Rev.com was a competitor that seemed to be pretty good. No, we'll just use yours. No way. Well, I shelved it, unfortunately. But yeah, the people for whom they don't need super high accuracy, they just didn't care. If it was with a clean recording, and no accents, they were probably going to get something in the range of 95% plus accuracy.
00:59:13
Speaker
And whether I moved the needle to 97% accuracy, they just didn't care. So I eventually set that aside and started thinking about other stuff.
Idea Journals and Business Ideas
00:59:26
Speaker
A few years now, I've kept an idea journal on and off, mostly off, but I do just keep any random idea, just write it down, and many of them are absolutely terrible. And in fact, in order to get a lot of them out there, you have to have plenty of terrible ideas, but some of the good ones will resurface. I got this idea from, what's his name? Derek Sivers.
00:59:53
Speaker
I'll figure it out later. But he suggested this. And one of the ideas I had had to do with a performance system. So this kind of came back to wanting to do something for an environment and also finding something that I feel like is kind of neglected and wanting not to waste CPU cycles, even though I'm using a high-level language. I also like making things randomly efficient.
01:00:22
Speaker
you know, upgrading my packages in the hopes that it'll be faster. I can't explain that. It's some sort of weird little compulsion.
01:00:30
Speaker
But so I thought, well, why aren't people keeping track of their performance budgets? At a high level, everybody knows they should. And for something to be interactive, it has to happen in less than like a tenth of a second. If you look at web pages like every 100 milliseconds added before the web page is ready means X amount of lost revenue or whatever if you're e-commerce, and stuff like that.
01:00:57
Speaker
But virtually nobody does that, at least not at the levels I've worked at, small and medium businesses for the most part. And my thought was, well, why not just turn
01:01:10
Speaker
unit test into performance test. It really shouldn't be that hard. And so what I'm working on, I've called it Speedarino because I'm a big Lebowski dork.
Speedarino and Performance Testing
01:01:24
Speaker
And the idea is that right now I'm building it for closure. I'm going to transform your unit test into performance tests. It's
01:01:32
Speaker
going to be built with Java micro benchmark harness. Turns out there's a lot of really deep subtleties to benchmarking correctly. Criterium is good for off the cuff stuff, but if you really need replicability like
01:01:48
Speaker
So what I want to do is, given a code base, I'll look at all of the commits and then turn your unit tests into performance tests and see how the performance changes over time. So I can identify cases where you added this, and it looks correct, but clearly something was inefficient because this function slowed down by 100%. You're using the wrong data structure, something along those lines.
01:02:18
Speaker
For the performance test, aren't there more of the context? Because I'm wondering, what is your control variable is going to be? And if there are too many confounding results, then you can't really pinpoint this is the reason. And there might be something that is masking, because I'm using M2 to do on my laptop with the performance thing.
01:02:43
Speaker
So yeah, so there's a couple issues to the replicability. Let me make sure we're talking about the same thing. Like one is ensuring that if you rerun the test later and want to compare the numbers that something in the environment hasn't changed. Is that what you're asking about? Yeah. Yeah. Yeah. Yeah. So because there needs to be reproducibility and there is a context that I need to know to know that yes, this is going to perform as expected based on the results. Yeah.
01:03:07
Speaker
Exactly. Yeah, replicability is something I'm still working on. The basic idea is just to get your own hardware box in the cloud, make sure nothing else is running on it, make sure it's always the same instance type.
01:03:25
Speaker
That's going to be the best bet. But I will admit, I still need to figure out the numbers on this to see just how replicable it is. Especially for the faster unit test, you're going to have a lot more variance just by sheer size alone. But that's part of why something like Criterium would never work. It's just going to have a little too much variability.
01:03:46
Speaker
You also need to do things like prevent save point bias. You need to ensure that function calls are monomorphic, like the JIT compiler itself can change performance numbers if you simply compile some more code. The way the Java microbenchmark harness JMH handles that is by essentially forking the Java process and only compiling and running exactly the code that you're testing.
01:04:13
Speaker
And so you can get around that if you want to defeat it, but that's not the point. But because literally nothing else other than the JMH and the code you're testing is being run, you know that the sites are monomorphic. Nothing else has been compiled that will interfere with that. So yeah, that's what I'm working on.
01:04:41
Speaker
It should be ready for beta soon, I think. I'm messing around a lot and rewrite CLJC and transforming unit test. Tricky Perch is getting the, handling essentially like all of the fixtures, your pre and your setup and teardown. Those essentially have to be moved and transformed and then also excluded from the analysis themselves.
01:05:11
Speaker
Matthew, coming back to your point about the transcribing and people not caring, what is the identified audience or what's the motivation that people have for caring about this stuff? The major reason for it is to avoid deploying performance issues to production.
01:05:37
Speaker
So you might have code that passes your test suite in terms of correctness. But if you weren't paying attention because your test suite takes 20 minutes to run, and then some part of it now takes 25 minutes and you've introduced some performance problem, you don't want to deploy that and then find out you have a problem. And production is always going to see the heaviest loads. You're not going to detect it usually on a development machine or a staging server, unless you're specifically load testing.
01:06:08
Speaker
Now turning unit tests into performance tests has some limitations for sure. Unit tests are not really designed for that kind of thing. But the alternative is nobody's writing benchmarks.
01:06:20
Speaker
testing. Nobody's doing this. No, this is certainly one dimension, right? Like the dimension that you're tackling is the unit as being converted because obviously if you zoom out, then performance would require the whole system to be tested. Like is the database that is going to be slow? Is the network or is it something else? Or my thread is waiting on something else to be available. So I think, but this is probably the one dimension. Yeah.
01:06:46
Speaker
Yeah, this is like CI for performance. With deployed servers, you're going to use something like APM, and you'll find out if a query is really slow or some server is being overloaded. But by that point, it might be too late to avoid the problem. And so this is maybe a way of inserting another step into your parallel step in your CI pipeline that can tell you, hey, this function is radically slowed down. Maybe you should double-check this.
01:07:15
Speaker
And this is your kind of like, let's say research, suggesting that that happens more than like, like dependency changes, or infrastructure changes, or like, like VGA says, someone writes about query, which you're not gonna like run in your unit tests. You know, what sort of percentage of performance comes from
01:07:42
Speaker
the unit tests, the sort of raw functions, essentially. My intuition is that not much, but I mean, you know, I'm prepared to be persuaded. Well, this could be more valuable. So, okay, I say unit tests, but a lot of people abuse unit tests for integration tests. That sense is more valuable.
01:08:03
Speaker
select your unit test. Yeah probably not but if you want other tests into your sweet then yes this will and hopefully those are more ecologically valid and will represent the code that you're actually running and give you a better idea like okay this whole you know retrieve customer or add product procedure has like slowed down.
01:08:27
Speaker
So, I mean, it's, you know, still a bit of an experiment. I don't I won't know for sure until see if anybody bites. My unsolicited MBA shitty advice would be that, you know, you don't mention unit tests, but you say that, hey, your test suit
01:08:45
Speaker
will be used as a performance test benchmark, then suddenly you're not talking about unit tests anymore. Because my assumption is that when you say unit test, then I'm thinking, wait a minute, when I write a unit test, then let's say I'm dealing with a smaller map in the unit test that I mocked. But in production, I've seen situations where a function taking like
01:09:03
Speaker
fucked enough time because in production, we didn't expect the map is going to have so many keys and so many things, whatever. But if I hear it's going to use unit tests, but rather than it says, hey, it's a CI for performance and it's going to take your test suite and then give you performance results. I think that has a different ring in my mind. Again, this is a completely unsolicited, you know, like random. I'm still learning things like how to write website copy too.
01:09:33
Speaker
I think I do mention unit tests, so maybe I could just say test. We'll see. Yeah. Because when you explained it in a way, obviously when Ray is asking, hey, is it unit tests? And when you said, wait, tests are not
01:09:50
Speaker
Not everybody just sticking to, let me write unit test, this particular small function, passing one or two. Because the quote unquote performance for those things can also be done with, at least my understanding is that using generative testing, you can actually run like a heuristic of different sizes of data.
01:10:10
Speaker
and then reduce the tree and then see what is the minimal thing that is failing. I mean, it's not testing performance, obviously. It's only testing the correctness of the function using QuickCheck or something like that. But if you say test suite, then like, oh, that makes sense because usually we
01:10:25
Speaker
One of the projects that I'm working on has Redis and all of these things. So even in the test, we bring up Redis because it's super fast to bring it up and then just test it again as that one. So it's not a unit test per se. It's actually doing the work that we want to do.
01:10:41
Speaker
makes it actually quite i'd say i agree with beach i think that's quite common you know what when you're doing like because the way docker works now you can kind of do it so it's relatively simple in these like ci systems to bring up a docker version of the database of a search engine or whatever.
01:10:57
Speaker
and then essentially just write a bunch of integration queries using the real thing. Even though it's not the same scale, I think you would generally find quite a good performance. It's a good source of performance measurement there because you will at least see the stack, even if you don't see the whole suite of data, you will see the whole stack through. I can buy that one for sure.
01:11:26
Speaker
I won't use the word unit in my sales copy. Free consultancy for the Deaf and Boise. God damn it, you don't say free. I was starting to negotiate the percentage in the startup. Okay, it's my bad. Yeah, we'll look into the deal sheet later. Anyway, but I'm curious about the technical stuff, though, because you mentioned Criterium, you know, to do the microbenchmarking and
01:11:55
Speaker
Do you see any challenges there? Because performance would mean that you need to run it multiple times, some statistical stuff that you need to deal with. Yeah. So it is essentially going to be some multiple of how long it takes your test suite to execute to begin with. So I'm not sure that it should necessarily. I've been toying around with, how do you report this in your GitHub PR? Should it be a blocker? Probably not. Or maybe it could be, but it shouldn't necessarily
01:12:24
Speaker
block before it's returned. But essentially, yes. For statistical purposes, you have to rerun, even if you're controlling everything else and nothing else is running, you can be subject to the whims of cache and memory and who knows what. And so you do have to rerun it multiple times. And typically, there's a bunch of warm-up runs and then a handful of straight-up runs. And then you just record the timing. So yes.
01:12:54
Speaker
Nice. So staying on the on the same closure side, so how did you start picking up the aleph and manifold and you know, this is something like so just this came about because so this goes back to when I was still messing around the transcription tech on the side. I had used core async very heavily on the closure script side of things. But
01:13:20
Speaker
It was the best of a bunch of terrible alternatives in JavaScript. But I wasn't feeling quite the same way about it in back-end closure. In particular, I never really had an error model. One collect-in-a-tech actually told me Corey's think is beautiful nonsense. And I haven't forgotten that.
01:13:44
Speaker
So I started playing around with Manifold, and it did things a little differently. But for the most part, it worked, and I like how I didn't have to recreate the same Take macro that would then throw if it got an exception for the umpteenth time.
01:14:00
Speaker
And so I was building that transcription tech very heavily in Manifold because there's a lot of like, it seemed to be a good fit. There are a lot of streams. I was like retrieving a URL from the browser side. And then I was like streaming the actual video from wherever the browser was getting it. And then I was then had like a processing stream where I would extract the audio and then stream that out because all of this had to happen in real time. I needed it to happen before you reached the point in the video where you needed to see
01:14:29
Speaker
that subtitle. And so I couldn't wait for the entire file to download. I had to like do everything streaming as fast as possible. And then I would be streaming out the audio to like AWS and GCP and Azure and then collecting all of those streams.
01:14:45
Speaker
And if I'd gotten a little bit further, I was going to have to make some decisions about stale data. If you hadn't reported back by, I don't know, 200 or 300 milliseconds before the subtitle was due to appear, I was just going to have to ignore that service. I never quite reached that point. But the point is, I was doing a lot of streaming work, and so I was getting very familiar with Manifold.
01:15:10
Speaker
And then at a certain point that kind of just posted that he was moving on from closure and maintaining all of his closure libraries and would anybody like to do something with them?
01:15:23
Speaker
And I raised my hand, as did like a dozen other individuals and companies. And I was the only one who showed up, actually. Really, I just sort of fell into it, because I needed to do stuff and
01:15:40
Speaker
I believed a lot in giving back to the community. That's for me what Open Source is about. And I've been the beneficiary of so many lovely libraries over the years and I wanted to share. And so I started working on Manifold at first, but it quickly became clear most people were less interested in a pure streaming library and more interested in ALOP as a web server.
01:16:07
Speaker
And so I hadn't really done much aleph before becoming lead maintainer. And it's still like a massive code base. And Zach's a cool guy, but for a fellow who wrote a book about closure style, it's very obvious that manifold and aleph were written in the early days of closure because they don't follow a lot of his style.
01:16:35
Speaker
But you have to, you have to write, you know, about your mistakes. Exactly. But like, there's a ton of death in line usage. I've never seen that in the wild. There's, I mean, and some of the Potemkin usage is also bonkers. There's some really interesting stuff going on.
01:17:00
Speaker
And luckily, it was a bit much to handle. But luckily, as people started to see that Aleph was, things were starting to slowly happen again, some people have stepped up to help out. Mostly, I forget his name. So there's Arno from Exascale, and Derek Moritz is his handle. I'm blanking on his name for a second.
01:17:27
Speaker
But they're both at Exascale and they've both been helping out a lot. And a few people have stepped up for other libraries like Manifold and Bloss and Byde Streams and Eric Assam at
01:17:42
Speaker
CLJ Commons has been really helpful. So yeah, I kind of just fell into it. And then I think, did Arno suggest you talk to me or was it somebody else? It was a colleague of Arno, Wouter, who does the audio. He works at Axelscale. Okay. Yeah, I was a little surprised, but a welcome surprise. Thank you for having me on the show.
01:18:08
Speaker
pleasure. We're always looking for interesting discussions anyway. So it's always looking for much chum to feed the audience. Chum for the sharks. But is it just the maintenance or do you have like, of course, the code quality, you know, it seems like a continuous perpetual work in every project like
01:18:37
Speaker
Yeah, none of the projects, you know, are following any guidelines to the T. Yeah. I mean, a web server is a huge piece of tech. And even though it's built off of Netty, I mean, there's just a lot just for, you know, dealing with bug reports and squashing bugs. Netty itself is quite a complicated beast, isn't it? Yeah, for sure. And it's evolving. So, you know,
01:19:03
Speaker
We'll see how that goes. So, you know, I was looking with, what's his name, the missionary guy, Leo Noel, I think his name is.
Future of Concurrency Libraries
01:19:13
Speaker
I forget who I was talking with last fall about, you know, what does Project Loom mean for us during concurrency libraries?
01:19:21
Speaker
And it's interesting because Manifold would almost certainly not be built the way it is if virtual threads had panned out a decade ago. Netty has kind of the same problem. If virtual threads really do work out the way people hope, then the entire event-driven programming is less appealing. If threads are literally that cheap, then we would just use future for everything, maybe.
01:19:48
Speaker
I mean, Netty has a lot of value in the network protocols that it has, but it would almost certainly be built differently. I've toyed around with altering Manifold to speak virtual threads, but I'm not sure if it'll be worth it, honestly. We'll see. I have so many other things I need to do anyway. I got a grant from Closureist together to upgrade Aleph, and so we're going to be bringing HTTP2 support to it.
01:20:18
Speaker
And if we have enough time, I'll try to add HTTP 3 and see how that goes. But what is the main difference then between 1.1, 2, 3, 4 for people like me? So the biggest thing HTTP 2 brings is multiplexing over a single TCP socket.
01:20:40
Speaker
So in the HTTP 1.1 days, you were typically, you would form multiple connections to a server because you couldn't reuse a connection while you're waiting for one piece. So if you're getting a video on one socket, then you can't simultaneously
01:21:00
Speaker
be requesting some JavaScript or stuff like that. HTTP2 allows you to multiplex and you just designate, you know, everything has a stream attached to it within the socket. And so you can like send multiple pieces if the image isn't ready. It doesn't block the whole socket. You can start sending some JavaScript and also request that kind of thing. I think for most of the users, it'll be pretty invisible, except that it will probably
01:21:29
Speaker
involve less socket contention, that's for sure. I don't think most people will really see a difference. We're planning on maintaining backwards compatibility, which for the most part is a ring map. You get a request map, return a response. The only thing Alif really does different from standardized ring is you can return deferreds or streams, and it'll slowly stream out the bodies that way.
01:21:52
Speaker
And I plan to maintain that. If we have time, I would like to expose a lot more of the HTTP2 bells and whistles. Prioritization and flow control are the big ones. So you can say, I want this stream to be higher priority than that stream. How well that will work out in practice and how many people will use it.
01:22:17
Speaker
I don't know. I know there was an HTTP feature called push promises where you can essentially send something that doesn't really have a request for it yet. So if you're sending like a web page and you know like I know they're going to need this JavaScript, I might as well just send it right now, then you can do that before the browser actually parses the page and then makes the request.
01:22:41
Speaker
In practice, that has worked out terribly. There's some very long-winded Google analyses where they show that it is extremely difficult to get right and that it can make things worse if you don't get your... Well, for starters, it's very cash unaware. And so if you're spending time sending something that's cashier than he has,
01:23:08
Speaker
It's an interesting analysis, but the upshot is Chrome actually disabled push promises. So I'm probably not going to implement that, or at least not with any high priority. Flow control would be the other big HTTP2 feature. So you can basically say, I'm willing to accept this many bytes and then stop if you're not ready. And maybe that'll be useful for some streaming.
01:23:33
Speaker
HTTP3, I haven't looked into too much yet. I know the big change there is the underlying transport mechanisms. So it's not using TCP, it's using UDP. And the protocol is, it's inherited from Quick. I don't know if there's going to be any, like I said, a lot of this is going to be invisible to the user. Hopefully they'll just see that it will be a little faster.
01:24:03
Speaker
I think it has to be three to me. You can prove that, right? Using speed arena. I am going to unleash it on Aleph when I get the chance.
01:24:16
Speaker
Isn't the idea of HTTP 3? I agree that the substrate is different, but the idea of changing the substrate is to make the substrate more programmable, actually. So you can innovate on top of that substrate, because that's the problem with HTTP right now and TCP in general.
01:24:37
Speaker
is that versions are stuck on the routers and they're stuck in the clients. And if you basically want to change anything, you have to change the whole world, which is very slow.
01:24:51
Speaker
For sure. Yeah, that's part of why Google invented it. TCP has some long-standing issues dating way back. Things like the slow start, had a blind blocking issues. Yeah, this was falling back to UDP, which essentially doesn't present a stream interface. It's just these packets show up whenever. We just know the destination. Quick replaces TCP, if I recall correctly.
01:25:18
Speaker
So it's going to essentially replace the TCP part and handle streaming in a more intelligent manner that, I don't know, I haven't crunched the numbers on that one.
01:25:30
Speaker
I do see Quick showing up because I use a program to block all the network connections, so I whitelist every individual thing that my computer is making, which is really, really tedious. But I see, hey, this program is trying to connect to this server with this protocol and all that shit.
01:25:53
Speaker
Sometimes when I get suspicious, I just install like MITM thing, you know, like Charles or something like a proxy to see what exactly they're sending for no reason. Because if you install any software these days, it just makes connections to continuously
01:26:09
Speaker
phoning home and shit. But I've seen this quick thing show up a lot in my logs these days. So I think people are upgrading to the protocols in the newer versions, I think. Yeah, it's a very slow rollout. I think HTTP2 came out in 2014, I think.
01:26:30
Speaker
And then HTTP3 even more recent. But it's probably going to happen faster with the Evergreen browsers. Yeah, I think. Yeah. Yeah. Because that's what I meant by in the background, just they keep updating and something is running continuously. And every time you start the program, it's a new version. It's kind of a weird feeling to think that your computers aren't necessarily strictly under your control or yours anymore. I mean, compared to growing up, the computer did nothing I didn't tell it to.
01:27:01
Speaker
And now I'm like, oh, this computer's got, I don't recognize half of these background processes. What is this doing? Yeah. And also I think what I noticed is, you know, because first of all, when I started like you folks as well, like there is no internet to the computer. Like there is nothing. It's just computers, just standalone thingy.
01:27:19
Speaker
And now every fucking program I start, there is a perceptible delay because they have to talk to some service to start and especially this shitty JavaScript based electron things and everything. They have to phone home continuously.
Integration with Third-Party Services
01:27:36
Speaker
This is just ridiculous. And and even all the software that they build, like we use sort of a wiki program. I don't want to name names, but
01:27:45
Speaker
The desktop version is built by some service, so you call it to desktop.com.
01:27:52
Speaker
Like that's nothing to do with this application. And then somehow it connects to Stripe. Why? Why are you connecting to Stripe to check my company is paying for the thing? Yeah. And the search is delegated to Algolia. There is no search happening on their own servers. And then it connects to Algolia because all the data has been shipped to Algolia and then there's a search in service.
01:28:15
Speaker
Yeah, God, I'm like, it's probably waiting for micro payments. And they're gonna charge for introducing a car that can repossess itself. Yeah, yeah. Let's take what your computer is doing as well. You know, that's what exactly. Yeah. It just decides you can't use this anymore. Because you know, it's gonna send someone to collect itself. You have to be able to
01:28:44
Speaker
Did anybody float that idea past a vouch? I mean, when I talked with David about it, they were building keys for virtual keys or for cars.
Developing Internet-Free Car Keys
01:28:54
Speaker
It seemed like it would be very easy. Can I go to the bank account and then check, do you have enough money? No, I'm going to block your car, not even payment. We're going to check your bank balance. Yeah. Actually, the biggest problem we were solving for, and it is an interesting problem, is that
01:29:09
Speaker
is that keys can keep the problem of keys is that if you want to check their validity and you have to phone home it's a problem if you're out in the woods.
01:29:20
Speaker
And a lot of people go for camping holidays, or they go, you know, whatever they go across country. And there's lots and lots of famous kind of examples, at least in that world, of keys sort of, you know, essentially, brick in your car. So actually, we'll do the opposite. We wanted to make sure that actually it would work in any circumstances so that it was completely free of the internet.
01:29:48
Speaker
Once you were back on the internet, it would check to make sure your keys are still valid, but none of the car companies ever wondered and they don't want to do this. They don't want to brick the car with a key. Now, that's not to say that they won't eventually like come and get your car collected or go to drive back to home.
01:30:07
Speaker
You know, it's like, that's different. That's a different, that's an old fog and old problem, you know? But they definitely don't like the idea of like the key expiring as you're driving along the highway and then stopping the engine. That would be super bad, you know? I think the health insurance people might have something to say about that, you know? So what I'm hearing is I should steal cars from camp sites? Definitely. Well, it's difficult to do it. That's the problem.
01:30:34
Speaker
Well, I just have to steal it where there's no connectivity and then yank out or just chop it up for parts. Yeah, but they don't give keys. That's the problem. If you have a key, then you're kind of golden, aren't you? But they don't give physical keys. So yeah, you have to wire it up. Hot wire of the car on a campsite, then rip out all the infrastructure, then you're good. Yeah. Okay. I'm taking notes. Keep going. Keep going.
01:31:02
Speaker
He's writing in his idea journal. This is not an endorsement by Deaf and not by Vowchider. We're completely independent media.
01:31:22
Speaker
This one weird trick will get you a free car. Drivers. But it's an actual problem where you've got kind of like, yeah, you've got all these kinds of problems with like, there's keys connectivity, and then you've got problems with batteries, because if you're in the boondocks and your phone like goes out of battery, what are you going to do? You know, you can't get into your car anymore. Yeah. There are ways around this, but maybe it's just a different podcast.
01:31:54
Speaker
That's going to be our other podcast, like how to hack your car. That's a different podcast. I listened to that. The batch people are interested in working on that stuff. We have a very low battery, very, you know, very like one year, two year type battery, small modules that you can use to unlock a car rather than a phone. Anyway, that's a separate discussion, but it's fascinating for sure.
01:32:20
Speaker
So anyway, let's get some lunch. Come on, guys. So just to just to wrap up the, yeah, I think he's, he's too late. I skipped lunch. It's like, it's got, I was very clever. I'm very sorry. I was very clever. Don't worry. No worries. Yeah.
01:32:44
Speaker
But I think just to wrap it up maybe, you know, um, so you started from PHP, moved to psychology, came to closure, and then now you're basing your startup thing in, in, in closure.
Clojure: Love and Frustration
01:32:58
Speaker
So as a conclusive thought, like what are the things that you find like, you know, exciting about closure and then what are the things that are shitty enclosure right now for you?
01:33:07
Speaker
Maybe the other way around, you know, let's close it on the positive. So we'll start with the shitty stuff and then close it, positive or not. I mean, I don't know. There's some decisions that I would just do differently. Like, I think the better decisions in closure are ones where a particular way is blessed or encouraged, but not mandated.
01:33:33
Speaker
So, for example, immutability, it vastly simplifies my reasoning about stuff. But if you need escape patches, there's transients, there's atoms, or job interop.
01:33:46
Speaker
I wish there were more of those that weren't necessarily prescriptive. I've never been a really big, big fan of the whole records and protocols, and some of it is that the underlying Java classness leaks through, and I've never liked that. But there's all sorts of random little impedance mismatches that have always bugged me.
01:34:07
Speaker
Maybe the underlying Java parts make that impossible to change. You can't use helper functions with mutable fields and depth types because they can't access the field and change it. And then occasionally I button to code reuse issues. A fellow I'm collaborating with on Manifold is trying to bring the Manifold protocols to a completable future. It's been a long-standing request. People have had three years.
01:34:35
Speaker
And there's an abstract, completable future type that we could use in Java, but we can't do it in Clojure, unfortunately. So I don't usually need abstract classes that I want to inherit from very often, like maybe once a year, but it irks me that I can't do it. What else? In that case, can you write one in Java that does it and then use that one? Is that the kind of a scare patch that could be useful?
01:35:03
Speaker
We've thought about that at the moment, the guy is playing around with Potemkin's death abstract type, which is a totally crazy macro that will essentially store a lot of function bodies and then move them over to another definition. So we'll see how that works.
01:35:24
Speaker
I guess my one major complaint about closure, I feel like it's kind of coasting a little bit these days. I wish it were a little more vibrant. And maybe, you know, obviously it's not a young language anymore, but I don't know. I wish there were more going on.
01:35:45
Speaker
Not all of us got brought up by New Bank and are set for life. So I think about hiring and other issues. I do wish it were growing a little bit more. I think the fact that the core team grew by 25% in the last two years is pretty good, but it could do with a little bit. It could do with a slight increase in that number. But in terms of what I love about the language,
01:36:14
Speaker
I mean, it is still a deeply elegant language and I really enjoy programming it. I don't know that I never really felt that way about like another language. But that being said, you know, I've also been programming a closure for a while and sometimes you start to get an itch to see what else is out there. So I keep thinking, ah, you know, I'm gonna play around with Rust for my next side project.
01:36:38
Speaker
Maybe I'll check out this elixir thing. We'll see. But it hasn't happened yet. Yeah. Nice. Cool. I've got to look at Rust. I think VJ has as well, and I don't know. It definitely doesn't bring me any joy, but yeah.
01:36:55
Speaker
I mean, I don't think it's, yeah, exactly. I don't think Rust is specifically joyful, like closure, like the tight feedback loop. But I think the switch of the mental model that I have to make is a bit different. Like I need to rethink and
01:37:13
Speaker
I keep hacking on smaller tools or some other shit. And I also write Python, a lot of Python as well. Which is OK. The language that I like to look off, and I've only done a little bit of programming in it, which is, in my opinion, I'm an old C programmer, is Zig. Because Zig is definitely a better version of C in terms of its memory model and its type systems.
01:37:39
Speaker
But it's not as mind-fuckering as Rust is, with this borrowing model, which is incredibly annoying. It's like the maybe of memory models. It just expands everywhere very quickly. Those of you reading the transcription, you do not say mind-bleevery.
01:38:08
Speaker
We have to use the right transcription service. Even the GBT stuff is just saying, no, I can't do that. Or I am not going to tell you how I joke about a specific thing, because there are so many guardrails around AI now. So I don't think our podcast is transcriptable that easily anymore. We have crossed that correctness boundary a long time ago.
01:38:34
Speaker
Anywho, Matthew, thanks a lot. Thanks a lot for taking the time and I know you're on the other side of the world. But, you know, we should have really, really big success with the speederino.
Announcing Speederino
01:38:46
Speaker
I'm not sure if you have announced it already or if there is a website or anything. I have not. This is sort of the soft launch. Anybody who wants to check out speederino.com and add themselves to the mailing list. And I will let you know when it's ready to be inflicted. I mean, tested on you.
01:39:06
Speaker
You'll be ready to make things faster for people. If you have the same compulsion to make things fast, look me up. Sounds good. We'll sign up. We'll sign up today. Thanks for having me on. It's been a pleasure. Yeah, thanks a lot. Great work on maintaining the manifold in Aleph as well.
01:39:33
Speaker
I'm just trying to keep the lights on. Yeah, I think we can all say that's excellent. Yeah. Yeah, yeah, totally. Good, good. Yeah. Cool. On that happy note, let's say bye-bye.
01:39:50
Speaker
Thank you for listening to this episode of DeafN and the awesome vegetarian music on the track is Melon Hamburger by Pizzeri and the show's audio is mixed by Wouter Dullert. I'm pretty sure I butchered his name. Maybe you should insert your own name here, Dullert.
01:40:08
Speaker
Wow, sir. If you'd like to support us, please do check out our Patreon page and you can show your appreciation to all the hard work or the lack of hard work that we're doing. And you can also catch up with either Ray with me for some unexplainable reason you want to interact with us, then do check us out on Slack, Clojarian Slack or Clojuverse or on Zulep or just at us at Deafened Podcast on Twitter.
01:40:37
Speaker
Enjoy your day and see you in the next episode.
01:41:04
Speaker
you