Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#45 - Moe Aboulkheir
53 Plays7 years ago
Links of interest:
https://datopia.io
https://nervous.io/
https://github.com/nervous-systems/sputter
https://github.com/replikativ/datahike
Recommended
Transcript
Introduction and Origin of 'Mo'
00:00:15
Speaker
Welcome to Deaf and episode number 45 today. Hey, welcome Mo, Apple Care. Yeah, Apple Care. Your guess is as good as mine.
00:00:30
Speaker
OK, I was trying to pronounce it, but I think more is the easiest thing to say. Yeah, it's not even my real first name. It's just a lot easier to say. You're doing like what Vijay and all the Indians do, making concessions to the English idiots.
Ethnic Names and Fitting In
00:00:50
Speaker
In my defense, it was decided before I could speak. My parents gave me this very ethnic name and I was born with blonde hair and quite light eyes and it just didn't play. It wasn't going to sell.
00:01:10
Speaker
But, I mean, you could argue that my name is ethnic. It just happens to be white ethnic, you know. So, of course, I win, you know. So, there we are, you know. So, what's more short for then? Salah. Salah. Yeah. How is more short for Salah? Or is it more Salah? So,
00:01:35
Speaker
The family mythology is that there was some, I think there was a female tennis player called Little Mo. And I don't know what she has to do with me, but my parents just- I think she was wearing the same cardigan as you were. Probably something quite similar if she was a tennis player. Yeah. So yeah, I think they just figured I looked a bit more like a Little Mo than a Salah, you know? Okay.
00:02:04
Speaker
And I agree with
Mo's Programming Journey
00:02:05
Speaker
them. But I must say, you're the most well-dressed guest we ever had. So I think the people who are listening to the audio probably can't see, obviously. But yeah, this is the first time ever we have a super well-dressed guest on the podcast. Well, if the content is a disappointment, at least we'll have to ask.
00:02:26
Speaker
Yeah. And also the David Lynchian style curtain in the background to give the extra spooky effect for the YouTube video. I mean, this time it's really worth paying that Patreon cash guys to get access to this YouTube video. I encourage it. I was going for some kind of Eldritch luxury.
00:02:49
Speaker
the final throes of disaster capitalism. Right. Yeah. It's like a hammer horror movie. I think the hammer horror movie is very on brand for us, actually. Yeah. Yeah. Oh, yeah. I have no idea what you're talking about.
00:03:07
Speaker
I'm down with that. Could it be a vulnerable Dr. Fives or whatever? Was it Dr. Feibs? Cheap horror movies, basically, that aren't really scary. OK. Anyway, yeah, carry on. Yes. So we'll get back to the cheap horror movies later, I think. So Mo, again, welcome to the show. Can you give us a bit of introduction about your work, your work in Closure? How did you come into Closure?
00:03:37
Speaker
Yeah, so I've been writing software since I was 12 or so. Was this after you gave up tennis or before that? I was actually playing tennis at around the same time. Reluctantly. And I've got a very funny story about that that I couldn't possibly tell.
Career Highlights and VR/AR Role
00:04:01
Speaker
It's okay, nobody listens to our broadcast.
00:04:06
Speaker
Um, so I, I, you know, just, I, just cause I was a loser and I, and there was a computer in the house, right? So I decided to figure out how it worked. Um, and, uh, you know, I was, I was, um,
00:04:21
Speaker
I wasn't the best behaved adolescent, so I left school early and was faced with the prospect of either living with my parents or deceiving some start-up into paying me money. I went with option two.
00:04:46
Speaker
And that was when I was 17, I think just before I turned eight. And then the internet died. I turned 34 yesterday. Oh, happy birthday.
00:04:57
Speaker
Thank you. Thanks.
Embracing Closure
00:05:00
Speaker
Since then, it's mostly been uninterrupted software development in a conspicuous number of languages, paradigms, operating systems.
00:05:16
Speaker
business domains, you know, online casinos, investment banks. Is there a difference more? It's like putting them in the same. I don't think of them as different domains, but you know, I've worked in both myself. But I think the dress code is perhaps the most salient difference. I was going to make the joke if you didn't.
00:05:43
Speaker
But and then, yeah, a bunch of sort of open source consultancy. I started my own, started my own after a bit and currently work for Samsung in sort of VR and AR stuff. So mostly around.
00:06:11
Speaker
high performance kind of pipelines for processing, transforming videos.
00:06:21
Speaker
And I first got into Lisp, I guess, a long time ago, maybe when I was 18 or 19. I was a kind of closet scheme developer for years. I had no idea. I could never have imagined that circumstances would evolve in a way that would allow me to get paid for bashing out the parentheses.
00:06:49
Speaker
And I kind of got into closure a little bit late because I read some of the docs and I wasn't
00:07:01
Speaker
overly enamored with the JVM at that time, although my opinion of it has improved. I just didn't have a huge amount of experience with it. And the interrupt kind of put me off. So I didn't really sort of get the kind of maximum leverage approach that typifies a lot of the decisions that guide Clojure's development.
Closure vs Other Languages
00:07:31
Speaker
I've got a terrible subjective sense of the passage of time, but I would say the last four or five years, I've mostly been writing Clojure, but I've also been a C programmer, a Python programmer, GLSL shaders, F-sharp, C-sharp, the whole deal.
00:07:56
Speaker
So how do you compare Closure to those things? Because you had experience with Scheme already and then with all these languages? Yeah. So I mean, with Scheme specifically,
00:08:09
Speaker
I think it makes sense to look at Scheme as a kind of construction kit for programming languages. It doesn't include any features that are extraneous beyond what you would need to implement an extraordinarily powerful runtime.
00:08:32
Speaker
I think its downfall is that it didn't really take a firm stance on packaging systems. So they do have a kind of a proposal amendment process.
00:08:54
Speaker
SRFI, or I think some people pronounce it phonetically, like, not phonetically, but, but SRIFI or whatever, you know, for advancing the standards. But if you look at the kind of landscape of scheme implementations, it's very much not, you know, write once, you know, run anywhere. It's very hard to share the code basically.
00:09:24
Speaker
Yeah, absolutely. Everyone's got their own prelude that will kind of coerce the code into running on that pet two or three top scheme. I mostly worked with chicken scheme.
00:09:40
Speaker
My claim to fame is participating in the logo design. I think it's got a pretty neat logo. But yeah, in terms of comparing closures to that stuff, I mean,
Open Source Contributions
00:10:00
Speaker
closure strengths, I think, are in its
00:10:08
Speaker
the abstract of powers that it confers on developers. And I think those are truly unparalleled in any of the programming languages I've used in any depth.
00:10:23
Speaker
Certainly, scheme has, the lambda is the universal abstraction and cons cells are used to inefficiently implement all manner of data structures.
00:10:47
Speaker
I firmly believe that closure is the best list. It's not the best list that's existed. It's not a compromise or a list that's easy to pitch to management because it runs on.
00:11:07
Speaker
you know the jvm i think it's i think it's immaculately uh you know well-designed and and uh well-considered you think it's the like the functional aspect of it because obviously with scheme you've got various options with lisp in general you've got lots of options lots of paradigms you can choose from
00:11:27
Speaker
But with closure, essentially, it was a functional beginning. I know that you can do with the bits and pieces with it. Everything can be imperative to some extent, but is that what you would like? Am
Ethereum and EVM Critiques
00:11:40
Speaker
I playing that back to you correctly? Yeah, I would say more that the different options you have for kind of runtime polymorphism, I think
00:11:56
Speaker
I didn't have as big a problem as I think others did with purity or immutability by convention in lisps that did permit it. But yeah, I also never really wrote a
00:12:15
Speaker
substantially sized system in Scheme. So I'm sure that stuff becomes a problem when you're incorporating multiple libraries, all with different approaches to doing things. But yeah, I think stuff like multi-methods, protocols,
00:12:36
Speaker
keyword hierarchy, even really simple things like namespaced keywords. I think those things are incredibly powerful features that just give us options.
00:12:51
Speaker
Yeah, particularly protocols, I think. Compared to, because there is this discussion around refactorability, if there is a word, of a large code basis, if you use like a statically type programming language, because you said you are also familiar with, or at least did some projects in F sharp. So how do you see the contrast between these two then?
00:13:16
Speaker
So I don't like statically type programming languages personally, like I have a grudge against them. Welcome to the show.
00:13:29
Speaker
I just don't like the compiler taking that tone of voice with me. I don't know if either of you have tried to write a program in Rust. Yeah, I did. But it's like going through customs in a country that's at war.
00:13:50
Speaker
Yeah, I always gravitated towards dynamically type programming languages. Yeah, you know, refactability is a problem, but, you know, like with anything, I mean, you know, types only get you so far. And, you know, some of this hyperbole you see from the static typing advocates, you know, I've heard it claimed that
00:14:18
Speaker
you know, we're morally deficient. We own the morally deficient podcast space, so it's okay. No, no, I'm certainly in the paragon of moral virtue myself, but I wouldn't I wouldn't ascribe much of that to my use of dynamic timing. No, I think
00:14:48
Speaker
Yeah, so I can be productive in statically-typed environments.
00:14:59
Speaker
I think, you know, in terms of languages that have achieved some kind of traction in industry, I would say, you know, C sharp is a relatively comfortable language to work in, you know, far more so than Java, you know, setting the runtime differences aside. And then, you know, I've done some messing around in, you know, SML and OCaml.
00:15:29
Speaker
And, you know, I found that stuff interesting, but it didn't. And Haskell as well, you know, to a lesser extent. But I just didn't feel like I was getting the super power that people were telling me I was going to get.
00:15:48
Speaker
So I think this is a nice comparison because we usually hear like, you know, having a compiler or statically typed languages is that they're going to help you in terms of writing your program and guiding you through the stuff. But I think we're going to settle for it's going through the customs to a country that is in a war zone or something. Yeah, absolutely. Can I see your papers?
00:16:13
Speaker
Yeah. So you said you started using Closure almost four or five years ago. I think so, yeah. And you have a couple of open source libraries that you already built. Do you want to give us some idea about what they are?
00:16:30
Speaker
Yeah, I think the most popular one is the unimaginatively named CLJS Lambda,
Blockchain Trust and Inefficiencies
00:16:40
Speaker
which is a collection of projects, a runtime library, and a line again plugin, and also a
00:16:52
Speaker
piece for integrating with serverless that allows ClosureScript functions to be deployed to AWS Lambda, which is their offering for precisely metered sort of deployable functions. The kind of
00:17:15
Speaker
premise being that even with AOT compilation, at least at the time that I was writing the library, it just didn't make sense to use closure because the startup times weren't acceptable. And it's difficult to predict whether a particular invocation is going to reuse
00:17:41
Speaker
the resources from a previous invocation or if it's going to restart a fresh one. I mean, I've seen people do crazy things like write functions that repeatedly invoke their function to ensure that it's always kind of hard. Sort of not the point of serverless then. No, yeah, totally not.
00:18:04
Speaker
I I'm gonna build a management infrastructure around the serverless environment. They're stopped right now, please. Yeah. I think serverless is great. And I initially wrote, you know, a line and gun plugin that does all of the AWS
00:18:19
Speaker
you
Introducing Datopia
00:18:20
Speaker
know, stuff in terms of, you know, creating the functions, merging their configurations, you know, uploading the zip files. But I would strongly recommend anyone who's interested in that Closure Script on Lambda library to use it via serverless. There's just a serverless CLJS plugin that you can add, and it'll just create a zip file.
00:18:47
Speaker
via the ClosureScript compiler interface. And there's an option to use. I forget about it because I forget even what the name of the...
00:18:59
Speaker
that the compiler is, but there's some non-JVM compilation option. And then more recently, there was a project called Sparta, which is a partial implementation of the Ethereum virtual machine.
00:19:23
Speaker
It doesn't do everything, and I'm not interested in making it do everything because what was mostly fun about writing it, aside from learning about the EVM and the decisions that went into its development, some good, some not so good,
00:19:47
Speaker
It was that it was the first project I'd worked in in a long time that wasn't principally focused with IO. And it turns out that it's really fun to write programs that are kind of self-contained and are mostly translating data. So stuff like writing compilers and so on. So before you get a bit deeper into that one, can you give us some idea about Ethereum and EVM? What it is?
00:20:17
Speaker
Well we can make money like everybody is jumping onto that shit right now. Yeah, I certainly, I'm not the person you should ask about how to make money in that space.
00:20:33
Speaker
Ethereum is a project, it was started by a gentleman, goes by the name of Vitalik Buterin or Buterin. It's considered the sort of, say, regardless of.
00:20:49
Speaker
you know, where it is in the market cap rankings versus Ripple on any given day considered to be that it certainly has the greatest amount of developer mindshare of any of the blockchains. And its distinguishing feature is that it allows
00:21:08
Speaker
I mean, in practice, it's not sure and complete because there is a fixed cost to each instruction. But assuming an unlimited budget, you can execute unbounded programs, deterministic unbounded programs that are written in higher level languages that are then compiled down to EVM, the Ethereum Virtual Machine bytecode.
00:21:38
Speaker
I think
Closure's Suitability for Projects
00:21:40
Speaker
they very much follow the DIY aesthetic. I would say that none of the higher level languages are particularly well defined, particularly well designed rather. Some of them are sort of actively catastrophic. And I think there were some awkward decisions made in the design of the virtual machine itself.
00:22:04
Speaker
But it is interesting. We can dive a bit more into that more actually. So just tell us a little bit about what you think about where those issues are. Obviously, there has been a big discussion around the fork of Ethereum.
00:22:24
Speaker
because of the fuck up on the contract. They're claiming it was not a problem with Ethereum, it was just a coding error. So, you seem to be suggesting there's something deeper there.
00:22:38
Speaker
So, I mean, I didn't have that Dow hack in mind specifically, but I think that kind of re-entrancy probably should be inexpressible in a language that's designed for financial computing. I would say I have some low-level quibbles with the EVM, the design that probably aren't worth.
00:23:03
Speaker
delving into in too much detail, like the native word size is 256 bits. So even if you want to twiddle a single bit, you're paying the cost of the other 255 that you don't care about. Despite the fact that in modern programming languages, it's trivial to represent variable width.
00:23:35
Speaker
In terms of the high-level languages, I think, you know, clearly, Solidity is the language with the greatest, I mean, you know, the greatest number of contracts have been written in Solidity. It's, I mean, it's, from my perspective, it's a catastrophe from the perspective of programming language design. It's, it's, it's,
00:24:01
Speaker
I think the kind of guiding architectural principle was let's make it look as much like JavaScript as possible. But of course, it's doing things that are very different to JavaScript. But I would say my critique in general would be that
00:24:24
Speaker
Clearly, Vitalik's inspiration for all this were Nick Szabo's early essays on smart contracts and the idea that we could have
00:24:39
Speaker
you know, deterministic, executable, financial instruments. We could, you know, represent title deeds, you know, dividends, you know, all derivatives, you know, all manner of complex instruments.
Datopia's Vision and Comparisons
00:25:00
Speaker
And, you know, I think that term smart contract
00:25:05
Speaker
It has a lot to answer for, and so does ledger. Yeah, the legal contracts, they're sequential, but I would argue that they're more declarative than imperative.
00:25:24
Speaker
I've read through a lot more, I've read through a lot of salinity contracts and my impression is that they're not really executing, they're not really performing logical operations, they're brokering access to data that is ridiculously expensive to store. So my fundamental contention would be that
00:25:52
Speaker
You know with a theory and is is is actually a database and it's a very badly designed one and I would say the same the same even about Bitcoin and I mean we use this word ledger and I and I think.
00:26:11
Speaker
That's kind of determined the way we think about these things. I mean, a ledger is just a special case of a database with a particular disposition towards the semantics of the values it's storing. So I'm interested in looking at blockchains from the perspective of information systems.
00:26:40
Speaker
read through any a solidity auction contract, an escrow contract, all the methods are doing, brokering access to pieces of data that could just as well be exposed directly if there were a general means for doing so and a means for declaratively expressing access controls.
00:27:10
Speaker
you
00:27:11
Speaker
So before we get onto like, yeah, I take your point. In terms of like logical like comparisons, would you say, cause one of the things that I say about, the thing that I hear about ledgers is that, you know, from a closure perspective at least is that they're a bit like the atomic in the sense that they're like time is a first-class citizen, they keep all their old records and you know, so you can like history is built in and that seems like a good thing. So there's some advantages around that.
00:27:41
Speaker
But obviously, the smart contracts are more about, let's say, trust issues and trustlessness rather than the semantics or the behavior of a particular time series in a database. Yeah, so one point I'd like to make around that is that while certainly
00:28:07
Speaker
A lot of these networks can be described as immutable databases, vaguely similar to Datomic. In their programming models, the history that all this computational effort and organization and in some cases power consumption has been devoted into maintaining is completely discarded. I mean, a solidity contract doesn't have access to any value at any proceeding block height except the one at which it's being invoked.
00:28:37
Speaker
unless it specifically made the effort to persist those values in data structures. But the only data structures that are available, completely immune to any kind of structured querying, or
00:28:58
Speaker
Yeah, and they're ridiculously expensive, right? Obviously, it fluctuates with the gas price, but I checked a couple of months ago and the price per gigabyte for data storage on Ethereum, it's in the millions, the tens of millions of dollars, I believe. I think it would have fallen propitiously since then, but it's still completely unaffordable.
00:29:29
Speaker
And I think, you know, datomic is interesting because it's saying, hey, immutability is great, not just as a kind of means of attaining auditability, but also if we embrace it in our programming models.
00:29:49
Speaker
It gives us superpowers like the ability to time travel, the ability to compare databases at different times or with different sets of transactions applied to them. It's astonishing to me that blockchains put so much effort into maintaining
00:30:16
Speaker
an exhaustive history of every possible state, and then they just drop it on the floor as soon as any code is being run. Yeah, I mean, obviously it's not dropped in a persistent state, but yeah, take your point. But one of the other things that people talk about, though, is the, like I said, the trust aspect of it.
00:30:42
Speaker
which is the more tricky part, whereas if you're in a kind of environment where your administrators can make all kinds of changes to the database, potentially they can subvert the data. Isn't that the biggest, really the biggest value, if you like, of the Bitcoin's near theorems in the sense that there has to be a very large quorum that's almost impossible to gain in order to modify the data?
00:31:10
Speaker
Yeah, I think so. But Amazon have filed patents for, and I'm sure I have engineering teams working on implementing those patents that will offer all of the guarantees of these public blockchains just without
00:31:35
Speaker
the counterparty risk of having all of the nodes located on a kind of homogenous network. And I would argue that that's a risk that would come across as extremely eccentric if it were expressed in a boardroom. So I think if this stuff attains any kind of mindshare
00:32:03
Speaker
For the majority of use cases, Amazon is just going to eat everyone's lunch. You could get that. Recursive cryptographic hashing and get all that good, tamper evident, good stuff without having to run a public network.
00:32:32
Speaker
And I think that that's what's going to happen. I think these public blockchains exist both at the mercy of or at the caprice of regulatory agencies and at cloud providers. I mean, they're at cloud providers have unprecedented levels of infrastructural centralization.
00:33:00
Speaker
that would result in throughput that would require a really powerful kind of counter argument to a business adopter in order to turn away from. So you don't think that the end of kind of like cloud providers as a monopoly is imminent then?
00:33:26
Speaker
No. I mean, I heard some rhetoric in that question. But no, these guys employ armies of astonishingly talented people.
00:33:44
Speaker
and they could out-compete the development teams on everything except just one or two line items of trustlessness that businesses gave up a long time ago and I don't see ever caring about.
00:34:08
Speaker
So, I mean, but that said, I do think there are cases where you do want to pay that cost. I mean, the cost boast it, but the kind of throughput cost of running on heterogeneous networks outside of cloud providers. And obviously while it's, you know,
00:34:37
Speaker
I'm sure, in practice, impossible to trustlessly prove that the kind of geographic distribution of a valid data set or whatever in a given network. I think there are...
00:34:56
Speaker
or maybe an awkward way to bring into the discussion, I think there are sort of commons-oriented projects that could justify paying that price, where it just wouldn't make sense to run something so ambitious and with such a long time horizon.
00:35:25
Speaker
On on on a ws and you know the stuff i was saying before i will be in a wasn't wasn't necessarily a defensive of the cloud providers i mean i think i did that that that's. You fun you have some fun really unpleasant asa asymmetries in the relationship between cloud providers and their customers.
00:35:48
Speaker
You can throw hundreds of millions of dollars at AWS, potentially even per month for a Fortune 50 company. And that doesn't buy you any foresight into their plans to end of life.
00:36:10
Speaker
software that you're depending on or to prioritize the features. There may be a champagne room that I don't know about, but I've certainly worked in companies that racked up substantial bills and mostly just got us prioritized support access.
00:36:37
Speaker
Well, didn't Samsung in the end just buy their own cloud company? They bought Joyant, didn't they? Yes, yes. Because I guess they couldn't get that champagne room access. They just figured, OK, screw it. We'll just buy something that essentially will give us control over it. I think instead of buying software, you just buy software companies. And then instead of getting into cloud, just buy a cloud provider and just be done with it. In the long run, it's probably cheaper, I think.
00:37:08
Speaker
I guess it's just rewinding to running your own data centers. Yeah, at a certain scale, yeah, that's what it kind of feels like. So just getting back to your project, the splutter, splutter, splutter, splutter, splutter. So what sort of state is that in now, would you say? I mean, what's your kind of commitment to it or what's your thought about it?
00:37:31
Speaker
I've taken it as far as I think I'm going to go. It can run a couple hundred of the, you know, Ethereum has an official test suite. You know, it can run a couple hundred of those. It implements the persistent storage.
00:37:52
Speaker
memory, I got a little bit nuts with it and wrote a Java project that significantly outperforms big integer for 256-bit types, numeric types. A lot of that stuff was good fun and good kind of
00:38:20
Speaker
uh, wood chipping, I guess the phrase, but, but, but, you know, I'm not, I'm wood shedding rather. So, but, so I'm not, I'm not, or camel shoving or yak shaving. Yeah. Yak shaving. Yeah. Um, the, the, the national sports of my people, um, so the, the, uh, so, so yeah, I think, I think it's, it's, it's, um,
00:38:44
Speaker
I've moved on to working on an open source project called Datopia, which is a distinct blockchain.
00:39:00
Speaker
That's centered around this idea of so hey what what would happen if we embrace the you know it means ability of block chains i availability of all historical values we embrace that in the programming model.
00:39:19
Speaker
So basically, rather than submitting financial transactions, you just submit database transactions. So it's basically like a permissionless multiplayer data log driven database. And it's
00:39:47
Speaker
We haven't made the testnet public yet because there's a couple of pieces of novel functionality we want to add to it before releasing it. But we've released a bunch of the auxiliary, we've been trying to sort of cut it down into components to try and encourage
00:40:09
Speaker
Blockchain development in the Closure community. So I released some of that stuff over this weekend. There's this project called Tendermint. I'm not sure if either of you are familiar with it. But it's a Go project that implements Byzantine fault tolerant consensus or classical consensus.
00:40:32
Speaker
and offers the option of a sort of a cross-process interface of a socket.
00:40:43
Speaker
we released a closure library that exposed like a sort of ring-like interface, you know, sort of functions that take maps and return maps that interface with Tendermint. It handles all the consensus details and, you know, your job is to just implement a sort of deterministic finite state machine that then
00:41:09
Speaker
You know via the by the magic of ten to men is is you know replicable across you know an arbitrary number of validated. So i think that i think you know the that may be useful for people.
00:41:28
Speaker
The project itself in an example project were released today and I hope to finish writing up a blog post about it in the coming week. We also implemented a
00:41:44
Speaker
a protocol buffers three implementation that doesn't involve any interop or a code generation. It's this one project that just generates Eden maps that describe the underlying protobuf messages and seconds
00:42:10
Speaker
Zero dependency project which interprets those schemas and Decodes or encodes either, you know byte arrays or streams Which is which is what we use in the ABCI host which is the
00:42:33
Speaker
the first project I was describing because the wire protocol for the Tendermint stuff is protocol buttons. So why do you think Closure is particularly suited for these kind of things between, you know, Sputter and the new projects that you're working on?
00:42:51
Speaker
It's on the one hand, like if you ask the COBOL developer why he thought COBOL was like, he would have like a water tie answer. So there is an element of subjectivity. I would say,
00:43:08
Speaker
Embracing immutability is great. I mean, you can run an operation. And if you don't like the results, you can walk it back or just retain the initial value. That type of stuff is hugely useful when I was writing Sparta.
00:43:36
Speaker
But did it change your opinion about, because when you were explaining your first look into closure, you said you're put off by the whole interop, the JVM thing that is baggage that closure is carrying. Yeah. So how did your opinion change, or if at all it changed? It definitely changed. It changed a long time ago.
00:44:05
Speaker
You know, I'm still a little bit pissed off that you can't pass a string into an integer without, you know, importing or, you know, string value off or whatever. Yeah. Yeah. But you know, I understand the trade off and I get that
00:44:26
Speaker
There may not be much benefit to replicating that functionality. Ultimately, all code is a liability and just having this kind of cutesy intermediary layer.
00:44:43
Speaker
it may not be worth it. So yeah, I totally changed my mind about that after I used it more seriously and saw that what was mostly a kind of aesthetic revulsion initially for me was totally disproportionate.
00:45:03
Speaker
to the abstract of powers conferred by the language. And also, just the level of consideration that was put into its design is singular, I think, and pretty astonishing. I mean, not that there are things I wouldn't change about it, but yeah, I'm happiest working in closure, for sure.
00:45:32
Speaker
Yeah. So most of your experience in Closure, is it web-related stuff? Or do you play something with Closure script as well on Node or something like that? Not really web-related stuff, mostly like
00:45:50
Speaker
You know kind of a synchronous message passing by like sort of you know multi-stage What's multi-stage pipelines that are doing you know incremental transformations of some sort asynchronously. Yeah And
00:46:13
Speaker
professionally, that stuff's mostly been around doing extremely computationally expensive and hardware accelerated stuff on videos and compositing images and that kind of thing.
00:46:33
Speaker
And yeah, I've tried to turn as much sort of, I mean, I'm the only Closure developer at Samsung to my knowledge. I mean, we have Closure, a ton of Closure in production, but the bus factor on it is zero. Well, one, so you're still there.
00:47:01
Speaker
Yeah, right. Sorry. My natural pessimism. Is it because of Brexit or something?
00:47:17
Speaker
So are you using the graphics acceleration stuff in Closure? Or is it just stuff? Because we had this... Dragon. Dragon, yes. Enter the Dragon. Neanderthal. Sorry? Neanderthal or something. Neanderthal, yeah. Which has all the interfaces into the... OpenCL. The graphic accelerator stuff.
00:47:43
Speaker
So you can do all of your nice matrix multiplication, et cetera, on the hardware level. Is that something which you're doing or is that something else? No. So I looked at that and that's super cool, but it's just that there wasn't really a place where we could fit it in. So most of the work that I've done has been around writing
00:48:11
Speaker
Uhm, you know, hardware accelerated FFmpeg filters. FFmpeg is, it's, you know, it's monolithically structured, but the fact that it abstracts away so many of the.
00:48:28
Speaker
you know, the details, the codec and container details makes it a really good place to embed those kinds of transformations. And then it's the project that I'm working on the moment, which is more kind of
00:48:50
Speaker
what's to do with AR, we're relying quite heavily on third party libraries that we're optimizing by shipping stuff onto the GPU, but they're mostly C++ libraries. So we couldn't, there's no real opportunity to jam in some of Dragan's wizardry there. It's,
00:49:19
Speaker
It's less fun than it sounds. It doesn't sound that much. But you've got to pay the mortgage, so it's okay. Yeah, or the rent in my case. The clothes, the clothes. You've got to pay for those curtains.
00:49:42
Speaker
So coming back to Closure a bit. It's my jam. Sorry? I said, that's my jam. Yeah, exactly. So what do you think about the general documentation and the community around Closure? Because we have been in a lot of discussions these days around the features in Closure or how they're being built, how they're being delivered.
00:50:14
Speaker
I don't have a huge problem with it. I mean, I don't read it very often these days, except when we get like, you know, some new feature announced. I remember having a conversation with somebody who I deeply respect as a software engineer who I'd worked with previously. And, you know, he was put off just,
00:50:36
Speaker
Yeah, it's on the documentation.
00:50:44
Speaker
by the kind of impenetrability of the language and the documentation. You know, I can be quite florid in my speech and writing, but, you know, I think it was, I think it was Rique specifically that, you know, that there was a form called Rique that just, you know, killed him.
00:51:09
Speaker
And I think there are some cases where closure, I guess, is specified by implementation, but some of the documentation is
00:51:27
Speaker
You know, is it like a mixture between, you know, casual community source examples and then some quite formal descriptions that are very, very unlikely to be helpful to newcomers. You know, variadic functions isn't a term that's, you know, popular with the kids these days. It should be.
00:51:55
Speaker
Hey, you know, maybe one day, but not today. So almost four years of closure, mostly doing all the, of course, the latest project being in the Ethereum stuff. So emacs or some other shit? Emacs. Ah, OK.
00:52:16
Speaker
Yeah, I was a VR user for years. It was CIDR that got me to switch to e-mix. I'm a much better e-mix user than I was a VR user. Something about it just made me want to learn.
00:52:40
Speaker
I'm sick, I've got my own themes and just terrifyingly organized dot files. They would be used as evidence if I come to the serious crime. You have your mark on them, the serial killer evidence. Yeah, no, absolutely. We should have known this all along. Yeah, warning signs.
00:53:08
Speaker
I can hear the discussion, like two agents talking about, did you get his dot files? Yes, I did. Awesome. Now let's put them on our machines. Nice. So are you best in London more? I am. Do you go along to the closure meetups there? I don't know. I mean, I really don't like people at all. Oh, OK. OK. It's not so much that.
00:53:38
Speaker
you know, like, I mean, what are you going to do? Like, oh, what library did you use this week? You know, I thought you might have been presenting something, you know, because you've got some interesting tech there. Yeah, I'd like to present daytopia this this this blockchain that I'm working on. But I just I'd like to get
00:53:58
Speaker
I'd like to get the testnet out the door. I'd also like to appeal to your listeners. I don't know if either of you are familiar with a project called Data Hike, but my, effectively, the only other contributor to Datopia, which is this sort of database.
00:54:22
Speaker
Blockchain that we're building as Christian Valbach the guy the guy behind the Replicative or at least that's how it sounds in his incredibly authoritative Teutonic accent Is a series of projects which is quite ambitious? distributed systems approach to building distributed systems and so he He
00:54:49
Speaker
There's a project called the Hitchhiker Tree, I think by David Greenberg, which I would argue is, it's a right optimized B plus tree. I think it's a fractal tree. I don't really see it as a distinct data structure.
00:55:08
Speaker
But it's a perfect data structure to use to maintain database indices. And Christian had this genius idea of, hey, what if we built on top of the hitchhiker tree and used an assemblage of trees to service data log queries?
00:55:38
Speaker
And, you know, it can do that. And it's I think it's an extremely interesting, compelling project. And that's what we use in Datopia as the underlying data store on the validators.
00:55:57
Speaker
He recently started a PhD program in Vancouver, and I would implore the audience, if anything I've said sounds interesting to you, and you know a thing or two about databases and distributed systems, both Datopia and DataHike.
00:56:28
Speaker
could do with some help. Is Ditopia is already on GitHub or is it announced already or are you still working on it right now?
00:56:39
Speaker
On Datopia, some of the peripheral stuff, like the components that'll be useful in isolation or on GitHub, the kind of network itself is not yet on GitHub. I'm waiting on the integration of one feature that perhaps we could just talk about briefly because it's quite interesting. So I don't know if either of you have, how familiar either of you are with
00:57:10
Speaker
Yeah, reasonably familiar. Yeah. So I've never used it, right? I mean, like in a project, but I've messed around with it. I've read the documentation. So we've sort of extended the concept of schematizing attributes, you know, with like cardinality and type information.
00:57:31
Speaker
with the ability to attach logical constraints to them, which are themselves data log queries that are required to unify in order for a transaction, so a submission of data into the network to be considered valid. It's like the database procedures and the schema merged into one.
00:57:57
Speaker
Yeah, and it's all declarative. It is just a data structure describing a schema. Sorry, describing a
00:58:14
Speaker
the logic program that receives a bunch of inputs, the state prior to the block, the data in the transaction in isolation, and so on, and has an opportunity to figure out whether it makes sense.
00:58:38
Speaker
The reason why this is interesting is if you build a network that only allows you to execute data log queries in a kind of distributed manner, and let's take a step back.
00:58:57
Speaker
That allows you to insert arbitrary structured data.
00:59:10
Speaker
validate that data using these logical invariants, you can build something like you can build sound value transfer just like Bitcoin and Ethereum have. But you wouldn't need to embed it into the core design of the system. You could just deploy an attribute that has a sufficient, we've already written up the constraints for value transfer.
00:59:39
Speaker
So that idea of, you know, I hoped I wasn't going to say de-complacting. You failed. Yeah, the story of my life. So I think it's a particularly elegant design that even something that's considered, you know, intrinsic, I mean, it's basically the only thing Bitcoin can do.
01:00:09
Speaker
But but you know so something that's that's often special case in these networks we can just implement as a logical invariant over a over a selection of attributes that are defined in in the genesis block. So how do you deal with because one of the problems with with with blockchains in general is.
01:00:28
Speaker
you know, finality ordering, those kind of things. So in the atomic space, and in cases like hyperledger, there is a sort of central ordering system. And in the blockchain, the ordering is more, let's say consensus based. So how do you, in your environment, how do you deal with like sequencing, ordering, making sure that things are, you know, that the value is transferred in the right sequence, because obviously that's important in any transfer of value.
01:00:57
Speaker
Yeah, so Tenderman deals with the actual serialization in the sense of ordering the transactions in the blocks. And we just process them as we receive them. And if we've already accepted a transaction, which
01:01:18
Speaker
you know, has created conditions for which this subsequent transaction can no longer succeed, then we reject that transaction. And then the other interesting part of it is,
01:01:32
Speaker
We disseminate segments of these index structures over a peer-to-peer network, either DAT or, which is a project I'd suggest you guys check out, it's not really enclosure, but it's called DAT or BitTorrent. So we get the same kind of
01:01:59
Speaker
Read scalability is they tarmac in the o'clock client starts out with you know, maybe a single trusted root hash it issues a query and it has a date log query engine it's it's it's it's it's
01:02:15
Speaker
It notices, hey, I ought to retrieve this segment from the peer-to-peer index segment distribution mechanism, and then eventually ends up with a fully authenticated subset of the entire database state that only contains the data that is required to service the queries that that user has issued.
01:02:45
Speaker
rather than every, every network participant having to replicate the state. Nice. Okay, so we're almost one hour.
01:03:01
Speaker
And we're all still away. Of course. Bitcoin is- I definitely thought VJ had gone at one point, to be honest. Yeah, it was looking dicey. I'm just shaking, nodding my head and then blockchain. Okay, moving on.
01:03:22
Speaker
So I remember looking at data hikes some time ago, but it is not a full replacement for Datamic, right? It is mostly like a single machine sort of thing, or is it the scalability thing?
01:03:34
Speaker
Yeah, no, so it is. It doesn't cover the same use cases as Datomic. It may be worth thinking about it more like something like Datascript.
01:03:53
Speaker
But the fact that you can trivially replicate these indices in the manner I just described allows it to be multiplayer or authenticate. And it is authenticated. It's a markleized data structure. So it's recursively cryptographically hashed and so on. So I think it's an immensely powerful
01:04:22
Speaker
building block for distributed systems. Even though we're not planning on doing it for Datopia, you could certainly use schemes where the state was sharded in a particular way. But we're just going to require that all validators maintain the entire state in a local data hike instance.
01:04:51
Speaker
OK. So you're thinking about re-implementing Git on top of it? You know, we got a lot of quite abstract comments in our Telegram channel. And one was that we should commit the code to. On the data hike. Yeah. Otherwise, what's the point? Well, I mean, that's beautiful.
01:05:20
Speaker
But yeah, this stuff isn't really too different to Git. No, I get it. It's a sort of joking point, but one of these days, the concept of building, because it's like Git
01:05:36
Speaker
is one of these things which again started off as a very nice sort of decentralized concept, but has ended up in here kind of bit ridiculously being end up in the hands of Microsoft. Yeah, you know, and all your code up along towards now. So it's like Jesus, you know, nice one Linus, you know, it's like,
01:05:57
Speaker
Yeah, for sure. The soft irony there is quite long-winded, I guess, in the end. Yeah, no, no, it is. All the operating systems belong to Linus, but all the code belongs to Bill. First is tragedy, then is fast. Yeah.
01:06:12
Speaker
But just one last point, because you made a joke about the use of the network. Look at something like DBpedia, right? So that's semantic export of the data that's contained in Wikipedia in the form of RDF triples, or consider any large-scale data collection effort.
01:06:40
Speaker
So, DBP is basically a bunch of GZ tar files, right? It's ridiculous. Just imagine how many, you know, Balkanized AWS accounts that data has been instantiated into.
01:06:58
Speaker
And because of a lack of a lack of strong identity, these people can't collaborate on deployments, even if they've got no intention to change the data. If you're really lucky, you'll get a rate limited API or a paid API. And I would argue that the
01:07:18
Speaker
overwhelming majority, anecdotally, if back in development basically consists of mediating access to a database, to clients. Our idea is that that data should just be schematized and put in this single global replicable database. It would work for stuff like DBpedia, it would work for
01:07:44
Speaker
you know, experimental data from... I guess all kinds of public data sets from government and stuff like that. Yeah, so we would seed it with, I mean, there's a ton of amazing data sets we would seed it for and we would want to pitch it as well to, you know, people who were interested in taking a more kind of open approach to science.
01:08:10
Speaker
and being able to authenticate the intermediate results that go into producing the papers. I know we're running a bit late now, but just one final question is, how would you compare that to, maybe we're getting a bit blockchain. Well, it's a blockchain show, so fuck it. If you're still here, you won't mind this question. How would you compare the topia stuff to IPFS?
01:08:40
Speaker
Sorry, I'm not an expert in IPFS and I've read some quite compelling bearish cases on the idea that block storage is ever going to be
01:09:08
Speaker
that a permissionless network will ever be able to compete on price, that there are game-theoretic reasons why that's impossible, given something like S3. But like I said, I'm not an expert, so I would say the main difference is structure.
01:09:26
Speaker
I mean, we even talked about using IPFS as the index distribution mechanism rather than something like BitTorrent. But yeah, the point is this can do everything Datomic can do, right? It's a structured inferential database. It's not
01:09:57
Speaker
Right, so you could say IPFS is a bit more like S3 or a bit more like, it's a kind of medium for you to store stuff rather than as a kind of, what you're doing is what you're doing is putting essentially kind of like facilities and affordances on top of that storage capabilities of any kind of file system essentially.
01:10:21
Speaker
Yeah, absolutely. I mean, it is just a database, right? And yeah, I think the ability to query and like, you know, forgive me for saying this, like, you know, potentially synergize. Oh, God, no. Stop the recording.
01:10:43
Speaker
We're all triggered. This may be a little bit optimistic, but through collaborative schematization and so on, you could have different organizations structuring common bits of their data in similar ways.
01:11:01
Speaker
And i think that out there there is a you know aside from from you know NGOs that collecting data you know with universities that are running experiments and projects like tb pd a corporations generating a ton of non competitive not identifying information. That they just they just can't afford to release because it means they're gonna have to hire a bunch of developers to build an api.
01:11:26
Speaker
or they're going to have to shit out some tar files every month, which is no fun. You think basically more you can make the world a better place? I've an acute ear for
01:11:50
Speaker
patronization. I think I could make the world, we could make the world a more convenient place. I suspect the gentleman that I'm working with thinks that we can make the world a better place. But I'm just interested in building a database that has useful information in it.
01:12:18
Speaker
No, it's really good. It sounds like a really interesting project, actually. I'll definitely be having a look at it. And like you said, hopefully the listeners will as well. It sounds really good. By the way, good work, Matt. Thetopia.io. There's not much there, but what's there is good. Thetopia.io. OK. Yeah.
01:12:41
Speaker
All right. Thank you. Thank you both for your time. I appreciate it. Yeah, of course. And thanks a lot for joining us and giving us insight into all this new tech that at least I'm not familiar with. I hope that's what I did. You did. That was really good fun. Thank you very much. Really appreciate it.
01:13:00
Speaker
By the way, we have an event called Dutch Closure Day. It's a one-day free conference about closure in Amsterdam. It's on April 6th. So if you want to join and then present, it'll be super awesome. I know by the time if you
01:13:17
Speaker
Amsterdam's great, so I'll totally keep that in mind. The weather ought to be decent. Yeah, of course. It's going to be really nice. I think it's usually springtime and it's a just one-day conference, totally indie sort of conference. So it'll be amazing if you want to join us and talk about Ethiopia and other things that you're working on. That'll be great. Excellent. Yeah.
01:13:40
Speaker
So that's it from us. One last thing though, Vijay. Before we go, I want to just give a quick shout out to some of the awesome Patreon characters that are helping us out. And I'll just give the most recent batch. We try and do just four or five episodes, and we've got four here. So I want to shout out our gratitude to Lubin Manilov.
01:14:09
Speaker
Thank you very much, Lupin. And then next is Dimitri Palamarchuk. I think it is. I think I got Dimitri right. The second bit I'm not so sure about. The next guy is Johnny Eradis. Now, it could be Eradis Eradis, but I'm pretty sure that Johnny is correct. And then the final one is a previous guest of ours, Mr. Daniel Compton, who's doing the closureist together.
01:14:37
Speaker
He's a solid guy. He's a totally awesome bloke. And in the end, what we're thinking is that whatever kind of remains we have from the Patreon sponsorship, we will actually put it into the Closureist Together pot because that's the best and most effective use of that money, I think.
01:14:57
Speaker
So if you want to join us on Patreon, then obviously we'll cover our costs, we'll keep going with the podcast. You know, we don't take anything out of it. And like I say, we will donate the remnants into Closureist together. So I really appreciate Daniel making that contribution. And it will definitely be repaired, not to him as well, but to the Closure community.
01:15:20
Speaker
So thank you very much again to the Patreon people and you can go to patreon.com slash deafen anytime and chip in a few quid. It's always good to have a sustained volume of income to help us out. Thank you very much for that. That's it from us. So thanks a lot more again and hopefully some of you listeners, I think we will see you at Dutch Closure Day or at some other event.
01:15:51
Speaker
Alright, thank you for having me.
01:16:25
Speaker
We can definitely burn some heat. Release some gas. That's a good one.