Transcript
Speaker: Welcome to Duffer, episode 16. Here we are with Mr. Malcolm Sparks. Hello, Malcolm. Yeah, hi, Ray. Hi, PJ. Hello, Malcolm. So it's Ray McDermott here from Belgium. And I'm Vijay from Holland. Where are you from, Malcolm?
Speaker: I'm from London and I'm currently in Milton Keynes in the office. Still working hard. Working hard, working late. Yeah. And it's Sunday. Is this your open source day? We still have about 400 conferences from the XD16 conference, so it's a nice place to come on Sunday night. No, I'm never here. I only came here because I need to get away from the kids because
Speaker: If we were at home, they would be coming in every five minutes. Anyway. And we'll have more special guests on the show. Yeah, very special. All right, fantastic. Cool. So, Malcolm, so let's just get stuck right in there. So what was the journey to enlightenment then? What were you doing before closure and how did you get to the parens? My road to closure.
Speaker: I was working on a Java project in a bank, and we were going to create something new. But we were working on this big distributed Java system using coherence. And I had a friend there who was using Groovy. And he was using Groovy to create a kind of REPL experience into this distributed system. And it was really, really good. And I thought, well, I'd love to have that level of interaction. And I was already playing around with a scheme
Speaker: called Cower and creating a build system with it. So I was really into getting back into this, you know, back in the day and getting back into it and just wish there was a good list on the JVM and one came along.
Speaker: So I got into it first. And then as soon as there was an excuse to kind of write this REPL for a new distributed system, I said, yeah, we've got to do this, but let's not use Groovy. Let's use Closure. And because it was only one little tiny thing, people said, yeah, that would be okay. And then we gradually wrote the web front end with it. And it wasn't too long before Closure became, well, making inroads into the Java app. So there's more Closure, less Java.
Speaker: And then we got a chance to write an application from scratch called Fandango, which was one of the, to meet one of the problems, the support problems in the bank. And we built it in about two weeks and it went.
Speaker: sort of live, kind of non -official live. It was very popular and very successful and it's still running now. So that was a good experience about how you can create stuff pretty rapidly with Clojure and
Speaker: it's pretty reliable. But how did you was at that time you were working with the bank itself or were you as a consultant or? I was I was working for the bank full time. Yeah, I was okay. But how did you convince them to say, Okay, I'm going to try to build something in a different language, because usually banks are like one of those very hard to change your heart, you know, the resistance to changes is very high. Yeah, it was it was a combination of doing something that was non critical, but
Speaker: doing something that absolutely couldn't be done in Java. So we wanted to create, you know, build a REPL on top of an existing Java system. So it had to be something dynamic. And then comparing how quick it was to build websites in Clojure, as opposed to Java.
Speaker: And then I set up a management thing where my manager left and went to join another team and they needed some other managers. So I put my hand up and said, I'll be the manager. Then you could set the direction. Yeah, I was the only volunteer. And it was all Excel sheets from there instead of IDs. Yeah, there were upsides and downsides with that.
Speaker: Okay. So how long did you stick with the closure in the bank? Because I know that you and John were also, I think he was in a different bank, wasn't he? Yeah, he was in a different bank. I stuck there for, it was about three years. I started doing some stuff in 2009, got really serious in 2011. That was when we built this.
Speaker: this support system. And then at the end of 2012, yeah, things were changing in the bank and decided to move on. Got in touch with John. We started going to lunch every week to talk about closure in banks. And then one thing led to another and we decided to start a company. Excellent. So, and that company was juxt, if I remember rightly.
Speaker: Yeah, I was going to call it something like simplect or something. John said, no, that's a rubbish name. We'll call it juxtap.
Speaker: So that's a nice, nice, nice function, right? I mean, you have all these people as the first parameter, like changing code, and then you have a list of programs coming in, and then you convert them using Clojure, and then you get different set of programs. So that's a nice name for a Clojure company, at least. Yeah, it's kind of a quirky function in Clojure company at Clojure Core, and we're a kind of quirky company. Fantastic. So it was just you and John, initially. Yeah, well, yeah.
Speaker: Initially, I went off and worked for a startup, a job I would have got if just contracting really, but I decided to go through Juxton and John picked up a contract at MailOnline and was instrumental in that for in MailOnline's decision to refurbish their website in Closure.
Speaker: And that was really kind of what got us really started because, you know, everybody has heard of the mail online. Yeah. Yeah. Yeah. So then you, what did you do then? You, you, you basically, uh, from the, from those other beginnings, you just brought, as you, as the, you needed more people, juxed, added more people to the staff or I guess with a blend of the, of the, of the company staff as well.
Speaker: Well, we really got a break with an old colleague of John's who approached John and said, look, I really need you to get back into this property portal business because we want to build a property portal. We've got to build it in 10 months and we've got nothing and
Speaker: Would you build us this thing? And John said, yeah, but I'm not going to build it in Java. I will only write it in Clojure, which is kind of a brave thing. And they went away and I think a couple of weeks later, they came back and said, well, okay, we've looked up Clojure, it looks okay. Go ahead. And so we built a team, John and I built a team together, and seven or eight people and started, well, you know,
Speaker: there were different individuals involved, but it got up to about seven or eight people. And that came on themarket .com, which was built. And we, I mean, I wasn't too involved in it, but John and his team did a great job and they delivered on time and did an amazing job. Yeah. And now you've got conferences and all this other stuff. So what's the state of jokes at the moment? Yeah, we've just released a
Speaker: Our first, I think it was a promo video that we got done showing the XD16 conference. I think John put that up last week. We'll put a link to that then. Yeah, that was our experiment into having our own conference and we really enjoyed it. It was a good experience. It was a bit
Speaker: fraught. It was a lot of work. It was kind of one of those ideas that was really in a pub talk, let's do our own conference. And yeah, that's a great idea. We've been at conferences. Can't be that hard. Yeah, it's just getting people together, isn't it? I mean, that's it.
Speaker: It was a lot of fun. We were very lucky. We had good weather. We had great food, great supplies. We had a fantastic event organizer that really helped us. So yeah, we just sort of managed to wing it. And it went OK. Cool. Maybe we should get into the more interesting stuff. The more interesting stuff. This is already interesting, Vijay. That's why I said more.
Speaker: I added a quality player there just to avoid. Your playing is shitless, Malcolm. Now let's talk about the watch level.
Speaker: Yeah, yeah, yeah, company, et cetera, et cetera. Let's talk about code now. Enough of this stuff. Yeah, get all the fluffy stuff out the way. So let's talk about one of the major projects, that open source framework slash library that you released. So what is the motivation behind it? And how did it come out? I mean, how did you get to the idea? Because in Closure Web Development, we have different kinds of
Speaker: I don't say competing, but, you know, different kinds of libraries filling different niches or like kind of community kind of settled down on composure driven stuff or at least some of them, I think. So what was what was the primary reason or motivation for Yada? Well, the primary motivation for Yada was that we were building on the market .com and it was a review, I suppose, after we went live.
Speaker: And it was talked to Martin Troyer, and he was very keen on ensuring that every web library has to be very defensive about what comes in. There has to be good parameter validation and 400s, which is indicating to the client they made a validation error.
Speaker: And there has to be strong defenses against injection tacks and all that, all that good stuff, which was already done in farmhouse. And I think some pedestal interceptors. And then we had somebody else on the project. Frankie was very keen on Swagger. And we had a bit of Swagger on the market that actually enabled the mobile app provider to access and see our APIs and document them.
Speaker: And then we had a bit of ring and composure and a bit of bitty and a whole lot of different things in composure API too. And we had quite a lot of different moving parts and it was an idea to consolidate. But at the same time,
Speaker: So that was one kind of checkbox, which was let's have something that can do parameter validation. Let's have something that can do swagger. And I was also trying to figure out how to make liberator asynchronous. We had David Hume
Speaker: had contributed a pull request delivery to maybe six months before that to make it all asynchronous. And I don't think Billy was able to accept the pull request for technical reasons that, you know, that this was a kind of idea that we've been talking about making.
Speaker: an asynchronous liberator. So it was an experiment to try and start again, because when you've got those three check boxes, those three constraints, it's really hard to then build that on an existing library. So I just went and had a blank piece of paper, an empty namespace, and just started playing around and very, very quickly discovered manifold, which was Zach Tillman's asynchronous library or kind of, yeah,
Speaker: It's a lowest common denominator library. It's a kind of interface that allows different asynchronous implementations to plug in together. So I very quickly discovered the deferred chain and started putting functions that could link up asynchronously. And it was that discovery that really propelled the direction of YADA. Okay.
Speaker: Yeah, it's good. I mean, I think there's a lot of, I think the time with async, HTTP, there's a lot of motivation for that, isn't there? Because I think people look at Ring, I mean, Pedestal, I think had some async stuff, but seemed to have lost its way a little bit, the Pedestal stuff.
Speaker: Um, cause, uh, I dunno, the adoption rate wasn't very strong with some of the documentation. I don't know if the people didn't work on it hard enough, but there was a need for async stuff coming around, uh, from the web teams enclosure, especially after core async was out and people were interested in node JS and all the kind of memory footprint type things that node was claiming engine X, you know, so that whole model of like, uh,
Speaker: Bloc non -blocking HTTP servers was definitely catching fire wasn't it? Yeah.
Speaker: Yeah, and enclosure was sort of late to the party because we were waiting a long time and sort of saying, well, async doesn't matter. We were on the JVM. We've got lots of threads. Yeah. Yeah. Martin Troyer had to credit again for kind of explaining to me and through his blogs about what async really meant and what it was and the advantages of it. So I was reading his blog one day and it kind of dawned on me. But, you know, to be fair to pedestal, pedestal came out before core async.
Speaker: And so then Coruscant came out and they had to refactor a lot. And I think that was still going on a couple of years ago and it wasn't clear to me that they had actually figured it out. Or at least asynchronous interceptors is something you have to do explicitly in pedestal. So you can do it and you can do it through Coruscant. But I wasn't aware of how
Speaker: how you did it. I wasn't playing with pedestal at the time. I was aware of it. I knew how it worked, but I wasn't trying to build on pedestal. No, I wasn't trying to criticize pedestal at all. I mean, I think for whatever reason, it just didn't catch on. I don't know. Also, the pedestal at that time was also like a full batteries included sort of thing. It has the front end as well and the back end. I think after we got enough of
Speaker: Closure script react related things pedestal guys decided that there is no point in pursuing it as a part of the whole framework anymore So that pedestal became purely back -end framework. Yeah, so ditching all the front -end things. So I think the front -end was
Speaker: Well, I think it's one of the iterations of the ideas, isn't it? It's like you take an idea and then you iterate on it, and then new things come in, and then they build up on the lessons that we learned from the previous one. Yeah. So that's what my view from pedestal is. Anyway, so coming back to Yara, so when did you exactly start this project, by the way? Yeah, it was two years ago. I think the question was about this time last, eight and two years. Okay.
Speaker: And this is extracted from the code that you have written or just the ideas that were taken from the work that you have done. Yeah, it was just it was a I started from scratch. Okay.
Speaker: So the patterns were there in different projects that you saw, these different patterns you applied for on the market .com and other places? Yeah, it was very much a kind of retrospective on a number of projects, the primary one being the on the market project, but other projects that we've done with Liberator.
Speaker: Yeah. So one of the, I think, critical things in a web framework is essentially routing or routing. I don't know. Is it routing for British people? It's routing. Yeah. Routing for Americans. Routing for Americans. Yeah. Okay. Well, screw it. I'm from a British colony or ex -British colony. Let's not get there. But anyway, so let's call it routing. So can you give us some idea about how the routing works in YADA?
Speaker: Like the bidirectional routing, you know, yeah, URL generation. Yeah, I mean, there's there's no routing in yard and that was an explicit decision. Yeah. And I got that idea from from Billy who I spoke to Berlin, your enclosure in 2013. And I was asking him,
Speaker: Billy, could you put, you know, when are you going to put routing? Because every web library has a router. Why doesn't Liberate have a router? And he was very clear about the separation of concerns here that a web resource is one thing. And the name or identity that a web resource has is another. And you shouldn't confuse the two. And if you want to simplify the web, you need to understand that these identifiers for resources are completely opaque
Speaker: strings, they don't actually have any meaning apart. Once you get beyond the scheme and the host and the port, the path is just a string. And so many web frameworks kind of get into looking at the path and breaking it up into segments and saying, well, this bit is some JSON resources and this bit is some HTML resources and this is some dynamic stuff. And they kind of
Speaker: I don't want to use the word complex. But they do that. Oh, no. Oh, God. He's imposing a vocabulary on everyone. I think we need to check the Google Zeitgeist or something. They publish the trend of the words that are used on the web. So complex was zero for some time. And then at one closure call, and suddenly everybody keeps saying complex. Yeah. You're not allowed to say it anymore.
Speaker: Let's say it braided the things. I think braiding is allowed, isn't it?
Speaker: Complex is a bullshit word, isn't it? Because no one's heard of it. But people have heard of mixing and mixing concerns. And I think that's really what he means. So screw complex, but we'll go with mixing concerns, because that's what you're really talking about. Well, I should say that Billy had another kind of key contribution, which is he said if there was a routing library, it should be a separate library, and it should have this quality that it was bidirectional.
Speaker: I completely agreed with that because one of the things that is really common to writing websites is broken links and you have text to make sure and people are always writing libraries to kind of string concatenate links together and it's a really it's a thing that people do by hand
Speaker: and they get it wrong or the URIs change or something and you get broken links. And that's one thing if you're a human and you're trying to click on a link and you get a kind of page not found. But if you're a machine and you're trying to follow out some kind of financial transaction, then that's a complete disaster. So we felt it was necessary to do something much, much better in that area.
Speaker: So do you want to just define what bi -directional means in that sense then? Because I think not everybody is familiar with that concept. Yeah. So bi -directional means that you have the URI and some algorithm that can look at that URI and turn it into a handler. That's typically the routing aspect of it. And then there's another algorithm that take the handler and generate the URI, which would target it.
Speaker: and that gives you a string. And of course, when we do routing, we have these things like parameters, which we might call them path parameters, which are kind of variables within the URI. And so when we generate a URI that has a parameter in it, we need to result and give the value of that parameter in order to generate the URI again. And the reason it's important is because when we create a web response, we might want to indicate to the browser
Speaker: all the other links, all the other web resources that we might want them to go to. We'll maybe talk about that later, about that being a key aspect of REST. And if you can't do that very easily, then you don't bother. And if you don't bother, it's a shame.
Speaker: Well, let's come on to that then because we talk about REST and page GDP and maybe is this hypermedia as the engine of application state. And, you know, I know when we were talking at a Euroclosure
Speaker: You were very motivated by this concept in general. And I think a lot of people that are real HTTP nerds or geeks or aficionados like yourself are very sworn to this way of doing things. And yet, for whatever reason, it's not very strongly adopted.
Speaker: So what can we say about that? What do we want to talk about with this benefit of rest? Because a lot of stuff we're doing these, according to most people, nothing is rest. And that's why Fielding says so. And he doesn't say it very often. So I guess he won't say that Yadir is rest. I don't know. Who knows? It's a fucking one -armed bandit as to what he says. But what do you say?
Speaker: Well, I know I'm trying to dig out a quote that I had from oh, yeah. Yeah, so There's a quote that I've got from Roy in one of those info queue interviews and he said rest was originally created to solve my problem How do I improve HTTP without breaking the web?
Speaker: It was an important problem to solve when I started rewriting the HTTP standard in 1994, 1995. So you've got all these people using the web and HTTP 1 .0 and they're having a great time. You've got surfing and lots of people building apps and businesses and stuff on it. And this guy's kind of tasked with writing the next version.
Speaker: HTTP 1 .1. He's got this really difficult requirement. How do you not break everybody else's fun? Can you turn off the web and get people to adopt a new one? He decided that wasn't palatable. He came up with
Speaker: I don't know if it was Roy, but I think a lot of, you know, you would credit other people with these kind of ideas, but he distilled all these kind of things in his dissertation about how the web worked and how to keep it working.
Speaker: And so when Roy sort of talks about what is REST and what it isn't, what he's really getting at is how do you put services out there that don't break the web? And I think that's a really valuable thing to do at an application level. How do you evolve an application, not a huge system like the web, how do you evolve a service and keep all of your other clients, who may be other businesses,
Speaker: How do you keep them working without breaking them all the time? And that's something that we still struggle with in software, you know, and we struggle with that locally, let alone globally. So I think it's a really important problem to solve. But well, maybe so you can say another way though, what is the actual problem? Because, you know,
Speaker: What's the problem with just putting out these APIs? Why isn't that a great enough answer? People are doing all these APIs. No one's crying. No one's breaking the web. Yeah, what is the problem with SOAP? I don't understand. Exactly. Yeah, come on. What's the problem? OK, don't troll him.
Speaker: But actually, it doesn't really matter whether, I mean, one could argue that soap is a bit more annoying than Jason, but it doesn't seem like it's changed that much in that respect, you know? Well, you know, I mean, it's just a bit of stuff, one of which is slightly more amenable for us to read or to pause or whatever. But the fundamental concept is somewhat similar. But the problem with soap, of course, is that it was not verb based.
Speaker: You didn't use the HTTP contract. Is that what we're kind of talking about here, is to reuse the web verbs? I think we're talking about how brittle an interface is. If it's very sensitive to, like some SOAP contracts are very sensitive to the parameter types and it's possible to evolve a SOAP interface in a binary compatible way.
Speaker: And we're going to get onto Richard's talk quite soon, aren't we? There's a lot of echoes. It's in the air, so why not? Yeah. I mean, these are really important ideas. And if you create a brittle interface, as people do, where they say, right, you've got to call this URI, and then you've got to call that URI, and then you call this URI for doing this and this action. As soon as you make a change, you break everything. You say, right, now everybody, you've got to upgrade.
Speaker: everyone's got to upgrade by Christmas because I'm going to turn off the old version and then we're all going to move together as one to the next version. And the world just doesn't work like that because everybody has different deadlines and release schedules and stuff. You can't just demand that everybody upgrades and that is the problem.
Speaker: But it's also the problem, again, maybe as I'm misunderstanding the problem, but some of the problems were that SOAP added a whole load of...
Speaker: imperative verbs, like a lot of the stuff that you did in the contract in SOAP was essentially above and beyond the web itself. It kind of went over the top of the web and had its own language essentially to specify contracts. It didn't reuse the underlying infrastructure of HCP where REST seems to want to do that. That seems to be a critical difference in my understanding anyway.
Speaker: Yeah, I think another critical difference is that in SOAP, it isn't well -defined what are mutable operations and what are mutable ones, at least in my understanding of SOAP. You can do anything and that thing that you do could be idempotent or it could be side -effecting, could launch missiles or it might be just getting the status of the missiles.
Speaker: separating reads from writes is really, really key because if you do that, you can then cache your reads and you can scale your reads out. If you know that you're dealing with immutable data, then that's very good knowledge for a cache to have because then they know that data is still valid.
Speaker: Well, isn't it the fact that if you use REST, then you can use all of the infrastructure of the web? So all of these kind of caches, if you set a not modified, or if you set an expiry date on a resource,
Speaker: to be a long -term future thing then all the intermediary caches can look after it and the browser knows about these things. We're in SOAP, it's an extra protocol essentially. All of those things are ignoring the fundamental infrastructure of the internet, of the web, sorry. Yeah, I think so. In that people didn't write
Speaker: in hardware proxy servers and things that would have knowledge of SOAP built in, maybe because I don't know enough about SOAP to know whether those qualities of knowing when something is mutable or immutable or idempotent, I don't know whether that's...
Speaker: Also, the primary thing with SOAP is that you have the SOAP with RPC. That means everything is a remote procedure call, so you're calling functions. And SOAP happens to be that you can have SOAP via SMTP or any other protocol.
Speaker: Stupid discussions to have now Interesting part about it is that is that or as far as I'm concerned the interesting thing about it is that this this hypermedia as the engine of application state Is is really kind of the apotheosis of rest? And yet it's it's it's kind of seems to be a bit elusive at the moment. So
Speaker: And we talked a little bit about that, about the fact that maybe people are, I don't know, fast food junkies. But it seems to me, I just put this in the discussion. There was a guy called, I mean, you probably know this already guys, but there's a Richardson's maturity model of rest. I don't know who this Richardson guy is actually, so maybe he's...
Speaker: As long as you can tell me. But here's this bloke who says to everyone, oh yeah, okay, we've got this maturity thing. And I love the fact that the level zero I think is pox, which is a plain old XML. Pox is a great word for it, you know? Especially for the XML files are small.
Speaker: Yeah. But, but I mean, what I was going to say was that like there has these level one, level two, level three, uh, things. And as a colleague of mine, uh, starts to write these in the APIs now and essentially judges people's, uh, judges people's APIs by their Richardson's maturity model. That strikes me as a bit, a bit harsh, but I don't know. What do you think about that, uh, Malcolm? Because we're, where are you supporting or how are you evaluating those kinds of things in jokes or in yada? Well,
Speaker: that one of the things that is making hypermedia elusive is we talked about the ease of generating links because it is all about the links. So that's number one. So that's really something that we're trying to tackle with Biddy. Number two is that you have to have an agreement between the client and between the server about where those links are or that the document has links and
Speaker: if it has links, where are they? And you have to be a bit explicit about that. If you have a link in front of a human and you color it blue with an underline, they can see that it's a link and they can click on it. If you do the same for a machine, you've got a market, you've got to say this is the next chapter of the previous one, or you've got to have some convention about what next and previous means, if you name all links.
Speaker: And those conventions are meant to be encoded in what's called a media type, which is a document which explains what the contract is. And it's meant to be a document that you publish and you say, this is what media type is. This is the contract, version one of the contract. And then the clients then can code that contract and so they can be looking for these names. So even if you change the URIs, then there's a degree of
Speaker: There's a degree of flexibility. Your API doesn't break just because you've changed the URIs because you've kept those link names the same. So as long as you have that media type agreed, you can then evolve it. You can have a new media type called version two and version three and version four. And what you're meant to do is then let the client and the server negotiate as to what
Speaker: version of the media type they've been coded against. So the server would typically, it may support version one through version four, it may no longer support version one and a version one client would then not be serviced.
Speaker: or a client might be coded to version three. So they negotiate and say, this is what I can read and this is what I can provide. And if they come to an agreement and they say, right, this is what we're going to talk about, we're going to talk version three, and then the server produces version three of the data and the client then can read it because it's been coded by programmers who understand what the media type version three means. And so you need to have
Speaker: everything in place. You have to have pluggable media types, you have to have links, and you have to have content negotiation for all of that to work. And when you've got that, then you have the ability to version and move and evolve services without breaking anybody. And I think that is the
Speaker: That is the holy grail. But what about these because I mean, yeah, we looked at this with the media types with things like atom and stuff like that. Because obviously, I think most people would like to use standard media types, if they possibly can. I mean, just an API, for example, is a kind of media type where there are well known links.
Speaker: What kind of, you know, what other examples would you maybe cite as, as, as kind of like usable media types in the API in the kind of service interaction space? You know, I, I, I really don't know that many. I mean, I want to get into that area, but I, you know, and perhaps I'm explaining what you might do.
Speaker: coming up with your own media types. And I'm sure there are good initiatives to create reusable media types that others can use. But we're not just talking about JSON. We're talking about something beyond that. To say, well, this is in JSON, but that's just the format the media type might say, well, it happens to be in JSON, but it is actually a more specific declaration.
Speaker: I think the reason behind JSON API is just to sort of essentially assert some of these things. It's essentially to say, here are a bunch of, here's a media type, here is the kind of links it will provide, here is the kind of status codes will provide. So although it happens to be using JSON, fine, but there could be other APIs like that, and those RAML and stuff like this.
Speaker: Like one of the, maybe it's a bit of a digression, but one of the, of course, our entire podcast is a digression anyway. So when you're building the REST API, so one of the biggest challenges is to design the API or the URI structure in such a way that you can reach all possible resources properly.
Speaker: And especially when you have an object graph as a model, then it is very difficult to specify at what level I want the data to be fetched. For example, if you see the Jira bug management software,
Speaker: If they have some sort of REST API, so they have a hierarchy of objects, they have projects and users and projects has issues and issue has more detail, blah, blah, blah, something. So it has always been like a contention point when I'm designing such a REST API to decide how much data to return and then how to give the control to the client to say, when you ask for a project, what kind of fields that you're interested in, let me know. And then we are going to respond with the appropriate
Speaker: nested data structure that they can use. Because otherwise, in just one query, they can get the entire data model. That's not going to be happy. That's not going to make their lives happy. And the interesting development in that one was the GraphQL. So you give a query. So what is your opinion on this one? How do you solve this kind of problem? Do you think GraphQL is one of the way to solve this?
Speaker: Two points. Designing an API where you list all of the different URIs, that's found a point in REST. You're meant to go to the only URI you really ideally have to know is the root URI. And from that, you get all the information about what's available and how to navigate it.
Speaker: And that means if you change your structure or you change your object graph, you're not going to break a whole ton of clients. The second thing is that, of course, a given URI can take query parameters. I mean, that's part of what a URI is. And URIs are designed to take queries. Google .com is a very good example. So there's nothing to stop you.
Speaker: defining your query language in a media type and saying, if you send me a get with this type of query string, I'm going to send you that stuff. And that's not really, people think, well, that's, you know, rest is dead because query strings are useful, but, you know, we know query strings are useful and people should use them more.
Speaker: Yeah, so it's not it's not a competing thing, but it is complementing the rest way of looking at the things. Yeah, I think I think what it competes against is the practice of defining all of your
Speaker: different URIs that are reachable and providing those upfront in a document, which is kind of what Swagger does. The other one of those was from our good friends at IBM and SAP and Microsoft, wasn't it? The OData standard.
Speaker: they allowed you, in your query strings, they allowed you to filter the objects and to search for objects. So they gave you a kind of query language on the front end. But the problem with that was that actually it wasn't discoverable. The vocabulary wasn't discoverable. But I don't know how you, on the other hand, order is a published standard. So you could discover it by reading the standard. Yeah, I mean, it's things
Speaker: either discoverable or they're written up in a document in a media type. And between those two things, you have rest. So I think as long as every data says this is the syntax, and these are the things that you can query for, then that should be enough.
Speaker: Yeah, okay. Right, so I think this has been a really good discussion of the various bits and pieces around that. What I was going to ask you about next was this something we're interested actually, it's kind of like this query thing is the
Speaker: the server sent events aspect of yada and it's something which which I've been playing with a little bit recently with Kafka hooking up Kafka topics and then essentially publishing out for given clients and
Speaker: It was really weird because, you know, I only found out about SSE, I don't know, maybe it's a year, 18 months ago. And yet they've been here around for quite a long time, hiding in plain sight, this kind of model for notifying the clients. I don't know, you guys at YADA support them. Do you see a lot of use for them in your work or is it still kind of coming? No, I think it, well, SSE has been around for
Speaker: years and years and years, it just hasn't. It's been part of the HTML5 set of standards. It's an event source, yeah. And it's kind of taken a backseat to WebSockets because WebSockets used to have all the media attention and people will use WebSockets for stuff. But then, of course, you have to reinvent your own security and cookies and all of the kind of security
Speaker: like cross origin resource sharing and all that sort of stuff that's being built on top of that safety net that's being built on top of HTTP, you then lose as soon as you upgrade to a different protocol.
Speaker: So people Yeah, that's just TCP IP basically, isn't it? Yeah. Yeah. And people laid that under the kind of misapprehension that the only way of doing notification was web sockets. But there's this kind of small thing. So the center events really simple doesn't take up a lot of documentation to explain it. And now it's pretty much available everywhere. And even in
Speaker: old browsers, you can get polyfills that essentially give you that JavaScript event source. And then of course, you get all of the benefits of the security, the cookie passing authorization, headers are passed and that stuff. It's not perfect, but it does the job. And if you're not writing a game on the web, but you don't need that really, really low latency, then it's good enough. And I think it can make some really nice websites with interaction and
Speaker: Well, you see it everywhere. I mean, when you close an issue and push it on GitHub and you're looking at that issue, you can see it close in front of your eyes. It's a nice, nice experience. I think the difference is, if I'm not wrong, is that like WebSockets are bidirectional and SSE is unidirectional. In other words, it's just only from the server to the client. Yeah, but you've already got the other direction because you've got post. Yeah, exactly. Yeah, exactly. Yeah. So.
Speaker: So screw you! Whoops, on it. Yeah, don't do it. Yeah, I think the security thing is pretty awesome. Talking of security, actually, this is something that when I was reading the documentation of Yada, I found very interesting, which was that you say that security is part of the resource, not part of the URL. Yeah. Do you want to talk a bit more about that?
Speaker: Yeah. I guess a lot of it is kind of post -factor justification for the way things are. But really, there's a lot in HTTP specifications about security, multiple realms, and different authentication schemes and stuff. And it's
Speaker: it all applies to a given web resource. And the specs just assume you just have this one web resource, but of course they know you're going to have many. And the way that we group web resources
Speaker: in most web frameworks is we group them with a single URI scheme. And I guess that, I can't, what can I say? What do we say, braids and tangles? So you can only really have one hierarchy of security if you've welded it to your URI scheme.
Speaker: then you've got that, instantly you've got that parallel hierarchy. So security might be, it might be in certain circumstances that you want to create different types of groups or you might want to do security based on, not on the URI, but on something about the user's credentials or something about the type of web resource it is.
Speaker: So I think any time that you look to create groups of resources simply through the URI, you lose the opportunity to create richer divisions and different categorization schemes and different hierarchies. So if we separate those two and say, the URI is the URI, leave it alone, and then you define a different hierarchy for security. So we do this in YADA by,
Speaker: you can tag resources and say, this is private, or this can only be accessed by Frank, or whatever. You put that in the metadata. And then what I tend to use is post -walk, which is Stuart Sierra's library, and close your call. And once you've got all of your web resources in a data structure, you can then walk the tree and then look at every web resource in turn and say, oh, it's got this tag called Frank, or it's got this thing, and I'm going to then
Speaker: adding my security policy for that thing. So it gives you more flexibility, which may be valuable.
Speaker: Yeah, because I think the classic one is where you have this URL that starts with slash admin. Yeah. And what's the difference between the slash admin view of a page and the normal users view of a page actually pretty zero, except you might want to you might want to sprinkle in a few menu items or something. But that's kind of annoying that essentially you have the same actual resource.
Speaker: just have a slightly different view of it, and it shouldn't really be through the URL that you express that. Exactly. That might be, you end up with a parallel hierarchy just because you need that bit being protected and that bit not being, and actually the protection should be something about the client's credentials, not part of the URI, but part of what request headers are being sent.
Speaker: And that probably, again, comes back to how people make routing complicated because they have this parallel hierarchy. Okay, cool. What about other authentication schemes like OAuth 2 and stuff like that? Does yada support for those kind of things? It does. It's quite hard to find, but it's in there.
Speaker: We use it on the website. It's an exercise for the reader. It is an exercise for me to document it. It is very high on my to do list. People have asked about JWT and other authentication schemes.
Speaker: And JWT is part of OAuth 2. The only part of the implementation of OAuth 2 in Yada is using JWT. And OAuth 2 is quite easy. And we have it working. We have a login in our own website where we can log in with a Google credential or a GitHub credential. And we just let people are in the drops to your organization. They get a login automatically. And they can log in with their GitHub. So that's much better than holding databases or passwords.
Speaker: All right, so I think we've done a good job there. I mean, I'm just wondering whether there's anything else in the current instance of Yada that we've missed, Malcolm, that you want to... There's quite a lot more, but there is a lot in there and partly that's because I'm really trying to implement all of the specification regardless of whether people want it or not. So one of the things I'm working on is
Speaker: something called reactive.
Speaker: negotiation. So when you do content negotiation, there's two types. There's proactive, which is where the client says, these are the kind of things I can deal with. And the server determines, says, well, I just pick one, right? There you are. And reactive negotiation is when the client goes to the server and the server just sets out its stall and says, these are all the stuff I've got. These are all the different types of representations I've got for this resource. Which one do you want? And the client will say, I'll have that one.
Speaker: And that's quite a nice, there's definitely, I can see some circumstances that would be useful. So I'd like to implement it. So there's a thing on range requests and partials that I don't implement in Yado and I'd like to. Not many people use range requests, but, you know, cause it's in the spec, it's kind of fun to, you know, not really worry about
Speaker: the fact that people don't want it. I don't have anybody sitting at me. I don't have a manager with the artist telling me, I've got to do this, I've got to do that. The spec is my manager really. I invested in this for years. I want to carry on working on the other for many years.
Speaker: you know, it's one sometimes you say, well, I just don't have time to implement at all. But actually, you know, we've all got forever, you just have to think in another dimension and just say, well, as long as I keep tapping away for the rest of time, then eventually I'm going to get around to it. Another other people working on it, Malcolm, or is it just a Malcolm Sparks labor of love? Or if you have you got contributors, adding stuff to the other as well?
Speaker: Well, not really invited, but there is 26 contributors in the project, so not a huge number, but there have been some amazing contributions that have, there's one contribution that
Speaker: improved performance in one aspect of Yada by a hundred times. There have been others like that and it's been transit support added. I should acknowledge all these people. I do try to acknowledge everybody who's made major contributions on the Readme and there's quite a long list.
Speaker: Yeah. All right, cool. OK, so we did something new for the show. Usually, we don't pre -announce who or when we are recording. But we did this today. And we asked the internet to come up with questions. And to our surprise and merriment, we have at least three questions from different channels.
Speaker: So here is one. I really feel like we're like a legitimate show now.
Speaker: OK, I'm going to open our audience from South Africa, Mr. Rob with a phone. He's asking. Friend of the show. Friend of the show. So anyway, so his question is, at Jext, how do you hire closure developers? Do you train them up? Or do you hire seasoned professionals? Or if you train them, how does the training look like? Hello, Rob. I'm fine.
Speaker: Well, what does the training look like? Yeah, we do both. So we hire seasoned pros if we've got projects ongoing and we need help. So we have a whole network of people that we can call upon, some of whom are available at any given time. And that's really great. And we've worked with them before, a lot of them. And we do have quite a
Speaker: quite hard to kind of vetting process for people to join in existing projects. But we also take on new people who don't know closure and train them up by, well, we often, I think the first thing we do is put them on our website. And so our website is kind of the project where people can make their mistakes because it doesn't matter. Or we can clean it up or stuff. So that's one way. And another one is to get them to, you know,
Speaker: join either an internal project or a project where we do some projects where we deliver and develop the whole system and they're ongoing and we support and maintain them. So we get people to maintain them for a period of time and people pick up closure pretty quickly.
Speaker: Okay, so speaking of your website, there is another question from Slack from Anne Montero. So he was asking, well, not exactly a question, but I'd love if the show touched a little on Malcolm stock at Clowish tray in Finland, what they are doing with jext for their website. Yeah.
Speaker: Yeah, so what is a closure trail? I explained that we use this library called Skippy McStipface, you know, in our website. What Skippy does is
Speaker: There's a whole bunch of state in our website. There's markdown articles that we have for our blog. There's config, and there's closure code, and there's some ASCII doc, and there's some HTML, and there's some SAS for CSS, and there's a whole bunch of stuff. And you can create a whole graph. And then what happens is when a request comes in,
Speaker: we check that that graph is fresh. So that whole graph is in this derefable thing. The root of this graph is derefable. So when the request comes in, we deref that graph, and that freshens the whole lot. So if a file is then noticed as being the date on a file looks like that it's changed, it's a blog article that somebody's edited, then the system knows that it has to repars that markdown
Speaker: and it might have to turn it into HTML. And it might be that the title of the markdown has changed, the title of the blog article has changed. And so it then has to update the blog index. And so it might have to change the menu page. And that bubbles up all the way to the top. So that's one part of it. And it means that every request comes in. They do quite a fast refresh, which is optimized. And they then get a single view of state.
Speaker: it's deref. Very much like you do in ClosureScript, you deref an atom and you get this kind of thing. We try doing it on the server. And also we have another thing called push to deploy, where we push to a Git repository, which is on our server. And it's a working tree that's expanded. And we do this little Git config thing where it updates the working tree. So because we're running the website site on that working tree, then every time we push something that it makes
Speaker: a live change to the website. As soon as somebody visits the website, they might see a slight delay. Then there are these watches that are looking for changes and they're tied up to the Java 7 file watching service. They're being notified by the file system if anything has changed so that they can in the background try and update this, freshen this big state graph.
Speaker: So all of those concepts are there to try to build a system which is very much like a REPL where you don't deploy it. Everything's deployed all the time. So it's continuously integrated all the time. There are no builds. You just make a change.
Speaker: and it's instantly visible. And so we develop on it as well. We have a beta website and you can push stuff and you get that REPL -like experience. ClosureScript, too. We built our own ticketing system for X2 .6 .0, and we've got ClosureScript and payments and all kinds of stuff in the website. I mean, it's a really, really
Speaker: kind of like lots and lots of commits are really old. We've had it right. The history of the website is the history of jokes. We've had it from day one. And it's just it's one of those things with so much technical debt, but then so much refactoring and getting out of digging out of the technical debt. And it's a good
Speaker: experiment to see how mucky a closure project can get and how much you can kind of clean it up and keep it going, which is really what most systems are. At the end of the day, you can't really keep out of technical debt. So it's an interesting experiment how good closure is at getting mucky. So is it like you basically use Git as the CMS? Yeah, exactly. Yeah, so you push and then that's it. That's gone live. And that's why we have a beta. Of course you couldn't then
Speaker: unpush, you can always go back to the work of init. Okay, so final question from the community then, Sean Mahood from Slack. So he's asking, did he, did you look, well, I'll just read the question. Question from Malcolm. Has he looked at spec much? And if so, does he have any thoughts on how things would change in Yada if Yada were to replace schema with spec?
Speaker: Yeah, I have had a look at spec and I've been working on a branch for some time about how I would do yada with spec rather than schema and some things definitely do change but I still like the aspect of yada where you can write data structures that are in shorthand and we call this data macros where you write them in a shorthand and then they get expanded to
Speaker: What would be a canonical form so that so that yada only has to worry about one canonical form of the resource and not all these different variations But the canonical form is a real pig to author so it's nice to have these short forms
Speaker: And I think that that aspect of yada will be replaced by something that would just do a tree walk with post walk and expand it that way and stay out of spec. I know there's conformers and stuff in spec. I try to emulate the same thing, but it gets difficult and I don't want to braid the two ideas together.
Speaker: But yeah, there are quite a lot of things that I would like to put spec into. The request context, for example, doesn't have a schema behind it. I'd like to get every interceptor to define
Speaker: What they need out of the resource rather than having one big spec like I have one big schema for yada at the moment and doesn't feel very modular and I don't think it really accords to what rich is saying about what spec should be used for and I think spec is about declaring the
Speaker: the requirements that you have as a piece of code on a structure rather than pushing. It's really a pool -based thing rather than pushing. So I'd like each interceptor to say that these are the requirements that I have on the resource model and for that alone to be enough. So there's quite a lot of experimentation to do and I'm definitely not going to experiment in the master branch. So I'm just playing around. But definitely I want to figure out how to do it before I
Speaker: before I go live with it. And that might be sometime next year. Okay, I noticed as well that like when I was doing the resources that you essentially you kind of provide you're asked to provide a schema.
Speaker: Yeah. For each of the items, so like you've got ID1, ID2, they're both, you know, you have to specify that they're both a string. So is that an area where also like the surface of the adder might change a little bit? Whereas you could be asked to provide a spec rather than a schema. Yeah, definitely. That's the other usage of schema, which is much more in the domain of
Speaker: So I was just talking to my own internal use of schema in Yarder. But yeah, of course, for all the Swagger properties and the Swagger spec generation, then that's all in schema right now. And Tommy Riemann and others in Metacin and that community in Ring Swagger have definitely made big inroads in turning, creating a spec version of that. And I was talking to Tommy a couple of months ago in Closure Trail.
Speaker: about how that was going. And it's definitely possible. So that will, you know, I'll have a look at that. But at the moment, I'm trying to separate a cleanup yarder a bit and separate the dependency graph. It's got a very big dependency graph at the moment, and it's much too big. So there's a lot of stuff like JSON and swagger and stuff that
Speaker: that are in the core and I'd like to sort of tease Yada apartment, kind of untangle a few things. So I'm doing a lot of cleanup. There's a lot of code in there that's sort of a bit old and needs a bit of a review. So I'm on quite a bit of a cleanup exercise and doing more documentation and trying to make Yada much easier for people to use.
Speaker: looking at the code myself as there's bits of it that I just think needs some work. So onwards and forwards. But if we were going to shout out to the community, then maybe these documentation pull requests or other kind of cleanup pull requests might be welcome. Well, actually, I find it's quite good just looking at what people are moaning about on Slack. And that's often an indication of where the, you know,
Speaker: Yada had a really busy year of development last year and then this year I mostly tidied it up and got to 1 .1 and then been playing with and using it on projects and seeing how people have
Speaker: implemented it and seeing kind of what has come out of the trenches and what people find difficult. And so I've kind of slowed down development and just sort of gone into listening mode and watching and seeing where things are happening. So I'm picking up on that. But I know I don't really I don't really need to feel the need to sort of ask people to write my documentation for me or that sort of thing. I think right. Well, the level of community interaction and the raising issues and just being patient with me while I think about stuff.
Speaker: I'm still in the mode of having to try and solve a lot of the problems in parallel. So I'm not racing ahead. And so there's a lot of issues that I haven't touched and I know about, but I just haven't fixed and I just kind of need the community to bear with me.
Speaker: Well, I think we will for sure because it's already a very, very useful and fantastic framework. I've done some stuff with it recently and I was very impressed. So I think you've done a great job, Malcolm. So kudos to you, hat tip.
Speaker: I think we should maybe start to wind it up a little bit Vijay, what do you think? Yeah, of course. Thanks a lot to Malcolm, to take the time on Sunday evening and walking all the way to your office and spending time with us. Before we wrap up the show, of course, we want to
Speaker: touch upon a couple of important things. First of all, as people in the closure community might have heard, Anthony Grimes, who is known as Raines, or IO Rain on IRC, has passed away.
Speaker: which is a pretty sad news for the community. I think he has been, well, I haven't met him, but I was on IRC most of the time when he was on IRC in the early days of closure, and he was also the guy who made tri -closure.
Speaker: And he was also the one who made Lazybot on IRC, so you could actually try out Closure Code IRC bot. And when I heard the news, I was looking at all my Closure Code and grabbing for Reins, you know, my io .reins. There's plenty of libraries. I was just looking at the GitHub, and there are 45 Closure projects that he made. Some of them probably everybody is using these days.
Speaker: The code lives on, I think. It's a hard time. And there has been a GoFundMe campaign for helping out his family. And I think Chas Amorek is leading sort of an online, a memorial service or something for him. I think the GoFundMe thing was very well supported. Actually, I had to look at it at the end.
Speaker: So I think his family will be able to, because I think there was a problem getting him back home or something. So that should be covered, which is great for the family, I think, but very early, very young man. And so it's very tragic for him and his family. That's for sure.
Speaker: Yeah. So yeah, I mean, like every now and then, life gives you some sort of a perspective on what is important. So that's a very heavy news. On the interesting, well, less heavy news, I would say. So we had Closure Conch, and Closure Conch videos are up. We are still going through the things. I think, Ray, you already started looking into the videos now a bit.
Speaker: Add a little bit. Obviously, I think we probably all watched the rich talk about versioning and being nice to clients. Basically, you're saying that Semver was flamed. Because if you have a minor or a patch version, well, nothing changes, so what's the point? If you have a major version, then you're basically an asshole, so don't do that.
Speaker: That's my TLDR anyway. I think one of the joke was Semwar has a version. The semantic versioning has a versioning thing. I don't know what that means anymore. So it was interesting. But anyway, I think we'll go through the stuff. And the other interesting news is that the datomic licensing model has changed. And there has been much
Speaker: Merriment and joy in the community, but there is no free datomic licenses. Of course, if you can download datomic and you get it for perpetually on the same version or something. So there are more details to it on the blog post. I think it is free still. I think the only limitation was that the starter kit
Speaker: Used to be renewable every year, but now it's the same. Yeah, but otherwise it's still it's still an awesome thing. The thing I didn't realize as well is that they have a new client as well. It's only an alpha stage, but they have a new client library which is meant to be a lot simpler to use than the.
Speaker: than the peer library. So yeah, so we'll see how that works out as well. Yeah. But yeah, as far as the kind of videos are concerned, I want to call out one more thing actually that I've watched, which was this proto REPL. That was goddamn phenomenal.
Speaker: Um, this guy's done a really great job, uh, just at the beginning of it now, but, um, it's very, you can make a lot of very visual, um, stuff. And he used another project that was in one of the talks called side, which allows you to have tracing. And so it was very, very interesting how, you know, his work, and I can't believe the guy's name, unfortunately, but their, their work combined will, will give you a lot more visibility into, um,
Speaker: into your closure code and a lot more power when you're editing it as well. So although it's a REPL, it's actually built on ATOM. So if you edit your closure in ATOM, then you can kind of have this REPL and it's much more visually interactive than typical REPLs. So yeah, very, very interesting stuff. But I think, like you say, we'll probably pick up with more videos as we go through them in the coming weeks.
Speaker: Yeah, of course. And we'll go through the videos and we will update our audiences and then help them in picking which videos they must watch and which videos they should watch. And well, that's the two categories, that's it. And of course, there are a couple of events that are coming up. So there is a closure D that is going to be in February. And the other event, which is, I think, much more
Speaker: Interesting. Which has more gravitas, at least in our podcast and as well as for me. It's a Dutch closure day, which is going to happen in Amsterdam on the March 25th, 2017. We just started the Necessary Preparations. It's going to be a free event just like the previous one. That means I'm going to spam everybody who says closure anywhere for sponsorships.
Speaker: So last time we did the same thing because we want to make sure that this is a community driven event. So it's only by donations by the sponsorship that we were able to pull off last time. And also Dutch Closure Day 2016 was the reason why Deaf and podcast exists. So we started the discussion on 2016 when Ray was there.
Speaker: so who knows you know and this year you might have another podcast or something else or we never know and the call for proposals is open so we'll put a link in the in the show notes so please take a look at it and submit your talk it's going to be another amazing day that we want to plan
Speaker: And if you know anybody who wants to sponsor this event, let us know, or at least let me know. Tweet me up. I'd like to talk to you. That's it for the events, I think. Anything else that we missed? I don't think so. I think we just...
Speaker: Just say thank you to Malcolm again and just quickly say thank you to the people that help us produce the show as well. We have Pizzeri doing the intro outro music with his melon hamburger so if you can look at his soundcloud that would be very good. We have the logo was produced by Love Off Sultan and we provide some information about the work that she's doing. She's a freelance designer as well so
Speaker: If you want some great work from her, then hit her up on Twitter. And we have a friend of mine, Wouter, who's helping with the mixing and fixing of the audio. So thank you very much.
Speaker: So those are the ragtag travelling thespians and we help as basically all respect to those people.
Speaker: We can insult each other, that's fine.
Speaker: you
