Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
3d32130bbe3d Drew Raines
734 Plays5 days ago
Where Drew comes to the parentheses and how he was stretched and has risen above.
Guest
- Drew Raines
Hosts
- Josh Glover
- Ray McDermott
MP3 SHA
3ff7608352b4181e562dddcfc85b06fa9e4a8d524753f8b800b16da8a95c4319
UUID v5 from SHA + People
d858274c-711f-58f4-b148-3d32130bbe3d
Recommended
Transcript
Introduction and Guest Welcoming
00:00:15
Speaker
Let's just say welcome to the next episode, Deflin. And we've got a, it's kind of a peculiar story, this which we'll get into maybe as we start, but you'll see from from the north that we've got Drew Reigns on the podcast. Hello, Drew.
00:00:32
Speaker
Hi, guys. How are And I've got myself and Josh as the hosts. Hi, Josh. Hello. Thanks for having me, Ray. It's a pleasure, always.
00:00:42
Speaker
know You're here to be pilloried, and you serve that purpose very well. Yeah, thanks. Thanks.
A Delayed Invitation: Drew's Story
00:00:50
Speaker
Yeah. um So yeah, um maybe as we can kick off with a bit of a story about like how I invited you onto this podcast, Drew, four years ago, I think.
00:01:01
Speaker
think it was four or five years ago. have to check the Slack site. It's something like that. It's really a crazy timeline. This is such a deafened thing to do, you know, professional podcast here. Yeah, but it was... There's this weird thing where I was working voucher. I think I was working at, and we were looking at doing some mail stuff. And I came across your library that did mail with closure and it was really impressive. And I was thought, Oh,
00:01:27
Speaker
this guy is you know fabulous. It would be really great to to have him on the show to talk about this stuff. And I reached out and there was absolute crickets, nothing, tumbleweeds.
00:01:38
Speaker
It was like, okay, well, that that you know some fell on stony ground. My joke earlier on about the joke waiting for an audience, this was a a message in a bottle out to Drew, you know?
00:01:49
Speaker
And then finally, like with a long beard shaggy clothes, this bottle finally arrived at the beach or you came back to the beach. I don't know. And you you hit me back up and it was like, wow, okay. Yeah, it was ah it was quite something. i I was not um a, ah I wasn't,
00:02:11
Speaker
in the Clojurian Slack um instance very ah very much. And ah I had some reason to to check in there and saw your message. And I was actually just looking and it was it was two and a half years later.
00:02:27
Speaker
Oh, okay. Yeah, I responded. And yeah, i you know you were still interested. So here we are. and this is fantastic Yeah, it's great to have you here. And and like you were kind enough to send us over some kind of ah background notes from your life, which 25 pages long. So thank you for being so thorough. You're welcome.
00:02:50
Speaker
Yeah.
Shared Backgrounds and Humor
00:02:52
Speaker
looking forward to getting it published at some point. Yeah. Well, 10 out of 10, no notes. This is, you gripping reading, but I think actually we don't even need Ray on the show for this. Cause it sounds like okay you and I could just talk about how we did the same stuff as kids and as adults and are the same person. I think proof that that cloning works. Is that, is that what it is? Yeah. uh,
00:03:21
Speaker
I wrote up some kind of highlights and think possible things to talk about. And, and Josh was like, wait a second. I'm from Virginia. uh, got into closure at the same time. i ah also have worked in mail administration and all kinds of fun.
00:03:40
Speaker
So yeah, it was, uh, it was, it was fun to see that.
Exploring Country Music's Diversity
00:03:45
Speaker
Yeah, fellow lover of higher order Pearl, which is like one of my absolute favorite programming books of all time.
00:03:53
Speaker
Yes. Just brilliant. Absolutely. Yeah, yeah. MJD, if you ever listen to this. Yeah. and I'm sure he does. I'm sure. Yeah. Yeah. I shouldn't say if, when you listen to this, uh, Mark, your, your, uh, your fingerprints are all over this podcast.
00:04:11
Speaker
This episode is also a message in a bottle, you know? So who knows when it's going to reach him? Yeah. Uh, maybe, uh, maybe we could email him. Um, yeah. Oh yeah. Yeah. Yeah.
00:04:24
Speaker
Sorry. That was looking for an audience. It's still, still looking, I'm afraid. Yeah.
00:04:31
Speaker
Yeah, so you're in your you were you are from Virginia, but you were telling us you're in Nashville now. Yes, Nashville. Which is the home of country music. That's right. Music City, USA.
00:04:42
Speaker
um many Many of your favorite artists um either live here or pass through here quite often. so Yeah. Have you been to the Ryman Auditorium, I hope? I have. It's a wonderful venue. Have ever played there?
00:04:57
Speaker
I have not performed there. I'm not that good. But I have been there. um It's a wonderful, wonderful place. I encourage everybody to see a show there at some point in their life.
00:05:09
Speaker
Absolutely. Is Ryman a person or is it a company or something? That too, Yeah, it is. No, Ryman, he was... yeah now it is ah no riyman he was
Discovering Clojure: Drew's Journey
00:05:23
Speaker
um
00:05:25
Speaker
What is his story? He was a captain of ah of a ship or or owned a shipping company back in the the eighteen hundreds Right. Um, and I can't remember the story. That's about it.
00:05:37
Speaker
I know. Fine. yeah this this is This is great content. you know yes i did not adequately prepare for national trivia questions. I mean, you don't mention things you don't know the exact history of. them Yeah. If you want to pause the recording, I'll go. and Okay. Maybe you can just open a Wikipedia page now and, you know, just read it out.
00:05:58
Speaker
Really good content. That's right. Uh, Anyone that's interested can look it up. here Actually, I was thinking when you first said it, I was thinking almost like Nashville. It's Ryman. like It's Ryman. It's Ryman.
00:06:09
Speaker
It's Ryman. It's kind of like it's Ryman. It's Ryman. It's Ryman. It's Ryman. It's Ryman. You know, maybe I shouldn't mention Hyman. Okay. find but Yeah.
00:06:21
Speaker
Walter, fix that. Thanks. yeah No, but the Grand Ole Opry that you've heard of, Ray, I guess. Pardon me? You have heard of the Grand Old Opry, I guess.
00:06:32
Speaker
Me? No, I have not heard any. I mean, I know about Nashville and country music and stuff like that, and I've heard. And I was i always, to be honest, i'm I've kind of got probably one of my remaining prejudices is is a kind of like, eh, don't like country music. um But i um I've kind of been enlightened in recent years um that country music isn't necessarily all bad.
00:06:57
Speaker
The D-I-V or R-C-E e is not the ah the, let's say, the... the the the pinnacle of all country music. um Tammy Wynette is not the greatest ever, but there are you better people.
00:07:11
Speaker
and Maybe that's the wrong way of saying it, not better people, but there are you know there's a more varied artist group in the country music set. I just have this feeling, I don't know why, but I have this like feeling that like in the 50s or 60s, I mean, let's just get into it. I think country music had a reputation as being a white supremacist kind of bullshit a bit of music.
00:07:32
Speaker
justified or not. um And I think that's i think that' being rehabilitated somewhat. um Although I feel like ah swift Taylor Swift, for example, came up in the country and then everyone was annoyed when she left and it was like she was going walk or some shit.
00:07:47
Speaker
you know So I feel like there's still something problematic about country music. so So that's my reasoning for being hesitant about country music. So I'll let you guys you know tell me why I'm wrong.
00:08:04
Speaker
It's a culture podcast, but we're going a bit, we're going a bit, going a bit country here. Yeah. Well, you know, country is varied, ah as is every genre.
00:08:15
Speaker
Um, you know, there's kind of the, the Appalachian bluegrass arm of country music. There's the pop end of country music, which is more,
Transition from Python to Clojure
00:08:24
Speaker
you know, the like today that would, that's definitely what I would call it.
00:08:28
Speaker
Um, and then there's, you know, just kind of the, the, um, somewhere in the middle. I don't know the rural kind of farm life, ah country music.
00:08:39
Speaker
Um, I know it does have a history of being, um of being fairly white. Um, there, uh, but there have been, you know, several, um, black country artists, Charlie prides, one of the most famous ones, um, excellence, um, artists.
00:08:56
Speaker
Um, yeah, I think it's, um, it, it, Yeah, the you could you could definitely have a whole podcast just on and on country music. there's a lot of moving on There's a lot of really good country music is the point, Ray.
00:09:13
Speaker
Yes. And some terrible country music performed by terrible people. Sure, sure. I mean, I think Beyonce's country music album recently kind of tried to reclaim country music as a non-white space, which is interesting.
00:09:28
Speaker
Yeah. She had a country album. This is news to me. well, yeah. exactly Yeah. Keep up, Josh. Come on. We'll go check it out immediately. so Hold the recording. Next on this podcast about Clojure, Josh and Drew listen to Beyonce country album.
00:09:48
Speaker
I did look up Thomas Ryman. So he was not, it was, it wasn't a shipping company. It was a riverboat company. you what a riverboat is? Freshwater, not seawater. I know what a riverboat is. yeah Yeah. Yeah.
00:10:02
Speaker
Okay. I mean, I've seen some examples of them Yeah. Oh, is it a steam one? Is it always a paddle? Yeah. Well, that yeah, there's steam and they have big old wheel in the back and, um,
00:10:16
Speaker
i I've read Mark Twain, you know? Yeah. Okay, good. Well, there you go. Yeah. But some so anyway, Emacs is another thing that you're... but I know about as much of Emacs as I do about country music. Or riverboats. It's going go great.
00:10:37
Speaker
I've only used it for 26 years now. so Yeah, exactly. Yeah. I think you're a couple years ahead of me. What year is it? I started using Emacs in 98.
00:10:49
Speaker
So whatever year. Yeah, I was about 2000. I think you're you're a little more. and Yeah. Look at me go. 99, 2000. ninety nine
00:11:03
Speaker
It's coming up to your 30-year anniversary, Josh, of being an Emacs user. You're going to break out some new keyboard cord or something.
00:11:13
Speaker
Exactly. I'll write a beautiful ELISP package for Emacs to give it to on our 30th anniversary. It's going to be a beautiful ceremony, I'm sure.
00:11:24
Speaker
yes Wait, do people have ceremonies on their anniversary? I don't know. Anyway, whatever. It's going to be party. think could call it a ceremony. You know what mean? Slightly weird, but you know. ah Okay, but. Right, let's get back on track. Yeah, I mean, because this podcast, by the way, mean, you've probably heard a couple of episodes of this podcast, Drew, but it's famous
Clojure and Data Processing at Sonion
00:11:49
Speaker
for us to do. It's usually much more organized, I know. You guys have some tight parameters that you operate in before our field.
00:11:57
Speaker
I get it.
00:12:00
Speaker
Oh my goodness. So maybe we we should kick off with the the the old classic of, you know, yes, you're an Emacs guy, but you know, when did you learn about the the parentheses of closure, you know, cause that's a, that's the, that's the, the real interest. That's what everyone's come here for, you know?
00:12:17
Speaker
That's right. That's the good stuff. Yeah. Yeah. Um, well, I, um, Yeah. So Emacs, I think, well, my first exposure list was actually in, in, uh, college in the late nineties, um, where I had a friend, uh, Dave, who was a computer engineering major as was I, but he was, um, he was a few years ahead of me.
00:12:44
Speaker
And, um, in the computer engineering, um coursework back then you took an artificial intelligence class and that class was taught using common lisp and he would complain about it.
00:12:59
Speaker
Yeah. the Poor Dave was not, uh, an enthusiastic computer programmer. He, I'd really don't to this day understand why he was in, that major.
00:13:09
Speaker
um he's a wonderful person. um but I, you would not talk to him and be like, this guy really should be, you know, a software engineer. Yeah. Um, but so he would complain about it and that was about all I knew. w I was sort of in the systems world doing Pearl and things like that.
00:13:25
Speaker
And I was like, Oh, okay. So Lisp must be pretty, pretty grueling. Um, and then, you know, years later I was, uh, I was, uh, getting into Emacs for reasons we can go into, but, um, that was kind of my, my gateway into Lisp.
00:13:43
Speaker
And i just really fell in love with, um, really the homo-iconicity of it. um Just being able, just cutting straight to the chase, writing just writing code in a data structure, i'm like, this is beautiful. Like, why have I never heard of this before? Yeah, that's crazy.
00:14:01
Speaker
The scales fell from your eyes. Yeah, yes. The clouds parted, life made sense. And um so really, I... um but But I wasn't like shipping software with Emacs. I was just going using it and and hacking on my my dot Emacs instead of you know getting real work done.
00:14:21
Speaker
Yeah, we've all been there. and right and then you know And then I got into functional programming um and from there,
00:14:32
Speaker
A couple of years later, it was probably around 2007, this guy named Rich Hickey posted a message to um this forum back then called Lambda the Ultimate.
00:14:42
Speaker
which I think it's actually still around. I'm not sure how how much it's maintained, but ah he he posted that he had created this language that was on the the JVM, which I had not had great experience with Java because back then Java was was terrible. And um it was ah functional in nature, mutable data structures, um and it was a lisp.
00:15:08
Speaker
And I said, ah well, this seems interesting. um ah me Let me give it a try. At the time, I i was also um working on a um sort of a reporting tool.
00:15:23
Speaker
that was It was in Python. It was pulling data from um from a database. and And i i was sort of aggregating and in counting a bunch of things and ship it to different people in the company I was working for.
00:15:38
Speaker
And it was super slow. This was back when like, um, I mean, Python's never been like crazy fast, but it, it, it had even fewer, you know, sort of, um, async and, and, and parallelization primitives. And, and so, um, writing concurrent Python was not really fun back then. And so,
00:15:58
Speaker
um I was like, well, why don't I try to try this Lisp out? I dabbled in it and like, oh, this actually works, you know, the REPL works and like I can do things in it. So then I just kind of progressively started doing more things in it. And so i was like, okay, why don't I try porting over this, um you know, this Python that I'm working on literally, you know, day to day right now.
00:16:17
Speaker
And i mean, I kid you not, i probably, um i had a i had a fully concurrent, you know, version of it Um, working and in probably a couple of days, like I was just blown away that, that, um, I could write something so non-trivial, trivially, like, like yeah yeah the impact, like I, the, the type of program that I wrote,
AWS Challenges and Elasticsearch Transition
00:16:41
Speaker
right. It was not hard to do in closure, but it was, it was.
00:16:44
Speaker
when I look back and I'm like, um you know, being able to stream data um in parallel from a database, like back then it just, like even Java, you know, you didn't have, this is before Java 1.5, like you didn't have Java util concurrent, anything like, if you wanted to do anything with with with ah anything in parallel, i mean, you were breaking out threads and you were like sinking your own state and you were doing a lot of painful things. And so this was, um you know, this was really, ah eyeopening for me. And, and, and I just want to say, like, I realized Rich is standing on the shoulders of giants too. It's not like he, should these things, but he, he really, um he, he, he really solved a major, um, problem, which is he, he, he decided to take ah a list, make it a hosted language on a ubiquitous platform.
00:17:32
Speaker
and, and make the interop seamless and, and use functional primitives to like make it super easy to reason about and make it super useful and practical. Um, I think it really just is, is such a, a wonderful, wonderful creation.
00:17:47
Speaker
So I was hooked from that point on, and um we ended up writing some more closure at that company. And, and eventually I, um, joined a the first sort of like large, and i'm I'm doing air quotes, because you don't really need a large closure team, but this is like, it was like eight or nine or 10 folks at a very early startup that um ah basically they were, you know,
00:18:13
Speaker
um building a closure team to solve ah a hairy back end data problem. and And several of those folks are still being good friends today. I mean, it was a wonderful team. Thank thank you, Tim Dysinger for pulling us together. um Love those guys.
00:18:32
Speaker
Nice. And what was that company doing Where you had the enormous closure team of eight or nine people. Yeah, right. ah So that company was called Sonion and it was an early Amazon funded ah startup in the information retrieval space for e-discovery, which is basically discovery, like legal discovery. you know if you're going to litigate in some fashion and you need to find a bunch of documents related to your litigation, you you go through a discovery process.
00:19:04
Speaker
um What that meant as far as software goes was a it was basically a search bar and you you go in there and um and it searched you know a bunch of email attachments that we had indexed.
00:19:16
Speaker
um that That's the short version. um But on the back end, they sort of did a... a ah So this was the early days of AWS. um I think they started around the same, maybe a couple years after, I think AWS started.
00:19:30
Speaker
Uh, you know, started in, in 2004 or five, something like that. And this company I think started in 2006. Um, a top five user of AWS. Um, we had almost a petabyte of data when I joined, which was just a massive amount of, uh, data for that time.
00:19:45
Speaker
um So we were really, you know, we were um really stretching S3, stretching EC2. You know, there weren't a ton of, that's about that. I think SQS, maybe a few other products back then. AWS was not as as full of a platform as it is today.
00:20:04
Speaker
um And so we, so they had built a um before I joined, kind of got it off the ground and bootstrapped it with um with, I think, just like a Ruby on Rails app. And they had used Ferret, which was this Ruby library, which is, ah I think, an abstraction over over Lucene.
00:20:25
Speaker
And um that wasn't scaling in the way that they they wanted. So the the VP of engineering at the time, who's Tim I mentioned earlier, he said he also found Clojure and had a similar reaction. He's like, i'm goingnna I'm going to go all in on this and try it out.
00:20:41
Speaker
Um, and so he started hiring, a few folks from the closure community and, and basically to rewrite that back in, in was essentially a distributed, you know, um, uh,
00:20:57
Speaker
indexing engine. um we you know We had workers that we, again, this like imagine like cloud computing is brand new, like launching and an EC2 instance with code. ah you know This is just like,
00:21:12
Speaker
you know what are the things you can possibly
Choosing Elasticsearch Over Hadoop
00:21:14
Speaker
do with this? Right. Like that was everybody's reaction. Right. And so, you know, dynamic workers, like, you know, firing up a machine and, and you know, firing up hundreds of them in the cloud at once yeah and distributing your workloads. I mean, that was like so fun, right. Because it was so possible.
00:21:30
Speaker
I wouldn't say it was easy, but it was possible, which was fun. yeah um And so, yeah, we had this whole system that would, you know, it would, ah We had collectors that would that would collect the email and then we'd fire off workers to, you know, actually um parse the attachments out and and, you know, extract text from the PDFs. and um and And then we would throw all that stuff RabbitMQ and then we had other machines that would take those, um ah take the ah text and we would index it into um basically, you
00:22:05
Speaker
we we We started just talking to Lucene directly, and then we were experimenting with Solar, which was a an open source search ah library that was very popular at the time. And well, it's still around.
00:22:18
Speaker
um and ah but But its ergonomics were were a little ah you know rough around the edges back then. um And so that actually led to the next you know sort of phase of my journey, which was... um ah Elasticsearch because we were essentially distributing Lucene and that's essentially what Elasticsearch is um and with with some some really great APIs on top of it.
00:22:44
Speaker
um So yeah, and so all of that was made possible with Clojure. um Everything from the text extraction to the indexing to the um you know to the Well, I guess that's about where it stopped because the front end was still, i think, Rails.
00:23:01
Speaker
um All of our ah sort of provisioning and systems ah stuff was Chef at the time. So it was still very very ruby Ruby land.
00:23:12
Speaker
um So we hadn't gotten, we kind of hadn't spread closure further than you know the kind of central system. So how did that, um Thinking about, maybe it wasn't around at the time, but I feel like it was.
00:23:25
Speaker
um How does that compare to the Hadoop architecture? I'm pretty sure Hadoop was around about that time. Oh, yeah. It definitely was. Hadoop was very much around, yes. Yes. because Because this is, you know, obviously Hadoop was this sort of whole MapReduce thing.
00:23:41
Speaker
That's right. Which was itself designed to do exactly what you said, you know. So why did you... do this rather than that? You know, I'm kind of always interested in like, you know, uh, well, yeah. Why did you like, the thing is sitting right there. It's a JVM yeah thing, you know, um,
00:24:03
Speaker
But obviously it was that had was issues, that's it. um but it was so it was very popular you know um amongst a certain, yeah well, amongst the industry really. It was a popular thing.
00:24:16
Speaker
um So yeah, I'll stop blah, blah, blah, and just let you answer the question as to why you went a different way. That's good, actually. Thanks for the but that, because it gave me some time to think of an answer.
00:24:28
Speaker
Yeah. I actually don't know. So, um, so we started that journey, ah pretty soon, I think around the time I arrived. So, um, and, and I was, ah you know, I was more,
00:24:42
Speaker
junior, more junior, uh, you know, person back then. i wasn't asking those kinds of existential questions. I also didn't know much about Hadoop. I knew it existed. Right. I mean, this was still in the days where like, you know, the, the Google like paper about MapReduce wasn't that old.
00:24:57
Speaker
Uh, so I think, I mean, Hadoop existed, but you know, like solar, like a lot of those early Java projects, like just the ergonomics were terrible. Like you couldn't,
00:25:07
Speaker
It was so heavyweight. And um so i would I would venture to guess that, you know,
00:25:16
Speaker
ah people People don't come to Clojure because they love complexity. I'll just say it that way. yeah um So I i i would, ah you know, Phil Hagelberg and Tim and folks that were there a little bit before I was, i Steve Gilardi, Dan Larkin, these guys, i'm I'm sure they were like, I think we can probably do something slightly simpler than then Hadoop. and in it And to be honest, like it probably was. I mean, it wasn't a trivial code base by any extent.
00:25:43
Speaker
any ah stretch the imagination. But you know distributing the work across the across machines like really wasn't the the hard part of the architecture. Right. mean, if you're using SQS, you can do it like that. And then you can just pick up pick up the work and report back, et cetera.
00:26:00
Speaker
Yeah. Yeah. Well, we were using a Rabbit. So actually, that's a question I still have. I need to ask them. It's like, why did why didn't we just use SQS? I don't know. Yeah. Well, it kind of depends on what you were doing because robert sorry rabbi robert robert Rabbit can be fully persistent as well, unlike SQS. So Rabbit has a lot a lot more capabilities.
00:26:26
Speaker
Yeah, I suspect that is a big part of it, plus Erlang. Plus Erlang, yeah. You just got to put some Erlang in there. Maybe SQS was still a bit also a bit new and rough around the edges and stuff like that. Well, I think SQS was... um Was it the second service? Was S3 first and then SQS or something like that? EC2 and S3.
00:26:49
Speaker
yeah. Well, that the very first AWS service was none of those. It turns out it was a product catalog service as the very first Amazon mobile website actually consumed these this XML API.
00:27:05
Speaker
and used ah excessl are too wait our amazon archive yeah whatever XML transforms to generate you know WAP and HTML. It was it was good stuff.
00:27:18
Speaker
That sounds beautiful. Are you allowed to say WAP these days? I think you can say WAP. It doesn't surprise me that it's like a ah you know basically a a service registry. Yeah.
00:27:30
Speaker
But yeah, that went away ah long time ago and S3 is the first like official AWS service as we know AWS today. That sounds right.
00:27:42
Speaker
Yeah. So ah Elasticsearch, you said you eventually yeah Okay. Yes. Yes, Ray. Thank you. Ray is just texting away here in the background, making fun of me for being ignorant about music. That's, that's cool.
00:27:59
Speaker
The important question is you said you eventually moved Alaska. You're not breaking the fourth wall there, Josh. You should just break it now. All right. Don't worry.
Partnering with Elasticsearch
00:28:07
Speaker
Walter will fix that. None of this will be in the episode. I'm sure it never is.
00:28:13
Speaker
Elasticsearch though, we wanted to talk about that, right? Yeah, definitely. Because you mentioned it and you also spent quite some time there, I guess? Yeah, i was ah I was at the company for 10 years, plus the two years before the company was founded.
00:28:31
Speaker
um So yeah, but basically what I said before was that we were you know distributing Lucene um and Around the time that we were you know sort of figuring out how we would do that, and we you know wrote a few POCs and and played with it, um this guy, Shai Banone, who's an Israeli developer, he had released um this thing called Elasticsearch, and it was actually February of 2010.
00:29:00
Speaker
exactly 15 years ago. it was exactly fifteen years ago um And it you know, brand new thing with, you know, one guy wrote it. It's like, this is, um you know, ah doesn't, it seems like we need to let it bake a little bit before we, you know, start using it. But it's pretty compelling. I mean, Shai spent a long time building.
00:29:26
Speaker
ah putting a lot of useful APIs into it. that he you know He solved some like sort of core distributed systems problems, but um but he really put ah such a useful veneer over it. He, he you know, having like,
00:29:39
Speaker
the The power of a system like that with with RESTful you know consumable JSON-based APIs, like that that actually was innovative at the time. And it was all non-blocking I.O. It was super fast.
00:29:52
Speaker
um and had you know had geospatial ah features i mean it just It had It had facets and all kinds of things that exposed from Lucene. so it was um, a very, um, it was a very robust piece of software that just commanded us to really take a look at it. And, and so eventually we decided, you know, speaking of engineering trade-offs and decisions, we decided that, you know, if let's invest in this rather than, um, you know, trying to, to make our own thing while this matures, uh, we can, we can sort of partner up with shy and, um,
00:30:31
Speaker
and ah benefit benefit both of us. And so that's what we did. um And we um you know basically were the first kind of large user of Elasticsearch. We weren't the first users.
00:30:45
Speaker
That title goes to a guy named Clint Gormley, who's a longtime friend and and early Elastic guy as well. um but But we were really the the the ones that um ah really put it through its paces and, and also stumble upon, I should give them some credit. They, they were also big users at the time. I think they were a little bit after us, but, um but we, ah yeah, we, so we started indexing all of our, our stuff into Alaska search, which,
00:31:14
Speaker
as I said, was almost a petabyte of data. By the time we actually got it all in there, it was probably a petabyte. um And that was, you know, of course we used several clusters for that. It wasn't, wasn't one cluster, even though we could have put that in a single cluster.
00:31:27
Speaker
um You know, any, any just operationally, um you know, any large system, we, we kind of sharted it out further from, from the ah single cluster. um But we it was great. we we you know We found a lot of bugs in Elasticsearch, found a lot of bugs in EC2, and we were using um EBS, like the magnetic EBS back then. Yeah. um Because that's all there was, which you know was notoriously... i hate to say unreliable because like it's cloud computing, right? I mean, you know ah you kind of expect machines to come and go.
00:32:01
Speaker
But with really large stateful, heavily stateful, um you know, instances of of of a system like that, you know, um interruptions and I.O. problems are can wreakha it can wreak havoc, especially with the ah the JVM, because um you know if it hits latency or it runs out of memory, then you get like weird network errors where nodes like sort of are half up. They they like don't die all the way, but they're not responding to requests. and
00:32:33
Speaker
um So we had all kinds of fun ah fun things that we found and um and got to to work closely with Shai to fix. and So we, and and we contributed a lot back to the code. So um it was really a great partnership. It was kind of the the the the beauty of ah you know open source and and commercial adoption, right? Kind of bi-directional value.
00:32:59
Speaker
It was great. So what was the ah closure part of the Elasticsearch story then? So that was largely um that was largely the same as what we were doing before. So that we we we were just indexing into Elasticsearch instead of ah you know directly into Solr or Lucene. Yeah.
00:33:21
Speaker
Yeah. But then you joined Elasticsearch itself, the company, right? At point. did. Yeah. Yeah. So it was founded in April of 2012, May of 2012, something like that. And then I joined in November.
00:33:34
Speaker
um That's right. And there was... Did they have a closure story at Elasticsearch? Elastic? They didn't. um Yeah, and i it's it's funny because um i I don't think Shai really actually liked Clojure that much.
00:33:54
Speaker
um i think he was I think he was intrigued by it and and certainly the the community um you know adopting it. um and And we actually wrote an Elasticsearch client that we in Clojure that we never released, which is kind of an one of the ironic parts of of my journey. but um ah wrote wrote a lot of um kind of internal tooling with it in the in the early days. Some of that stuff I think is still around the today.
00:34:22
Speaker
um a lot of our, um the way that we, ah some of our KPIs were around downloads, of course, because that's, you know We were an open source business at the time. um And so like the way that we aggregated that data was was some Clojure that I wrote.
00:34:38
Speaker
um some Some of our first downloads infrastructure was actually Clojure. And I used Pallet. So shout out to Hugo and and and Tony for a great... um ah An amazing ecosystem of libraries.
00:34:55
Speaker
um you know i wish i wish it would have taken off and and taken over the world, but um but it's super fun. We ended up migrating that to to Puppet eventually when we hired some other folks. but um But yeah, Clojure was a ah power tool um in the early days of Elastic and and really um helped bootstrap the company, i mean to be honest.
00:35:17
Speaker
One of the tools we wrote was um a ah So we use we used um Jenkins for our CI, but we... didn't love like the Jenkins ecosystem of plugins. And like, you, when you upgrade Jenkins, it's like a nightmare.
00:35:35
Speaker
So, um, it's, it's really reliable for just like, I need to run a thing on a schedule. It does really well at that. But once you start trying to write plugins and stuff, it's just, um, it it's, um, you know,
Transition to Management and Leadership
00:35:47
Speaker
it's, it's hairy.
00:35:49
Speaker
So, um So we we said, well, let's just rely on the sort of shell out functionality of Jenkins, and then we can do a bunch of fancy stuff after that.
00:36:01
Speaker
And so um we we had this wrapper that was essentially, it was a closure program, and it was essentially a... um You know, it would ah it would spawn a process to ah to run the whatever the build script was for the thing that we were building.
00:36:19
Speaker
But then while it did that, it captured telemetry from the the child process so and then shipped that to Elasticsearch. So um it was this shim in between, you know, kind of the the of the scheduler and the the the real build. So the thing that, you know, you would run on your laptop or something.
00:36:39
Speaker
It worked really well. And we used that, I mean, I think we finally migrated to, to well, we we expanded to like 16 or 17 different Jenkins clusters for different slices of the company.
00:36:51
Speaker
And that was running a lot of it. And ah and then eventually we had adopted Buildkite, which is a great company. Shout out to the Buildkite folks. Imagine that your CI is on 16 clusters though. I mean, that's where you're starting to scale stuff. Yeah, definitely.
00:37:11
Speaker
Definitely. Yeah. The, we hired a wonderful guy named Aaron Bull Schaefer, who, who was just like a, a Jenkins and an Ansible wizard. And so like you put those together and he was like, how many clusters do you need? And by when? That's great.
00:37:27
Speaker
Yeah. my Oh So it also in your bio, and there's another thing you and I have in common, which is some point you ah decided to become a manager. I guess you lusted after power and wanted to- That's exactly right. um I always wanted to tell other people what to do and it's been great.
00:37:49
Speaker
How's that working out? People love to be told what to do. It was a great- great yeah, ah especially programmers. Yeah. um so we um i you know we yeah So, the the the Elasticsearch um company and community is full of brilliant people and i was not one of them.
00:38:13
Speaker
So, were like, what are things that Drew can continue to do that are valuable at this company? Because we really don't want to write any more code. Please don't.
00:38:24
Speaker
um um And he's not a great Java programmer, so ah you know don't ah don't let him touch Elasticsearch. um Although i did i I did inspire the CAT API, which is probably my most famous contribution to Elasticsearch.
00:38:40
Speaker
Shai actually did a lot of the heavy lifting as far as turning into ah um an API in Elasticsearch. But the the idea and the format and all the stuff came out of... um came out of ah the way that I was administering the clusters in at Sony and at the company before. and so what What is the CAT API?
00:39:01
Speaker
Oh, so the CAT API, yeah. So um I'll get back to your other question here in a second. The CAT API um is a, it it was ah the answer to like, ah an somebody who does a lot of um command line sort of Unix administration really doesn't love JSON.
00:39:22
Speaker
like it's great It's great to have at least some structured format, but um you know us Unix people, like we want we want white space and and lines and and pipes and all kinds of things. right So um I had basically written some ah some translation code to take the JSON-based Elasticsearch APIs and translate them into line-based ah column columnar kind of looking things so that I could use tools like you know sort andre and grep and fun things like that. So Shai being very much you know an expert and in in Java and distributed systems, not as much an expert on you know Unix and large scale systems administration,
00:40:09
Speaker
He's like, I think we need to add that you know as a feature, as an API in the in the ah in the software. i was like, that's a great idea. Let's do that. so So normally like when you see, you know it's it's a very common tool um now in order to so just do basic tasks, like you know seeing how much data you have, how many shards do you have, how many nodes do you have?
00:40:34
Speaker
What's your JVM heap? You know, all these kinds of things that you just are pretty common um ah to do while you're ah you're looking after your Elasticsearch clusters. Because I was thinking that it might have been something to do with like cats and dogs because they're the most popular things on the internet, you know?
00:40:52
Speaker
So it was like a specific API to to search for cats. Yeah, right, right. In elastic search, but yeah, that was funny. touch i think it should I think it should do that. Although if you go to just the cat endpoint without anything else, it will it will show you tasky picture of a cat.
00:41:08
Speaker
All right. course. Excellent. But yeah, it was just kind of a nod to, to the cat Yuto and Unyased. It was a nice short name too. Yeah. Shy, Shy was brilliant at, at the branding.
00:41:23
Speaker
Yeah. Yeah, so your question was about management. Let's see. So so yeah, they were- Heard of if you will. Yeah, so you're like, okay, clearly that's the best thing this guy can can offer from a software standpoint. So let's see if he can you know actually um talk to people and and and manage a team.
00:41:40
Speaker
but Basically, we had um you know we had our product teams um like Elasticsearch and Kibana and Logstash. And ah you know as we grew, we had more and more products.
00:41:52
Speaker
um There is a, ah but but but there's a ah lot of engineering that happens underneath the ah surface to glue things together. And so we had kind of, you know, I mentioned the tools a minute ago that I had um ah developed. And then there were some other folks working on, you know, other sorts of um tools in the infrastructure space.
Leading Elastic's Infrastructure Team
00:42:14
Speaker
um So I, yeah ah so my my boss asked if I wanted to sort of gather all of those folks together and and lead a team around that. So we called it the infrastructure team or the infra team for for short. That was um ah kind of ah a home for a lot of like polyglot, multi-discipline folks.
00:42:41
Speaker
um yeah And so we we solved like all kinds of problems um from running a ah globally distributed tile service for our maps functionality to continuing to like expand our build infrastructure to making sure that users could um download Elasticsearch, um running the release process. And eventually, we we collapsed all of the products into a single versioning scheme and so we would release them all together.
00:43:12
Speaker
um So that um process was was kind of built out of our team. And then, um yeah, we were the do-all team. if you had ah If you had an issue, if you needed a site ah kind of ah a supporting service for a product, um we we were usually the team to so make it happen.
00:43:33
Speaker
So we're kind of ah kind of a, ah you know, very much a and an SRE-like team. yeah We build, but and and then eventually we, I mean, we really, we didn't call ourselves at the time because it wasn't as much of buzzword. We were we were a platform team.
00:43:50
Speaker
um So, we um you know when Kubernetes became kind of the the clear winner in the container wars, we um started offering that as ah as an internal service.
00:44:03
Speaker
um So, we had a ton of um of Kubernetes clusters that we you know ran different things on and and and and you know eventually ran part of our build structure build infrastructure on that. and We would give different clusters to different different product teams for for them to write their own automation and you know that kind of thing.
00:44:22
Speaker
Smaller side there, Drew, since um since you said it's the clear winner in the container wars. I have a feeling that those wars are not over.
00:44:33
Speaker
Yeah, was that too provocative of a statement? No, no, no. I think you're right that it felt that way. Definitely it felt that way five years ago or maybe even seven or eight years ago.
00:44:43
Speaker
But it feels like... like It's just growing and growing and growing to the point where it's bloated. It's almost gone topsy. you know Right.
00:44:56
Speaker
Yeah, there's a lot going on there. And I don't even pretend to to act like I am up to speed on it, but... um yeah i mean this is probably 10 years ago right and it was you know kubernetes and nomad and and um you know some of these other kind of orchestration suites um and um you know Kubernetes was was clearly gaining a lot of interest just with its the kind of the Go influence and then obviously Google being but beneath both of those.
00:45:32
Speaker
um You could kind of tell where the wind was blowing. And and then, um so we were all we were actually big Google Cloud um users. And they ah came out with GKE, their Kubernetes engine, and it was like the most vanilla Kubernetes of of of the you know of all of the platforms, all of the cloud companies. So um they did a really good job of of exposing Kubernetes as kind of a hosted service. And so we were able to really do let them do a lot of the heavy lifting. We could just kind of provision clusters at will.
00:46:04
Speaker
um And so I would say that's you know the biggest reason why why we got hooked on it was just Jiki. It was a really good way to to meette meet our users' needs.
00:46:16
Speaker
And did you so did you actually run your own clusters eventually or was it always on the cloud? So the company went through an interesting transition.
00:46:27
Speaker
um We acquired a cloud, basically a startup called Found, a nor Norwegian company that um had built a container orchestration engine with with Elasticsearch at the center of it.
00:46:44
Speaker
um Sorry, I said that wrong. Elasticsearch was not the the orchestration engine. um it It was built for hosting Elasticsearch nodes, which you can you could assemble into um you know a hosted Elasticsearch platform. And so they would that eventually became what today is the Elastic Cloud product.
00:47:04
Speaker
um And so that was not at the time written in Kubernetes. um So we had sort of two internal orchestration platforms for a long time.
00:47:17
Speaker
um And Ray, I can't remember your question now. what What did you ask? Why did I start talking about that? It was about whether or not you run your own Kubernetes clusters or whether you always use the cloud versions of these things.
00:47:28
Speaker
That's right. so So for my team, we ended up ah just relying on GKE because it worked so well. um But yeah, the reason I started talking about the um the the um the cloud platform, the Elastic cloud platform, is because that was not written in Kubernetes initially, but later um became more more Kubernetes-like.
00:47:52
Speaker
um and then um And then in recent years, there's been ah kind of a replatforming um to to put Kubernetes underneath it.
Open Source in the Cloud Era
00:48:02
Speaker
Now, Elastic is famously upset about cloud providers, including its revenue.
00:48:08
Speaker
um Is that something that is open to to to you talking about? Or is that something which is like above your pay grade or ah interesting?
00:48:19
Speaker
Yeah. Yeah, I mean, i we can try it. i like i can um you know I can vamp on it for a minute see what we end up with. So there was...
00:48:33
Speaker
so there was um and Yeah, I don't know how, yeah. um amazon Amazon liked to host open source ah software and and make money from it. I mean, you know, legally they could, right? I mean, it's a pretty shrewd ah business move.
00:48:51
Speaker
And to be fair, in some cases, they were pretty active in communities and and and gave back. um But they they certainly... um It felt to us at the time that that they could have partnered with us more than then just sort of cannibalizing it. And I think um you know that that ended up in in kind of the fruit that you see, right? There was kind of fragmentation and and you ended up with the OpenSearch fork and and all kinds of things.
00:49:21
Speaker
um Yeah, it's... ah in the And the... the re-licensing of Elasticsearch. It's a very interesting thing, right? um And it it really was made possible.
00:49:36
Speaker
ah Open source enabled cloud computing and then cloud computing definitely kind of ah you know starved open source in some cases.
00:49:45
Speaker
Yeah, guess, I mean, now you have Elastic Cloud, right? So you have your own kind of SaaS offering for Elasticsearch. But I mean, that I have some experience with Elasticsearch, enough to know that, like as we say, one does not simply deploy an Elasticsearch cluster. right like It does take a little care and feeding.
00:50:07
Speaker
And so you know I can see the benefit to people of just wanting a hosted one that at least takes some of that stuff off your plate. um of course yeah absolutely the the i the the functionality provided by both services is is is wonderful like if you want to use elastic search um i yeah having a hosted version is is uh is wonderful um i think the um uh
00:50:39
Speaker
can't remember now what you said in your in your answer there, Josh, that made me think of what I was going to say next. Well, you can just say what you were going to say next. It'll be fine.
00:50:51
Speaker
I'm wondering, maybe maybe his's bit maybe he's like as a sort of segue to something else, is like or a more general point, is whether this is sort of prefigurative of the um the the thing that we had the problem with recently with HashiCorp.
00:51:08
Speaker
where they wanted to have terraform. Um, and and again, terraform was this thing where they wanted to keep it to themselves and not have third parties, like essentially eat their lunch.
00:51:21
Speaker
Other people argued, well, you know, actually we're, we don't really steal from you. We're not really, we're giving back and blah, blah, blah. And it's kind of like this balance seems to be, or it's an interesting balance as to whether users of the software or anything to the creators of the software.
00:51:39
Speaker
Um, Whether fairness matters. you know I think thats that's the kind of like essence of this problem, I think. It's this concept that where you give you you give out this open source, you explicitly say it that it's as is, it's and it's available and usable for commercial purposes. So how can you complain?
00:51:58
Speaker
um And it's a fair question, I think. It is. but but i But I think also we people do put a lot of effort into these things and to see ah third party kind of like just essentially take advantage of that situation rather than, um let's say like you say, on ah honor that the inventor or kind of like at least share some of the benefits.
Commercial Aspects of Open Source
00:52:24
Speaker
It's cruel in some ways, but it's kind of like a mafia type thing. It's just business, you know? But it really is. I mean, that's exactly how country is run these days. mean, don't if you've seen, but, you know, it's just business. It's just business. I thought you were going to say would be a shame if something happened to your Git repo. Sorry, you had a serious point. Yeah, well, I mean, it's...
00:52:54
Speaker
Yeah, thinking about HashiCorp and their decision, and it's it's similar. i mean you know the The best thing for software you love is for it to live. And um you know sometimes that happens because you have a benevolent dictator that can, uh, you know, fund their life in some other way and they can donate a lot of their time to continuing to offer the, the software that you love.
00:53:19
Speaker
Um, in other cases, you know, uh, a project dies because it doesn't have that support. And then the other option is, um, you know, sort of, you have this, uh, relationship, a corporate relationship where,
00:53:33
Speaker
either a business is, um sort of donating that work to the community, um, or it is, you know, requiring some kind of, um, you know, payment to, to use it.
00:53:46
Speaker
Um, so it's, it, it, it's not inherently evil or good in my, in my mind. I think it can be done, you know, poorly or, or well. Um, I don't really have a, um,
00:53:58
Speaker
yeah I think it's, I think when you see changes like with Terraform and with Elastic, it's because businesses are wrestling with this, right? I i know very well that the the heart of Elastic is deeply, and you know, an open source um yeah ah DNA. And like if every every founder and developer of that company would love to give it away, you know, for free, but But at the end of the day, we have very large important segments of our economy that run on software and these enterprises come looking for a commercial relationship that they can make sure that the thing that they depend on has some security behind it to keep living and to be able to get support when they need help and all those kinds of things.
00:54:50
Speaker
um so and that's a beautiful thing because you know it provided me meaningful meaningful employment and it was a fun fun company to work at so like you know i'm i'm grateful for it um i think the other aspect of it is And I realize you know the cloud providers are are unicorns here. like it's it's it's hard to um It's hard to generalize too much, but but because they control so much of the sort of the IT t spend market share, like like they are very relevant.
00:55:18
Speaker
And I think the question is, you know can AWS effectively offer every kind of product? you know, can they be experts in all of these disparate platforms? And, you know, I i think, you know, I mean, anything I guess is possible with enough um with enough time and and money, and and certainly Amazon has has deep pockets. But when but when when you are when When you are trying to host ah software that already exists, the community and the users and and the the the dependencies or the things that depend on it, you know they're they're all um they're all needing it to fulfill not just ah its sort of software contract, but there's
00:56:03
Speaker
you know, all kinds of things that go into actually helping software live in the wild. And it's, you know, comes down the community, it comes down to how you get support, it comes down to, you know, conferences around it. I mean, all the kinds of things that, you know, you you and I know so well, just being in the Clojure community, um you know, even and even a even one of the most valuable companies in the world, can it effectively do that for all of these products? And I think you know, I don't want to, yeah I don't want to disparage the open search folks. Cause I think they've put in a lot of their own, um you know, labor into that, but, um, but it's it's really hard to, um, it's really hard to kind of keep a thriving community going with, you know, um a sort of segment of a large company, right. We've just got kind of ah a product champion and maybe a few developers and like,
00:56:57
Speaker
that that thing could go away at any time. I mean, how many how many you know things that we, ah you know there's the famous Google examples of Reader and everything else that are in the graveyard now.
00:57:08
Speaker
um So you know i think I think that they're realizing you know that you can't just um you know offer all of these types of disparate products with without um potentially losing the fidelity of and the quality that that folks are expecting from it.
00:57:27
Speaker
Yeah, but i think I think to your point, they wanted to own the space. um They saw that search as valuable because obviously Google is a search company.
00:57:37
Speaker
Amazon is a search company as well to some extent. Their catalog is basically a search engine. Yeah, I worked on the search team at Amazon as well. I have some friends on that team, ah which is separate from from AWS and some brilliant, brilliant search engineers.
00:57:55
Speaker
um
00:57:58
Speaker
i I think um that, oh shoot, i got of I've got it to like take my ADHD medication or something. I had a point when Ray was talking and then totally lost it.
00:58:14
Speaker
um Well, I was talking about the fact that they want to own the space, you know? Um, and it it feels, and and, you know, they have their own search products and they, they've, I'm just kind of wondering, again, from a commercial perspective, probably elastic felt a little bit like, like they were closing down the possibilities for elastic to exist.
00:58:39
Speaker
And that, that, that probably feels a little bit wrong in inverted commas, even though it's just business, you know, It feels like, well, you know, how are you going to, like you said, how are you going to like keep this product alive?
00:58:54
Speaker
if we can't actually survive. Right, right. Yeah. And I think, um i don't I don't necessarily think it was anything nefarious. I mean, I think i think it was, um you know, some some folks that that felt like they could build a successful product for AWS using, know, standing on the shoulders of Elasticsearch. and Sure, sure. And they had some level of success at that, even though it it eventually um shut down.
00:59:24
Speaker
Um, I think that, um, that they, this is, this is an example of, of, you know, in large corporations where, you know, the right hand and the left hand don't really, um, either talk or or necessarily need to talk.
00:59:40
Speaker
Uh, but, you know, Amazon was making ah does make a tremendous amount of money from Elastic Cloud. So the actual Elastic Cloud product. Right. um One of the providers is is AWS and people who are AWS customers come to Elastic and they want to buy elastic hosted Elasticsearch because it's in AWS and you know it's it's in their it's in their region and and they can talk to it with ah with low latency and all those kinds of things. So um you know there's a very healthy relationship on that side of the of the ah house that have just...
01:00:16
Speaker
you know they couldve just you know, left that one to flourish and and and not tried to go in these other, in these other directions. But again, you know, um large, large business is going to, of course, have as many revenue streams as as they can. but But it feels a bit like, like it, and again, maybe as I'm sort of over dramatizing a little bit, but like open source, you know, because we talked about, you know,
01:00:42
Speaker
When Red Hat went IPO'd ah for Linux, it was like the biggest IPO in history, I think, at the time. um Clearly, that's been superseded, but it was a huge thing.
01:00:52
Speaker
um And, you know, now Red Hat is on by IBM, and um it's, you know, it's weird. But, you know, so but Linux became like an enterprise thing, and then,
01:01:08
Speaker
you know an enterprise company bought that thing and so it it feels a bit like there's this sort of commercial open source and open source like the obviously amazon make a lot of money out of linux you know ec2 is basically linux you know um they have their own version of linux even but it doesn't really matter the main point is that the linux community doesn't care you know they don't They don't care that Amazon are making any money. Google are making money. Android, but the Apple, everyone, that Linux doesn't care about the commercial aspects of this thing. um
01:01:44
Speaker
But companies like HashiCorp, Elastic, um ah think there was another one, Redis, I think recently as well. that you know There are various companies looking at this model.
01:01:56
Speaker
It just feels like once you go into like the commercial open source world, which I mean, you know, to some extent, people don't like to have code that's not open source anymore.
01:02:07
Speaker
So it's it's almost like ah obligatory thing to have your code open source. And then you're going to have it potentially, you know, um used by people who are going to compete with you.
01:02:22
Speaker
Yeah, well, I think i think technical... um people, I think developers are just ah always going to default to open source because it's you know it's the thing that gives us the most control.
01:02:36
Speaker
we're under you know if we're part of a business, we're under tight deadlines. If we're doing stuff on our own, we just want to build cool stuff. And like you know open source is a great ah a great way to solve both of those problems.
01:02:50
Speaker
um View source. Yes, please. yeah Yeah. Right. So, you know, yeah democratization of software and and and just going straight to the source. i mean, it it just alleviates so much extra, ah you know, diversity.
01:03:07
Speaker
it shrinks the feed feedback loop down so small that it's, it's, it's just an efficient way to do things. Um, it's also, even if you, even if you don't fix things or contribute, you know how to work around the problems cause you know what the problem are. You know what problem is. It's also wonderful way to, uh, to run an engineering organization. That's how we, um, know, that's how we organized at elastic.
01:03:28
Speaker
Um, and, um, the, but, but, you know, at the end of the day, like somebody has got to fund things and, you know, um, I, I think the, I think founders that are technical, you know, they're, they're gonna want to have as much open source as possible.
01:03:48
Speaker
Um, but at the end of the day, you know, you've got to raise money, you've got to sell a product. Um, and so, you know, how can you sort of let your, um, you know, your source itself not be the product. I mean, that's really the, it's why services became, you know, a money stream and we stopped selling, you know, boxes of CD-ROMs, right? Because, um because people just want stuff on demand. They don't want to pay, you know, necessarily big contracts. They just want to ah pay for something and start using it.
01:04:21
Speaker
um And so, yeah, And so, you know, open source kind of um is somewhat ancillary. And I think a lot of, at this point, and so a lot of companies that get started from the open source thing, ah you know, very quickly, you you you have to figure out what your what your revenue streams are going to be.
01:04:43
Speaker
Man, that's a heady discussion. I just want to say that I'm 100% on the side of HeshyCorp here. like I think there it's fine for them to want people to pay for licenses for their stuff. But you know that's because I am a capitalist to my last bone, as as every everyone.
01:05:02
Speaker
Everyone knows. i mean, it's it's extremely impressive. They they you know have been so through so many generations of the company with so much stuff being freely available.
01:05:14
Speaker
yeah I mean, when you look at the landscape of of what they offer, um the vast majority of it is still free and source and yeah and very useful, right? Like you can get you can do a lot before you get to the enterprise Yeah.
01:05:29
Speaker
Yeah, and that's the thing. like I guess the... I'm looking at this through ah the lens of somebody who's worked at like, you know, very, very profitable companies. And it always just bugs me a little when these companies with more money than God are just like, oh no, we're not going to pay ah hundred dollar a year license. And it's like, come on.
01:05:54
Speaker
yeah So anyway, but, um, we're back to management maybe back management. Yeah. Speaking of management, Obviously, at the company where I'm a manager, we pay $1,000 day licenses for all software we use. you know That's just my policy.
01:06:13
Speaker
um But yeah, so like what what was it like to um go into management? ah like If you can kind of think back 11 years ago or whenever it was you said, um do do you recall of what that was like?
01:06:31
Speaker
Going into management? Yeah. Yeah. Like what was the mind shift you had to make? Yeah. um Well, I um i i did feel like I was you know the kind of
Philosophy on Leadership
01:06:46
Speaker
reluctant manager. I think you know Steve Jobs kind of describes it that way.
01:06:50
Speaker
um i i didn't i Like you we were joking before, like I really didn't want to have any sorts of sort of power. um and And I didn't necessarily think I was, um you know, good at it, but I did have this desire to give engineers a good manager and to solve problems broadly for, you know, the the company and and um provide ah input into, you know, hiring and staffing and and how we build, know,
01:07:22
Speaker
um, and and, and what we build. So I definitely felt, you know, like leadership was, um, or people leadership, I should say, um, was, was something that I, um, could, um, could pursue.
01:07:41
Speaker
um and, and, you know, it turns out that like a lot of, um a lot of management really is motivating people and it's inspiring folks. And and um I think that I had and enough um ability to sort of connect the art of programming, the discipline of programming with you know business problems and and give ah you know connect with engineers on a deep level, the deep technical level, but then help them map
01:08:12
Speaker
you know, map business problems um to, to ah you know, interests and abilities. And so, um that, you know, once once I sort of tasted some success at that, I i felt almost like a burden to do it.
01:08:29
Speaker
um and and And probably is while i'm why I'm still doing it today. Yeah. Yeah. I think that's that's interesting because i think about my job really is being an information conduit. So trying to capture the context of you know whatever is important to the business and then talk to engineering teams with that context so that we can you know find the right trade-offs basically.
01:08:58
Speaker
Yeah. I'm thinking back to there have been so many times when I was an engineer and I i had i was just so focused on the technical solution. right like We have to do it this way because it is obviously better, but obviously better for what? right and that was um that was That's why I think a good manager can actually help by giving engineers the type of information we actually need to make good decisions.
01:09:29
Speaker
and so Otherwise, you're kind of in a vacuum. but Yeah, I mean, being able to you know translate um requirements into two you know just enough specificity that they're clear with enough freedom to where engineers really can um can have the the space to explore you know really good solutions to problems, it's it's it's really gratifying. And I realized that not all of that's like an EM's role. I mean, that's you know product managers do that as well. and in different aspects of leadership. But um I think the thing that I still just love about engineering management is, um um in in management in general, is the the team aspect of it. I mean, i just, i can I can shepherd a team to solve way harder and bigger problems than I can can solve on my own. And and that's just extremely rewarding.
01:10:20
Speaker
um and And then I think like, as I was, you know, yeah doing it for longer and longer and hiring. and I've hired so many people over the years and just being able to hire people that literally are just exponentially better than you are at at something that you know how to do.
01:10:38
Speaker
It's so, it's inspiring to me and, and, and, um, being able to to, to be an enabler for, for those kinds of people and then watch junior people grow because you've put those people together and now you've got this beautiful team ecosystem that's just like thriving. And, um, and it's just, it's really quite something to, to, to be able to, to shepherd.
01:11:01
Speaker
Yeah, that's my favorite part, to be honest. I don't know if you ever played Baseball Stars on the Nintendo, but ah didn't Baseball Stars, it was kind of a RPG ah baseball mashup where you could draft a you know a new player and they arrived with basically no stats, right? And then you you developed their hitting and their running and all of this stuff. And yeah.
01:11:25
Speaker
I just see that you know as ah as a manager. I always loved finding this this person, whoever
Balancing Management and Creativity
01:11:31
Speaker
they were. you know Maybe they were a career changer. Maybe they were you know kind of first-time programmer.
01:11:37
Speaker
And you know finding just that the person who was right for the team, seeing them come into the team, flourish, grow, and ah you know now be way better at their job than I ever was. That's yeah i love that that feeling that uh ray sorry you were i cut you off with my uh no no no feelings of joy sorry about that good this is all good i think um no i think those feelings of joy and the sort of general positivity is is i think this these to me are the kind of uh
01:12:10
Speaker
the That's the attraction of management. um the thing The reason why I'm reluctant to get into management, to be to have come there to be completely honest, is because there's a lot of process around management and there's a lot of process around engineering, which you know I personally feel is generally total worth of time.
01:12:34
Speaker
um But there are certain cultures or a certain environments where some management teams will value numbers and, know, sort of like, I don't know, they, they, they value having, you know, a very, like they don't want, to want to go agile, but they don't want to have the spirit of agile. They want to plan for everything. They want, they want to have deadlines. They want this, they want that, they want the other, which is just the opposite of agile.
01:13:02
Speaker
So I'm kind of like interested drew and and Josh as well, actually, you know, both bo can chime in here. It's like... We're the same person, so... Exactly, yeah.
01:13:13
Speaker
It doesn't matter which one answers. Probably, you know, multi-channel is stereo. you have one sentence at a time, you know, left, right. We'll get out to do a stereo effect.
01:13:24
Speaker
Yeah, just, you Josh, you say the next word and then I'll... Yeah, exactly. No, but seriously, I mean, you know, what how do you like square that off between this sort of, this this but I mean, maybe as your're we maybe you're working in organizations which don't have those kind of cultures. So that that would be great. But, you know, you tell me.
01:13:47
Speaker
You go first, Drew. i have I have answers for that.
01:13:52
Speaker
Yeah, it's one of the reasons why um you know I my have not been as busy in the Clojure community over many years is because i mean, leading teams and and you know diagnosing processes and just management leadership in general, it's just such a fascinating um rabbit hole that I've completely fallen down. I'm at the bottom. If if there's a bottom of a rabbit hole, like I'm at the the bottom of it. You're in Wonderland. There's more hole below that, Drew, I find. Yeah, right. As you keep digging.
01:14:29
Speaker
found Found the cave underneath the rabbit hole, and somewhere down there. But I just fell down a rabbit hole and landed in Wonderland. So that's, yeah I guess, where you are. I think all of us you know have really you probably have more experience working for an inefficient organization than than we we have and and in ah in ah an efficient one.
01:14:50
Speaker
And so, you know I think you know that's one of the the things that I want to do as ah as a leader is is free up engineers to spend as much time actually solving the problems that they should be solving instead of dealing with the you know incidental incidental organizational complexity.
01:15:09
Speaker
um And so, ah it it you know there really is no perfect process. um It comes down to the right people um coming to ah good decisions and and acting on them quickly. I mean, that's really the only process that at works.
01:15:30
Speaker
Now, if time is not a factor, um then Which is true in a lot of organizations. If you don't need to accomplish any specific thing by a certain date, then it really doesn't matter what your process is because there's going to be a whole lot of other, um a whole lot of other things that, that, that muddy the whole thing. But, um,
01:15:51
Speaker
But when you truly have ah ah an objective and you um and you need to work toward that objective, um having the right people communicating with each other and you know doing that often is is really the the it's going to work every time.
01:16:07
Speaker
um And I think what you said about about Agile is really is is right. you know People have um basically put the Agile label on a bunch of things that are not not Agile. basically stated what Agile the whole i basically just stated what you know agile is Exactly.
01:16:27
Speaker
but If you go back to the original manifesto, I mean, it's basically like ship software often, like that's really all it is. um and, and, and have people talk to each other. i mean, it's, it's really not, uh, I, I, I think the problem with it is that it's not a process.
01:16:43
Speaker
And so people just want to put a process yeah on things that, uh, that they're ignoring, like, you know, having the wrong person in a position or not giving not giving feedback to people, um solving the wrong problem and not owning up to that early early enough. You know, there's all kinds of like weird things we do as humans to kind of, um in you know, not maliciously. I mean, it can often be, you know, subconscious. It can be things that we're not even aware of.
01:17:13
Speaker
um But we we we thwart things that probably could be a lot lot simpler um because we we try to make it more complicated than than it needs to be. um Yeah, that's my that's my hot take. Josh, what about you?
01:17:27
Speaker
Yeah, no, I think that's right. There is one thing that's inescapable though. it A company of a certain size, let's just say, don't know, 80 people.
01:17:39
Speaker
companies above 80 people will have processes like actual business processes that are happening. So this is like outside of the the engineering organization, right? Like bookkeeping has to be done, you know there's reporting and all of this stuff.
01:17:56
Speaker
And so a company at some point, even a technology company, there is this part of the company that has nothing to do with technology, right? So that it's a very different culture from engineering.
01:18:08
Speaker
um And there has to be some kind of interface between these business processes and the engineering organization. And I think like really good engineering managers can be an interface to that.
01:18:23
Speaker
So what you know a good engineering manager can create an agile environment for their product team, where it's exactly as Drew said, you get the right people pointed at the right problem.
01:18:35
Speaker
with the right support and kind of let them go. um And you know like I said before, making sure that you're feeding in the information they need. Like, look, the business wants to do this.
01:18:47
Speaker
Okay, so if the business wants to do that, then maybe like we need to do this other thing, whatever. um But also, people ask for you know people ask for data when they're worried about something. right like And we do the same thing as engineers. right like I don't know how to solve this problem. Give me a bunch of data so I can stare at it and you know have inspiration.
01:19:12
Speaker
well, guess what? like you know The C-suite is the same way, right? If the CEO is worried about something, the CEO probably says, like show me all of your process data for you know how many stories per minute you're delivering or whatever.
01:19:27
Speaker
and And that's where you get into trouble. So again, like as a manager, your job is to have that conversation with like the the you know senior management of the company to help them work through you know whatever it is that they're worried about.
01:19:43
Speaker
Because you know maybe they are right to be worried, right? And maybe maybe you and your team need guidance, but you as an engineering manager have to be the person who can translate between that.
01:19:56
Speaker
So the team isn't just like, ah, the CEO doesn't get us. It's like, well, no, but you don't get the CEO either. So that's kind of why I'm here. That's right. Yeah. i So I said sort of the pro engineering answer, right? like Like get some smart people and like great get things out of their way. um I'll now you know give the pro management response, which is, i mean, engineers are often terrible at providing information.
01:20:24
Speaker
We all are, I think. for a four yeah that's true. um you know for For such a technical discipline, um we often do not use data to drive decisions, to run our systems, um to yeah you know do do a lot of things. I mean, if you want to look at a data-driven organization, I would typically tell you to to go to the sales department rather than engineering department, which is um which is quite a condemnation on you know our our slice of of the industry. um so i you know i And I do think you're right that EMs are sort of that translation translation layer but
01:21:04
Speaker
you know, as managers, we can't do it all. We really have to coach engineers to to to really provide them the the substance of of those justifications and and and reasons and roadmaps. And, you know, like, even though, yeah, I'm gonna give you a working thing in two weeks or a week,
01:21:20
Speaker
here's where I'm going. And like, here, here's a couple of paragraphs about, you know, the narrative of this thing that might take six months or a year. And writing that is, is, is challenging. I mean, a lot of engineers really struggle to fit the thing that they're working on into a larger story.
01:21:38
Speaker
And, um, and, and businesses really, ah they, they really deserve that because, um, you know, you've got, Uh, decision makers at high levels that are dealing with an, an enormous amount of information.
01:21:52
Speaker
Um, I, I don't think, you know, like kind of engineers and their daily work can appreciate just how much, um, that, that your typical executive has to, to deal with.
01:22:03
Speaker
Um. And, you know, being able to concisely state why something is worthwhile, what its progress is, how much it's costing and, you know, what the value to the business is.
01:22:15
Speaker
that that i don't i don't believe that that is, you know, solely up to the the sort of quote unquote management. I think every engineer working on an effort should really be able to to to state that in some form or fashion if they were asked.
01:22:29
Speaker
Yeah. Yeah, definitely. i mean, I think I always push back whenever somebody is like, ah oh, you know, I want to be promoted to management. And I'm like, not a promotion, career change, um because it is a ah totally different job.
01:22:47
Speaker
Now, having said that, um so I think some of the things that really good engineers have do prepare them for that career change. And one of them is what you said, communication, narrative,
01:23:03
Speaker
um I think like the thing that has been more responsible for whatever quote unquote success I've had ah in my career has just been my ability to communicate in English, not not a programming language.
01:23:17
Speaker
and um you know And I always joke with people that like I am a programmer, but what I program is organizations and the language I use is English. And gee, it's that hard. Yeah. It's really hard to do. yeah Yeah, what I wouldn't get for a compiler right about now. Yeah, indeed, indeed. Well, you've got a dynamic language right there.
01:23:35
Speaker
Oh, that's nice, yeah. ah By the way, ah before we we do the context switch, common Lisp, there is a command you can run at the REPL that has it recompile itself without ever stopping the runtime. So, you know.
01:23:51
Speaker
Common lisp for the when. But to Drew, what are you up to these days post-Elastic? So I was going to take some time off and yeah spend some time with with my family. And then they said, please don't, or what?
01:24:10
Speaker
ah Well, this friend... They wouldn't let you. Yeah, they're like, actually, we don't want spend that much time with you. Can you please find a job? I got a call from a friend, Zach Smith, who had...
01:24:26
Speaker
she had created a um ah startup back in 2014 that was um focused on bare metal hosting. So basically take taking the cloud ah style of consumption, but um giving you literally a bare metal machine, not not like you know a hypervisor that just has all of the resources on the machine, but like um like like giving you like an IPXI interface into actually provisioning whatever OS you want on the machine.
01:24:57
Speaker
Um, and, and it was, uh, very successful.
Packet's Role at Elastic
01:25:01
Speaker
it's called packet. Um, and we had actually used it at elastic, uh, because we were, uh,
01:25:10
Speaker
i we were, making sure that we um covered our bases from an architecture standpoint. um And when ARM was starting to um really gain a lot of traction, this was before like Graviton and Tau and and sort of the cloud instances, um but but ah you know ARM hardware was was on the horizon. And so we needed to to add that to our CI pool. and so um Zach and and his his company, they had ARM machines through their partnership with Works on ARM and and um and the the Java builds of of um ah the JDK that worked.
01:25:51
Speaker
that that that builds on ARM. um So that's how I first got introduced to them.
Equinix's Acquisition of Packet
01:25:57
Speaker
um And then they got acquired by a data center company in 2020, this company called Equinix, which if you're like me, I'd never heard of them unless you, for some reason, buy co-location space and and you then you would know about them. But I had been in the public cloud for, you know, many years. And so I didn't ever need to buy Colo space or rent Colo space.
01:26:22
Speaker
Um, but Equinix is the largest, um, you know, uh, data center provider in the world. Uh, we have 300 some odd facilities and 70 plus markets. And, um, we are often the, um, we, we are often the building that you're, that probably this, this call is going through right now. um and, and,
01:26:43
Speaker
ah essentially ah brought Packet in to to be um ah to provide compute on their um on the network backbone of the company, which is um you know provides 70 to 80% of the world's internet traffic. so 10,000 enterprises, you know every industry is a customer. I mean, it's ah it's an amazing, amazing business and company.
01:27:10
Speaker
um And so, they ah so Zach talked me into joining to lead engineering for um two
Leading Engineering at Equinix
01:27:20
Speaker
products. One was their but what became their metal product, um which was their their business that they had before. and then um And then another product called Network Edge, which is essentially a um and a hosted um VNF provider, which VNF stands for Virtualized Network Functions, which is basically a VM that has like a Cisco router.
01:27:44
Speaker
um instance instead of actually buying a big expensive Cisco router. um And so um and and then the I am not a networking expert.
01:27:54
Speaker
I am not a bare metal expert. um so the The sort of strategy in that that hire was um for me to come in with a lot of ah you know distributed system, ah large scale software um development and and and operating you know systems at scale.
01:28:13
Speaker
Um, and so, um, because, you know, Equinix for all it's, you know, um, for all of its value as a data center company, it's still finding its footing as a software company.
01:28:30
Speaker
Um, and so, um, you know, having to, um, sort of incorporate these products into the, the, um,
01:28:41
Speaker
into the the company was was meeting certain challenges and we wanted to expand and all kinds of things. So um so yeah, that was how I got to the company.
Expanding Equinix's Products
01:28:51
Speaker
um And then through the course of trying to expand these products and meet different customer needs, um it was clear that um we really needed to tackle the the um sort of the tech debt is probably ah a good way to describe it, but the company has been around a while and it hasn't had a modern um take on sort of identity and and access management.
01:29:19
Speaker
um And so um really that is like kind of the substrate of any good service ecosystem is you've got to have that story pretty well defined. and um And so, I decided to kind of focus on on that problem and it really was a better fit for my you know sort of background anyway.
01:29:40
Speaker
um And so, that yeah, that's what I've been doing for for the past year or so um and ah And it's going great. where're We're consolidating sort of kind of cleaning up the the debt that that had been accumulated and and we're just going straight, you know, standards-based interfaces and, and you know, building what should be really, Equinix's story from a physical standpoint is is a rich history of physical interconnection. um
01:30:10
Speaker
You know, all of the sort of major interconnection points in the world where you have different networks that need to talk to each other, literally or are, you know, if AT&T wants to talk to Amazon, to ah talk to Google, you know, that is a, literally cables in a building somewhere. And it's most, you know, most often it's Equinix that provides that.
01:30:31
Speaker
Well, if you think about the the virtual space or digital space, we need to provide that same sort of software ecosystem.
Equinix's Federated Software Model
01:30:37
Speaker
um Customers want to be able to build things at Equinix. They want to be able to build software, not just you know build out their co co-location space.
01:30:47
Speaker
um And so building a federated first model where you can basically just speak standard you know auth with this with everything is um ah is what we're doing, and it's a lot of fun.
01:31:01
Speaker
Fabulous. Yeah. yeah And yeah, okay. So it's mostly, just to be clear, that that's something that's an internal project at Equinix. It's not some external product that you're developing.
01:31:14
Speaker
No, I mean, in the sense that, um you know, if you think of like I am in a cloud provider, like it's not really a revenue generating product, but it's very much a public, know, you've got public APIs. It's kind of the the thing that you have to do when you're stitching services together. i mean I mean, how many of us have banged our head against the wall trying to figure out like,
01:31:34
Speaker
how to appease the AWS IAM roles to make a multi-service thing work, right? Just use the policy simulator. It always gives you the same results as real IAM, right?
01:31:47
Speaker
um Exactly. exactly um But you know the fact that you can do it is ah is pretty amazing. and and the And the process is just not good at Equinix right now. um So that's what we're trying fix.
01:31:59
Speaker
Yeah. And that's Clojure stuff, is it? So we have, um yes, we do have sort of a kernel of ah of um functionality that is closure that's written in closure um it's not it's not dependent on closure but um basically how you register a service how you sort of um ah govern what can access what and how you give tokens out to be able to to um to um to authenticate that that and authorize that access um that stuff is all all written in closure
01:32:38
Speaker
And it's it's sort of like the service registry aspect of it, ah which is why, you know, it's interesting you said earlier that AWS started sort of with with that as its first service. um That really is the sort of the, if if you're going to build ah infrastructure and you really do want APIs all the way through, no then you have to have that. so I call it a kernel because you really do have to have like that seed where you kind of register everything. Because if you don't, then,
01:33:06
Speaker
then now you can't access it via an API. If you have any information that is doesn't participate in that in that ecosystem, then you know you've got to to figure out how to get it in the system, and you can't do it with APIs.
01:33:20
Speaker
um And so that's ah it's been really interesting. you know We're not trying to build a cloud or be a cloud provider, but there's some of the same primitives there that we've had to bootstrap. It's really fun.
01:33:31
Speaker
Well, I mean, if if closure is rock and roll and management is country, this has been a little bit country, a little bit rock and roll. And I think maybe that's a good way to start wrapping it up.
01:33:45
Speaker
I mean, maybe... um I don't know, what would what would be your kind of like closing thoughts, Drew, about about how closure has affected your work or yeah as as a sort of developer, as a manager, as a sort of individual? What's what's your kind of like, maybe we can like close on that.
Impact of Clojure on Career
01:34:08
Speaker
Yeah. Well, I, um, I just want to say that I really do owe my career to, Clojure. It's been, I really did not enjoy a lot of aspects of programming. um when I found Clojure, and that sounds, you know, kind of an extreme statement, but it's really true. think it trunks very well. Yeah.
01:34:31
Speaker
Yeah. i I, um, Closure Rich really did bring the, the, the, um, a lot of great things together in a very practical way.
01:34:44
Speaker
Um, and so it really, it really lit my, um, you know, it allowed me to focus on ideas instead of, you know, dealing with, um, weird, you know, just, Memory allocation. that's Programming issues, right? ah like i you know I can't tell you the number of runtime errors in Python or packaging your problems or just like ah all the stuff that that we could spend a whole other podcast talking about. um And so I just want to say that um and I really just, the Clojure community is still like my gold standard for for quality. I mean, it's just, you know since Clojure isn't the thing that gets taught and like
01:35:24
Speaker
boot camps or, you know, is, is backed by Google or, you know, like it's not, if you come to closure, like it's because you've, you have hit ah a wall somehow and you're like looking desperately for a solution. And, um so, so I feel like there's just that inherent bond and all in, in, in the, you know, between all of us in the community where,
01:35:46
Speaker
we we really have found some measure of of of happiness because we were able to ah finally, you know, do things in a way that that didn't make her us pull our hair out.
01:35:57
Speaker
Does not mean that closure is perfect. I mean, obviously, like anything you love, you could you could go on and on about the things that are not good about it. um But I mean, it's just- The weather, breathe, you know? Yeah.
01:36:09
Speaker
it ah It's just on another ah another plane as far as technology and and just community. and And it's still my, you know, it's still my ah my first love as it were.
01:36:22
Speaker
Wow. Beautiful
Conclusion and Listener Engagement
01:36:25
Speaker
words. I think with that, Drew, we should just thank you for teleporting yourself to Europe to be on DEFIN.
01:36:33
Speaker
It's been absolutely wonderful talking to you. Thank you both. It's been an honor and really enjoyed it. Fabulous. Thanks.
01:36:44
Speaker
Thank you for listening to this episode of Deaf Anne. And the awesome vegetarian music on the track is Melon Hamburger by Pizzeri. And the show's audio is mixed by Wouter Dullert.
01:36:55
Speaker
I'm pretty sure I butchered his name. um Maybe you should insert your own name here, Dullert. Putter. If you'd like to support us, please do check out our Patreon page and you can show your appreciation to all the hard work or the lack of hard work that we're doing.
01:37:11
Speaker
And um you can also catch up with either Ray with me for some unexplainable reason. ah You want interact with us, then do check us out on Slack, Closure in Slack or Closureverse or on Zulip or just at us at Deafen Podcast on Twitter.
01:37:30
Speaker
Enjoy your day and see you in the next episode. Music
01:38:08
Speaker
I'm Nipro. I mean, it's like, you know, two Indians and one chief here. Two chiefs and one Indian even. don't know if that's still allowed as a sort of, is that a... I don't know. Would it get cancelled for that?
01:38:19
Speaker
I wouldn't use it on the air. Walter, make sure we get that.