Transcript
Speaker: Everything was like manually configured as to where things lay. and We went from like less than 10 servers to 80. If we fell behind, there was no way to recover. That's terrifying. It was very much the pace and the nature of the work that made it drain people in the way that it did. You could run the script in a dry run mode for every single job that a team had. And then it would say, all right, there is like a 99% chance that this 90 jobs are just going to work. I would much prefer people did a ton of prototyping, like with those, you know, dry runs and all that kind of thing, or like just gather information about stuff and have that at their disposal.
Speaker: and then have like a more rough timeline and schedule for things and then use that information to to figure out the best way forward in web. Absolutely, I agree. Today on Turn Stories, we've got Tom Elliott.
Speaker: um Tom is currently the founder at Akiroot. But before that, he worked on the infrastructure at Yext, where he was he joined as a senior software engineer, um ended up as director of engineering, and oversaw ah ton of their transformation.
Speaker: we're going to talk about why they didn't use Kubernetes today and all the journey that happened along there and how they they ended up where they did. but um Welcome, Tom. um It's great to be here.
Speaker: So, all right, let's get right into it. Because i was looking at kind of the trajectory of Yext and trying to trying to put a couple of dates together. You talked about um the decisions you made and and the company. What I didn't realize is that Yext had raised their Series f shortly before you joined. $50 million on like half a billion dollar valuation.
Speaker: Yeah. And you joined, if I'm not mistaken, like the week that Kubernetes went to general availability. So you were at this like perfect storm of scale and like infrastructure churn in the world.
Speaker: Tell me a little bit about what it was like to walk into Yext at that point. And like, what did you see? Yeah, ah so it was it was interesting, especially because at the time i was coming from a really small startup.
Speaker: like and you know the i When I left the company before, there were like four people there. At most, they'd been eight. And then I joined this 300-person company. There's now like a 1,500-person company.
Speaker: um and it it was It was interesting because there was a lot of stuff that was already there. you know and a lot of history already, but there were also a lot of practices that were they were still trying to figure out and like sorting out the best um the best approach for things. They they weren't doing CI at that point. It was all like manual deploys from someone's laptop, which... Well, desktop at that time. Oh, everyone had desktops.
Speaker: ah You were considered really fancy if you got one of the trashcan Macs back in the day. That's how long ago this was. And... yeah Yeah, it's funny you mentioned the GA of Kubernetes.
Speaker: ah that That timing sounds about right. i hadn't put that one together myself. um But at the time, Yext was using um this orchestrator that they built in-house um because that sort of stuff didn't exist.
Speaker: um And they'd done the classic thing in that period of time of just hiring a bunch of people from Google. ah which was relatively easy for them because they were right across the street from Google at the time. So it was like, oh, hey, your commute doesn't have to change. Welcome aboard. So those engineers all went about basically rebuilding the tools that they already had full and being like, oh, I wish I still had this. So they ended up with a version of Borg that they called Khan.
Speaker: um And I thought it was beautifully ironic that we ended up using Nomad because that is yet another Star Trek villain. But... It's kind of a deep cut Star Trek villain, but still. That's funny.
Speaker: um Interesting. That's yeah. The desire to use ah your old tools is definitely real and especially real in people who worked at Google. No offense to anyone who's listening who worked at Google.
Speaker: Yeah. Cool. So what's when you got there, was like how were things going? Was the scale, like real handling the scale in a way that felt sustainable or was there there sort of cracks in the cracks on the woodwork?
Speaker: um I think it was it was generally working well at that point. There was definitely a sense that... you know People were starting to wrestle with legacy at that point. And there were like you know some things that had been around for a while that people didn't want to touch.
Speaker: um and But there was a lot of exciting new stuff going on. like and i I joined right when someone had finished up their internationalization project. So there was this whole fun internal framework for that.
Speaker: um Many years going around trying different ways of doing front end. and That was always a struggle until they ah they adopted React. um and um But it yeah, it was like there was like, what, six or so teams, relatively small, like four to eight people.
Speaker: ah That was a structure that worked really well. For Yextin, actually kind of kept going. It was a ah really big deal when they had added another layer of management on top of those teams.
Speaker: um But like each team had a really well-defined thing. They had a concept of owning your entire stack. So the you know each team would be responsible for a product vertical and would own database right to front end.
Speaker: So you had to do a little bit of everything. um I joined a team that compounded that because we were doing mobile as well. So we were like you um back-end database,
Speaker: all the stuff in the middle, front end, and mobile apps on the side. Interesting. how so How many engineers were you at that point? It was like somewhere between 30 and 50.
Speaker: Okay. Even at that scale, it seems like there is... there's all There would be a lot of like either duplication or confusion. was there how do you How did that model work for you all where you basically had teams that had to own the entire stack? like i see I could see the upsides, but it seems like there's maybe downsides to that as well.
Speaker: Yeah, I mean, I think it worked well because they were really good early on making sure that there were consistent patterns and that there were common ways of doing things. that you know They enforced a very small set of languages. It was a really big deal when they added Go as well as and Java.
Speaker: um They were very specific about you know where you could deploy, um what sort of tools you would be expected to use. um ah Over time, and this sort of like was kind of there when I joined, but got better as ah as it went, um the onboarding process. it was It was generally understood that most people would be using the same IDE.
Speaker: and They ended up gravitating towards JetBrains IDEs. um And there was a big push to like you know deploy code on your first day, which required quite a lot of consistency.
Speaker: um And so the the challenge of teams sort of doing things their own way or like diverging from that, that only really became a problem around about 15 teams.
Speaker: And when they started to introduce like groups on top of the team that then had their own way of doing things, um and the products started to get a little bit further apart. Interesting. So was that that growth from like, you know, four to four, six, eight teams up to 15 all sharing basically technology based on cultural norms?
Speaker: Was that the was that the trigger that caused you to kind of invent to go investigate something that wasn't con? Or was there another trigger? um there There were a couple of um contributing factors. I mean, one of them was that we we didn't really want want to be investing all the time into Calm at that point. And it you know it was like one or two people were working on it. And that was kind of sustainable for a while. But as things got bigger, you know you'd have to dedicate more and more resources to it, which...
Speaker: didn't make a whole lot of sense since there were options available. um And it was probably more around the 20 team mark where we started to see like the challenges there because then you were getting into places where there were features that we were clearly missing that would have taken a lot of work to put in place.
Speaker: um So we had a separate infrastructure team who were managing all the servers. like Everything was in a data center in New Jersey. And anytime they wanted to upgrade one of those machines, um they would have to go through this whole down. So it used to be that they would actually have to change code to move stuff off of those machines.
Speaker: and then change the code back to put it all back. And it's like everything was like manually configured as to where things lay. And, you know, there was a lot of work involved in adding a new server or that kind of thing, which you didn't have to do very often when it was small. But, you know my time there, we went from like, you know, less than 10 servers to 80 in one environment.
Speaker: And So that that was sort of a pain point that we saw. And there were like tools that were put in place to like help facilitate those moves and and all that kind of thing. But like we were itching to have something that was more powerful, but also a little bit more standard so it could rely on either another company or um And we had a couple of tries at Kubernetes.
Speaker: ah The first try suffered because we were hosting our own stuff in a data center. This was like before a lot of the tooling that made that more straightforward for Kubernetes.
Speaker: So we we really struggled to get the management aspect of it together and feel comfortable with that being able to trust that these collectors were going to be reliable enough for us in that data center.
Speaker: Yeah. and I mean, this was, this was quite a while ago. Sure. Sure. But what you what did you hit? Do you remember? um it was Yeah, there were there were a few a few incidents where um basically just like you know a whole bunch of nodes went down, ah things didn't get moved around correctly, and then like we were doing some funky stateful stuff for some reason, so like we lost things, and um and the the upgrade story was really the biggest challenge for us.
Speaker: um Interesting. yeah I probably shouldn't have said that we'd lost things, because I don't think we lost any data, but... Yeah, but you can have, i mean, this is similar at at Slack where like, if you lose a cache, you haven't lost data, but you have lost operational stability. Absolutely.
Speaker: Yeah. Oh yeah. That's a, that's a, that's a really great clarification because that was like, we'd have things like RabbitMQ node goes down, you lose that queue, you've got to recreate it from the database and that takes time.
Speaker: Yeah, so absolutely. there's Semi-stateful. We need a word for stuff that isn't like data loss, but really, really hurts when it gets rescheduled. yeah Yeah, absolutely.
Speaker: um Yeah, so there was there was that, yeah, the upgrade path it was a real pain. um like And yeah know it's still not the easiest thing in the world to upgrade Kubernetes. Yeah. but If you have it foist on you by your provider and all that kind of thing.
Speaker: But you know we we we really struggled getting a good cadence down for that, which wasn't previously an issue with Calm because we just upgraded each and every node. It was pretty straightforward.
Speaker: um So that you know we we sort of shelved that. um And then there was another project exploring using Kubernetes in cloud regions because we had a bunch of cloud regions for Kubernetes.
Speaker: for serving consumer data that was like read-only, API search, all those kinds of things. um And we we explored that for a while. I think we tried to but bite off more than we could chew with that. And you know we tried we tried to basically like reinvent everything all in one go, which was um unsurprisingly, in retrospect, very difficult.
Speaker: um So we ended up... ah This might be skipping ahead, but let me ask you a question that might be a little disrespectful. was Was this a problem with Kubernetes or a problem with the way you had deployed Kubernetes?
Speaker: ah In the first case, it was absolutely about how we deployed Kubernetes. um and In the second case, I think it was more of a project planning challenge.
Speaker: um it was It was a very big vision and there was a lot of great things about that vision that the we kept hold of afterwards. But the plan was like, hey, we're going to reinvent everything and it's going to be the world and this is going to be the best thing since sliced bread. And yeah I'm sure that's a pretty familiar story to a lot of people.
Speaker: um And so that that, again, it wasn't necessarily Kubernetes. It was like all the work that we thought that we had to do to get to Kubernetes at that point was sort of the the big the big challenge. And we sort of boiled that down in retrospect.
Speaker: Had a look back over that project and we thought, all right, well, what what do we need to get something that's just what what's going to drive us forward over the next 10 years?
Speaker: yeah um And we kept hitting up against the ah lift of doing a complete Kubernetes migration um involving containerization.
Speaker: um because we were running bare processes on servers at that point. And um we were relying on having things installed on those servers. um yeah but i'm I'm sure a lot of people would relate to problems with image magic.
Speaker: i've I've heard many people complain about image man magic at many companies because it's just one of those dependencies you end up with and you've got to remember to have it and relate. So there was a lot of stuff that was already set up on these servers. The responsibility was spread around.
Speaker: And by this time, we had like, you know maybe a couple of thousand ah jobs running. um ah At one point, I did the math and worked out that we had a higher microservice to employee density than Huber um where did, which is probably not the flex that I thought it was at the time.
Speaker: um In retrospect, certainly not.
Speaker: ah So, me ask you this, because you are you know i love this model where you have like tightly aligned product teams who own soup to nuts, the entire stack.
Speaker: I love a little bit less that maybe those people had to think about the bare metal in a data center in New Jersey, or at least when I think about working there. um
Speaker: Who cared enough to go work on Kubernetes or you keeping con up, like driving these decisions forward, because my naive assumption would be that any given product team doesn't care that much because got a roadmap and OKRs and a CTO breathing down their neck.
Speaker: Who is who is willing to go to bat for like we need a major platform shift here? ah Funnily enough, the CTO, who was really happy with me. We were really fortunate in that we had a few very, very technically strong senior leaders um with within within the org. And um yeah a guy here and I can't remember if he became the CTO before or after proposing this project. It was probably before.
Speaker: um he had been wrestling with these kind of problems for a long time. He had a big hand in both the previous attempts. Like the first one, the the migration part was like his baby. And he he cooked up a really good plan. Actually got us to a point when most things were running and in Kubernetes, which helped us later on.
Speaker: um And so he he proposed looking at Nomad explicitly because it didn't require you to run containers on. Interesting.
Speaker: um And he he made the leap that this was like this was the the big slowdown factor that was going to cause us problems. And just being able to effectively lift and shift most of our jobs um seemed a lot more tenable.
Speaker: Interesting. Very cool. That makes a ton of sense. You done you you could pick Nomad. like You could make that decision because you have done all of this work with Kubernetes and sketched out, like well, here's all the work and 90% of it's going to be putting stuff in containers.
Speaker: What if we don't do that? Yeah. And there was like you know networks and all all those other things. and um the The irony being that we'd effectively built something that was based on Google's internal tools.
Speaker: But the public open source project that they created diverged so far from that that we had to go and take a third party. That's funny.
Speaker: I have heard that before.
Speaker: Yeah, it's it's kind of interesting because it's like, you know, there's also the the time gap as well. Like, you know, and Kubernetes is way more complicated than it it was when it was originally.
Speaker: um but and as I recall it didn't even have deployments initially right it was like all pods that you were managing yourself in the first release yeah deployments are a later abstraction because who could have known that you need to deploy software yeah who who could have thought that you might want more than one instance of it I'm glad they figured it out honestly yeah yeah absolutely absolutely Um, but yeah, so the, um, so a lot of the advantages of, of Nomad was that it wasn't trying to be everything.
Speaker: yeah So, you know, you could add on console, you could add on vault and all these other things to do the fancy additional bits. But if you just want to run a process, yeah, Nomad's got you. Um,
Speaker: So you made CTO makes this decision. it's like, i've I've seen too much with Kubernetes. We're not doing that work. We're going to go to Nomad. We're just picking up Nomad, not the other the entire HashiCorp stack. what What was the goal? was it Was it to get everything off of Khan? Were there certain services or processes that were were more important? like How did you think about actually getting from here to the promised land?
Speaker: So the biggest motivating factor for us was enabling the infrastructure team to do upgrades of individual servers easily. So getting everything in the data center.
Speaker: onto Nomad was our primary goal. And we you know we did have these additional points of presence, but we weren't too worried about the upgrade story there because it was, um you know you could do immutable infrastructure just withdrawing the grid.
Speaker: sorry um So we start off with the plan. Let's look at the data center first. That's the big thing. And there were um we wanted to look at ways that we could order the migration to be based on business needs.
Speaker: And we'd considered like, okay, so we'll do the, you know, the least important things first, as in like those few services that like, oh, maybe they run a cron job for something that only really impacts, you know, 30% of our customers.
Speaker: If that cron job doesn't happen, we can do it manually. You know, we can, we can recover this pretty easily. And then there's other things like, oh, there's this you know this service that provides a front end for a product that a very small number of customers are using. So you know the the risk there is much lower. And then like doing the most significant things last. like Everyone relies on this. This is absolutely critical. If this goes down for more than 20 minutes, we're in trouble kind of services.
Speaker: um So we thought about doing that, and then we realized we couldn't. because this was in a data center and we did not have that much headroom. So budgets being what they are, I think we were able to maintain a headroom of maybe five servers at a time.
Speaker: um And but around then we had 50 or 60 servers to worry about. um So we ended up doing this like two to three machines at a time.
Speaker: and we would be doing direct migration of the jobs that were pinned in Khan over to Nomad. And so that had to be coordinated much more closely.
Speaker: um So in the end, we created a task force, I think we called it a task force. We might have had a fancier name. but um And that was formed of like a bunch of people from my group ah focused on DevOps.
Speaker: um It had a a permanent member in one of our architects who was like really good at this kind of thing. um And then we would rotate people in from the other groups.
Speaker: um And we kind of try and schedule it so that we do the migration group by group. um And because of the way Khan had grown up, we actually ended up with like dedicated machines for certain aspects of the product. So it actually made group by group migration kind of tenable because they effectively owned the whole of two or three servers each.
Speaker: um the The problem with that was that if we fell behind, there was no way to recover.
Speaker: It's terrifying. So it's like, if you've we've like scheduled this out, and we've said to all of these teams, this is the schedule for when your people are on deck, and when we need to steal someone for a sprint from your team, which, like you know, you you say that to anyone, like, you're not going to have this this person on your team for a few weeks, they' they're not going to feel good about that.
Speaker: ah and But every time we, like the first few weeks, we were like only getting like 70% of the way there. And then like something, we have an issue where like maybe we have to do a bunch of rollbacks or we discover something we hadn't expected. So weve got to spend time working on that. um you can't You can't meet your goal enough.
Speaker: So there ended up being like this kind of semi mopping up kind of process. Like we had to extend some weeks, which pushed the whole schedule out and kind of juggle things around a little bit.
Speaker: So, um you know, we were sort of forced into doing things in that kind of sequential way, but like um it, was was very, very difficult to maintain pace and to deal with those sort of like unmovable goals.
Speaker: Yeah, that but that kind of that kind of pressure coupled with what is effectively exploratory work. Like you have a process, but every service might be a little bit different. is i That is difficult.
Speaker: um What sort of things did you run into? Were there were there like particularly gnarly cases that, with the benefit of hindsight, you you would have liked to have known about so you could keep yourself better on track?
Speaker: ah The biggest problems tended to be where we discovered there wasn't problem particular package installed on the the host or there was a quirk of configuration that wasn't quite as we expected it.
Speaker: um Like, I think we had issues with like the number of available ports a couple of times. And and Anytime we discovered there was something missing one of those things, we had to go to the infra team and then be like, can you get someone to drop everything to do this for as quickly as quickly as possible, please? um we didn't have low-level access to one of these machines.
Speaker: Gotcha. And this was stuff where you would discover like you'd move a service over and it was sitting there listening on a port that does not talk to the internet? there was Yeah, but that was definitely a thing.
Speaker: Yeah. but um I think some of the migration did involve like standardizing ports. um But then there was like patterns where it would work fine in Khan, but the convention that we'd set up of like, oh, you have to set the port number by passing in this particular flag. If like one service didn't do that, it probably didn't matter that much in Khan. But soon as you move it over to Nomad and it needs to advertise, that that all goes out the window.
Speaker: Yeah. Yeah. yeah Yeah. Did you build tools as you were going through this, to like catch common bugs, try and make it, make it go smoother the further you got in the process? um A lot of it was scriptable.
Speaker: um And that was one of the really nice advantages of having all of these conventions and standard ways of doing things. Cause we could just for the, for a large part, like sort of automate the generation of the nomad conflict from the,
Speaker: um the Khan config. um We actually ended up having a ah wrapper written around the Nomad config to make it even closer. And so we had this sort like additional YAML BSL ah for managing all these things, which I wasn't wasn't the biggest fan of, I will have admit.
Speaker: um it did It did help with a couple of things. um So that that that was largely automatable. ah There was also you know scripts to move something over, wait till it came up, and then like report on a couple of common, yeah do health checks, make sure all the ports are open, all that sort of thing.
Speaker: um And we made sure that our monitoring was very, very good when we were doing this. So like if something suddenly went wrong, like it hit an edge case where it needed this you know one package or config on the server, and then if it wasn't there, the whole thing would fall over, we'd be able to catch that quickly.
Speaker: Got it. That makes sense. How did you think about these things are some of this this stuff is largely automatable. How did you think about the line between what was worth automating and what was just better to do manually, given that you were going to have to do this 5.7 times per engineer?
Speaker: Yeah, and I mean, the ah think the big thing was like figuring out once things were automatable in a repeatable way, how often you could batch things together to minimize human interaction. Because if like you know one team has 100 services that they need to worry about, you don't want to have to run the same script 100 times.
Speaker: ah You want to be able to batch things together and like yeah but we we ended up like usually doing maybe five services to a single commit, which seemed to be a ah good enough size that you could effectively review it.
Speaker: And then light rolling back wasn't too big of a deal, um but you weren't like flooded with all these different commits that needed to be reviewed. Yeah, that's an interesting that's an interesting way of thinking about it is like you you will move at a certain speed it would be Of course, there's there's some upside to having it all done tomorrow.
Speaker: But i the risk is probably too high. So finding like five per gets you like the risk profile you want. I like it a lot yeah makes so yeah and that I think that was a big part of selling it to the individual teams as well.
Speaker: um yeah Because if something goes wrong, they were probably going to be pulled in. um You know, things went wrong enough. So it's like, oh, we're doing a thing that might get you paged.
Speaker: ah This is what we're doing to try and avoid that and make it as easy as possible for you to help if you are in. Gotcha. how How did you work with those teams? you're You're about to go rip their service out and maybe break it.
Speaker: Like what kind of prep did you do with them? Um, so having, having a member of their team, like on the task force was a really big, a big help because that was someone who kind of knew what was going on and they, they could sort of be the face of the migration for that team.
Speaker: Um, but we, we talked about this plan for months ahead of kicking it off. Um, and the, like with this and sort of a follow on migration I might hint about in a minute, um it was really important to keep communicating and like keep all the teams informed about what was going on.
Speaker: And at this point, we did like a bi-weekly demo at the end of every sprint. So it was pretty easy for us to slot in a video for the migration program process to say, like hey, this is what has happened. These are some of the interesting challenges. This is what we're going to be doing next time.
Speaker: um And usually, I think, when we had someone scheduled to come onto the task force to do some migration projects,
Speaker: they'd probably spend the sprint before planning it and just double-checking things and and having an idea of what order they wanted to do things in. Cool. That makes sense. um was there as you As you went through that, you know you prepped all the teams, and you built the schedule, and you're going through it.
Speaker: How did... how was your team's morale over the course of the project? like Did it get better? or like Was there like a low point or a high point? um It ebbed and slowed.
Speaker: um you know the you know If we have have a bad sprint, people are going to feel down about that. um Getting it done felt good, but I think a few people did feel a little bit burnt out after that.
Speaker: and um and i like I think that it was very much the pace and the nature of the work that made it drain people in the way that it did.
Speaker: um we were We were fortunate because after we'd done this this migration to Nomad, like a year, two years afterwards, we ended up having to do something similar to containerize everything.
Speaker: You did end up containerizing everything. We did end up containerizing everything. And that that went significantly more smoothly because we could like we didn't have to do it machine by machine because Nomad helped us move things around a little bit.
Speaker: um And that from the automation standpoint, we were able to go through a process where you could run the script. in a dry run mode that every single job that a team had.
Speaker: And then it would say, all right, there is like a 99% chance that this 90 jobs are just going to work. If you run the script on this 10 have a problem. Yeah.
Speaker: And we may or may not know exactly what that problem is at this point, but these are going to need more attention. So we were able to sort of pre-roll parts of the migration so we could see where the attention needed to be before it became an urgent issue.
Speaker: That sounds huge. but That feels like it's the kind of thing that would just totally change the shape of the effort. Yeah, absolutely. it It made it significantly more comfortable and it also meant that we didn't have to centralize all of the work within one team. um So we got a really nice kind of side benefit of like, okay, we had to work closely with each of the groups and each of the teams to schedule the work for them.
Speaker: And usually it was like between two projects or you know they dedicated a couple of people to but they could do it how they saw fit. um but it became a race. We had a leaderboard. We had like very public list of every single team, how many jobs they'd migrated and how many they had left to go.
Speaker: And so no one wanted to be last. I love it. and So they were, they were like incentivized to schedule this work and just knock it out. And it,
Speaker: It was a little more, you know sort of up from planning and prep work, but maybe we did a month of building the tooling, building the leaderboard and all that kind of thing. And then we could just sit back and nudge people occasionally.
Speaker: That's if they asked for help, we could help. We could jump in, but it sort of became this kind of snowball community effect. That's cool. I mean, that i mean that's sounds qualitatively different than grinding through with with the teams under high pressure.
Speaker: mean, knowing what you know now, is there a different approach that you would have taken to the Nomad migration in order to get that same kind of vibe, for lack of a better word? It's a good question. I mean, the...
Speaker: having those um that restriction on the servers was was always going to be a big problem. um It would have been really nice if we could sort of like dual run Khan and Nomad at the same time. but And I think we talked about that with the infrastructure team and they weren't entirely comfortable with doing it that way.
Speaker: um What I probably would have done would have um explored tools that would have allowed you to done do that kind of dry run process. um And sort of maybe like, you know, you set up one server that um isn't, you know, has limited access. It's not going to be hooked up to the proxies or anything like that.
Speaker: It wouldn't be load balanced. um You just have one server sitting there and then just cycle through all the jobs and try to bring them up in Nomad just to see if they can operate on that platform.
Speaker: um And then you've got your list of like, okay, what needs tweaking? What needs fixing? Maybe we can move a few jobs around ahead of time before we start doing the machine version.
Speaker: Interesting. Yeah. That makes a ton of sense that if you've got, there's the, it, everything's a feedback loop problem. but Yeah. and And if you can push that feedback loop to the, Yeah. And if, if teams can help, teams can understand like this is broken and it doesn't seem like it should be broken, then you don't have to call them when the service is threatening to be down.
Speaker: ah Yeah. I mean, there's another very, no shifting lap. It's like, you know, the more, the more problems you can identify and anticipate early on when the risk is low, the easier it's on, you're going to have what is critical.
Speaker: Yeah, absolutely. um i i don't I don't think I caught it. um was there What was the impetus for the Docker migration? You decided to put it off and then you came back to it. like well yeah So, yeah, I mean, originally we wed thought, okay, well, you know, we can run bad processes in Nomad. That's going to take a lot of pain away.
Speaker: Great. we've We've at least bought ourselves some time. And we kind of wanted to do containers, but the original reasoning behind wanting to do containers was it's slightly less work for the infra team, which would have made them very happy, but it's hard to sell to the product engineers.
Speaker: thats Where's the benefit to me, they say. um But also it would have given us more flexibility um around you know how you configure each individual job and all that sort of thing.
Speaker: um um There's always the lingering, oh, it's best practice. This is how everyone else does it. Try to ignore that. But the motivation in the end was actually a security audit.
Speaker: oh ah We were using CentOS for a while and that got end of life. So we were like, all right, clock is ticking for us to change the OS on every single server so that we can pass this audit.
Speaker: So we've got a very strict deadline. um and And there's like a push to do it. You can sell that to the teams. You'd be like, hey, you know if we we lose this audit, we definitely lose customers, especially in sensitive industries.
Speaker: um You could ballpark a dollar amount that that would that would come to. but and So the reason for containerization was, well, hey, we've got to strip out the OS and start again.
Speaker: So we don't want to have to do this like whack-a-mole with packages again. So let's just containerize everything and just drop it onto this No OS. Oh, that's fascinating. um We saw something extremely similar at Slack, actually, where um every two years we had to do an Ubuntu upgrade. And the question and the the discussion with every team every time was it's going to be twice as hard for you to do a Docker upgrade to containerize your stuff now, but it's the last time you have to do it.
Speaker: You think you're going to work here for another two years? yeah
Speaker: Most of Slack is containers now. ah Yeah, absolutely. you know that's like Knowing that that is coming and it is something you have to deal with yeah is is is a huge motivator. and For us, um it meant that we we actually had to get to this deadline. you know like the The Nomad thing, the the pain of missing that deadline, I think we ah we ended up going over by that maybe a month.
Speaker: um the pain of missing that deadline was just, oh, we've stuck we've got to continue to think about this thing. Right. um Whereas, the yeah, the audit was a real push.
Speaker: Coming down from the top, everyone could get behind that. And we actually ended up finishing a month early on the containerization, which was very nice. Hey, there you go. that's That dry run script and having everyone, that's real.
Speaker: That's awesome. Very cool. I wanted to ask about your journey in this a little bit because you joined as an IC um you i as a senior engineer and and on this. By the end of it, you were director of this group.
Speaker: um what What was your journey like over the course of ah this time? um It was actually more complicated than that, because I was not just hired as an IC. I was hired as a mobile developer, ah specifically an iOS developer.
Speaker: ah ah That doesn't sound like this area of the stack. Yeah. yeah ah I was there for nine years. A lot of things happened. um So, yeah So I joined explicitly to work on this one mobile project that they were they were working on. that And then um I got moved on to another mobile project.
Speaker: And both of those got canceled within my first year. but Or at least so like you know we dropped them into maintenance mode. and not um Not a significant thing for us to work on.
Speaker: ah So I get moved to working on the back end on the Pages product. um That was when I started working in Go, fell in love with that language, still using it to this day.
Speaker: And um as I'm starting to work on the back end, I'm seeing that there's like these frustrations of dealing with you know all of these different services.
Speaker: And it was something that I never really had to deal with before because I was either like building, you know, singular mobile applications that stand alone or monolithic PHP apps with Laravel.
Speaker: So I have to start like 15 different services. on my local machine to be able to test one feature. I'm like, all right, there's got to be an easier way of doing this. And a couple of people had their scripts, but there were like one-offs and all that sort of thing.
Speaker: So I ended up like going through a couple of iterations of bash scripts to do it. um that would like do like terminal screens. So you had 15 segments of your terminal all spewing logs at the same time.
Speaker: um And that culminated in a tool um that would basically just, you would configure it to um have the startup scripts for all of your jobs.
Speaker: um You would group them together. then you just said, I just want to start this job group. keep it running, restart things when you change stuff. And within about six months or so, everyone in the engineering team was using Awesome.
Speaker: um And that felt so good. Yeah, bet. Yeah, it was it was like, there was always a bit of a separation between what I was building and the people that were using whether that was because they were anonymous people on the internet.
Speaker: I used to work for a VPN company, so they were very anonymous.
Speaker: um Or like there were layers of organization between you and the customer. um So you know you had sales engineers, you had support reps, you had ah account managers, you had the product team.
Speaker: um And so the only time you would really be guaranteed to talk to a customer was when they were very, very angry. Um, and it felt so good to be able to just look to the person sitting on my right and see that they were using something.
Speaker: That's cool. Yeah. It's a great feeling. That's really cool. And I, I can absolutely see how that would be more motivating than we ship the next version of the mobile app. And I hope these 10,000 people click this button more.
Speaker: Yeah, yeah, exactly. So like I got a real kick out of that, built a few other bits and pieces, ah helped with a couple of other people's projects. projects um And the thing that really sealed it for me was that I actually ended up replacing that original tool that I built with tilt uh which um not sure if you're familiar with that yeah okay recognition Daniel a Bentley right yeah yeah so yeah I think I know Dan so Dan showed me a demo of tilt very very early on I think they were like months into buildings he showed it to me on a bar at Gotham go mean it was like 2018 and I look at this and I'm like this is a much better version of the thing that I bought
Speaker: this is so um So we spent some time figuring out what we we needed. We entered a ah contract so we could like ask for a few um additional features, including like running things not in Kubernetes.
Speaker: That's one. needs to Yeah, we certainly did. um And so, you know, replace that tool and and I had to do a little bit of work to smooth that transition, but people were also loving this new tool because of a few of the additional capabilities that they got.
Speaker: And um that sort of sealed it for me that, like oh, so it's not just building things that like I feel good about the fact that built it. There's also...
Speaker: a world where I can just find things for people and make their lives better by facilitating their use of this. Um, and I think that was noticed, uh, cause we did sort of a bit of a reorg and, um,
Speaker: one of the VPs at the time comes to me and just says, all right, so it seems like you like building tools and you like you know helping people set up tools and all that kind of thing. Do you just want to run that group?
Speaker: We'll give you three teams. Um, I'm like, heck yeah, I'm in. Um, and that, that grew over time, added some responsibility, shifted some things around a bit, but, um, was doing that for about five years and, um, yeah, dev tooling. It's, uh, sort of my niche and it's what I love.
Speaker: Yeah. That's awesome. That's such a cool story. I had no idea that, that you sort of like found your way to it at Yext. Um, that that's fantastic um all right we're getting towards the end of our time so i want to ask just a couple of of quick quick questions quick fire do a lightning round um so let's start with start with fun and silly things well what's the most embarrassing workaround that is probably still running in production
Speaker: um Okay, so ah we had an issue with um basically making sure that everything was running the latest version off of the latest commit.
Speaker: um Because ah you know CI tooling, it was like sometimes easy to miss doing a promote or something. So we basically deployed the world every night. And they're still doing that.
Speaker: I'm genuinely shocked that that works. well it's I see how it solves the problem, but that's awesome. I think it's the kind of the kind of thing that you could only get away with in the sort of organization where like deploying to production rapidly is kind of the whole point.
Speaker: and like So you know for years, we were deploying 200, 300 times a day production Wild. that's fine Yeah, it was like small, quick iterative stuff, you know, feature flags to control bigger releases.
Speaker: um But because we kept up that cadence, like deploying the world overnight just to double check that everything was in sync. like was was fine. you know the The only kind of problems that it caused us were ah resource contention.
Speaker: um And um there was occasionally a situation where someone heard you know was rolling something out in a not quite backwards compatible way and they quickly learned not to do that.
Speaker: yeah Interesting. That's very cool. um All right. What migration advice does everyone give that is actually terrible? oh man, you're asking me to have a controversial opinion.
Speaker: hmm. yeah um so
Speaker: i I think that a lot of people would emphasize the planning stage of things. um And they would probably do, like for any project, like people say like, you know, oh, plan so, so much so that your plan is perfect and nothing can possibly go wrong.
Speaker: Well, it's going to go wrong. Like your plan will change. be okay with that. Like if you're doing a design ah for something, like when was the last time you look back on your design and had implemented exactly what you wrote down?
Speaker: Like it never happens. Absolutely. Yeah. but Like how checkpoints, be ready to change things, be adaptable and be okay with that. Absolutely. I was ready to, i was getting mad about this because I'm like no this is so much of what we care about at turn is like planning and shifting information left in migrations. And I absolutely 100% agree with you. The plan changes. That's the whole point. You need to constantly learn what has changed and what you're doing in the moment based on what you know.
Speaker: Yeah. Yeah, absolutely. And, and, you know, I'm not, I don't want to say that planning is bad, And I don't think people should like aim to do as little planning as possible. i think the more you know, the better. up But you know i I would much prefer people did a ton of prototyping like with those you know dry runs and all that kind of thing, or like just gather information about stuff and have that at their disposal, and then have a more rough timeline and schedule for things, and then use that information to to figure out the best way forward in the moment. Absolutely.
Speaker: I agree. That makes a It's 2025, so let's talk about AI. ah What would you do different if you had the AI tools of today?
Speaker: ah
Speaker: I mean, I'm still trying to figure out how I want to use them now.
Speaker: yeah Fair. um I think that like um using AI to help you generate um communication is useful.
Speaker: So like you know just coming off the and don't play too much, but also you've got to tell people when you're roughly planning to do stuff. So like communicating schedules, communicating like processes.
Speaker: um i'd want to lean more heavily on ai to come up with not just um you know summaries of things help you draft stuff out maybe come up with structure for things but also like taking the same message and rephrasing it a few ways because you've probably got to tell people something seven times before it sticks and if you can make it a little bit different each time
Speaker: absolutely i love that actually i gave a ah little talk to the product or get Slack about how to how to run big projects. And one of my pieces of advice was like, you got to have a couple of different stories because everyone cares about different things.
Speaker: um But maybe more fundamentally than that, like people get tired of you saying literally the same thing, but you have to keep saying the same thing, try and get them to listen. That's easy for AI. I had not thought about um Cool.
Speaker: um And last one, where where can folks find you online if they want to reach out? ah So I post most frequently on LinkedIn and Blue Sky.
Speaker: um So, you know, LinkedIn, I'm just Tom Elliott. Two L's and two T's. A lot of people make the mistake of leaving out one of those. um I'm also at telliot.me on Blue Sky.
Speaker: Very cool. Oh, and tell me a little bit about what you're working on now because you're not at Yext anymore. That's true. Yeah, I left Yext almost exactly a year ago. So I'm currently working on a tool called Okiroot, which I like to describe as a layer on top of CICD ah to help you manage large numbers of environments, to chain together all your different resources um so that you've you can have like much more complex behaviors but automate them much more successfully.
Speaker: um So in the case of doing things like the Nomad migration, in theory, we could have put together a ah plan that basically described all the relationships between um each of the different servers, which ones were Nomad hosts, which ones were servers, ah which ones clients, and the packages that you installed, and then create a map of all of those dependencies so that first of all, you can visualize the structure of your um environment, and then you can completely automate its creation.
Speaker: Very cool. um Definitely something near and dear to my heart and I'm sure a bunch of listeners as well. So um Tom, thank you so much for coming on the show. This has been fantastic.
Speaker: um And hopefully you can have a couple of good conversations with with folks about Nomad Kubernetes and what they should do with their CICD. Sounds great. I had a great time. Thank you very much.
