Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Surviving High Stakes Code Migration Without Breaking Everything
29 Plays3 months ago
Code migration is one of the toughest challenges engineers face at scale. In this episode, we dive deep into a real-world code migration that powered millions of queries daily.
You’ll hear how Sourcegraph rebuilt their core parser, the testing strategies that prevented disasters, and why “unglamorous engineering” is often the work that matters most. From differential testing to temporary safety nets, this story shows how code migration can be done without breaking everything.
Whether you’re planning a refactor, facing a large-scale rewrite, or just curious about engineering under pressure, the lessons here will help you approach your next code migration with confidence.
Get Tern Stories in your inbox: https://tern.sh/youtube
Recommended
Transcript
The Unsung Heroics of Backend Engineering
00:00:00
Speaker
It is almost as if you had Google search, but instead of looking for websites, you're looking for code patterns. you have millions of queries hitting this backend and you basically have to go and say, okay, we don't want to break anything because this is literally the most sensitive part of our search product. Unglamorous engineering.
00:00:16
Speaker
This is work that is not going to result in a UI change. It's not going to be super flashy, but it's nonetheless really impactful and it touches a lot of people. It can move faster. You can execute faster, but you have to keep up with the speed that
Introducing Renard Van Tonder and His Work
00:00:26
Speaker
you're executing now. So your mental model has to keep up with the code All right, welcome to Turn Stories. i am your host, T.R. Jordan. And today I have Renard Van Tonder with me.
00:00:34
Speaker
um We're going to talk about his time at Sourcegraph, where he rewrote the ah parser underneath their query engine. So... so Welcome to the show, Renard.
00:00:46
Speaker
Hi. Well, great to be here. i'm I want to start with just like... i look I was looking up a little bit of like where Sourcegraph was at the time. You did this work towards 2020, 2021, peak COVID
The Scale of Sourcegraph's Query Engine
00:00:55
Speaker
times. And the time... um peak covid times and ah at the time Sourcegraph had tens of thousands of users.
00:01:04
Speaker
You had folks at Amazon, at Uber, at Yelp using Sourcegraph. And the core query engine, which is what we're going to be talking about today, was doing millions of queries across their code bases, which were tens of millions of lines or bigger. that These are of It's a lot of usage across real code bases.
Challenges in Enhancing a Query Engine
00:01:25
Speaker
um Give me a sense of, tell me a little bit about what you you were working on there. um And in particular, I'm curious, the you mentioned the features you were trying to add around like Boolean operators.
00:01:38
Speaker
um What was it like to work on the the sort of core query engine at a company at that scale? Yeah, I mean, it's it's like you said, right?
00:01:49
Speaker
You're coming into this company, you're aware that you have operational customers and they're using this thing daily. So every single search query is hitting your backend. It is almost as if you had Google search.
00:02:01
Speaker
um But instead of looking for websites, you're looking for code patterns and and and things you want to fix, things you want to find, understand the code base. So everything is happening all the time. You have millions of queries hitting this backend. and Now comes this challenge of like, you need to change something here in order to extend the features, right?
00:02:23
Speaker
And that's an intimidating thought because you basically have to go and say, okay, um we don't want to break anything because this is literally the most sensitive part of our search product.
00:02:35
Speaker
And at the same time, we need to grow. We need to expand our feature set. So the feature that that you're talking about, um basically, we had a plain input string that you would search. ah But you couldn't say, hey, search for this string or that string without using regular expressions. So you could use regular expressions to do some of that.
00:02:54
Speaker
ah But one thing regular expressions are not good at, for example, is saying, hey, um I want to find a
The Critical Role of Testing in Engineering
00:03:00
Speaker
file that contains both this string search term and another search term. So maybe a function and some return type.
00:03:08
Speaker
And you can wang you can wrangle regex a little bit to get that right. But overall, it's just you're adding user friction to do that. And although we had backend search components that could support that, we couldn't do it cohesively across the entire backend by just having user type that in.
00:03:26
Speaker
um So that's the challenge, right? Somebody comes to you say, hey, here's a super sensitive part of our product. um It needs to be better. It needs to be performance. And we we don't want anything to break.
00:03:39
Speaker
um Please do this for us, right? um I mean, the person saying, like, please do this for us is also me because I'm using this product every day and I want this to happen. um But...
00:03:50
Speaker
How do you go and ensure that is the challenge, right? What are all the the roadblock cones that you put up everywhere and say, OK, careful here. There'll be dragons there, this kind of thing, until you kind of remove the feature flags. So that's one one component is develop it iteratively, have feature flags in place to revert things, and then see how how how you roll it out and how things go.
00:04:18
Speaker
Yeah, i I love reading through the post you you shared about this. I'm really interested to hear a little bit more about the the sort of production side of things. There's one part early on that you glossed over that I did want to hear more about, which this is a full rewrite of...
00:04:36
Speaker
the query parser, which, like you said, the the search box is sort of like a unique place within then ah search product that is the product in a lot of ways. And ah it's not a trivial decision to implement that.
00:04:49
Speaker
um Why did this feature require a rewrite? When did you realize that? Yeah, that's a great question. So there are a couple of things. that going to a search query box, right? And I think it's important to distinguish expressivity in this case, which is for someone to be able to express something that they're searching for versus a UI or UX concern.
00:05:14
Speaker
So maybe you want to highlight some search terms, make it obvious that some syntax is in a regular expression syntax and this sort of thing, which I ended up implementing later and actually using um this implementation for.
00:05:25
Speaker
So the impetus really was that the existing a query parser was quite simple in its assumptions, and it was prone to basically falling over if you tried to extend it and in a in the fashion that that we were looking for, because basically what you have or what we had at the start is you had a list, a list of terms. And those that list of terms is going to be run through the backend and searched.
00:05:57
Speaker
But fundamentally, what you need when you have expressions is a tree. right So our query went from being a list to being a tree. And you can certainly go and write a tree transformer on top of this list-based representation by going and saying, OK, well, we'll just keep the sublist components of the query that we had before and then organize this into a tree, which is actually what happened initially.
00:06:24
Speaker
um But the trouble came in actually transforming this list into a tree. You need operator precedence, right? so Ors and ands. And these things are the things that imposed a rewrite, so to speak. So we actually still kept using um some of the same helper functions to identify, for example, quoted strings and this sort of thing, even though we had like some bugs and a lot of confusion around that initially. yeah um So there was some lexing issues and there were some basically tree transformer issues. So
00:06:58
Speaker
Taking these things as a whole, it was kind of like, OK, let's rewrite a bulk of this. um But we have a reference implementation for the list part of our query that should work. right So that was always the the starting point was to say, hey, this is the situation. But we have a reference point. We have existing tests.
00:07:19
Speaker
And so go from there. Yeah. Okay. That makes a ton of sense, right? if the If the search box is like the whole product, it's not, but you know, it's kind of the whole product.
00:07:30
Speaker
Then the string you type into the search box is the whole user input. And that foundational decision of we should parse it as a list is just incompatible with operator with adding, you know, Boolean search and respecting reasonable operator precedents.
00:07:45
Speaker
Yeah, if you parse it only as a list. I mean, at some point, everything and a parser compiler lexes and turns your tokens into a list, but then you have to go from that to a tree, right? And if you don't have a flexible enough um framework to do that, then it just becomes difficult to reason about.
00:08:04
Speaker
Yeah. No, that totally makes sense. um Well, a lot of a lot of this work... you know doing the Doing the work itself, you know you've got a fairly straightforward product spec that you need to and you need to implement against.
00:08:17
Speaker
You spend a lot of time talking about your testing approach. i think and I'd love for you to talk a little bit about the the theme. I hadn't heard this phrase before, and I really like
The Underappreciation of Non-UI Engineering Work
00:08:26
Speaker
it, of unglamorous engineering. um that this is This is work that is not going to result in a UI change. It's not going to be super flashy, but it's nonetheless the less really impactful and it touches a lot of people.
00:08:37
Speaker
um And it also, it touches a lot of the way that the work is done around that particular software component. Tell me a little bit about how you think about like unglamorous engineering and how that led into like how you approached this particular project.
00:08:53
Speaker
Yeah. Yeah. I think at the core of a lot of software engineering and on it honestly other kinds of engineering is this kind of work done behind the scenes, right? So um maybe you just care about driving your car across a bridge and you don't really...
00:09:11
Speaker
care about the underlying structures of what's going on, and maybe that could be different. But the point is, like a lot of people work um you know the structural integrity of a bridge or the back end of some great varsity thing.
00:09:23
Speaker
And that is a lot of work. And I mean, if you look at product updates today, anything like and Google Sheets or any product that is kind of consumer facing, a lot A lot of engineers are sitting behind those changes, and you usually don't see much in the way of that. I mean, you see a tiny little feature, right, or something in the changelog. And for us, it was like, hey, you can now search you know and or and not operators in the search. And a lot of work goes into that in the back end. And I think that is...
00:09:55
Speaker
That is something that resonates with a lot of engineers. Now, the thing is, like to what sense is that gratifying and satisfying, ah this kind of flavor of work, right? Because ah it's always nice to have a UI or UX improvement, and it's so visual, it's obvious, um and sometimes also very open to krista criticism, right?
00:10:12
Speaker
Yeah. So my my take on this is that I think it's it's core to to the discipline. And there there are aspects that come with that, which is basically testing as as a core um interest, because it is the thing that that can convey to you that things are correct when a lot of this is really invisible people.
00:10:37
Speaker
to the majority of users and even other teams, right, on on your product. um So that's my high-level take on what this is. And I think it's, in some cases, very understated.
00:10:54
Speaker
And in some sense, because it's understated, it's difficult for engineers to really go and say, hey, I did this amazing thing, right? What do I get? Do I get a little trophy to on my desk?
00:11:07
Speaker
and And a lot of the times it was like, no, no, there's... This this this is... You said we needed this. And so... um It was kind of in this sense, right, ah this kind of work.
00:11:21
Speaker
ah But it is gratifying in a very, um I would say, you know, towards yourself kind of way because you know that you have the responsibility and the ability to ah make a big change and ah execute well, right? So today I think people are all about...
00:11:43
Speaker
Talking about you know execution is the only moat and ah you know executing and being a first mover in any space is so important now. And ah this is this is really a question of execution as the North Star, right? ah Nothing else really matters in terms of
00:12:04
Speaker
I don't know, ah like, oh we love this UI, you know, or we, we how do how do we turn this into an ad, right? Like, nobody's going to look at it from that perspective.
00:12:18
Speaker
ah True.
Personal Satisfaction in Engineering
00:12:19
Speaker
Do you think, do you think that, you just mentioned that unglamorous execution is personally gratifying. Do you think that's something that only applies to certain engineers or like only certain engineers find this kind of work gratifying?
00:12:32
Speaker
Or is this something that is just part of the job and everyone does it in at least part of the time? I think it's part and parcel of being an engineer because it's a so kind of unavoidable in some senses. But I think there is a lot there's different intensities and and blocks of work, right? So you might you might be faced with like, oh, I have a refactoring to do. And that's also a kind of unglomerate engineering. But it's so well recognized, I think, by doing like, oh, we need a refractor.
00:13:03
Speaker
One day we'll restructure this, right? it's always that there's always this... thing in the back of people's minds where they're like, okay, yeah, this could be better, right? And I think that ties into the the what is unglamorous about doing that kind of work. And that ah when you decide to act on it, that's when you really engage in it, right? So to the extent that you are comfortable and able to engage with refactoring and work that's changes something that's not so visible, you're you're doing unglamorous work.
00:13:36
Speaker
um And I think for some people, that's 90% of their work. And for some, it might be, yeah, it's much more spotty, 10% or something. That makes sense. that It's definitely all over the place. So maybe a provocative
AI's Role in Engineering and Automation
00:13:50
Speaker
question. Isn't AI going to do all our refactoring for us?
00:13:53
Speaker
I love this question. ah Because...
00:13:59
Speaker
To some extent, I think, I think i can't remember who said, I think it was the founder of OpenRouter. said every company will be a model routing company in the future.
00:14:12
Speaker
And ah part of that resonates with me, which is basically to say ah anything that people query or want to do, some tasks they want to take is going to be routed to some service somewhere that eventually ends up with an LLM input output in that pipeline, probably multiple, right?
00:14:33
Speaker
So from that context, I think you know refactoring and restructuring code fits into that architecture and model. And the the direction we're going certainly seems to to be something like that. But the the I think the underlying question is to what extent ah do you have LM's reasoning about it versus making tool calls and dedicated tools for actually transforming your code in a way that convinces you it's correct and better, right?
00:15:03
Speaker
So we'll solve everything. It depends how you chop this. I think it's going to be a question of ah what tools are you plugging into that pipeline and how are you letting the LLM drive those changes?
00:15:14
Speaker
But to say that you all have tooling to do refactoring completely without the need of LMs, I think is a far cry away from the reduction rejection from the direction we're heading into. I think it's going to be part of it.
00:15:31
Speaker
what's What's the line there? Like, if we've I've seen you know reports of folks firing off LLMs to just like go do refactoring with a very little oversight, very little um and you know very little direction, sometimes very little success.
00:15:49
Speaker
But what is the what's the the handshake there? Do you think that that's an evolving thing? Or is that there is there any bright line around you know what is the engineering work of doing? What is the engineering work that happens and what is the LLM doing? And how much overlap is there between those two buckets?
00:16:06
Speaker
Yeah, it's interesting because I think people are asking that question and saying, how can we be a little bit so scientific about it? How can we go and benchmark something like this, right?
00:16:19
Speaker
And one one example that you could run with is something like, hey i I'm experienced in my field or in this code base, right? And I know it's in this state. And let me...
00:16:33
Speaker
do a thought exercise or a coding exercise where I go from what I have to something that's more desirable, right? And now you basically have something to benchmark against that you personally can can evaluate, right?
00:16:48
Speaker
And I think a lot of things here is like, how do you evaluate this? And the best way probably right now is very vibey, as people say. But that might just be the starting point, which is to say like, Okay, so I'm going to evaluate this.
00:17:01
Speaker
ah ah Or you can use you know yeah your LLMs to evaluate this. So you can say, well, which LLM, if I tell it, hey go ah which model? like Go and refactor this. um what What comes closer to your expectations or what is considered better...
00:17:19
Speaker
by some model's judgment, right? Evaluating kind of the change in the code. we I've seen, you know, the ah you our on X, people saying things like, oh, you know, I um ah have this model, check that model's work, and this model went further than another one, and then the model actually says...
00:17:41
Speaker
it was very impressed by the other model's line of thinking as if it couldn't come ah up with that by itself, right? So you have all these kind of interesting interactions, but bottom line is you got to you got to do the the exercise to see and convince yourself, hey, this is better. If you go in blind, I think, you know, what what are you doing?
00:18:03
Speaker
You're blind and you only have yourself to rely on in that and that sense. If you're not experienced, then that's going to count against you. That really resonates that you said you said in the middle there of like, does this thing meet your expectations? I think that feels really key, right? Is that at the end of the day, you can use all the tools you want, but like you, the engineer on the hook for the thing, you did the refactoring, no matter how much you've routed to an LLM and allowed others others to do it that...
00:18:32
Speaker
Does it match your expectations? Is it doing the thing you want? Like refactors are in some ways suggest ah right subjective in what is their their output.
00:18:43
Speaker
I think that but that makes a ton of sense. Yeah. And if I comment on that, I think what's interesting about that is knowing what you want, right? Yeah. a lot of... ah if You can go into this and say, hey, restructure code, please make pretty.
00:18:55
Speaker
But then... It's going to do something. he wants It's going to do something, but what want? Right? ah Which is much different from like, hey, i this function or this functionality is being reused everywhere. Can we please put this in a helper function?
00:19:08
Speaker
No, please don't put the helper function in a file called utils.ts or whatever. because I know you have lots of training data that says that's the right thing to do. Right.
00:19:19
Speaker
but Right, right. And so, I mean, that's a special that's a special thing, utils files. I don't know how you feel about that, but yeah okay. there Not great. so So what you want is important. And then, yeah, and then being able to evaluate, hey, does this meet the expectations that I want? And of course you can express these things and prompts and stuff, but, um ah you know, at some point you need to look at the code.
00:19:43
Speaker
You need to look at the code.
Advanced Testing Techniques in Development
00:19:46
Speaker
Yeah. Coming back to the source graph thing for for a second, um the i think one of the things that was really interesting is that you had this sort of multi, this layered testing approach.
00:19:56
Speaker
And I'm going to say, I've got a bunch of questions about the production part of it, but sort of imagining um the unit testing to integration testing to fuzzing they that you did. and Looking back at that, I thought one of the things that was really interesting is that your fuzzing, really it,
00:20:14
Speaker
great r Highlighted a couple of bugs, to which your reaction was like, no biggie, good, fix those bugs. But your reaction to getting CI stood up so you had like both internal representations running at once in code, both of them running against a set of like known good test cases, that you said that was super transformative.
00:20:36
Speaker
Tell me a little bit more about, as you were going through this, it sounded like you had actually kind of made a change to your plan midstream about, like that impacted how you were approaching this project.
00:20:48
Speaker
<unk>d I'd love for you to talk a little bit about what impact that had of getting ah fully separate copy of the Sourcegraph parser stood up and running in CI on the regular. Yeah, so I think the traditional way or the required way of a lot of functionality today is unit testing. So, you know, testing things in isolation and then your integration tests. So you're running things more end to end and making sure nothing breaks along the way.
00:21:14
Speaker
um What is cool about having a reference implementation, so you know what you want or you know what you don't want to lose, right? um Is that you can test that against, you know, v2. So,
00:21:28
Speaker
so
00:21:31
Speaker
the The thing here was I wanted more assurance that like things were not going to break given him this this new functionality. So it was really a confidence and expectation kind of thing. um And so fuzz testing is a fantastic way to basically go and say, hey, look, um just try a bunch of inputs. Here's the structure of ah the input or here's the in input box. And nowadays you can go you know can go deep down the rabbit hole of all kinds of different fuzzing techniques. You can teach the with your grammar and have it ah do kind of much more structured or produce structured inputs for something like a query parser. um
00:22:11
Speaker
Very popular fuzzing targets, our compilers, and and this class of tooling. And um honestly, that can be run independently of a of a reference implementation. so After all that, I was like, okay, we can do some differential testing, which basically amounts to saying, okay, we take the exact same input, feed it to both systems, and see if there's any difference in results, if there's any difference in ah how the compiler parses things or outputs this tree, and make sure that's consistent. Obviously, there's going to be a difference...
00:22:46
Speaker
Like this is modular, the new functionality that you added. So the whole parser is not going to produce anything worth a tree structure, but that's something you can recognize. um So once you do that, I think that's, you know, you're testing against ah your gold standard in terms of being regression free.
00:23:05
Speaker
um And so I think that is something valuable today. in a general sense, right? If you have some reference that you're working against, it is challenging to put up the infrastructure to actually do that. I mean, I was doing this mostly locally.
00:23:21
Speaker
So you have a lot of, you have a lot of work that has to go into making that, making that happen, but it does give you the confidence. So it's, it's like this period of principle thing, right? If you want to if you want to put in ah 80% of your results come from 20% of the effort.
00:23:40
Speaker
I think general testing is your thing, but then you have a long tail of ah potential issues, and that's when you kind of get a little bit more sophisticated and a little bit more um ah bit more effort to to get that last last bit out. so Mm-hmm.
00:23:56
Speaker
Yeah, that makes sense. And and especially if you're if you can build a system like that that sort of like lives for a long time like over the course of the project, that that live feedback is so much more powerful than just like, I have i have something.
00:24:10
Speaker
One idea, this is this just occurred to me. I have heard this idea from a bunch of people of the gold standard during any migration is that if you have a migration-specific test suite, historically, this...
00:24:24
Speaker
you only do this for like the most horrible migrations, like the stuff that you know is going to take five years up front because creating test suites is like not cheap. um You know, you writing unit tests, I imagine those like lived with the code since, you know, once once the code is shipped, that's like, that's not migration specific. It's just you're testing your new code. You're writing integration tests.
00:24:46
Speaker
Did you do any migration specific tests that... were sort of only for this period where you had both implementations living in the code base? Oh, yeah. So this this is exactly the last phase, right? Which is this deferential testing phase. So you're you're creating inputs, creating fixed inputs, right, to to run through. i mean, you could do these dynamically, but the point is that like this was a short-lived phase in the sense of um Okay, it works. We don't need this infrastructure anymore because it's, I think, like immigration. it is
00:25:21
Speaker
we We make a big leap. We make a big jump. we're switching over a system, a component, right? Kind of wholesale. And so when you do that, but then you have this kind of short-lived testing phase that I think fits into this picture of what you're saying.
00:25:38
Speaker
And to the extent that that's useful afterwards is is limited because you don't know which, it like there there's there was no, you know oh, we need to support something more in the query after this point that would justify this amount of like,
00:25:52
Speaker
testing and so on, you can have unit tests or like smaller things to to detect your regressions at that level. um Now, question is, I think, like a five-year migration or, you know, six-month migration is a lot longer timeline than this work happened across. I think, you know, this was maybe one release that we did the differential testing was like a month worth of just stability and everything else before that. So timeline will also, I think, dictate the extent to which you can like...
00:26:28
Speaker
Yeah. Yeah. if you If you only have a month where you're going to really do testing, like that that constrains your budget. yeah you You only want to spend so much time writing tests that you're going to throw away afterwards. um but it's not But it sounds like those are really valuable, which is, I think, a really interesting push because most of the time when I talk to folks about migrations, they don't write these tests regularly.
00:26:51
Speaker
even if they have they have substantial time to work on the migration that you dragged out over several years. um it's Did you have a ah sense of why? I'm curious what you saw that made you think ahead of time, like, no, we need to do this, even though it's going to be throwaway work, even though it's not going to prove anything once the feature is shipped.
00:27:14
Speaker
Like what what What pushed you to say, like, just testing the feature itself is not enough. We have to test against the old stuff. I didn't want to piss off tens of thousands of engineers. It's that simple. They get so mad.
00:27:28
Speaker
how They do. But honestly, that's it, right? It's like, okay, you know, we put a lot of effort into XYZ. You know you eager could be launching a new product at your company and all this stuff, and you're going to allocate, like,
00:27:46
Speaker
couple weeks or months to do that um why wouldn't you do that for like the most critical part of your your system like ensuring like it doesn't break um so that that's the bottom line that's like if you're if you care about your work if you care about what you're putting out there and you really don't want it to break then like you you you kind of see what what's available, what what the tools are that are available to you. And I think to to parallel with migrations, you know there's a payoff, right?
00:28:19
Speaker
And so you'd rather get the work done and done well. And I mean, it's not just, it's not just all one thing at a time. We had a lot of feature flags along the way to kind of gradually introduce this, right? So you can have people um try it. And I think a lot of migrations in that sense can be done you know partially, for example, like Python 2 to Python 3 migrations can sort of be, depending on how your backend is structured, like these things are are possible. So it's not like everything at once, but at some point you're like, okay, this is the last part, right?
00:28:55
Speaker
and Yeah. um I learned there's a bunch of people um who... are thinking have thought about the Python 2 to 3 migration. Their current focus is I need to get rid of the library 6, which is the Python 2 to 3 migration compatibility library, which is now being deprecated as part of 3.10 or something.
00:29:16
Speaker
ah like Once you get that shim in there and you're like, oh, we can live in both states, sometimes that just incentivizes you to live in both states for a really long time. That's true. And i yeah that's that's that long tail um because yeah you have maybe 80% of your code is working, but you have to put in the work for for but the long tail bits.
00:29:37
Speaker
yeah Absolutely. um So as you went through, do want to talk about this differential thing. that You pulled a bunch of use cases from production and built up this, this you know,
00:29:48
Speaker
temporary, highly valuable throwaway test suite. At what point did you know, yeah we got enough? Was that, did you go all the way down to zero errors or did you have some floor where you're like, look, let's just, it's time to ship this thing?
00:30:04
Speaker
Yeah, I think it was timeline-based, right? In the sense that you can do a fuzzing campaign on projects or differential testing and you run it for how many any any days you want or weeks.
00:30:18
Speaker
But at some point you're like, okay, it's good enough. Maybe if we ran this longer or maybe if we refined it a little bit more, we'd fight more issues, but intuition is like yeah you did well.
00:30:30
Speaker
um And one One comment on that is that typically with testing, at least with fuzzing or any kind of dynamic testing or differential testing or these kinds of things, the bugs tend to be shallow for that testing method, right?
00:30:47
Speaker
So it's kind of like the more varied and the more different kind of ways that you test your implementation in, um the more likely it is that you'll find bugs quickly with that method. And then not so much afterwards, which was the case for this work.
00:31:02
Speaker
um As I remember, it's like, okay, we'll quickly find a ah couple issues or ah flags in the implementation when we did the differential testing thing. And then after that, it's like, seems to be smooth sailing.
00:31:18
Speaker
Are there more things could tested potentially? But i think like having a reference implementation and testing and against that really is like one of the most powerful things you can do. That makes sense.
00:31:28
Speaker
yeah did you Did you have a sense beforehand you know, we're going to get to, you mentioned it was deadline based. Did you have a sense of like, we're going to reduce it to like, no more than one in a million errors or no more than this type of error? Or was it really just like, we're going to do this for a month. And if it doesn't feel terrible at the end of that month, we'll just go.
00:31:50
Speaker
Yeah, it's more more in the second bucket. There's no way to know kind of the the the whole surface area of like how how many things can go wrong, how many bugs.
00:32:01
Speaker
I mean, there were no existing bugs at that point that were due to this this change, right? Plenty of like bugs and issues where people saw different results based them, but these were like not mount central to the actual like syntax and AST changes.
00:32:18
Speaker
So it's kind of like give give yourself enough room to convince yourself this is working. um basically give enough room for it rolling out to crop up with issues, maybe in production users or so on after you've tested, like, okay, as are things running smoothly, right? Because that's the benefit of having something that is actually used a lot. It's like, it'll show up pretty quickly if people are ah running into something. Yeah, there are obscure queries that might ah not, but but you know you you want to not break anything for the majority of people.
00:32:55
Speaker
Cool. um So, okay. you found all You found all the bugs. I'm sorry. Just to go back. i think that makes sense I think that makes sense. Like, that feels right to me in some ways. That I...
00:33:11
Speaker
I feel like I have a lot of conversations, especially around turn where I'm pushing the idea of like plan your migrations harder. You can use AI to plan it I do believe that,
Learning and Adapting in Project Development
00:33:20
Speaker
but it sets up this, this dynamic of, well, you can plan your way out of everything. Like that's just not true.
00:33:28
Speaker
You can, you should run these experiments in production. You should run experiments around your code. And like you said, you run it for a month, you gain some confidence, you stare at it and you say like, look, we're, we're good here.
00:33:41
Speaker
And and that feels that feels like a more honest description of how any major and engineering effort is done. It's not that you knew every detail ahead of time. If you did, you'd be done already.
00:33:54
Speaker
you but But you do learn as you go. And it's just a question of, can you get the right information? Can you change techniques? Can you pick the right technique next? Yeah. um Yeah, I think at a higher level, like it's about correctness.
00:34:07
Speaker
So people are like, okay, does my system behave correctly? And however you want to poke and prod at it to see if it is correct is is up to you. But I think that's always going to be a ah ah time-based exercise.
00:34:22
Speaker
it's not It's not so much... ah have ah you know i have You have a checklist for executing ah migration, and that the result of that is like it behaves correctly.
00:34:36
Speaker
But there's no kind of like um discrete, I think, way to go about saying, okay, if... ah Because basically, if you don't have a time-based thing, it's basically an an unbounded timeline, right?
00:34:51
Speaker
And so I think that's problematic. Yeah, nobody wants that. No one wants to work on the same project forever. and no one wants to wait around for a feature that's never going to ship. Yeah.
00:35:03
Speaker
So maybe realistically, it's like, okay, if we have a migration or or an effort like this, like I knew this was like a multi-month effort, right? Overall, ah you could you could get out in two weeks or four weeks, you could get out your new query language and that's fine. People can run with it and do things, but maybe that will break things or maybe it's slow. So performance was another piece of behavior that ah we needed to revisit with.
00:35:27
Speaker
Yeah. higher expressivity. And so that might kind of migration scenario as well, but you have to know whether it was just like, is this a multi-month thing or could it be a multi-year thing? And then if it's a multi-year thing, okay, well where where does our checklist um address ah you know how how we bucket up a multi-year migration?
00:35:48
Speaker
It's my intuition. I've never engaged in a multi-year migration exercise. One day, I'm sure. ah Do you think that timeline would have been different if you'd started the project in 2025, given the tools available today?
00:36:04
Speaker
Yeah, probably quite a bit different in the sense of
00:36:09
Speaker
just developing the supporting tooling around this, right? Because what I would, like, if I had, if i had ah you know, a LMS creating my code, some code agent doing things, I would i would tell it like, I wouldn't tell it, go and test whether these two implementations are the same, right? Like, what what what am I telling it to do fundamentally? But I would tell it like,
00:36:32
Speaker
can you write a script? Can you write the script for me that does this, right? It takes two inputs and then runs it against these functions. ah Basically, the bit that I did manually
AI's Impact on Productivity Dynamics
00:36:40
Speaker
that was kind of temporary. And then maybe I figured like, okay, can you come up with some interesting test cases, right?
00:36:49
Speaker
ah Some interesting test inputs that maybe, know, specifically test things like dangling quotes, dangling parentheses, things that would basically ah push the parser into corner cases.
00:37:01
Speaker
And that would be more of a manual thing. I mean, that's what a fuzzer would do and discover are kind of on its own, but as ah because of like and feedback direct to fuzzing. But essentially, I'd have another tool my toolbox here, which is to say like, okay, this thing kind of can can generate useful ah directions based on this code.
00:37:23
Speaker
And I know what I want, so I can tell it to like make my work easier and I'm not going and spending four hours writing a script is like pop it out in 15 minutes and then not have that tooling.
00:37:37
Speaker
So that's that's how I think it would change. Now, overall, how much time would it have saved? I think ah it might have, um like if I was running this up today, probably have a quicker sense of confidence of the implementation correctness compared to to back then.
00:37:52
Speaker
But that would also be like more intense, right? Because you are doing the same amount of work in a shorter period of time, and that's no less cognitive me cognitively like draining, right? but it is But it is more efficient in the sense that, okay, you're doing a lot of automated stuff much quicker. So I think that's important today. It's like, yeah, you can move faster, you can execute faster, but you have to keep up with the speed that you're executing now. So your mental model has to keep up with the code,
00:38:18
Speaker
And it is more cognitively draining if you are able to test things quicker, right? Because now you're your mode switching, just everything is happening more rapidly. So that's my that's my sense of what is different today.
00:38:34
Speaker
That's a really interesting observation. and i've I've felt that as well, is that... if you're really picking up a productivity increase, and that's not always true when you're using AI.
00:38:44
Speaker
um But on the days that it does feel like it's really working, it's exhausting. Do you think that that is... Do you think the amount of cognitive effort that...
00:38:58
Speaker
Let me ask that another way. Do you think AI is actually saving you any cognitive effort there? Or is it just compressing the work that you're otherwise doing? Compressing is an interesting word, I think.
00:39:09
Speaker
ah Maybe compression is the right word or condensing. um it's I'm sure some researchers are going to study this or have studied this and they're looking into like, what is the cognitive effect of you know all these things? That'll be interesting. but But my intuition is that
00:39:29
Speaker
but There is there is that kind of there's more intensity, i would say, cognitive intensity. But i don't I don't know what you know what that what that really ah means. But I think the result of whatever that is, is earlier fatigue or mental fatigue.
00:39:46
Speaker
um but you get the efficiency out of it. And the other part is that you're thinking about things differently, right? Like instead of going and saying like, okay, I'm going to write this function now and I need to take care of this case, this condition, i forgot to add this condition, all this stuff.
00:40:07
Speaker
Now you're going and... ah formulating with prompts how you want something. And that's a different kind of thinking. And I think that kind of thinking is potential potentially more load-bearing than just... Okay, I'm going to...
00:40:30
Speaker
be a little bit on autopilot, you know, okay, put in a nice little left brace here, that's where that goes, and and then, and ah okay, we'll have a return at the end of this function. so So going slower, you know, in some sense, doesn't doesn't cause that high-intensity activity.
00:40:52
Speaker
process Right. So it's interesting. The machine took away all the palate cleansing work. Yeah, there you go. The hard stuff. There you And also like, it's also interesting in the sense that like a lot of these things, you you know, the more you,
00:41:06
Speaker
I think to use your word, compress ah what you want in the beginning and write a prompt to some extent. like The longer it will take and you can go and make your coffee or be distracted for five minutes and then come back, right? But then you're back in the like, okay, um focus and spend two minutes really intensely thinking ahead and then like enter and then leave. So it just...
00:41:28
Speaker
There's also that, right? The contrast of intensity and then like be distracted as opposed to, again, when you're coding, it's kind of like, okay, well, can't be distracted now because the characters literally will stop right there in my editor if I don't continue um writing the code.
00:41:47
Speaker
After four cups of coffee in two hours, that the the prompting gets a little worse, I suspect. Yeah, that's true. i think so. um right on.
00:41:59
Speaker
Well, um I think,
00:42:06
Speaker
sorry, thinking about, um I love this train of thought. I'm just like spiraling. I'm trying to consider like how it how it plays out in my own life and how it it plays out. I think that there's ton of there's There's a ton of things that we're going to see here. and i've you know I saw one study the other day around how um a lot of people using AI actually can turn off a lot of various parts of their brain. like it's It's almost the opposite of what you and I are talking about where like you quit making those decisions. and It feels like both of those things can be tripped. You can use the tool in a way that
00:42:43
Speaker
gets you out of doing the hard work and you simply defer, or you can use the tools in a way that create the work and focus the work in a way that you're really, you are really working on the hard stuff. And I i suspect both of those things will happen to depending on the person, depending on the task at hand.
00:43:01
Speaker
Yeah. And another thing that's, that I find in my own work is that like, And again, to contrast, I think a big engineering effort like this or migration in the past, right, for any big company is like, okay, we're we're investing in this, we're going to do this, we're spend time on this. And then, you know, you're you're a couple weeks in or a couple months in.
00:43:21
Speaker
And then maybe along the way you discover, oh, hey, this could have been different or this would have been better. But you know that like, going back and changing that is just not an option, right? Like that door is closed.
00:43:32
Speaker
yeah you You've gone past that point. It's like not an option. But today I think it's more a question. Like I experienced this myself, right? It's like, oh, okay, I've implemented something in this way or I have this code generated by the model.
00:43:44
Speaker
ah but maybe this other thing works. Maybe I could tell it to do it this way. And then you end up with like a billion like branches, all kinds of different things. And at some point it messes up, but then you're kind of like, well, that was so easy to kind of explore. Why don't I explore some more?
00:44:01
Speaker
ah And the extent to which that's productive or unproductive, I think is is very up in the air. Like this is the thing that everybody's... kind of grappling with, right? Like, did it really save me time? Or because I knew her if i ended up just coding this, like maybe I wouldn't.
00:44:18
Speaker
Like, just by virtue of like not having the option to explore so many things, you will arrive at some status fire like some satisfying result.
00:44:30
Speaker
But because you this option to play with that, that is either a distraction or it is ah something that can maybe advance your your um your objective, right? Because you do find something better on the third or fourth try.
00:44:46
Speaker
So this is this is, for me, like at a personal level, very difficult math. Like, okay, at what point do i stop like tinkering and and just say, okay...
00:44:58
Speaker
um I don't want to go back. no No going back to that branch. No no forking that branch again. um that that is That is, I think, we one of the like biggest ways to to understand like how do I use LMs to be effective.
00:45:17
Speaker
Yeah, absolutely. There is there's a world of difference between using using LLMs to you know polish a feature into oblivion because you can explore everything and using it to take the work off your plate that you don't that you're that you weren't going to do anyway or that you had to do but you were trying to get through so you can focus back on work. It really does force you to think about what are you trying to get out of the tool, back to your point of like what did you expect from this thing.
00:45:45
Speaker
it's It's going to keep writing code as long as you keep typing things into that prompt box and you need to be really clear in your own head about what you're trying to get out of the tool. Yeah. And I think with migrations, it's especially true because you can you can go at it from different angles, right?
00:46:04
Speaker
um Just looking at some of the other stories and more stories that you you guys posted. And I think if you if you ask people during a migration today, it's like, which direction would you go in?
00:46:16
Speaker
maybe are you know upgrading some C++ plus plus library or something like this, is you the dedicated teams at Google for this, it's like a lot of decision-making goes into, okay, where are we going to start doing this in the code base?
00:46:28
Speaker
Who are we going to contact and all this stuff? And maybe there's this high-level kind of orchestration ah role that the LLMs will also take on, right? Not just code changes, but understanding the architecture of the system and all that.
00:46:43
Speaker
um And then again, you're faced with auctionology and that can make it more challenging.
Connect with Renard Van Tonder
00:46:50
Speaker
Yeah. There's, I think, a a ton of upside in letting people climb the abstraction ladder and focusing higher and higher on where are the points of leverage? What are the architectural decisions?
00:47:00
Speaker
um We are unfortunately running out time. um So one final question, um or I guess two final questions. um Where can folks find you on the internet if they want to talk more about this stuff? And is there anything in particular that you would like folks to reach out to you about?
00:47:16
Speaker
Cool. um So I am rvtond on Twitter. um I try to just read about tech things as as I do things, so ah which is kind of fun.
00:47:29
Speaker
I am now working on LMs. I had ah since some blockchain tech and I am now kind of like formal verification is a very cool thing that's been in the works for decades and historically just very...
00:47:45
Speaker
ah very much requiring a lot of human effort to to get awesome results. And so I think now a lot of the the mechanic parts of formal verification, so that you know the code that's running Airbus, the things that are not you know causing your critical systems to go down, this is this is like fascinating code and the and the history of the development of this those pieces of technology are amazing. And I think basically on the Caspware, we can generate a lot more of those kinds of very strong ah guarantees in software, but without without as much effort, right?
00:48:27
Speaker
With LMs being able to recognize patterns so well being able to like mechanize things a little bit faster. So that's my current line of thought.
00:48:37
Speaker
um Basically, imagine if I could have a query ah parsing migration, but instead of all the things I did, have formal specifications that say, this is what happens with this kind of input.
00:48:51
Speaker
So if it's ah only a list input or or a tree input, and this is what guarantees like the behavior of the system. um And you prove that at a mathematical level and say, hey,
00:49:02
Speaker
it's impossible for this thing to do anything else, which would be a fantastic thing that you're going to go and roll out. Right. So um that is my, my Lerp.
00:49:13
Speaker
That's awesome. i I love the idea of unlocking formal verification for more use cases. It's just such a it's such an expensive thing to do. And it it makes so little sense in so many cases, but if you make it cheaper, you can use it. Yeah. I mean, look at migrations, right? It's like,
00:49:30
Speaker
Again, if you're migrating like system critical code, like you want it. And then all the effort that you put in, yeah, all the effort that you put in for like specifying the behavior of the system pays off because now ah now you don't need concrete examples. You don't need to generate these like concrete strings that go into your system. You just like prove it at the level of like, this is the structure, this is the semantics and that's it.
00:49:55
Speaker
so Awesome. Cool. Laurenard, thank you so much for coming on the show. This is fantastic. Thanks for having me, TR.