Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
"Observabilitying" the Future of Software with Charity Majors
5 Plays3 days ago
Charity Majors is the co-founder and CTO of Honeycomb.io, where she pioneered the concept of modern observability for distributed systems. Before Honeycomb, she spent years at Parse (acquired by Facebook), Facebook, and Linden Lab building Second Life. She is the co-author of "Observability Engineering" and "Database Reliability Engineering" (O'Reilly). She writes at charity.wtf. Whatever she and Scott are planning, they are gonna observability the heck out of it.
Check out https://textcontrol.com for industry-leading document editing and PDF processing SDKs for .NET developers
Recommended
Transcript
The Humor in Fast Code Generation
00:00:00
Speaker
because everyone is suddenly generating code faster than they can understand it. So people are saying weird stuff like, oh, the code is free now and tokens are free and they're not free. And what's more important is that I'm making code faster than I can personally absorb it. And some people are saying, just let the code wash over you and I'm not there yet. You don't vibe into production. Most are not on call.
00:00:21
Speaker
No, they're not on call because ultimately, yeah, if you're creating a bunch of code and trying to sleep through it, and when production goes down, it's going to be a problem.
Community Support by TextControl
00:00:28
Speaker
Hey, friends, you probably knew that TextControl is powerful library for document editing and PDF generation, but did you also know that they're a strong supporter of the developer community? And it's part of their mission to build and support a strong community by being present, by listening to users, and by sharing knowledge at conferences across Europe and the United States. If you're heading to a conference soon,
00:00:50
Speaker
Maybe check if TextControl will be there. Stop by and say hi. You'll find their full conference calendar at textcontrol.com. That's T-E-X-T, control.com.
Charity Majors: Leadership Transition
00:01:04
Speaker
Hi, I'm Scott Hanselman, and it's another episode of Hansel Minutes, and it is the return of Charity Majors. She is the CEO and co-founder of Honeycomb and the world's most decorated engineer around observability and distributed systems. I know of no other person who, when someone says observability, they say Charity Majors. How are you?
00:01:22
Speaker
I'm great. I'm the CTO. o The CTO? Oh my goodness. I was the CEO for the first few years, but I was a terrible CEO. Oh no. My co-founder and I swapped places few ago. have to update the about page on charity.wtf. Oh my God, seriously? Yeah. i got the fresh I thought I was going to the fresh source.
00:01:41
Speaker
No, that's a very legitimate assumption. yeah yeah yeah, yeah. Go to your blog. I got to go update my about page as well and make sure it's not saying things about me that are not true. Right, up that's on my to-do list now. Oh, excellent, excellent. Get one of your agents to work on that. Yeah, exactly.
00:01:55
Speaker
So let me just dive
The Evolution of Observability
00:01:57
Speaker
right in, right? You've been talking about observability for a decade. What do you think people finally understand now? And what are people still like getting wrong?
00:02:07
Speaker
Oh, boy. You know, it's been really fun in the last two months. It feels like people have realized that you don't need metrics, logs, traces, all of the different signals. Actually, you just need traces. You just need structured data that is connective, connectable. You could call that traces with spans. You could call that, you know, wide structure logs, whatever. What you need is there's actually so much more power in having your data united, not fragmented. And metrics and logs, and you they will always be with us, but they're they're really, they're the exhaust pipe of your infrastructure, right? it's for It's for the telemetry that you don't control. You can't change it. You just have to operate it. Right. That's those your metrics and logs. But for the for for your software, the software you do control, your telemetry is a product problem. The trace is part of the product. Right. Being able to validate is part of the product. And it has been so fun the last couple of months as you just see people like on X and on LinkedIn going, we it wait, wait, wait, wait.
00:03:21
Speaker
Have you ever considered that maybe all we need is traces? And I'm just like, duh, duh, duh. This is amazing. This is one of my superpowers and one of the things that's worst about me is my ability to make really bad analogies. But I feel like observability and then putting in the context of agents is like got here debugging at scale.
00:03:44
Speaker
Before debuggers, before attaching debugger and stepping through a debugger, we would just put like console.writeline, got here. yeah. Control-out writing got here, number two. And then you just your whole life was just a series of bisects as you're like, it's between here and here. And I was recently doing some agent work on a Windows app, and it was very visual. And someone said, oh, you should use computer use. And I was like, no, you're going to take screenshots of pixels, and you're going to burn carbon and waste energy.
00:04:11
Speaker
This feels like a debug.writeline situation. So I just observability the hell out of the entire system and then made an MCP server that
Using Observability as a Verb
00:04:20
Speaker
pointed to it. And then the agent looped on it and solved it within minutes.
00:04:23
Speaker
It was very much a got here debugging at scale problem. I respect the use of observability as a verb. That was well done. Thank you. I think that's a thing. We should make that a thing.
00:04:34
Speaker
I do too. I observability the hell out of it. I will use that first. That is the new title of this podcast. I observability the hell out of it. Also, by the way, I'm a sucker for a title that is a pun.
00:04:46
Speaker
Hansel Minutes. Yeah, you like that? You know, that's actually because I'm a really bad estimator. And 20 plus years ago, someone was asking me how long something was going to take. And I was telling it was going to take about 30 minutes. And they went,
00:04:59
Speaker
Really? minute Those minutes are Hansel minutes. And they were right because it was about six hours. Yeah.
Simplifying Software Issue Resolution
00:05:05
Speaker
So that is what people are getting right, a think. Okay. At at long last.
00:05:10
Speaker
ah the The other part of that that i don't think they're they've really gotten... yet is just wrapping their heads around the part where this is a product problem and where there is actually, it's like one of the smallest investments you can make in your future with the biggest payoff is just like devoting just a little bit of attention to what you're putting in and what it's, you know, what it's,
00:05:33
Speaker
what it's married with contextually, you know, because once you've separated the bits of data that all describe the same thing, you can never knit them back together. But if they're knit together, you can tear them apart and put them in different dashboards and all these things to your heart. And, and, and, and, and graphs and charts and visualizations and yeah whatever makes you happy, but you've got to bring all this stuff together. And 30 years ago, when I was in banking, we were, we were FTP log files around and grepping our way to glory. Yeah.
00:06:02
Speaker
and it was a nightmare and observability and otel would have solved all of that for me and with otel and auto instrumentation now i think it is genuinely possible to build faster with telemetry than without which is huge and i think underappreciated you know i don't blame software engineers for I have this theory that like the entire 20 year journey of DevOps was really all about us trying to create one feedback loop that included both people writing the code and
AI and Feedback Loops in Observability
00:06:35
Speaker
the code in production. And it completely failed because it was just too hard.
00:06:40
Speaker
It was so like, imagine you're a software engineer. It's the year 2020 pre You are staring at your code and you're thinking about, okay, I need to instrument this. All right. I've got a bit of data.
00:06:51
Speaker
Does this go in a metric, a log, a trace, an exception, an error, a profiling, all of the above? If you decide it's metric, does it is it a counter? Is it a gauge? Is it just all these different things? And then what's the data type? Is there going to be high card? Like just the the bench of deep knowledge that you need. That's so true.
00:07:12
Speaker
And then if you get it in there, you get it deployed, you got the right data types, all these things. Okay, it's out there now What are you going to do? Okay, now you go and you you have to like try and find it and craft the right dashboard and do the right. to Oh my goodness.
00:07:26
Speaker
Two, three times. you You are the amount of labor that you unload and cognitive load. You're doubling, if not 5X. And ironically, you're going to be putting it somewhere and squirreling it away. Like on Windows, when I was working in Windows like 30 years ago, it was like perf counters.
00:07:41
Speaker
And we would throw all this data into perf counters and we would never look at it again. Yeah. Until there was a crisis. Yeah. Yeah. And if you're not used, if you're not fluent in it, looking at, it's just it's just been so hard. And one of the most exciting things about AI to me is the ability to close this loop and to and to be understanding as pretty much as you're writing it, you know? Mm-hmm.
00:08:05
Speaker
Yeah, yeah, yeah, yeah. Okay, so recently, this the second of ah the second edition of Observability Engineering. Yes. It's happening. ah Is it out now or is it coming out in a week or so? it can be downloaded as of Wednesday. the heart The Dead Tree ones won't be out for another month, but it it can be downloaded now. Very exciting. Okay, so second edition. But I would argue that it's arriving at a kind of a weird moment because everyone is suddenly generating code faster than they can understand it. So people are saying weird stuff like, oh, the code is free now and tokens are free and they're not free. And what's more important is that I'm making code faster than I can personally absorb it. yeah And some people are saying, just let the code wash over you and I'm not there yet. You don't vibe into production. Most on call.
00:08:48
Speaker
No, they're not on call because ultimately, yeah, if you're creating a bunch of code and trying to sleep through it, and when production goes down, it's going to be a problem. So what changed enough that the book had to change?
00:09:00
Speaker
Oh, that's such a great question. So first of all, it's an O'Reilly thing. If the book was relatively successful, then two, three years later, they want a second edition because technology changes.
00:09:11
Speaker
But so the first book, right The first edition it's about 250 pages and it took us three and a half years to write it because we were chasing a moving target.
Updating Observability Engineering
00:09:23
Speaker
like The definition of observability when we started writing changed two or three times over when we finished, which was deeply frustrating. and and At the end, you never want to say that you're not proud of a book, but I, there was no point where we were just like, we feel great about it. It was just like, oh my God, i can't look at it. Please just get it. I don't care what it says. i can't read it anymore. Just get it away from, that was the sort of,
00:09:51
Speaker
I'm grateful to the people who read it because I just cringe when I read it. So I was really excited to write it a second time. I was like, yeah, it feels like the definition of observability is stabilized and observability is now the sort of first principles thing. And the rest of the world has gone batshit crazy out of their minds. So we started with,
00:10:15
Speaker
It's over 600 pages. And I'm a little sheepish. Hang on. First edition, 250. Second edition, 600. Yeah. And I cut at least 200 pages. like This is a new book.
00:10:25
Speaker
It's it's isn't entirely rewritten. The whole thing has been rewritten. There's 27 new chapters. Only like four chapters even carried over any material. It's a whole new book. Oh, my goodness. Okay, wow. Which is was funny because I just said that people are generating content and code faster than they understand it. So here I am now. I'm going to have to absorb this book. No, know you don't. So let me give you a little bit, just sort of bird's eye view.
00:10:48
Speaker
There's six sections. The first section is just sort of like, so I wrote section one and section six. ah My co-authors wrote two and three, and then we have just a wide range of incredible guest authors who contributed like sort of deep dive and use case chapters in four and five.
00:11:02
Speaker
So the first part is sort of like, To me, one of the differences between monitoring and observability is monitoring was very pegged in you know, here here are the grooves we've worn in our systems. This is how we know if it's up or down. But to me, observability is reasoning from first principles about how to understand our code.
00:11:24
Speaker
And so the first chapter is kind of like acknowledging like what you said about this is a crazy moment. The possible futures seem way wider open than usual. And it's a little scary. It's a little terrifying.
00:11:38
Speaker
But like at a moment like this, I think first principles matter more than ever. And so what we hope we can do in this book is equip you with the tools to navigate this scary time.
First Principles in Rapid Tech Change
00:11:49
Speaker
When you said first principles matter more than ever, I was presenting at a very, very, very large company. We do these things at Microsoft called EBCs, executive briefings, where they fly all the C-suite people and everyone wears a suit. And then I feel awkward wearing a t-shirt.
00:12:00
Speaker
And then they ask questions about, what do you think about the future of engineering or whatever? And I just kept saying first principles, first principles. This was literally yesterday. I was in Seattle. It's like, you've got, and I've always said this on the podcast, people are sick of me. You've got to learn how to drive stick shift. It doesn't mean you have to drive stick shift all the time, but your relationship with the car is fundamentally different if you drive stick.
00:12:19
Speaker
I love that. If you can't observe your system, you don't know what's going on. yeah And AI is now making code so quickly. Yeah. Leaders hear that and they go, oh, we can ship more. But you you seem to be saying, now we need better feedback loops.
00:12:35
Speaker
Yes. You know what creates a feedback loop? Observability. its that Otherwise, you just have a bunch of things happening and a bunch of things causing things. Right. It's only if you observe the connection between the two that you have this feedback loop. So it's funny that you say that because i I was trying to explain like LLMs to a bunch of muggles and they weren't getting it. And I talked about the a million monkeys with a million typewriters will eventually write Shakespeare, but only if those monkeys are, you're actually observing the output and evaluating it against Shakespeare, right? You can't just randomly pick one of those monkeys and say, look, Shakespeare.
00:13:14
Speaker
Right. The feedback loop and the the reinforcement learning effectively to extend the analogy is
Pitfalls of AI Without Observability
00:13:20
Speaker
required. So that would imply that if someone is trying to go and implement AI agents and agentic loops in software on brownfield systems that have no observability, they're probably screwed.
00:13:32
Speaker
Oh, so deeply screwed. Deeply. All right. So part one is sort of the context setting history, first principles. And then parts two and three are about, for practitioners, software engineers, how to instrument your code and how to understand your telemetry. And we've got two tracks. There's doing it kind of, I think of them as ah Coke Zero and Coke Classic, doing it with ah AI and doing it you know without ai they're both they're They're both really solid. I had nothing to do with them, but they're I think they're pretty good. And then we have part four and part five. One of them is deep dives into like use cases and the other one is into use cases and what should we call it? They're system studies, right? Like a deep dive into agentic development and deep dive into use cases, right? Into like the storage engines that power high cardinality telemetry systems, that sort of thing.
00:14:25
Speaker
And then in part six is, it ended up being a third of the book.
00:14:33
Speaker
It's, all right. So if I can take it as a tiny detour, because this was supposed to be my little three. I'm like, at the end of the book, I think I'll write a new section for observability engineering teams on governance topics. And I wrote it.
00:14:47
Speaker
And then I ended up rewriting it twice. And it turned into something completely different. um And it ends up being, Governance, yes, but it starts with an open letter to CTOs and why all their hopes and dreams for AI are blocked behind their ability to understand their systems. And and it kind of starts at the top of the org chart and goes all the way down to like, observ but like what are VPs? What are distinguished engineers? what do What do people need to know? So like it starts with the open letter to CTOs and then there is two chapters of just systems thinking for software delivery, all about feedback loops and and and how this works and, you know,
00:15:24
Speaker
And how that sounds like a book of its own, honestly, it it really is. if You could spin that off into like a little gold finger mini plane and it could be its own pamphlet. Kind of should. and And we might do that, but it, yeah. Cause what CTO is going to like start two thirds of the way through the book. and Yeah. Don't bury the lead. Don't bury it.
00:15:45
Speaker
So it's, it's systems thinking. And then there's some stuff about like, how to drive change in an organization with influence, not formal power, how to how how to partner effectively with... My actual favorite chapter in the book comes almost at the very end.
00:16:04
Speaker
And it's my favorite, not because I think it's brilliant, but because I don't think I've ever heard anyone write about this, seen anyone write about this. But it's when I think about how improbable Every transformation is like most of them fail. The vast majority of ah of to technical transformations fail.
00:16:23
Speaker
The ones that succeed. What do you need to make them succeed? Well, as as a very senior engineer, you need trust and credibility inside your or organization. People need to believe that when you say something, it is true.
00:16:36
Speaker
It will happen. It will be backed up. That cuts through bureaucracy like like a hot knife through butter. It's one of the only things it does, that deep credibility. And when it comes to partnering with vendors, like when when it comes to you partnering with people in your team, you need to understand that you're all in the same team.
00:16:53
Speaker
Everybody has their needs, but you're all in the pursuit of the same goal and you need to keep reminding people of this, right? But when it comes to partnering with vendors, you need to you need to build trust and reciprocity.
00:17:03
Speaker
You need to understand you are not on the same team. You both want different things, right? And that the trust of believing what they say can be built over time, but you should not start out believing what they say because they're trying to make a sale. You're trying to do what's best for your company. And these things are just in natural tension,
Trust and Reciprocity with Vendors
00:17:22
Speaker
right? And so like, but if you're going to partner with a vendor, you want influence over their roadmap. They want to be able to believe that you will invest in, you know, for us, it's like, are you actually going to instrument your systems? or are you just going to like buy it and then it's going to sit there and you're going to be unhappy? You know, so like setting the the trust and credibility versus trust and reciprocity and like building successful long-term partnerships. These are durable skills in the era of ai for very senior engineers. And I tell a little story about
00:17:51
Speaker
Mark Callahan, the best engineer I ever worked with and how while I was in Facebook, he's in databases, right? I watched him alter, permanently alter the trajectory of multiple billion dollar companies just with a few conversations. And I was just like, mind blown. Same incredible idea.
00:18:08
Speaker
I have to send you this paper. I did this paper that Mark Russinovich and I have been working on about how if we don't double down on investing in the early in career people. I read it. Did you read it?
00:18:20
Speaker
Yeah, I did. so So when I'm trying to explain to a 25 year old about a Mark Callahan type person, I'm they seem to think that that's a coding problem and it's not. It is a human being problem. And as much as the AI grifters are out there selling AI and I'm one of them, I am selling it with a humanistic perspective that like, if it's if this new power tool doesn't make humans talk more and better, then we're doing it wrong and we've got to like stop it.
00:18:46
Speaker
So you're absolutely right.
The Role of Communication in AI-Driven Future
00:18:48
Speaker
Someone told me actually, i at at this company I was talking to yesterday, they said, what programming language should we teach the young people? And i was not trying to be snarky, but I just said English.
00:18:59
Speaker
Like really, really, like it's not the language that we deserve. It's just the one that we have. It's like JavaScript. It's just happened. And here, now we all speak English. But if you can't do it clearly and crisply and be persuasive and be credible and be thoughtful, then you're pretty much screwed. And you could call that prompt engineering, which I still don't think is a thing. Or you could just call it being a good communicator. But that is the programming language of the future. It's clarity.
00:19:22
Speaker
Yes. Writing is thinking on paper. Content creation and writing are not the same things. They're not, they're really not. And neither is yapping on TikTok as much as I do love TikTok. Now you, you have written in the past that the unit of work is changing, right? Like it's conversations, it's workflows, it's retries, it's tool calls. Does that change the old trace model? Do we need to observe different things at a different levels?
00:19:48
Speaker
You know, I, so I think that for most of my career, the unit of work has been transactions. and And it just isn't known. it it's The unit of work is is usually the trace, but the trace is just structured data. One of the things that I find really frustrating about people who have been in observability for a long time is that they get really wrapped around the axle about signal types often. And it's just data. They're just... differently enveloped wrapped bits of data right and the more fungible those data types are the more powerful it they typically are and so i think that you know we just shipped something i think it went like yesterday into ga called timelines which is so cool because you know you can imagine if you if you if if you're if you're like intercom or fit you you build like this chat product right the interaction begins when someone asks a question. And then you might have like a supervisor agent that kicks off and spawns many different agents. And each of those agents does a bunch of API calls and MCP calls and storage calls and everything. And then, you know, at some later point in time, they converge, they bring back an answer and they and they return it.
00:21:00
Speaker
And then the user says something else and it kicks off another and like knitting together all those bits, it's not a transaction that might take place over hours. Right.
Evolving Software Workflows
00:21:10
Speaker
And so the primitives have changed and our need to like go up the stack has changed.
00:21:16
Speaker
I had a really interesting conversation yesterday, and I think it'll be a show with Nathan Zobo, who worked on Zed, and he's now working on a thing called DeltaDB. And DeltaDB has a perspective on top of Git that Git is committing the before code and the after code, and it's missing all the interesting bits.
00:21:34
Speaker
So one could argue that it is observability in the development of software process where they are committing the conversations so that if you look at a line of code, and let's say that that got refactored and moved somewhere, you could imagine putting your cursor on some code and then being jumped to another another temporal location, which is a physical location, or the conversation where that code was conceived of.
00:21:59
Speaker
So drawing a direct line from conversation into code, into moving code. So I feel like as much as people are lamenting that like the, the crafts, that the the Japanese carpentry of bespoke assembly language, like, you know, 6502 assembler like that, those days may be over the the days of us slapping the keyboard may be over, but there's still really, really, really interesting philosophical work in the practice of software engineering. Oh God. Yes. This is still an engineering problem.
00:22:28
Speaker
We need see this is more rigor, not less. One of my spicy takes, and it's I don't know if it's good or not because I don't know how many people have this degree, but I don't have a computer science degree. I have a software engineering degree.
00:22:40
Speaker
oh, I'm just a music major dropout. Well, music music is also a valid major if you want to become the CTO of a major company. But the reason I'm pointing that out is that I don't have a good background in computer science. Like I did compiler class and I did one and I was done. yeah So I'm not a researcher. I don't write papers and research, but I do know how to ship. And when I started in the early 90s, test-driven development, the Gang of Four, and like the manifest Agile Manifesto was just getting started. And we called them build servers and now it's DevOps.
00:23:09
Speaker
So they, they, my, my university was like, software engineering is a valid practice and shipping is a thing that is different than computer science. And I think that people need to just always, if there's a question, first
The Indispensability of DevOps
00:23:22
Speaker
principles.
00:23:22
Speaker
Yeah. Do we have a good but good build server? Talking to these giant companies at these executive briefing centers and then finding out that their DevOps is a hot mess. Yeah. No amount of AI is going to save you. No, no. In fact, that is a trap.
00:23:35
Speaker
It really is. And they're going to be absolutely screwed. And then they're going to be sad when their teams collapse. Yeah. Yeah. You have also been associated with test and production, oh which some people have said is kind of reckless.
00:23:48
Speaker
Yeah. See, look, test and prod or live a lie. So some people call that reckless, but these are the same people who might be using AI to just kind of yeet code directly into production. So and what is reasonable? Yeah. What does it look like in production it's only ah The only question is whether or not you acknowledge that that's what you're doing and build tooling to do it safely. there is no There is no replacement for reality. No matter how much you think you have tested or examined or validated,
00:24:17
Speaker
Every single moment of intersection of software, infrastructure, deploy process, time, data, users, every single one of those is unique. oh There is no test for reality. there is there There is only practice runs, right? You're always going to be you're always going to be tested by- It's Schrodinger's production.
00:24:37
Speaker
It really is. You just have to open it and find out the cat's okay. and And none of this is an argument to not test, but I feel like we all have limited attention cycles. We all have limited engineering cycles, even with, you know, even with AI. And I feel like most people get so wrapped around the axle of making sure it's perfect that they're under investing in the other side of it, which is how am I going to find it? How am I going to trace it? How am I going to react to it? How am I going to fix it?
Feature Flags for Safe Testing
00:25:07
Speaker
Yeah, that's a really great point. So every deploy then is just a production experiment, whether you choose to admit it or not. and And the thing is that you don't have to do this dangerously. We have so many tools now for ah canaries or progressive delivery or feature flags or you're deploying the code and then turning it on for one person and then turning pranking up their ratio. Like nobody's saying this. few people do that stuff. So few people do that stuff. Well, you know, there are lots of contexts where that would be over-engineering, but if it's not, you should do that. Yeah, yeah. Feature flags are like...
00:25:41
Speaker
it's so funny that like, what's the most sophisticated, most powerful thing that I can do to really enable production and like experimentation. It's like a bull. You could put a bull, you could put a bully in there and just maybe like turn that feature on or off. And if it goes bad, maybe, you know, like, yeah, just throwing it out there to spit, spit balling, maybe a debug.WriteLine, a little printf and a Boolean. yeah It's so funny though, because it's, it is the core of what we're doing.
00:26:07
Speaker
it's Yeah. Yeah. yeah That's so funny. Like the combination of feature flags and the kind of the kind of high cardinality observability data that Honeycomb does, like it's, they're greater than the sum of their parts. I know a lot of people have used feature flags when all they have are like metrics and aggregates and log lines and feature flags are still useful, but they're not like superpowers, but like it is a superpower when you can break down by one in a million apps and enable just that app and then compare it against the baseline of all the other apps and trace every request. Like it's just,
00:26:41
Speaker
Yeah, you can test in prod. Then it's it's like, why wouldn't you test in prod? It's the easiest way to test. Yeah. All right. In the and the in the remaining couple of minutes, let me get a little bit rapid fire. But also, I want to explore some of the spaces where you and I, I think, deeply
Value of Hiring Junior Engineers
00:26:55
Speaker
agree. Because you have argued that not hiring juniors is a problem. If a company says we only hire seniors now because AI does the easy stuff, what do you say to It tells they're trying to automate all people away. Okay. Okay.
00:27:08
Speaker
You don't like that? Totally unacceptable. I mean, i am a person and I would like to have a job. So yeah, I don't think that's very smart. Good. Okay. So if an AI is an ai more like a junior engineer or an intern with bad judgment, or should we not anthropomorphize it at all? What do we, how do we think about it? I'm anti-anthropomorphizing it because it's just, I love the definition of consciousness. That is, there is something it is like to be that person.
00:27:33
Speaker
it's It's about like the way that we develop taste and judgment is through repeated exposure and experience. And that is just, it is categorically different from what we get from agents. Okay. So that's awesome. Cause my next question was going to be how do we teach judgment when the visible artifact in this case code is so easy to produce, you got to read more code.
00:27:52
Speaker
we We create the conditions where people can fail and experiment and try things safely. I have a lot of faith in humans. There are a lot of problems with AI. Don't get me wrong. I am not willing to take any of them away. And I suspect that a lot of the problems of how do junior developers learn these things is going to come from them, not from us.
00:28:10
Speaker
I believe they're learnable and I believe this is solvable. But I just, I fundamentally believe that you decide to take a risk on people, you support them, and it pays off.
00:28:21
Speaker
How important, I think it's hugely important. How important do you think like empathy is and how did being on call early teach you about empathy? I'm just kidding. I mean, speak it's how important is empathy is kind of like saying, do we really need oxygen? Let me rephrase. How did being on call being on prod teach you about empathy?
00:28:43
Speaker
That's the real question. That is the thing that I love about being on call is how much it aligns you with the pain that your customers are going into. We were talking about feedback loops earlier.
00:28:53
Speaker
and I have told many people that the way that they get their developers to care is by putting them on call. Because when you align, when the pain is separated and you don't, you're not a good developer, but when your pain is aligned with your customers, it pain is nature's teacher.
00:29:09
Speaker
Yeah, it really is. Like it has to hurt a little bit. And I was talking about this. If you work out, your muscles need to hurt a little bit. yeah if you are studying, your brain needs to hurt a little bit. I'm a little bit sad that like long form learning is now becoming tick tocks is now becoming snackable. And people add friction.
00:29:27
Speaker
Every interaction with a person has friction and that's why it's valuable. Yeah. Yeah. And that builds it that that you put them into the forge and you you turn them into something special. Absolutely. yeah That idea of being in prod, but also letting the early in career person, like I use the analogy of hold the scalpel.
00:29:47
Speaker
Like we're about to do a heart surgery. Hey, have you ever, you ever held a human heart? Here you hold the scalpel. We're going to go into prod right now and let's do that. Yes. Do you have, ah how do you treat early in career people at your company? Like, do you have interns? Do you call them interns? We have a few interns and we have a few early career folks and we try to make sure and never have just one. You don't want there to be one person. It feels like, you know, you need to hire in pairs. We're not a super big company. Sure. And
00:30:20
Speaker
Watching them learn has been so interesting because everyone uses AI in a different way. But like most of them have just this running dialogue all day long with they have a 24-7 tutor personally devoted to them, you know, and and it helps them figure out how to ask interesting questions and interrupt their more senior counterpart. Yeah. I just, they're, people are, people great.
00:30:44
Speaker
Yeah, i'm I'm really looking forward to spending the last kind of thousand days of my career spending as much time as I can with early in career people and then going and teaching again at community college. and i was just talking to Jay Gengelback from Vercel and he was saying, you know,
00:31:00
Speaker
He was expressing the same thing. He's like, man, I see so many more bugs that I can fix. I see so many more problems that I could i can't build all the things that I see. and And I can't ask agents to go build those things or solve those things because they're too they're too big. they're too They're the right size for a more junior or mid-level engineer. And I could have so much more impact if I just had people do this. And I was just like, I think that's the best pitch for junior engineers that I've ever heard. I honestly do. They've got to have exposure. Well, the second edition of Observability Engineering is available as a PDF. Dead Trees coming soon. It is fantastic, apparently, because it's 600 plus pages. but I feel like I shouldn't have said that. Now nobody's going to want to read it. No, it's a whole new book. Hey, when I hear 600 pages, I hear value.
00:31:45
Speaker
I think this is, that is exciting to me. That is the hook. It is the war and peace of technical books. There it is. The war and peace of technical books. And we're going to observability the heck out of it. So thank you so much for chatting with me today, Charity Majors.
00:32:01
Speaker
This has been another episode of Hansel Minutes, and we'll see you again next week.