Transcript
Speaker: This is the Silver Bullet Security Podcast with BIML. I'm your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses on machine learning security.
Speaker: For more, see berryvilleiml.com slash podcast. This is the 161st in a series of interviews with security gurus and I am really pleased to have today with me, Christophe Kern. Hi.
Speaker: Hello. How's it going? Glad to be on the podcast. Christoph Kern, widely known as X2F, is a principal software engineer in Google's information security engineering organization, where he leads research and engineering initiatives focused on scalable principled approaches to software security.
Speaker: A founding member and steering committee member of the IEEE Center for Secure Design, where I met him, Kern has played a central role in shifting the industry paradigm from reactive patch-based bug hunting towards secure by construction design principles that eliminate entire classes of defects at scale.
Speaker: His influential research published across the ACM and IEEE focus on type-safe abstraction, static analysis, and secure developer ecosystems that systematically prevent vulnerabilities like cross-site scripting and injection attacks across massive industrial code bases. Recently, his work has expanded into establishing structural security frameworks for AI agent architectures and modular software safety.
Speaker: which is totally great. You did the exact same segue that I did. So Christoph, welcome to the show. About a decade ago now, you helped fundamentally reshape web security with your seminal work on killing cross-site scripting at scale, moving the industry away from the endless reactive cycle of finding and patching individual vulnerabilities towards a secure by construction thing.
Speaker: um By leveraging type safe APIs and hardened web frameworks and compile time guarantees, Google essentially proved that you can systematically eliminate an entire class of bugs before code even hits production. So looking back at the progress made since those early framework hardening days a million trillion years ago,
Speaker: How has the security by design methodology evolved and what are the major engineering hurdles teams still face when trying to enforce these hard structural guarantees across modern heterogeneous software attacks?
Speaker: Yeah, that's a good question. I think ah it's become perhaps more mainstream accepted. um And one thing that was actually really nice to see, I think, is that a lot of the underlying philosophy of how to do this ah evolved separately and independently ah in the Rust community.
Speaker: um I really like sort of seeing the work there and seeing like, oh this is exactly how I think about the problem, which is really nice to have sort of independent cooperation that this is a um sort of a practical ah path between the spectrum of like full rigor of formal methods and just sort of like um you know ah ah software crafting without any rigor at all, right? And it seems to be kind of ah a sweet spot that gives you ah high confidence outcomes at a very, very reasonable cost in practice for large scale practical software projects.
Speaker: um And since you were asking about challenges, I think where we've seen this to work really, really well is for ah new projects, for greenfield projects that ah basically start on the ah secure by design stack. Because then the extra work you need to do do to stay on the within the guardrails of the secure stack is really very, very minimal. right As a developer, you hardly ever notice, ideally, if it's designed well. And this is sort of part of the usability ah a question of these these stacks, like the developer friendliness and ergonomics aspects of these. And yeah this this is you know manifested in things like you know Rust has really nice error messages that tell you what to do, right? And we've done similar things in in our tooling where if you hit one of those ah compile time checks, you get ah you get an error message that tells you what to do next and how to get rid of it. And then once you've done it, you know how to do it. And then from then on, the friction is is very, very minimal, right? That's very cool.
Speaker: Where it gets tricky is if you have a very large code base that is very far away from this style. We've actually had some successes with existing code bases. For instance, in the early days of the web security work, one of our first sort of large early adopter um programs was was Gmail, which was a very large code base, a very large complex web application, probably one of the most complex ones you would find out there at this time. right at that time And we were actually able to refactor it in place to adopt these secure by design APIs ah with a, you know, considering the size of the project, a very reasonable amount of effort. Right. It was, you know, on the order of a couple of sweet years, maybe, which is which is a really small cost compared to the size of that project. So in that case, it was possible.
Speaker: i know that I know that DARPA was working on you know some Cedar Rust and and so on projects too, because Dan Wallach is a friend of mine. so he's yeah Yeah, exactly. right And so this this was going to be my example of where this is much harder. right like We have, you know I don't know how many like millions of lines of C++ plus plus code and ah getting that transfer to a secure by design ah ah platform with respect to memory safety in this case is much, much, much harder, right? um
Speaker: And so the interesting thing, though, was that the the Android team has made that transition for their code base over the years, and they've I think last year, announced that they have basically crossed the threshold from where most new code is written in a safe language, even though there is still a lot of existing legacy code that's written in T++. plus plus it doesn't change much. And that has actually resulted in a disproportionate drop in the number of vulnerabilities that are being found. right So there does seem to be sort of an effect that vulnerabilities have a half-life. And if you don't touch the code, you're not introducing new ones. And so eventually, you'll kind of shake them out. right So you're actually getting a lot of progress by just focusing on new code. But you still have residual risk in all that old code. And this is sort of, since we're talking in the context of ai this is where it gets interesting now because we now actually have models that are capable of doing high quality rewrites of existing code in different languages, right? And I think so there is actually a ah unique chance that that the the cost of migrating greenfield ah brownfield code that is
Speaker: ah not very close to a secure by design style is becoming maybe much more palatable, right? Or much more practical. So so that's a really exciting opportunity. So I agree with you, but I think there's some flaws that are still really hard. Like, I don't know, deep logic errors, broken access control, subtle time and state mistakes. They're hard to constrain through language or framework design alone.
Speaker: So yeah from your experience, what structural properties determine whether a bug class is amenable to being wiped out by architectural correction? And where do traditional type systems and frameworks sort of hit the wall?
Speaker: Yeah. Yeah. um So that's a very good question, right? like So I think it kind of depends on how far you lean into it. um An expressive type system like Rust can actually encode a remarkable a degree of are ah of of correctness in variance that you want to express. There was a really interesting talk by a a colleague who was working on the Fusha network stack rewrite in Rust. um And they ah they really sort of leaned into this use the type system to encode invariance approach. And they i think I recall, it's been a while I've looked at this, but I think I recall that they were able to express ah a um a lock ordering discipline at the type system level where basically yeah taking out locks in the wrong order becomes a type error. And and this relies on like Rust linear types to actually express. right And so they basically were able to statically ah eliminate at compilation time ah the possibility of of deadlocks. right That's very cool. I mean, time and state has always, to me, been the next big thing that we're all going to have to deal with.
Speaker: Yeah, and and so they found you know that they ah when they were like when they finished the code, they were going to roll it out internally for team fooding. They expected to get like you know tens or hundreds of bugs because you're writing a new TCP stack, there's going lot of problems. And they reported they had like three bugs. Oh, my God. So I think there is a lot of scope for doing this, but you really have to think about... i don't know. All you're saying is that they rolled three D20s and got, you know, 60. Yeah, yeah. So its it is there is... There is, I think, um you know, really...
Speaker: a pretty significant possibility. um the The other question is, you know not only what can you do, what but what can you do cost-effectively, right? And so I think what's been, in the past at least, without the benefit of um highly capable automation through ai what's been the determining factor is whether or not the bug class is common to a large number of applications of a similar archetype, right? Like a similar structure. and And typically this relates to bugs that are actually at the sort of underlying platform level, like web bugs, you know, web application bugs apply to every web application, irrespective of what it does, right? Like it doesn't matter if it's a banking app or a note taking app or a photo app. Sure, like a system call NC, for example. Exactly, right? and so And the same applies to memory safety, right? That pretty much applies to any program. And so then if you make this investment in the platform,
Speaker: um it essentially ah amortizes across all the applications of that archetype, right? So our work on secure web stacks got amortized across hundreds of web frontends. And so then it became ah very, very cheap. And if the of course it only works if the the sort of variable cost of it is also small. Well, I mean, you gotta to be able to shove stuff into types, right? So now now we're moving towards stuff like RAG or it's everywhere now.
Speaker: yeah you know So you got like search grounded AI systems. We're seeing kind of untrusted external content fed directly into model contents. so So what do you do in that situation? Like how do we take lessons learned from securing traditional search architecture, say, and apply that to LLM context pipelines? Yeah.
Speaker: yeah Making sure that retrieved data actually is the kind of data that you want with the right types and never got gets interpreted as a control instruction. Yeah.
Speaker: So that's, I think, a very, very difficult problem. And as far as I can tell, with the current architecture of LLMs, essentially unsolvable, right? Because there is no rigorous process.
Speaker: uh, a delineation between data and control in an LLM. It's in the end, it just becomes one big vector of tokens yeahp and you're relying on the, uh, the, the, um, the weight matrix of the model to kind of separate them, but it's always going to be an imperfect,
Speaker: ah level of assurance, right? Well, but then we get stupid things like filters and system prompt instructions and guardrail models. but But from a secure by design perspective, those feel like the old days of like trying to sanitize inputs instead of finding underlying you know the underlying paradigm. so But I think you're getting somewhere with your architecture, like CAMEL, for example. Exactly, right? So I think ah basically ah for programming, properties that we want to be assured with very high degrees of confidence.
Speaker: um As far as I can see, the only way to do that is to ah essentially treat the LLM itself as a fallible component of a larger system and then enforce the property through something that is implemented as classical, understandable, ah trustworthy code outside of the model. right yeah So though this is where then like policy engines come in and and things like Camel, where if you want to track and reason about the provenance of certain data, um you have to keep it in a separate context and and use a classical system like a
Speaker: in the camel example, data flow aware ah execution engine for the code that is emitted by the main planning model to, to, to actually keep track of. Right. Right. And, and, but I mean, honestly, it's just the separation of control and data and back in the um center for secure design days, we had a lot to say about that. Here we go again. It's like back to the future. yeah,
Speaker: Let's kind of push on that. you know write that When things get agentic, it gets even more complicated because right now developers can wire LLMs directly into APIs and code interpreters and file system. We have a way of handling that. But when you look at designing tool abstractions for agentic systems to use, How do we construct capability based kind of security boundaries around an agent so that even its ah if its internal reasoning is hijacked by, say, some but some adversarial input, it still has this untime run underlying runtime environment limitation.
Speaker: um And so it can't execute really terrible stuff. Yeah. Yeah. so I mean, this is basically the crux of like ah AI security and agent security. Right. yeah um And I think the the the really underlying challenge is in finding the right trade off between adequate bounds on risk and ah and utility and usability. Well, the good news is humans are so good at risk management, I tell you. Exactly. Okay, never mind. They're bad at it. So this this is, I think, really the the trick. And I think actually, in some sense, a lot of um folks who come from a pure security engineering
Speaker: background have sort of a tendency to not ah see that trade-off. And you hear a lot of people say, oh, agents should have. i don't know. Some. I think academics for sure. But the people in commercial land, we got to build stuff that actually works. And we have to ship. that you know The Visa card has to ship.
Speaker: People have to carry that around. Exactly. Right. And so so I think, ah you know, yeah but you do hear a lot of like, security people say, oh, agents should have least privilege. Right. And then on the other hand, yeah mean ah agent people and like sort of ah like, you know, AI developers and users also kind of come from the exact opposite spectrum. What they do is they're like, okay, least privilege sounds great. Let me just get OpenClaw to build that for me. Exactly, right? Like they basically come from the position, you know, I want my agent to do everything that I can possibly do.
Speaker: And of course, neither one of those are practical, right? Like you don't really want your agent to, you know, ah like buy a house or sell a house, sell your house on your behalf without asking you, right? I mean, that's like, you know, or or liquidate your 401k or things like that, right? Or even make a major purchase, right? I probably don't want my agent to buy a car without talking to me. But at the same time, i don't want to be asked every time my agent like buys groceries for the week to approve that. And so I think the key is to find the um the the ah ah the place in that spectrum where you have a reasonable bound on tail risk. And you also at the same point have a good amount of utility. right So if you look at an agent that can make purchase purchases, right like what I kind of want is an agent that can make ah small purchases that are you know rate limited and limited per transaction. So it can go buy my groceries automatically. And if I tell it my toaster is broken, buy me a new one. It can go out and buy me a toaster and not ask me a lot of questions. But it can't just go and buy a car.
Speaker: Right. And that's me. Right. Like if you're if you're Elon Musk, maybe you're OK with your agent you're buying an airplane. Then it's a small country or maybe democracy. That's for sale. But I mean, I think there is an interesting lesson there is is that the what is an appropriate bound on risk is a very user specific consideration. Right. So exactly right.
Speaker: The system needs to be built where, as a user, I can tell at once, okay, this is what I'm comfortable with. And if you buy me the wrong thing you know once a month and I have to go back to UPS store and return it, ah I'm okay with that, right? If most of the time it saves me, it does the right thing and it saves me time, right? So I'm actually okay with a small risk of an incorrect action or an undesired action by the agent, as long as I'm very, very sure that there's a hard bound on how bad it can be. Okay, so we we've we've barely scratched the surface because agents that are by themselves, we still have a hard time reasoning about. Now scale it up from one of those to ah multi-agent swarms where specialized agents pass contacts and delegate tasks and trigger actions and have little societies and trick each other and use each other's privilege and sleep with each other's agentic spouses and stuff like that. So
Speaker: If you evaluate a single autonomous agent, its capabilities or failure modes might sort of seem bounded to those of us who are not supreme formal Methodists. But much like inspecting an individual ant, that's why it's possible. But when you get when you deploy a colony of tens of thousands of these things, you get emergent behavior that no engineer programmed into a single prompt or some sort of system instruction.
Speaker: So how do we shift our security architectures to govern those things, agentic swarms, where the primary risk is no longer just one compromised screw up or node, but the emergent dynamics of the whole dang colony?
Speaker: I don't know if I have an answer to this question, right? Come on, this was the fun question. Yeah, this is the fun question. i mean, one thing that does come to mind is that at this point or at that level, i think the fundamental considerations aren't even technical to start with right i think they're more sort of societal i mean basically you know this is the same thing if you have a country full of people uh that has emergent behavior and societies have evolved um mechanisms of governance and things like morals and and and ethics yeah kind of govern that in a reasonable way and it kind of well we have some sciences first right for studying those same things when we don't understand them like ecology. you know We can go to a new zone on the planet and figure some stuff out scientifically. Maybe some of those methods are things that we can use. Yeah. And then and then you know once you figure out at a high level what that governance looks like, then you have to build the mechanisms to actually implement. Right. And so, ah you know, maybe that takes a form of, ah you know, delegation mechanisms for capabilities or privileges or perhaps, you know, if the other agent, you
Speaker: is running in a different runtime, ah but it is actually running in ah on top of a trustworthy platform that can certify or a vouch for bounds that are placed on the behavior of that particular agent. Maybe then, as a user of that, that's enough for me, and I don't actually need to know the intellectual property that went into the agent and the prompt and whatnot, right? We sort of worry about atomicity and a thing like that.
Speaker: I mean, we we worry about time and state where we understand the atoms and they have types. But imagine that the atoms are so small, we can't really say what happens when you assemble them in new ways. Right, right.
Speaker: and that's That's the challenge. All right. So, all right. that was That was a little too much. let's Yeah, that's that's above my pay grade. Let's let's let's back it back a little, um sort of. So as these agentic swarms and self-organizing computational ecologies take on more of the routine software lifecycle, it frees up human engineers to focus on highest levels of design.
Speaker: So probabilistic engines are really good at sweeping through a vast pattern space. um But defining true architectural elegance and anticipating human context and inventing entirely new security paradigms still requires something fundamentally distinct.
Speaker: So when you look at how security engineering evolves alongside of these autonomous systems, how do these tools amplify the uniquely human thing that we bring our capacity for insight or architectural intuition or creative ingenuity to build ecologies that are more resilient than anything we would have constructed by hand.
Speaker: Mm-hmm. Yeah, so this this is that's a very good question, right? I think um i think the before we can actually answer that question, i think we need to be able to ah answer the question of how the human um ah observer and and sort of accountable person for the system that's being built can actually exercise this accountability in a meaningful way when the code, or almost all the code or all the code is written by an agent and really no longer consumable by humans because of just the sheer volume of it. Right. Right. And so um one one interesting way of looking at it, which kind of comes out of like the way I think about systems through ah sort of platform and and developer ecosystem enforced invariance, is that um essentially what you want
Speaker: the out the outcome that you want is um ah high confidence, understandable bounds on the behavior of the system that came out of it. And an important or ah maybe a valuable thought there is that not all bounds are equally important and need to be assured at the same level. ah So if I have a system and i'm very very I kind of want to be very, very sure that it doesn't have misbehaviors that would be very, very bad in impact and impact an outcome, right like security bugs or catastrophic data loss defects and things like that. um On the other hand, if that system has a behavior that's is prescribed in ah yeah the typical of sort of type of spec where you have a user journey that says, you know I fill out this form and I click a button and then something appears in the database, um
Speaker: if the system doesn't always do this ah and maybe it doesn't do it in in certain obscure circumstances that are never reached in practice and ah it fails in a safe fashion, it just nothing happens basically. yeah That's actually maybe an acceptable defect, right? Because we'll never hit it in practice. And if it does happen,
Speaker: frequently will know, and if it doesn't happen, maybe we don't care, right? Unless it's application of your breaks, but yeah. Yeah, and so so, exactly, right? So we basically need to identify, and and the the software equivalent is like, it doesn't ah let one user access another user's data, or or it doesn't crash and destroy your disk or your computer or whatever, right? And so ah so I think um if we can get to a way of building systems where these important invariants that if they fail or if they are ah if they are violated, we result it results in a defect that has a very high negative impact, if we can ensure those very, very confidently,
Speaker: ah but And at the same time, we have sort of, you know, high confidence, but not highest confidence assurance of the of less important properties or less, less high stakes properties of the system. We're actually maybe in a good place. Right. And so yeah this leads to sort of a. It frees up your creative juices as an architect to do some cool stuff. One would. Exactly. Right. So then, you know, you can basically describe the system. You can describe what it wants and you impose platform insured in variance that really bad stuff doesn't happen, you can maybe at that point actually trust both an implementing and a reviewing agent to do a very adequate job on on building the thing, right? And then, you know, what what ah where you then impose your judgment is sort of on architectural structure of the system. And you'll want some way of actually expressing this rigorously, like with a rigorous expression of what can depend on what subcomponent and things like that. Sure. But you can also have a sense of aesthetics that, you know, you might not be able to formally describe, but you know when you see it that that's a beautiful piece of architecture. And that's the part that's that ephemeral human stuff. So the good news is we're still going to need really extremely good architects forever, I think. Yeah, so i think one one one one thing I've actually been thinking about a lot is like, how can we encode ah sort of legible? How can we ensure that the architecture of the system as built is legible and understandable to the architect, to the human? yeah And we can be very sure that the system as built is actually...
Speaker: ah actually matches or its architecture actually matches the understanding that the human has based on whatever artifacts they see. think Think about this. It's kind of like growing a building instead of building it out of pieces. you know More organic structure. We can reason about those and think about those, but maybe we might have to throw out our crystals.
Speaker: Yeah. Yeah. That's ah the building all analogy is interesting, right? So you may have, ah you may have a very ah high confidence understanding of the basic structure, right? Like where it's the load bearing pillars or whatever. And then you have like some organic goo that grows the walls, but they stick to the scaffold. And so you don't really care exactly what they look like. Right. Exactly. stu and stuff Stuff like that. and then you end up with like 1970s planet of the ape architecture and everybody's. okay Yeah. Okay, let's wrap up by looking out over the next five years, which is an absolutely ridiculous exercise, but because this is a podcast, we have to do it. So we're standing at this unique inflection point where the speed of software generation is skyrocketing, and yet our traditional defensive frameworks were built for a way slower manual era.
Speaker: um On one hand, we risk flooding the ecosystem with probabilistic code that resurrects long vanquished bug classes. On the other hand, we have a rare opportunity to embed secure by construction constraints directly into the generative substrate itself. So ensuring that systems are safe by default as they're created is ah is our goal. um As you look forward like over the next five-year horizon, where do you see the most profound shift happening? Are we going to be integrating rigorous structural invariance with AI driven reasoning, finally allowing us to outpace the race of new vulnerability creation, or is it going to be fundamentally rewriting everything by system of correctness or dot, dot? dot What do you think?
Speaker: Yeah, I mean, one thing I'm quite excited about is that it seems to me that when we use models, it's very, very difficult to be sure that it won't do something undesired or that it always will do the right thing. However, if we let the model produce something that can be mechanically verified with a high degree of confidence, yep. then we're in a good place, right? And sort of does seem to be emerging. mean, we're a good place, but we also, we decided Turing completeness is what we want all the time. And I don't think we need make decision. Well, mean, maybe we don't, right? Like, I mean, so the interesting thing is that finding the solution might be very, very hard and it might take a very, very,
Speaker: a very, very capable model to do and you know burn a lot of tokens on. But once it gives you a solution, checking that the solution is right yeah is a mechanical thing, right? That's old Nebula Lee come back and at us. Yeah, exactly. And um I think this is there this is a really promising direction, right? Like, I mean, basically, I think doing...
Speaker: rigorously verified development used to be impossibly expensive, right? The verification was like orders of magnitude more expensive than just writing the code. But we are getting to the point, mean, like OpenAI just published that they solved, that their model solved like a Millennium Prize problem, right? That's an extreme, I mean, we'll see where that lands. I mean, it's a very, like, I think this is a very good example of where,
Speaker: um the, the formalization of what is proved is relatively compact and should be understandable to experts. yeah And then the proof is like insanely complicated and tedious. But if you trust your proof checker, if you trust lean, you can trust this whole thing, which I think is a very plausible place to be right. Because, because if you were watching video at this point, you'd see me going with my mouth. Of course this is audio, so you can't see that.
Speaker: Yeah, I mean, I'm i'm somewhat somewhat optimistic about this, right? Like, because, I mean, you know, maybe there is a a flaw in the in the core theory behind the prover, but I mean, that is like, they're so small and so well thought through by experts in the field that that a a discovery of a flaw in that would be like an earth shattering discovery. Of course. I mean, it's it's like choosing your right tools all over again and putting your invariance into the tools. Yeah, and and so I think it is perhaps not inconceivable that the use of rigorous machine check verification will become much more um expansive, right? Like it used to be basically what we had so far because it could only be done by a very small number of experts and was insanely ah labor intensive. We would get this for like a microkernel like SEL4 and they spent like 20 person years verifying it doing it um and uh and this is maybe something we can actually at some point in the not so far future afford to do for more of the core trusted components of our stacks right i like it so building building better crystals to to you yeah exactly right and then uh and then the things and those things then are what places very firm
Speaker: ah ah security boundaries of what is built on top of them and is a firm foundation for what ah for what is built on top of it. And then maybe those things on top of it don't need to actually be verified, right? Because that's actually very hard. Like verifying the property or like formally specifying the properties of a web-based application is is is really difficult, right? And ah maybe not worth it.
Speaker: That's because nobody knows what it's actually supposed to do. Exactly right. But i mean, if we can have foundations, again, this comes back to like, if we can if we have the foundations that can give us very strong guarantees that ah large classes of well understood very bad behavior really are ruled out with a very high degree of confidence, then what's on top of it is perhaps okay if it's not absolutely perfect, right? I love this. So so you're not going to believe this, but we've been talking for about 35 minutes, so we have to stop.
Speaker: But I could talk to you forever about this stuff. It's really, really very fascinating. Thanks so much for being the show. Yeah, thanks for having on. This was a great discussion. Good questions. Thank you. There were a couple that stumped me. Well, I want you to sit at the highest point where you are and take a look over the mountain. The rest of us are too short to see over there. So thanks for helping.
Speaker: This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security.
Speaker: You can find a permanent archive of all our episodes dating back to 2006. at garymcgraw.com slash technology slash silver bullet podcast show links notes and an online discussion can be found on the silver bullet webpage at barryvilleiml.com slash podcast this is gary mcgraw
