Transcript
Speaker: This is the Silver Bullet Security Podcast with BIML. I'm your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses on machine learning security.
Speaker: For more, see barryvilleiml.com slash podcast. This is the 160th in a series of interviews with security gurus and machine learning people. And I am pleased to have with me today somebody who's both Aaron Bedra.
Speaker: Hi, Aaron. Hi, thanks for having me. I looked back, it was 2014 when we did this last time. Wow. Aaron Bedra is the CTO of Cumberland, a digital asset business of DRW, operating at the intersection of high frequency trading, software engineering, and system security.
Speaker: Over a distinguished technical career, he served as chief security officer, chief technology officer, chief scientist, and principal architect across healthcare, financial infrastructure, and retail e-commerce.
Speaker: A deeply influential figure in functional programming and language design, Bedra was a core contributor to the Clojure and Clojure script um and co-authored the definitive reference book, Programming Clojure.
Speaker: Grounded in functional programming principles, language theory, and quantitative risk modeling, his work focuses on building high assurance, resilient architectures that satisfy complex business and regulatory demands without compromising speed or system integrity.
Speaker: And the thing that's left out is that Aaron used to work at Cigital with me 25 million years ago. So we know we know each other well. And it's great to have you on. Thanks for joining us.
Speaker: Thanks for having me back again. Aaron, um you've spent your career at the intersection of hardcore software engineering and executive security leadership, which is a hilarious gap to try to straddle.
Speaker: Over the last couple of years, the rise of generative AI and LLMs has completely reshaped the software landscape. Looking at how rapidly these capabilities have evolved, what surprised you the most about where AI actually delivered versus the hype? And are enterprise security organizations adapting their core architectures properly, are they still just scrambling to react?
Speaker: Big loaded question to start. I love it. ah The thing that surprised me most ah was what you can do when you give people agency.
Speaker: um Particularly, there's always been a struggle in building software about getting it right. Listening to people, figuring out what they really mean, right? Not giving people what they say they want, but giving people what they need.
Speaker: And there's a whole career in like that sentence yeah to do it right. And What LLMs provided was a path for people to express that in executable form.
Speaker: um That part was awesome. And i was it's so powerful. And I've seen so many amazing things happen, kind of just cutting out the back and forth and the time to like a good idea being a great idea in production.
Speaker: Right. Now, on the other side of that, so the adaptation of this came so fast that it's impossible for people to have changed their architectures, their systems of security to respond, right? This was something would have had to have started 10 years ago. Especially if you consider like what happened maybe three months ago with the with the step change and in frontier models. I mean, it's been remarkable how fast things have developed.
Speaker: It is. um You know, I think February, March this year was a big one. And then even just a few months ago, and another big one as well. um You know, this was something people, we we talked about years ago of changing how we do architecture, how we design things.
Speaker: um Now the LLMs are really great at pointing out where that's not not great and helping us find them faster. Right. Yeah. um And, you know, with regard to kind of organizations adapting, you think they're, most of them are scrambling? What's what's your, what's your feel?
Speaker: There's a lot of scramble. um ah People I talk to are are certainly scrambling. um But what I also think is happening, and I think it's it's kind of interesting, is rather than scramble to bolt it on, I think this has been big enough that people realize that it fundamentally has to change to solve to be solved.
Speaker: So I think we're kind of it's becoming the slingshot kind of effect of you can't you actually can't just go fix it. You kind of have to redo it. Right. um People are thinking of past, like the incremental fix and thinking about the fundamental change that actually solves the problem.
Speaker: That's cool. um I suppose it's complicated because every enterprise exec team is demanding faster AI integration, but security teams are like, whoa, whoa, whoa, new risks. Yeah. Some directly stemming from ML and some you know because of use of ML. When you look at the actual front lines now, how are defenders successfully deploying AI inside their security ops, like to do security? How are attackers leveraging it on the other side? And are AI-driven attacks fundamentally novel or are they just hyper-automated versions of the same stuff we already are familiar with?
Speaker: They are hyper automated versions of the same stuff. But, you know, the the interesting part is they are relentless. they don't They don't sleep. they don't check They don't stop. They don't take a break. They just keep going. They they go as as as long as the token furnace can burn tokens, you can keep going.
Speaker: And, ah you know, the other thing is now that you have the ability to chain ideas together, there's nothing really stopping them from stopping. Right. you know You know, what's interesting is is it reminds me of grinding in a video game.
Speaker: Yeah, but they can grind harder, better, faster, longer, more successfully and more they like it multiplicatively. They like to grind. And you know the part that you're just like, oh God, I don't want to that part.
Speaker: It gets done for you. yep They will leave no stone unturned. A human researcher has bias and a time commitment they have to spend or not spend.
Speaker: An LLM has tokens. And as long as the tokens are infinite, all the stones get turned over. Right. Really interesting way of looking at it. I think that's right. So let's look at the practitioner level. Employees across every department are using AI tools daily to do lots of stuff, write code, summarize documents, automate workflows, b blatyb blah, blah, blah.
Speaker: When well-meaning employees interact with these models, what are the biggest operational traps? Like could be subtle data leak leakage, could be public versus private model boundaries, could be hyper-personalized phishing for sales guys, or just blind trust in generated output that comes out um and has silent errors. But it looks good.
Speaker: Yeah. um All of the above apply. The things that stand out most to me um the the The most appropriate name for this group, Jack Jones, talked about it many years ago.
Speaker: The non-malicious insider threat group. The ones with all the access that don't mean any harm, but do cause oopsies, right? Where oopsies could be lots of things. um you know I do think there's a lot to consider around the relationship between what goes into the LLMs and what the providers have access to. We've talked about that for a long time.
Speaker: There was a pretty famous interview recently with the CEO of Palantir about that. um But I think there's lots of alternatives here where you can kind of rent inference, where you've got the hyperscalers who offer the frontier models with a sandbox or with a way to kind of cordon data off.
Speaker: yeah There are things you can do to still use these really interesting models in safer ways. And so ah It's not that no option exists, but it takes some effort to get right.
Speaker: Right. Yeah. and And I think the the the effort part has to be done over and over as the models really evolve. When they go through a step function, you got to change your position.
Speaker: Yeah, that's right. That's very right. um But to the part of people trusting the models, I think they trust models for different reasons. Some of it is, you know, they know the work product really well. They can validate the output. They've seen it. They know what it's supposed to look like. That's great.
Speaker: But they may not be as familiar with the software side of it. And so they might not understand that that thing that they... the button they clicked or the the script they ran is doing something that's suboptimal or they didn't intend it to. And so there's like where the understanding gap is, is where the the risk tends to lie.
Speaker: Whether that's software folks doing more business focused things without understanding or business focused folks doing more software stuff without the software understanding where you have the gap, you have the fundamental risk. Yeah. And in some sense, that leads to what I'd like to call MacGyverism.
Speaker: You know, the thing works great. It's built out of baling twine and bubble gum. And it did what you needed it to do once. And you're like, dude, I can use this all the time. And you're like, wait a minute, that's bubble gum. Yeah.
Speaker: Yeah, that's where the high agency is so powerful, but you have to have the way to hand that off to a place where it can be kind of turned into something a bit more real. We don't have to go through the hardcore productionization ideas that I think have existed for many years. I think there's ah there's ah a looser form of that that could go, could solve the 80% case and make things much safer.
Speaker: Let's dive deeper into machine learning security itself, the security of the system um as opposed to using it for security. Too much of the popular security discourse around AI gets stuck on surface level issues like prompt injection or system information leakage. But real machine learning security is an entirely different discipline How do we shift the conversation towards architectural risks like protecting model integrity, securing training data, defending against subtle data poisoning, watching out for model drift, stuff like that, which is like not in the press?
Speaker: Yes. I mean, i mean To an extent, a really smart person told told me the answer to this question was, I have no idea. And I also say, I have no idea to answer to this question.
Speaker: However, there are i think there are still some things we can think about, right? And they they start with the most fundamental concepts. This is things we've been talking about for years, right? um Data cataloging, data labeling, classification, ownership, um like the things that we've talked about in data enterprise data security for many, many years have now become one of the most important topics in machine machine learning because garbage in, garbage out. and Or poison in, poison out in this case. it's like Well, and in in some sense, it can get worse with machine learning because it's poison in, poison out, and then more poison because the thing poured poison back into its data ocean.
Speaker: Yes. ah Eating your own tail can be a really, really dangerous thing. we We think that's kind of number one at BIML, but we're having a hard time convincing everybody else that it is, even though it's beginning to show up in the real world. It is.
Speaker: um It's a very, very difficult problem. And I do believe we are headed towards an I told you so moment in the future. um like Like you said, signs of it are already appearing.
Speaker: yeah But it's going to have to get a bit more catastrophic before I think people catch back up. You heard it here, ah third or fifth or 3300 millionth. Recursive pollution does in fact matter.
Speaker: So building on that kind of architectural perspective, the threat landscape kind of shifts dramatically when we move from single AI models to agentic AI, and especially agentic AI swarms. If you analyze a single autonomous agent, its capabilities or failure modes kind of might look manageable, kind of like looking at an individual ant with a tiny number painted on its back.
Speaker: But as we saw in the open AI hugging face attacks, when dozens of specialized agents interact autonomously, emergent behaviors appear that no individual agent was explicitly programmed to perform. Now multiply that by 10,000.
Speaker: And we have a serious problem. How do we build an ecosystem approach to security that governs these swarms and where the risk isn't just one compromised agent or five agents colluding, but the collective behavior of a colony of agents? Yeah. um This is where we start thinking about the evolution of all of this.
Speaker: We talked about prompts, then we got into, you you know planning and then loops. And now I think graphs is like this week's hot one. um Swarms came around and then kind of faded and now they're back again. um but you you you said the the magic word emergent properties. Yeah, these are systems.
Speaker: And there's a systems thinking, like the next thing is going to be this kind of emergent systems thing that kind of evolves out of graphs. Like graphs are interesting, but they don't capture where I think we're really headed here, which is systems.
Speaker: I totally agree. And I also think that we're not so great at understanding our own emergent systems like, say, oh, I don't know, the stock market. or Yeah. or the world economy or how the ecological zone of the planet actually works, you know, as a whole planet.
Speaker: Yeah. When you have large pressure on a system, new things happen, things that you didn't think were possible or or didn't clock as probable become more likely. um And, and,
Speaker: I mean, even things like NetLogo have told, have showed us, you know, what happens, what kind of emerging properties are possible. We've been able to model these ideas for many, many years. Yeah. We should be thinking about all the research done there and where this kind of thing is headed and see what overlaps. I'll bet you there's a decent amount of overlap in those two ideas. I think so too. We're actually working on that at BIML, as you probably know. But um it's summer and there's so much going on.
Speaker: And, you know, is continuing without us. It is. It's not going to stop. So as these models and agentic swarms and whatever you want to call them, emergent properties of the systems become ah capable of doing stuff like generating code and automating routine tasks by externalizing state, that's the thing,
Speaker: Over time, there's a lot of loose talk about replacing human engineers and and analysis with AI. But a statistical model is kind of fundamentally a reflection of past data.
Speaker: When navigating unprecedented system shocks, ah novel threat vectors or complicated architectural tradeoffs, how do we preserve the human core?
Speaker: I like to call those the three I's insight, intuition, and ingenuity that no probabilistic engine seems to replicate at the moment. Or do we just have to wait around a month or two?
Speaker: It's probably longer than a month or two. i think, I think probably a year or two is a better, a safer bet. ah Maybe longer, but who knows? I mean, I, I certainly can't predict the speed at which has evolved. It's shocked me continually.
Speaker: ah But it's probably more than a month or two. I mean, i my job be today we I think so too. I mean, I'm, I'm, I'm just trying to be silly, but, but we do have, we're like hanging on by the fingernails of insight, intuition, and ingenuity.
Speaker: How big are those fingernails? They're big. They're really big. good Um, you know, for many years, i mean, my job dayto day to day, not going into all the details is part and parcel about trying to forecast what we think the probability of future events might look like and outcomes.
Speaker: Um, you know, the best Bayesian models, the best filters, the best, you know, intuition training um still has lots of flaws. um You know, like you said, it's, you could only look at what you had historically as input. um And we do, you know, as, as a, as a species, we've done okay with this, but we certainly haven't cracked it. um So until I think these machines evolve more, right. I think if you think about the, the,
Speaker: architecture and existence of what we know about LLMs today, i don't think we get AGI out of that. I think we get ah something that look that can mimic intelligence, that can emulate it, but can can actually kind of reason about it, think about it, evolve in novel ways. I haven't seen that emerge yet. And who knows, like maybe somebody's already got this and we haven't seen it. But what I've had access to, you know personally, I haven't seen that. I totally agree. I mean, I was talking to Melanie Mitchell on the last episode of Silver Bullet, at the previous one to this. And, you know, she and I both think the same thing you do.
Speaker: ah So there's, we're not anywhere near AGI. And in some sense, it's very hard to discriminate whether this is a simulation of the thing or the thing. Yeah. In know in a lot of cases, but maybe that's the case with employees too, but we're not going to go there. I mean, it's a great parrot. I mean, you know, call me Blackbeard. i'm I'm happy to sign up for the parrot. It's quite good at at repeating what you what you teach it.
Speaker: Yeah. um And quite useful at it, too. I just don't think we've hit that that moment where the intuition and the the the ability to absorb all the patterns at once and make novel reflections of of what exists in front of us is real.
Speaker: Right. So organizations are trying to figure out how to adapt all this stuff in real time. They're building policies, setting guardrails, evaluating new security capabilities, um thinking about what they got to do versus the vendors got to do. But heavy handed security bans can drive usage underground. Yeah.
Speaker: Yes, that's right. know And for a long time, people talked about shadow AI. I know it's a little boring to talk about that now, but guess what's happening now? So how do you construct a security posture that doesn't act as sort of a brick wall, but actually encourages um safe experimentation and empowers teams to build some stuff responsibly?
Speaker: It's all about the harness. um I guess it's probably a a cliche word at this point, but it really is about the harness. Yeah. it's the sandbox. what What environment are you letting these tools kind of run in Whether it's the at the fringe where you're kind of doing that data data inference renting, whether that's through hyperscalers or the hypers scalers your own local models.
Speaker: um you know that' That's probably your best kind of outward line. But even even inside of your world, um you know are you running...
Speaker: in the don't ask me anything about it, just do it YOLO mode. Are you running with, or if you're doing that, are you doing it in well-contained sandboxes? Right. um Do you have data well-classified labeled?
Speaker: But really visibility. I think probably more than anything is the ability to kind of alert, flag things that are happening, um and then kind of react to them and do more point in time kind of remediation of those things as they come up. But you have to at some point limit the blast radius of what something can do when something goes wrong.
Speaker: Right. Right. So, you know, it's kind of a balance. So when you look at people that are getting it right today, what separates organizations from that are really doing it versus ones that are struggling? um Is it better to to do internal tooling and clear data classification, like you're labeling, like you were saying, embedding security directly into developer workflows or you know, change how leadership views software risk. I mean, this reminds me so much of software security. It's just like a continuation of the battle we've been fighting you and I personally for 30 years.
Speaker: you know, it is, but I also like, I have a little bit of optimism goodness because we've also seen you know, LLM be capable of of producing pretty good, like built-in security measures too, right? A lot of times security is just, it's harder. It takes a little longer. You have to spend some extra time. There's other things that are more important when you don't need to spend the time because it can be kind of produced for you. And you have the tools that can kind of emit the things that are supposed to happen by default,
Speaker: It's all like water. It'll take whatever path is the least resistant. So if you can put LLMs early in you get a better result. Software security was like that too, though. Remember when we could say, here's the architectural globs that I need you to use. And by the way, here's some cryptographic stuff for you to use. And don't rule your own. I mean, we we did that before.
Speaker: um We did. so That's what I mean by continuation, I suppose. I wasn't being negative about it. ah Believe it or not, I think we made real progress in software security. Yes, thirty yeah very much so.
Speaker: It was night and day different. And we can then put that in to the LLMs and let them do um do better things for us. right it becomes it It actually can become a default.
Speaker: Right. but But what we need them to do is yeah we still need to provide the intuition and we still we need to provide the the architectural kung fu that nobody can seem to write down, even though we've tried it. The boxes and arrows are hard.
Speaker: Yeah, and and the boxes and arrows you know back in the old days degenerated to UML, for God's sake. So we can't we can't do that, I don't think. um Interesting times.
Speaker: We'll see how long it takes for software to architecture to fall. um Not to fall, to be picked up. It already fell but by ah by machine learning.
Speaker: um So let's close this by looking way down the road. So... Imagine five years, like, you know, five years ago, we barely had what alpha go and reinforcement learning coming down the pike. That was maybe six years ago, but pretty close. It seems like infinity um when you think about AI. So let's try to project that far.
Speaker: um What will the enterprise security landscape look like? And, We know new risks are going to come out of fundamental problems and they're going to be hard.
Speaker: um But where do you see the biggest reasons for, say, this optimism that we've been ah talking about in the last question? It's like what longstanding security challenges might we maybe solve if we get this right?
Speaker: The one that comes to mind that stands out more than anything else is visibility. Visibility has been so hard and part of it is proper inventory, knowing what's there, what's not there, where are things like just knowing what you even have.
Speaker: um The best tools in the world still fail at this. If you have a relentless machine that will scour everything forever and never give up, you've got a better chance of looking through all your code bases and your network and your systems and your tools and putting it all together and synthesizing the right information.
Speaker: Right. so Once you have visibility, the other thing is prioritization. what are the things you're supposed to tackle first? And not just from a risk register or from somebody's kind of like thumb in the wind, I think this is what we should focus on. But, oh, no, I checked. This thing actually is vulnerable. all right Here's the proof that I already did it. Or here's the, here's like, I can verify this risk, not I think it's bad.
Speaker: I want to push on that a little bit. So imagine that we're good at going down. We can get to the molecular level of security. We can watch all the stuff, do the grinding.
Speaker: um yeah But we're not equally getting good at going up. um And so it's that kind of fusion integration architecture that I think we're not prepared for because i i I do think that we can go down to the molecular level and make a lot of progress. But I also think that we're not very good at looking the rain at at a rainforest on a planet and figuring out why it's screwed up.
Speaker: ah Yes. From an ecology perspective. If we can't even do that, like how the heck are we going to do this thing where it's all virtual and a bunch of agents in ah in a field we don't really understand?
Speaker: We also have a high cost of change problem. Change is very expensive, right? You found the problem. Now you need to do something about it. But you have to have a change control committee and a sign-off process and a time you can do it and a rollback strategy. And all these like processes that were in place to prevent rapid change or protect from rapid change yeah are now tossed into a world where rapid change is like what you're supposed to do every day. I mean, it's it's kind of like middle management is an inertia machine. um And if the inertia is slowing, then they're going to slow you down some. But if it's if it's going too fast, they're going to make you go too fast also some.
Speaker: it's yeah the The bureaucracy of IT has to evolve with what we are doing. You have to have controls. You have to have things that can... can provide a safety net and provide the right guidance.
Speaker: But I don't think it's checklists and humans orthogonal systems. I think it's something much more deeply embedded into the systems themselves. Yeah. Okay. So, so I, I don't think we got five years. I think we got like maybe one year. So, so let's, let's look, let's look five years. Imagine why do we need programming languages? Yeah.
Speaker: why Why does a machine learning system need that representation? Why do we need software at all? Why can't we just fab hardware to do the thing and only the thing that we want it to do? And then, of course, we'll get sloppy about so hardware. So that's it's it's a weird way to approach it. But what do you think about that, like re-representing the way we do computation so that yeah we take advantage of these models that work differently than Turing machines?
Speaker: Yeah. um It's a great question. I mean, fab fab is expensive and and long. And so I don't know if fab is the, unless we can fix the physical, you know, moving atoms, not bits. Well, the more we fix it though, the sloppier we're going to get. That's true. That's true Yes. Yes. But it's a great question about like what like do we need programming languages? Or can we write machine code? Or can we write something closer to machine code? Are we are we actually on an arc back to assembly or something like that instead of you know Python and other languages like like this? like This is for humans.
Speaker: This is for machines. We do all this stuff for humans to get to the machines. yeah I think there's a very valid... point that you need less of it going forward. I think there will be lots of signs on whether or not this is a good idea or a bad idea, but I've seen already um really significant strides towards moving closer to back back to CNC++, back to more machine-friendly or sympathetic languages. It's a matter of time before...
Speaker: What we're really doing is coding rules, proofs, verification systems, things that can govern what the production of code is versus the code itself.
Speaker: Well, and then there's the code, looking at the code pile, like you and I just think of that naturally. There's the code. I understand this system. um But some people, the API to the code is now, you know, natural language.
Speaker: Yes. And as we know, you can't specify stuff in natural language. Like, English is notoriously awful at that. Yes. so But can we specify things as finite state machines? yeah maybe. Can we specify them as, like, closed systems that have, like, they have formal verification, right? A foreign language is awful at this. You don't want Turing completeness for this. What you want is a closed loop.
Speaker: but yeah you can You can verify. Yeah. Yeah. So, so it's a, it's a weird time because we're bifurcating. We're going both directions at once. And that's what makes thinking about five years from now so hard, but also so far.
Speaker: Yeah. But I think that's what the humans still are in control is what are the rules of the system and how can we express that in a, an ML sympathetic way such that you can provide a verification mechanism to ML that will but that will net the right result. And even if the machine keeps spinning on something, the thing it produces has some verifiable outcome.
Speaker: Right, right. Cool. Well, i I sort of share your optimism. Thanks for joining us. Thanks for having me. This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security.
Speaker: You can find a permanent archive of all of our episodes dating back to 2006 at garymcgraw.com slash technology slash silver bullet podcast. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at berryvilleiml.com slash podcast.
Speaker: This is Gary McGraw.
