Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
158 - Artem Dinaburg
122 Plays8 days ago
On Episode 158 of the Silver Bullet Security Podcast, BIML’s Gary McGraw hosts Artem Dinaburg. Artem talks about using Agentic AI to find, fix, and exploit software security defects. We talk decompilation, stochasticism and the tension between harness development and LLM nondeterminism, human intuition, and the hard parts of program analysis. We also talk about MLsec (machine learning security), recursive pollution, and tokenization. Do we need more people in software security in the age of AI? Yes we do.
Recommended
Transcript
Introduction to the Podcast and Hosts
00:00:11
Speaker
This is the Silver Bullet Security Podcast with BIML. I'm your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses on machine learning security.
00:00:28
Speaker
For more, see barryvilleiml.com slash podcast.
Artem Dineberg's Role and Insights
00:00:32
Speaker
This is the 158th in a series of interviews with security gurus, and I'm pleased to have to meet today, Artem Dineberg.
00:00:40
Speaker
So thanks for joining us, Artem. ah Hi, Gary. Artem Deneberg is the chief scientist at Trail of Bits, where he leads research and security engineering products for projects for government and commercial clients.
00:00:55
Speaker
Artem has worked on a large variety of projects, ranging from low-level software development to vulnerability research, reverse engineering, malware analysis, and program analysis slash transformation. His core interests sit at the offensive and defensive intersection, automated vulnerability discovery, exploitation, and patching. These days, much of his attention is on a topic now familiar to the whole industry, how AI best can be used for automated bug finding, reverse engineering, and decompilation.
00:01:26
Speaker
Artem holds BS in computer science from Penn State and an MS in computer science from Georgia Tech.
AI in Software Security Automation
00:01:33
Speaker
So let's start with the reality of software security automation.
00:01:38
Speaker
Over the last year, Trail of Bits completely overhauled its audit methodology, transitioning from widespread internal skepticism about AI to actively deploying an arsenal of dozens of specialized autonomous agents. and hundreds of customized skills.
00:01:55
Speaker
On the right engagements, your AI augmented engineers are now uncovering upwards of 200 bugs a week. So from your perspective as chief scientist, what's the fundamental difference between treating AI as a typing assistant versus weaponizing sort of a proactive autonomous agent loop to dismantle a code base?
00:02:16
Speaker
Essentially, you are delegating work, but you are delegating work to, you know, not to humans, but to, as you said, autonomous agents. And these autonomous agents are very good in certain domains, and they can cover a lot more ground than what you could as a human. There are certain categories of vulnerabilities, there's you know that we Obviously, we've seen you know with the recent releases from Anthropic and so on that AI agents are just extremely good at finding.
00:02:45
Speaker
And if you can you know unleash this on a code base, you can cover a lot more ground very quickly and focus your human evaluators on assessing higher level properties that need to hold that may require domain expertise that, um you know,
00:03:05
Speaker
an AI agent simply just does not have because it does not understand the business context where the software is running, the threat model that its creators are concerned about, and you know maybe something that crosses different abstraction boundaries that you know you as a human can think more concretely about.
00:03:23
Speaker
Yeah,
Shifting Focus in AI and Security Skills
00:03:24
Speaker
that makes sense. I mean, hopefully it will move the humans towards the design level in some sense, which has always been the hardest aspect of software security in my view. ah Yes, I think you're absolutely correct. you know As more and more mechanics of code auditing get automated, the bottleneck becomes other security skills, you know the ones that usually you know in base your basic intro to security course, now you you know you emphasize compartmentalization, you know secure design, isolation. and Then most of the things are kind of forgotten about and everybody's like, yearss now here's your you first project. your end the What you struggle on is you know mechanically writing a lot of code.
00:04:06
Speaker
Well, you know now we've solved that problem and the bottleneck has become, again, what is the proper security design for the problem you're solving? you know What threats are you concerned about and how are you going to mitigate these issues?
00:04:20
Speaker
Cool, I love that. That's a great answer. um Much of your research focuses on using LLMs to reverse engineer and decompile binaries. When you ask a statistical model to reconstruct control flow graphs or guess variable types or label stripped functions, are you actually extracting the original architectural intent of the software?
00:04:44
Speaker
Or are we in some sense generating a highly plausible semantic veneer that might make a human auditor's job harder? Like, is there some of that or am I just crazy?
00:04:56
Speaker
Oh, so I think this is ah a fascinating question. It touches deep into a lot of what I've been thinking about, but I don't necessarily have the data to completely validate my thoughts yet.
00:05:07
Speaker
um So I guess of to to step back a bit, if you know you said translated to a control flow graph and so on. And I think that you know this works for you know algorithmic-based reverse engineering and decompilation. This has worked great. And especially for you know the C family of languages. This has worked wonderfully.
00:05:23
Speaker
ah But if you notice, a lot of the current top tier automated reversing software or manual reversing software really wants to reverse everything to see. ah But this has not been the reality of the languages that people are developing. in i know your' you know If to reverse engineer you know of Rust or Go or Swift or you know one of the modern HLLs, you will get you know C that looks like it is pretending to be rust, which I think is currently worse than you know looking at bad rust. Believe me, I i came up in the C-Tran days. So I know exactly what you're talking about.
00:05:56
Speaker
um And you know my hypothesis that we can do we can do much better. um i So you know let's say like if the let's talk about you know a language translation problem that know we're more familiar with. Let's say you know you're translating you know English to Japanese or Japanese to English. you know You do not make a sentence diagram, reconstruct every sentence, every word, verb, translate it to Japanese and rebuild your sentence. you know This is kind of you know how more traditional decompilation tools work.
00:06:24
Speaker
ah you know you Yes, sometimes it will work. Occasionally you'll be able to get the message across. Maybe it'll work for certain classes of phrases, but you know modern models, you know they have an enormous amount of Japanese-English pairings. and you know the machine The machine model works at statistical magic, and out you get a translation that's you know really good and idiomatic.
00:06:43
Speaker
And you know why can we not do the same thing for effectively, you know let's say you're translating ARM64 to Go. yeah no you You have the same thing, I know, because you don't really need...
00:06:56
Speaker
you know You do not want to think about translating you know one function at a time or you know parts of a function and trying to recover the idiomatic Go language from that. This is you know possible. This is an extremely difficult problem, which I'm not sure it's even solvable because you know that original Go is gone and you're never getting it back.
00:07:12
Speaker
like you know It's went through several layers of intermediate languages and outcomes, you know some binary, which has nothing to do with necessarily the language semantics that created it. But, you know, you could take this neural translation approach and actually get some kind of normal, you know, Go or Rust or Swift or what have you back. And I think this is very promising, especially because, know, you have you have a compiler, so you can generate you know immense amounts of training data, and you have an oracle. you know, you take the thing, it just decompiles, and you build it, and you compare at the binary, and you have some level of binary similarity you can measure, and he'll climb. And so I think this is a very, very tractable problem for,
00:07:51
Speaker
you know, AI, ML and agentic based systems in general. And, you know, I think the reason it hasn't been solved yet is because simply, you know, it needs a whole bunch of work to put into solve it. You know, um and then eventually but eventually it'll it'll get solved. And I think it will be,
00:08:07
Speaker
much better than you know what humans currently do on HLLs. And it'll you know help kind of ease the gap between source auditing and binary auditing because you will be able to have some kind of quantifiable measurements with how close you are to the source code.
00:08:22
Speaker
You know, I think that's interesting, but I also think there's a real possibility we will one day realize we don't really need that source code anymore. But also very possible we'll see what comes first. That's that's too speculative to put anywhere but a podcast. So in traditional program analysis, we rely on sort of rigid compiler rules, deterministic oracles to prove that a bug exists.
00:08:47
Speaker
um If the core engine of modern automation is a stochastic guessing machine, the agentic harness is supposed to, in some sense, be our rescue mechanism, the deterministic wrapper that validates and executes and grounds those probabilistic guesses.
00:09:03
Speaker
How do you design a harness that's tight enough to enforce correctness on a non-deterministic model without completely suffocating the creative leap, or maybe more accurately, the stochastic generalization leap we want the AI to make in the first place?
00:09:19
Speaker
I think that's a fascinating question. And you know this is genuinely a you know ah difficult problem. you know If you do current you know AI generated bug finding, and you know we see this in our audits and we've seen this from like complaints from maintainers of various open-star software and so on, like, you get a vast amount of false positives. It wants to want to find critical bugs everywhere. And you need some kind of gate that, as you said, grounds us in the real world. um
00:09:53
Speaker
Typically, there is you know no better proof than actually executing a live deployment of whatever software you are running against and seeing the know the issue come up in real time. Sometimes this is easy when you know you can prove that you've exfiltrated some kind of in a flag in a CTF style setup or you know there is a segmentation fault and you have an obvious memory card. But sometimes this is actually very difficult. because you know maybe the issue is some kind of race condition only triggers occasionally and just didn't trigger it ah maybe the flaw is actually very difficult to understand and i mean currently right now at this point the solution is you know throw it back to some humans who think really really hard about it and try to determine whether this is indeed a problem and i know well at least we'll have jobs Yeah. Well, currently, you know that's one of the things that is you know where we are you know our minds are still you know produce better results than ai But I think you know there is no fundamental reason why you could not think about this at some deeper level or create some kind of tooling to help you think about this. like
00:10:58
Speaker
you know We have lots of there's been you know lots and lots of work done modeling distributed systems and you know normal computation. And a lot of the reason why you know we don't nobody does you know verified programming to begin with is because it is extremely tedious.
00:11:14
Speaker
the The development pace is really slow. And I know think there's ah you know a very good argument to be made. ah you know Had we went down this path in the very beginning of programming, you know if let's say you know Basic came with you know mandatory validation in the interpreter. ah Maybe if everybody look if you're kidding if everybody learned to do this, maybe you know we would have been in a much different place off the architecture wise and everything would have been a lot better. But you know that's not the way things ended up. And in in the reality that we have, you're going to lose development velocity to somebody who just does not have these constraints.
00:11:48
Speaker
yeah But with AI assistance, you know you don't necessarily have to lose a velocity with these constraints. And in fact, forcing your AI to work in a much more verified development mode helps prevent you know slob. It helps ground what it is actually trying to do. So know this is a fantastic use for you know those vast amounts of you know verified programming. like you know I think there's lots of stuff from like Microsoft Research about this that all of a sudden has very renewed relevance. yeah And now that you know the mechanics of writing code have been automated.
00:12:23
Speaker
Fantastic. So we're all going to become formal Methodists, which I've been trying to avoid my entire career. so hopefully i'll I'll retire properly before that. Well, I mean, you ideally, you know, you wouldn't, sure, you wouldn't you would need an understanding of how it works. Like clearly, you know, you will need an understanding of it, but you will not have to, you know, do the check the arity of each function to make sure that, you know, try to understand, know, obscure 500 long error messages about why your code doesn't validate. Ideally, you know, the AI does that.
Securing Complex Machine Learning Systems
00:12:52
Speaker
ah All right, let's turn to the security of machine learning itself. um When traditional security engineers look at machine learning, they usually try to map it into familiar mental models, like treating it as a standard application component with an unstructured API. or maybe as an esoteric pile of statistics. As we transition from hunting classic software bugs and flaws to auditing complex ML systems themselves, what's the fundamental architectural shift that engineers miss?
00:13:24
Speaker
And why are we sort of failing to isolate data from control flow in our standard designs today? ah Yes, again, another very fascinating question. um So there's there's several layers to unpack here. ah you know The data control thing that you mentioned, you know I feel that every decade has its, you know we put our control in the data, you know starting with the phone system to cross-site scripting and and so on ah Now, you know we have another decade where we are going to do prompt injection, where we put control inside the data. there's enough I think there's been some research to try to mitigate this to some degree where you know you have certain control tokens that you can reject at input time. And you know i've the model says, once I see this control token, I will stop processing this. And you know you have some kind of basic pre-filter that does not allow control tokens in your input at all. But then again, you know you have this arms race of how can you sneak this control token back in? Maybe you can work around this. So I feel that we will be stuck with this for a while without and this is necessarily a good, clear solution. The second aspect, you know how do we treat these systems? I think this is actually you know very fascinating in that
00:14:38
Speaker
with the first not the for look Some of the first places we're going to see a lot of ai use by default is inside existing workflows, ah but almost by definition, because you have lots of existing systems. you know You are going to plug these things into an existing workflow and you know with the obvious benefits of you are automating something that is very complicated that previously could not be automated and there is a bona fide business need to do this.
00:15:07
Speaker
ah there needs to be a very thorough revaluation of the security model you're doing this because you are going to have a thing that can take action and it is going to take untrusted input and it is going to be able to effectively, if right now you have to model it as, you know it can do anything that an attacker tells it to do and you have modular whatever external controls you have. And like, you know the latest example of this is the whole Meta Instagram support bot. where you could tell it to reset the password for you know if somebody else's account and it would bonu gladly do it because you know it is in its job to reset passwords. It is, I'm sure, saving you know a substantial amount of know support costs for what are effectively free accounts. this is you know Business-wise, it makes sense. You want it to have this functionality, but you know you did not carefully model the fact that anybody can ask for somebody else's password to be reset.
00:16:03
Speaker
And I think you know we're going to see a lot more of this in various contexts because the business case for putting these agents into legacy systems is very, very high.
00:16:14
Speaker
Like you definitely want to be able to do this. people It's not just you know that you know somebody wants to save some money by cutting employee costs. People want this because it's legitimately very helpful. They help you accomplish the thing you want to accomplish 24 hours a day, seven days a week. Right. There's there's a bunch of studies that you know obviously you know these are all come from companies that sell this, but I tend to believe they're correct where people prefer AI based support staff if they're not told it's AI based.
00:16:46
Speaker
Right, right. it's more helpful. I mean, it's more helpful. We've always had that trade off in security engineering between functionality and security. um And in some sense, you know the pressure on the functionality side is just a little bit higher now. So that balance is going to teeter totter, it's to shift.
00:17:05
Speaker
um I want to get into something slightly more technical. So when when people talk about the data corruption feedback loop, they sometimes use the terms recursive pollution, that's what we like at BIML, and sometimes they use model collapse um in academia interchangeably, as if the only thing that matters is the dramatic sort of in-state catastrophe.
00:17:28
Speaker
um We've been looking closely at the distinction ah and we believe that a model doesn't have to completely implode to become a liability. It just just needs to absorb enough of its own distorted output to degrade over time, often quietly.
00:17:44
Speaker
From an architectural defense perspective, how do we build an assurance framework to detect early stage recursive pollution before a system drifts into full blown collapse?
00:17:57
Speaker
I think at some point, you know you have to have grounding in bona fide facts that exist outside of the model that you have validated somewhere else. This is you know even obvious when, not obvious, but this is like and something that exhibits itself whenever you are trying to do work ah doing, let's say, source analysis on one hand, but honestly, like any kind of AI use, if you do not have empirical facts to ground it in, it will you know hallucinate, it'll make things up. And as the model gets worse, it'll hallucinate even more because you know it is continuing to operate. It loves its own hallucinations, of course. yes. from me so that you know Yes, it it does. um
00:18:44
Speaker
And so some kind of grounding is essential. And we see this in software security work, whereas this is kind of a symptom of you know it'll find high severity issues everywhere. And unless you have some kind of grounding in actual execution where feasible or some kind of Oracle that operates outside the model, you know you will find high security issues everywhere that don't actually exist. Or maybe you know the code isn't like and a lot of times this is something we see the code isn't even wrong.
00:19:09
Speaker
But it is actually impossible to reach the path because it requires you know two conflicting hardware versions to be operating at the same time, which never happens in the real world. But you know it is impossible. you know This is outside of the model's ability.
00:19:24
Speaker
and I know, I think in the computer domain, this is at least models usually quite good because they have been trained and there's a lot of reinforcement learning that happens on how actual computers work. And they're very good at this. If you shift domains to other things, they actually get considerably worse because their intelligence is you know jagged ah One thing I encountered recently in my personal life is, you know if you ask the even mythos, like not mythos, mean fable, I guess that's the the myth. like ah If you ask even the fable model ah how to get a 2003 Ford Crown Victoria out of park when the brake lights don't come on, it will make something up that is completely not real.
00:20:05
Speaker
ah And then if you tell it, you know hey, you know this actually is not in the car, it will take you down a path where it has you disassembling the steering column. where Whereas the real answer is you know on dozens of forum posts and in the official PDF manual from Ford that you can go and download and use. So the answer is ground. There's a grounded answer. It just you know will refuse to use it and it will take you down a completely different path. and I'm sorry to hear that this is a real example.
00:20:35
Speaker
ah So most AI people treat tokenization as a mundane pre-processing step, completely overlooking its deeper architectural and philosophical impact.
00:20:47
Speaker
By slicing inputs into arbitrary statistical chunks rather than meaningful semantic units, tokenization fundamentally decouples the model's emergent internal representation from the reality it's supposed to be processing. This is kind of getting towards grounding a different way. From a white box security perspective, how severely does this translation layer distort our ability to reason about system behavior?
00:21:13
Speaker
And can we ever truly secure an architecture whose very ontology is built on these crystalline non-human representations? ah That's a very good question because tokenization has been, you know, the root of evil for a lot of different problems.
00:21:31
Speaker
ah I think this is... ah a very, you know, kind of a difficult problem because, know, I know there's been lots of research about, for instance, doing some kind of byte level understanding, ah usually aimed at doing, you know, image processing or processing different kinds of binary structures. You know, I feel that my, know, I'm going to give an answer, but I always, I feel that my answer may be inadequate because there's a lot of frontier research here that I'm not very familiar with. And I feel like even if I had read it, I would not,
00:22:02
Speaker
produce a great answer. But, you know, in the model training that have attempted to do, I have run into the problem where, you know, you have a token, the token stays under trained because it's not appear enough in your training data. You know, we saw this, know, with the back end, you know, GPT three, three and a half with, know, solid gold, Magikarp and so on. You know, it's just, well, uh, trigger very, very strange results. I ran into this doing some local model training for some specific task where I accidentally left some tokens under-train and I realized my performance was horrible. One of the lessons, build evaluations first and do training, then check to make sure you're better than baseline.
00:22:40
Speaker
and know The reason was I had under-train tokens. yeah and you know i i do not know you know what tokens and frontier models are under trained how much this is affecting hallucinations uh how easy it is to trigger them or you know what potentially the impacts are of leaving certain states unrepresentable because they cross token boundaries you know or you know you you know, you have this pair of tokens and you need a pair of tokens to represent something or maybe more than a pair. And, you know, this particular pair has been not appearing next to each other very often. So would actually be very hard for the model to represent the thing you want.
00:23:22
Speaker
yep I know I think there's going to be certainly an emergent class of, I don't know if going say attacks, but ah things that cause undesired behavior in production AI agents that are possibly going to be based on you know looking at this and figuring out what kind of you know what can be insert that is going to make this difficult for you know difficult to process?
00:23:51
Speaker
Yeah, I mean, and it's it's an interesting area of research for sure. And I think that it's it's pretty much... you know generally ignored by most people working in the field so far. We need to fix that.
00:24:05
Speaker
I think, ah yes. so One, I think people making models understand that tokenization is a very, very difficult subject and have dedicated a lot of research to it. But also, you can't really... you know the The byte level schemes have not produced the same kind of performance that actual tokenization does. And you know you need to stay at the frontier and deliver performance you know in terms of model capabilities to your end users. ah In terms of you know security aspects, yes. I know i think there's many, many things, and this is definitely one of them that we have not really gone into and that reinforces the need to, again, have a
00:24:42
Speaker
proper architecture, especially if you're integrating one of these systems into an existing you know enterprise deployment of something, you know how are you going to put a perimeter around it and limit its actions and you know audit and validate its actions if you can't limit them? So at least you can go in the past and undo them.
00:24:59
Speaker
um based on know what you wanted to do and what you allowed to do. And going back to where we started, you know those very beginning security engineering principles of know auditing, logging, isolation, privileges, you know now they have enormously renewed importance because the you know actions happen at machine speed instead of human speed.
00:25:20
Speaker
And you need to make sure that certain actions are simply impossible. I will point back to some work we did in 2020 at BIML where we took Salzer and Schroeder's principles and we adjusted them for machine learning. um Check that out in all of your copious spare time. i want to um I want to kind of stick ourselves directly on the present, so into the immediate
Need for Human Security Professionals
00:25:42
Speaker
present. At at the Qualcomm Product Security Summit, you argued that we actually need more human software security people right now.
00:25:50
Speaker
not fewer, strictly because generative AI is allowing developers or whoever to churn out coded in unprecedented velocity, which I absolutely agree with. I used to joke that Gartner analysts could only ah understand one variable, bug cardinality, like whether that was going up or down. um while completely ignoring the two variables that actually matter, bugs per square inch and miles of new code. um If the AI firehose is just massively amplifying the miles of code we have to deal with, how do human security teams scale to find the architectural flaws, the the ones up the food chain, when we're drowning in an ocean of kind of AI-generated syntax, some of which we can automate away with other AI?
00:26:36
Speaker
ah Yes, thank you. i guess i do i have the i guess I don't guess don want to say counterintuitive, since at least I can tell it you agree with me, Gary. But it's certainly counterintuitive to some people who say that security is over. you know Why are you working in security? you know This has clearly got automated away you know you don't have any way. you know you should stop, you know do do something else. And I'm like, no. like ah The amount of code being generated means that there is going to be more bugs than we have ever seen. even if it can you know We are very lucky to that AI is good at bug finding because we're going to need it to find all the bugs in AI-generated code. yeahp
00:27:16
Speaker
the What we have to focus on is and this is you know going easier said than done, but the software development has to adjust to a much faster pace of code velocity.
00:27:29
Speaker
yeah And this means gating as many tasks as we can behind some kind of no machine checked or machine audited ah ah analysis and only pushing to humans what there is uncertainty about or what may require some kind of higher, as you said, higher degree of abstraction as to how to scale humans to be better at this higher degree of abstraction. I think, you know unfortunately, human we have not found a way to accelerate human learning. So that we have we have not found a way to do. But it's it's essentially just going to mean that changing prioritization and you know the kind of things that people who are entering security now, the kind of things they learn, are you're going to have to shift the focus. you know As you said you know previously in your security project, now you did a little bit of learning about policy, and then you had lots of learning about mechanically analyzing code and writing it to implement it. well, we're going to have to do a lot more of the learning about policy and how to assess this and how to you know interpret
00:28:34
Speaker
output from potentially automated systems and how to know when you know they are wrong and and so on. like We're going to have to change how we train. you know A lot of this ties into the fact, you what are we going to do about junior people entering security? like One, we're going to need a lot more of them. And two,
00:28:50
Speaker
yeah mean They're just simply not going to learn the same things that we learned. This is not necessarily a bad thing, you know fields evolve, but their knowledge is going to be valuable for where we are headed. and Where we are headed has a much, much higher development velocity than we had before, requiring different strategies to keep up with the newer volume of code.
00:29:12
Speaker
yeahp And you know I think we are seeing this, you know there's been an uneven adoption of these principles. you know I think a lot of, like if you look at some of the outputs from a lot of the frontier labs, like Anthropic and OpenAI, essentially they're saying, you know we use codecs or we use code code to you know write 80, 90% of codecs and cloud code. And you know we've increased our shipping velocity by eight to 10 times in the past month. And you know other places also do it. like know We have a lot of adoption internally and we can audit more lines of code and we can find no more bugs per project faster.
00:29:47
Speaker
and you know eventually everybody is going to have to adjust to higher velocity or be outcompeted by people who do. And this is you know going to require a shift change, both in software development and you know from perspective of people who do security, how you structure product and code security and what kind of things do you do. And yes, you're definitely going to need more people who who can do this because there's going to be so much more code. I know some of the
00:30:19
Speaker
know I'm sure you can remember you know the first time you wrote a computer program and you know the joy you get out of having the computer do something. You're not going to believe it, 1981. Yeah, and to lot of...
00:30:32
Speaker
i've talked to a lot of um are you know non-audit non-development staff and they have been onboarded with you know cloud code and codex and they get the same joy now because they can finally tell the computer what to do and the computer does it and it is like you can like hear how awesome it is that they can like they had these things that they wanted the computer to do and now they can tell the computer to do it and the computer will do it for them and it is like it is an amazing feeling so Now you don't you don't have to learn this wizard language and you know interpret these spells that the computer tells back at You can just like tell it to do stuff.
00:31:09
Speaker
I love it. i love it i kind of want to I kind of push on that for the last question. So you know let's close with where this is all heading, um which is obviously agentic which we've been talking about the whole
Designing AI-Augmented Architectures
00:31:21
Speaker
time. So we're rapidly building autonomous agents designed to write code, find vulnerabilities, make system level defense decisions all on their own. But as we rush to build the mathematical harnesses and formalisms to keep those things from going off the rails, we shouldn't lose sight of what actually makes a defender effective.
00:31:40
Speaker
ah Machines are great at scale, but humans bring what I call the three eyes to the table, insight, intuition, and ingenuity. um When we look past the hype of fully automated systems, how do we design high assurance architectures that don't just constrain the AI, but actually amplifies those uniquely human traits?
00:32:02
Speaker
So I think that this is actually one where defense is going to get a lot of value since traditionally the problem with defense has been you know volume.
00:32:13
Speaker
There has been an immense problem with volume. Like you have, you sure you can log everything and lots of sensors, ah you can audit things, but you will get a large volume of data coming in. You will get lots of alerts from your entire network and being able to this is it very quickly get human cognitive overload. Even if you have an intense amount of filtering, if you have a large enough system, you get cognitive overload from what are false positives.
00:32:40
Speaker
Right. and ah you, I mean, now we are, have the ability to where we can offload this cognitive load to somebody else. ah in order to In fact, we can offload this cognitive load to a swarm of agents. And those, and I mean, obviously we can say, you know, there's going to be turtles all the way down, but ah we can finally make sense.
00:33:03
Speaker
Like there is a thing that, like sorry, I'm like mumbling all over myself, but, There is at least a vision for how we can take all of this vast amount of data and process it either in real time or you know at some delay offline and make sense of it and bubble up only the things that have been you know past several gates of review that a human needs to use their intuition on to see, is this actually allowed per the business policy? Right. But it nowken it also emphasizes the fact that that human intuition is really important. We can't just get rid of that. Right. Yes, i don't know I think finally, like what you will be able to do is identify higher level policy violations from lots of lower level actions, which before you know took forensic analysts you know maybe months or years digging through things after an incident. Now you will be able to ah achieve this much faster because you actually have something that can
00:34:03
Speaker
look at all these sensors and correlate this data and actually intelligently make some kind of decision on it in you know either real time or close time and bubble up something to a human to say, know we have detected motion of data from here to there to there and finally exfiltrating out of VPN endpoint. you know Is this a permitted workflow?
00:34:25
Speaker
And somebody can say, you know yes, totally. you know Gary's traveling to Tokyo to this week and this is absolutely expected. Or they can be like, no, no, no, no. no this is This is a terrible problem. You know, the press the red button.
00:34:41
Speaker
Well, this has been an absolutely fascinating conversation. Fantastic. Thanks so much. um I can't believe we've been talking for 35 minutes. it It feels like 27 seconds.
Closing Remarks and Resources
00:34:51
Speaker
This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security.
00:35:03
Speaker
You can find a permanent archive of all of our episodes dating back to 2006 at gerrymcgraw.com slash technology slash silver bullet podcast. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at berryvilleiml.com slash podcast.
00:35:22
Speaker
This is Gary McGraw.