Transcript
Speaker: This is the Silver Bullet Security Podcast with BIML. I'm your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses machine learning security.
Speaker: For more, see berryvilleiml.com slash podcast. And I'm pleased to have today with me, Greg Kurtzer. Hi, Greg. Hello. It's great to be here.
Speaker: Greg Kurtzer is the founder and CEO of CIQ and president of the Rocky Enterprise Software Foundation. A high performance computing pioneer with over 25 years of systems software experience, Kurtzer is best known as the original creator of CentOS, Rocky Linux, Werewolf, and Singularity, now obtainer under the Linux Foundation. which transformed containerized environments across scientific and supercomputing infrastructure.
Speaker: A relentless advocate for open governance, reproducible computing, and supply chain transparency, he co-founded the Open Enterprise Linux Association, OpenELA, and leads OpenWALDO to bring auditable data provenance and open source rigor to modern AI ecosystems.
Speaker: Kurtzer holds a Bachelor of Science in Biochemistry from Holy Names University. Hi, Greg. Hi. Greg, you've spent your career building transparent, high-performance open infrastructure because you know that systems you can't inspect are systems you can't trust.
Speaker: Yet today, the entire tech industry is racing to hand over core operational logic to uninspected uninspectable black box models. Why are engineers making the exact same foundational mistakes with machine learning that we fought to fix in early operating systems? And how do we stop the industry from sleepwalking into a complete loss of architectural control?
Speaker: That is a great question. um i kind of been describing it as... there's there's there's There's two populations that exist. ah there's the There's the open source group who historically have been building and managing and and securing a lot of the infrastructure that that runs the internet, that runs enterprises, that runs um ah personal devices, and um they've been directly impacted by how open source has made, and not to sound cliche, but made the world a better place. Absolutely. And through that transparency and through that um ah community and collaborative effort on standards and everything else that open source brings to the table, they've lived to see it. They've been part of that and it has made their lives better.
Speaker: On the other hand, on the other side of the population, we have a number of people that have been doing research in AI. And the research and work that they've been doing in AI has been incredibly and massively ah just amazing. It's been impactful. um The innovations that they've been able to drive has been fantastic. and And a lot of the tooling around AI is in fact open source.
Speaker: But when we go to create these models, We are using various bits of data where the data side of the coin has never been quite so clear when it comes to open source.
Speaker: Yeah, that's a very kind way of putting it So, ah and and and to build these AI models, you have to have a tremendous amount of data, a huge amount of data. And and you're you're training on this data and you're organizing this data in such a way such that the The end result is going to provide logical outcomes. And and so people are are kind of confusing a little bit where are the IP is and where the innovation is. yeah And so we have on, again, kind of one hand, we have um you know the open source people who are like, you know everything should be open source.
Speaker: If you're gonna call something open source, the source should be open, kind of in the name. and And the source for an ai is it's the training data. it's It's the tooling. It's the build process. It's how you're composing that model. um But I think a lot of people are actually really confused on this one because- Well, I was going to you know really address that. like here' Here's what I think. Let's just cut through the marketing around open source AI. Yeah.
Speaker: right, so a lot of vendors publish pre-trained weights and they claim they've delivered open source. But from a security perspective, an open weight model is basically a compiled binary with no source code and no build instructions. Yes. So you can't inspect how it was trained, you can't verify if toxic or malicious data was baked into the substrate.
Speaker: You can't patch it or recompile a model if a fundamental flaw emerges later. So why is releasing static weights without full training pipeline provenance, including raw corpora, filtering scripts, and RLHF parameters, kind of security washing? It's sort of an illusion. And and why does this missing history make real risk analysis impossible for down funnel engineers? So I want to like address the technical aspects of this more than the social aspects.
Speaker: That's a great point. So when you're training a model, the data that you're using to train that model is what is going to generate the weights and the probability and statistics of what the next tokens that it's going to complete are going to be. And if you are training it, a training and AI model on, let's just say the Terminator series,
Speaker: Like, don't be surprised when it believes that, you know, machines are great and it should exterminate all people because we are in their way. um like And i'm being somewhat... SkyNet is good. i Yeah, I'm being like sarcastic about this, but at the same token, it really kind of articulates the point. And if you train, if you have enough data on an entire fic fictitious world...
Speaker: and you train the AI model on that, it will confidently describe and respond to questions as if you know it is it is built and based on that fictitious world. Absolutely right. I mean, you know that the artificial data idea um sort of forgets to tell you that part.
Speaker: and So I think you end up with like a Potomkin world. Yeah. ah So you're going through, you got your data that you, that you've made you made, know, it's fake artificial data and you eat it and you build a model of your artificial data and you have a little pretend world.
Speaker: You can definitely do that. And, and the people who are actually you know claiming that models are now conscious all of a sudden um are exactly like, there's certain arguments you can, you can bring to the table. um Although, as you know, with, with many religions, it doesn't matter what facts you bring to the table. it' I love it. They're going to believe it and they're going to believe AI is conscious. It's going to do what it's going to do. But but here's here's where this is really important. And it's the point you were trying you you were making, which is if you train the model on bad data, let's say, for example, you're building a coding model and that coding model has tons of code examples, all with similar backdoors. Mm-hmm. in the code and you train your model based on all of this data.
Speaker: Now, I'm going to say it's a hard thing to do because data is you need so much of it to train a model. yeah But if you take that, that, that code, those code snippets, those code examples, which are trained with malicious code in, in, inside of it, don't be surprised when the AI starts injecting some of that malicious code of into your own,
Speaker: projects that you're using it for coding. And if we don't have the capability of looking and auditing you, there's no way to go from the weights backwards to figure out what was the data it was trained on. Yep. You lose provenance. Yeah, that's, ah i'm I'm totally with you. um Do you know Dave Evans work on Trojan stuff from the machine learning security pile? i need to go back and research that some more.
Speaker: It's a good one. I think you would like that paper. So let's talk about data pollution. We're rapidly approaching a tipping point where the open internet is flooded with synthetic model generated text and code and media and everything else.
Speaker: And as new models scrape this synthetic noise to train future generations, we run straight into what we call epimel recursive pollution. which is a feedback loop where statistical artifacts and bias and poison get exponentially amplified in every training cycle.
Speaker: if and it eventually leads to model collapse, which is what some people call this phenomenon. If our computational substrate is being continually contaminated by the unvetted outputs of previous machines, how do initiatives like OpenWALDO actually wall off clean training corpora?
Speaker: And how do we prevent future machine learning systems from consuming their own digital waste? ah I love that.
Speaker: um So OpenWALDO, and I'll give a quick kind of background on OpenWALDO. So OpenWALDO exists really to be kind of the the open source ah infrastructure tooling and data around AI. And the first thing that I started with for that was the training corpus.
Speaker: And there's a lot of challenges with building a very scalable training corpus. As I kept kept saying, like the amount of data necessary to build a big model ah is is massive. yeah And so to do this, you need a ridiculously large amount of data and you need to be able to host that. And my goal of it was to be able to host that in a way that is compatible with the open source kind of way that we do things. Over the last 30, 40 years, we've created a ton of really great innovations, capabilities, what methods to manage provenance, et cetera, accountability, all within open source.
Speaker: And how do we bring that to something that is such a large training corpus? yes um So i I leveraged kind of some of my background in in you know how do you manage an open source Linux distribution? um That has tons of binary artifacts, tarballs and and ah other things where you can't just throw everything in Git.
Speaker: So the way we do it is we use look-aside caches and that look-aside cache is indexed in the Git, in git um but that index is pointing elsewhere to where those look-asides exist. Right. So what I created was a index that is all in Git and the appropriate tooling to manage that index so people can come in and and add you know their own material into this corpus or review or audit the material in the corpus where the index and the men the the metadata for this is all exist in the index.
Speaker: So, and the index is in Git. So it makes it really kind of nice for somebody to actually go and submit a PR to add more data into the corpus. and I like it. I mean, as an author, i feel like I want to give you my books.
Speaker: Oh, love that. And um so to to the question now directly is how do we kind of ensure that we're not bringing in a bunch of ai slop? And while we can never completely say we're not going to bring in AI slop, um what we can say is at least it's auditable and we can build a situation where the quality of the data that you're ingesting will get you better results and within within the AI. So in marketing, people have been talking about SEO for for decades. And SEO, search engine optimization, it's it's how a particular website or a particular page or landing page will end up ranking reasonably when you do a particular search.
Speaker: yeah um It's kind of the same premise that we need to solve in AI. Companies are like, how do I ensure that my company, my products, my services, and my my company itself is actually getting ranked in AI responses. And so the way everybody's approaching it is they're basically just flooding their own websites and their own um documents and whatnot that's public. and They're just flooding it with content, hoping some AI search engine or crawler is going to see it and then add it to the training data. It's actually a worse case even than um than than like SEO,
Speaker: And this goes back into kind of something we already touched on a little bit, which is the training data itself. Like what's the license of data that's on a website?
Speaker: I mean, we we all kind of know at this point, if you've been an open source for any amount of time, if there's no license, it's covered under copyright. And copyright law is pretty well understood.
Speaker: The question is, is this fair use? And that question is still open for debate. There's been a few suits on this lawsuit. The suits are ongoing. I mean, yeah you know, Anthropic apparently owes me some cash, but I haven't seen anything yet.
Speaker: Yeah. I know a few authors that have actually gotten eight or $9,000 checks in the mail from that $1.5 billion dollar settlement. So I haven't gotten mine yet. Why not? What's going on?
Speaker: sucks. going to have to go, you know, talk to Addison Wesley, those buggers. They probably pick it. So the question is going to come at some point.
Speaker: is what is this data trained on if the if the data is trained on copy left instead of copyright you know so if it's if it's gnu licensed software or gnu licensed data i mean that that those licenses are viral if a company is writing code based on a model that was trained on copy left gpl gnu code yeah like what does that mean for the for the code that they're creating is that a derivative is it not a derivative like These are tricky things. They're tricky things.
Speaker: Too many open questions right now. And I know a lot of companies that are, they're on the AI bandwagon, but they're like, we're not going into this until some of these questions have been answered.
Speaker: Yeah. Well, they better not hold their breath because it's going to be a minute. and And when and if it gets answered, it may be a catastrophic backlash on how we've been doing all of this. So inside of the OpenWALDO index, ah for every corpus of data that somebody brings in,
Speaker: you have to associate a license to it. And the contributor has to commit to that Git repository for the index, what that license is, and use the developer certificate of origin, the DCO, to properly take accountability ah for what it is that they're... That is so cool. I really like this. That's that's really nice.
Speaker: So i want I want to switch gears a little bit. Yeah, please. That's cool with you. So I want to talk about high performance computing a little bit. So you you built Singularity for that and you did it because standard container tools kind of failed to handle security correctly.
Speaker: because they were giving processes too much ambient power. Today, we're seeing the exact same mistake play out with AI agents, right? So developers are spinning up models and giving them direct command line access and file system privileges and execution tools without any real operational boundaries.
Speaker: um When you have a non-deterministic AI agent executing actions inside enterprise systems, why is trusting standard web container wrappers a massive security gamble?
Speaker: And how should we be thinking about privilege and isolation in an agent-driven world? Well, the first thought that comes up is, um goes back to the, there are these two populations of people. Yeah, back to the people. Back to the people. um On the open source side, it's not just the open source people.
Speaker: it It actually, that side also contains a lot of the infrastructure people, the people that have been managing large infrastructures for for decades. It also includes the people have been managing security for those infrastructures yeah for decades. And um i can't say this for a definitive fact, but I can tell you anecdotally, I know a lot of security experts that are fantastic at what they do. I would trust them with my in infrastructure. Heck, I trust them with the keys to my house.
Speaker: And they have been, and this is not just like one or two, this is groups of the ah of these capable individuals who have tried to get jobs,
Speaker: within a lot of the AI labs. Oh, I know. And a lot of companies that are focusing on AI first. And they're being ah discounted due to either, and it's silly reasons, but I've heard cultural misalignment, expertise misalignment, and- I think, you know, it reminds me of software security, which is a field that I helped to found long time ago.
Speaker: And the the same problem existed then. And we had to invent a new category of people called software security people. who were developers first and security people second. They weren't like, we didn't take guys who were good at firewall management and try to turn them into software security people. You could in principle do that, but it takes a lot longer than taking a software guy and shoving some security into his brain or her brain. And so I think we're in some sense in a very similar situation. And I feel in my own work,
Speaker: um in machine learning security like we are at the very, very beginning. It just reminds me of when software security was like, gosh, why is all the software broken?
Speaker: but Why are we just putting a thing in front of it? That's not a solution. And um and I think we're going to make rapid progress, but it's still early days there. Yeah.
Speaker: So I definitely have a point on that I'd like to make. um But I also want to really just kind of push forth the notion of the initial question, which was, we're in a situation where we're giving a black box, whether that black box is an open weight model that we've downloaded from from China, or it is a API that we're using from one of the major frontier labs. We don't know what it's going to respond with, and that's kind of the point of AI. yeah um And there's there's a lot of variables in the mix, and we're giving it tooling,
Speaker: for it to basically do lots of different things without thinking about the security access controls and sandboxing, reasonable sandboxing. So going back to my point on on security engineers and security you know software people, like we we are going to need, and this will also fall into my next point, we are going to need to be escalating up the people who understand how to do proper sandbox sandboxing, yeah proper um ah air gapping, yeah and proper controls and instrumentation to make sure that
Speaker: the The tooling is bound. It is being controlled. I totally agree with you. And we're going to have to stop saying the agent decided to do X because agents don't have intentionality. The last time I checked, it was somebody ran that program and it did a bad thing. And whoever pressed the button is probably who should be in trouble.
Speaker: Oh, the AI did it should never be a legal defense. Very, very concisely put in perfect. So so right real quick if real quick, there's a second point I wanted to make on this because it's hugely, hugely important.
Speaker: um We've been sitting on decades of human created code. um Jokingly, we can call it human slop as we call AI slap, slop but we've been sitting on decades of human created and human curated code. And the wonderful thing about humans is, well, we, we, we miss things and we sometimes make mistakes. We've got decades of bugs sitting in our software infrastructure right now that nobody has seen, nobody has found. And we are starting to see an increasingly prevalent um situation where all of a sudden it's like the AI is exploiting this, the AI is exploiting this, the AI is exploiting that. And And it's just like, no, we turned on the lights and the roaches are running around. Exactly. I love that analogy. and And it's going to get worse.
Speaker: We are going to accelerate finding the bugs that exist currently in our human created code. And um and that's going to it's going to increase. If you look at the Linux kernel, there's a graph that Greg Crow Hartman ah posted not too long ago, which shows the amount of CVEs that they've been doing for a number of different releases. and Like it it hockey sticks. Yes. It's a huge amount of work.
Speaker: It's going up and it's only going to get worse. And it's not just the kernel. It's the entire user space. PHP was just afflicted with this. ah you know I mean, it's your own fault for using PHP. What can you say?
Speaker: um I rolled eight servers. i'm Guilty. Guilty as charged. I am a huge fan of Rasmus. So I am. He did wonderful work. So I'm not going to say anything negative about PHP, but I will. i can't argue as well that um it's not just PHP. It's the way people also code in PHP. Of course.
Speaker: too easy of a programming language. So people who shouldn't be coding or coding and then putting, you know, I was once on a panel at using technical with Larry wall and Larry said, the great thing about Pearl is there's six ways to do everything.
Speaker: And I said, Larry, I have a different alternative perspective from security. The really problematic aspect of Perl is that there's six ways to do everything. And you only thought of three ways to block the six things. Six ways. So you're screwed.
Speaker: And Larry didn't talk to me anymore after that. It was a But Perl was my second programming language of choice after C. And I was talking with a Python developer at ah Berkeley Lab a while ago. And he said, you know what Perl looks like to me? If you press the shift button on your keyboard and drag your elbow up and down the new the number row of your keyboard, you will end up with proper obfuscated Perl.
Speaker: And it's like, yeah, okay, I get the point. All right. So let's let's ah let's hop back on track. Right now, the dominant approach in the market is to take that uninspectable black box and wrap it in some sort of secondary classifier models or prompt filters or external guardrails to block malicious behavior. But in traditional software, we learned decades ago that input sanitization and perimeter filters never fixed really fix vulnerabilities. They just create some sort of cat and mouse game and the attacker is eventually going to win or somebody is going to come and turn the lights on and they're going to go, oh, my God, look at all the bugs. So in machine learning security, why is treating architectural flaws
Speaker: like, you know, let's say mixing control and data together in the same channel, yeah um as treating that as simple input-output filtering problems, um a dangerous delusion. And from an open so infrastructure perspective, what does it actually mean to build an auditable system that's secure by design from the bottom up?
Speaker: and you know not just an open version of the broken design that we have seemed to converge on, but something better. In reverse order,
Speaker: Building the infrastructure, the architecture, it again comes down to you have to know what's going in. yeah So that if the source is open and you're building a model based on things that are auditable, we can easily say, you know, in this corpus of books and movie text,
Speaker: we're going to leave out the matrix and Terminator and other sci-fi examples in which bad robot behaviors. Yeah. It's, it's probably a good idea to leave out, you know, how to kill humans off in mass extinction. Like, um,
Speaker: or Or how to use us as batteries. It's probably a good idea just to leave that one out. And again, I'm being sarcastic about this to make the point. yeah But there's a lot of information in training corpuses that we probably could just leave out. It's like training know child. then it then it becomes Then it becomes a political question of what's right and what's not right and what's allowed and what's not allowed.
Speaker: And that's, so you know, as you can tell, humans are really bad at the political stuff, especially lately, like particularly heinously bad at that. So i agree completely. And this is why it's so important to have the open corpus of data, because somebody who wishes to do a um ah to build a model.
Speaker: that for example, has no medical information. We just had a situation where some poor girl um passed because the AI was telling her to that that the medication that she needed was in fact optional.
Speaker: ah and um and And she passed. Like, this is horrible. This is really bad. But I think this is your point too. Do we go and censor?
Speaker: And to me, the answer no. I do not believe in censorship. I believe that open access to data and and intelligence guided by wisdom is important. And with that being said, there are going to be models that you're going to purposefully want to leave things out. We've already talked about copyrighted material. We talked about copy left material. um And I'll give an example. As as a coding like a coding model,
Speaker: I personally would would feel more comfortable if I'm developing commercial code on that not to include copy left. Right. um Because i don't want to I don't want to have that argument or that that legal result blow up in my face later. Right. You don't want it to be legal.
Speaker: Yeah. So I would... i would I would be proactive on that one. um so it's but But the fact that we can have lots of people building models that are optimized for particular tasks means you can choose what out of the corpus you're putting in. Not everything has to be a world model.
Speaker: Totally with you. So um I want to shoot a little bit in a different angle. um A core thread running through your entire career is is reproducible computing the principle that if an execution state or a scientific run can't be deterministically recreated it can't be trusted so today model builders hide behind the idea that non-deterministic output is some sort of proof of quote creativity or quote intelligence but stochastic behavior isn't creative
Speaker: It's just pseudo random statistical sampling across a high dimensional vector space. So from a machine learning security standpoint, when an agent or inference pipeline produces different execution paths every time you run it, how do security engineers conduct forensic analysis or verify and variant enforcement or prove that a system didn't quietly break its security boundaries?
Speaker: So proper instrumentation on the agents, I think is, is fantastic. I think it's needed, but to go back to your earlier point, it seems we're building infrastructure around an architecture that is, i don't want to call it flawed because it's by design to do this, yeah but it is a, and I will kind of do what you said not to do, which is it is a non-deterministic system. So, um, uh, I, I get the aspect of the LLM with not an understanding, not knowing where the LLM is going to go at a high level.
Speaker: Um, specifics about where the LLM is going to go is 100% though dictated by its training. And it's going to. So, and, and there are ways to build infrastructure. Like this is what I was talking about with Christoph. Um,
Speaker: that you can think of as kind of like the skeleton and the the organs and the flesh and all the stuff around the skeleton, we can let that be stochastic and slightly non-deterministic.
Speaker: But the skeleton itself, maybe that should have some you know proof carrying formality with it. And that might be an answer. I know at Google that that they're doing some of that stuff, yeah at least in science land. Of course, Google's so big, they're probably doing the opposite as well.
Speaker: um So, you know, maybe that's an answer. So I hate it when people talk about comparing LLMs to like humans and the human brain. I i actually like I really have a like a knee jerk, but I'm going to do it.
Speaker: You know, you have to know i have a PhD in cognitive science too, so it drives me absolutely up the wall. So here we go. Let's see if we hit the kill switch. so But how I'm going to use it is not comparing it and and using it as justification for for one to the other. What I'm going to say is using the metaphor of of kind of the skeletal system and whatnot, I would actually say that ah LLM is probably most logically associated with the brain, which means the agent is most logically associated with the body, which means, because the body is the, ah the the heart excuse me, the agent would be the harness plus the LLM, right? So yeah yeah the body's the encompassing unit of everything. yeah um
Speaker: And then there laws. And then there's society. And then there's how are you supposed to act and how are you as a human supposed to be within society? And if I work for a company and I do something illegal on behalf of that company, who is responsible? The company is going to be first line of your dog bites somebody, whose dog is it again? i mean, you So responsibility is so incredibly important here. And I think that is the undercurrent of of so much of what we're talking about is we're seeing these issues happening. We're seeing exploits happening with very little accountability at this point, because to the earlier point, the AI did it.
Speaker: yeah And so like you can't, you that that is not a legal justification for why somebody or or a company should be not held accountable. We have to be holding organizations accountable. And this goes now directly into your point of,
Speaker: We need, companies need the appropriate tooling. They need the appropriate um ah sandboxing, instrumentation to see what's happening. And they need to be held accountable in such a way that they build this infrastructure. And this infrastructure now becomes extremely valuable for anybody who is using AI and part of their core infrastructure.
Speaker: All right, another thread. You helped form OpenELA because leaving core enterprise Linux infrastructure under sole control of one vendor created unacceed unacceptable system systemic risk. So today in AI, Western enterprises are caught between two structural extremes. One is relying on closed proprietary cloud monopolies or um deploying open weights models when and I'm putting a quote around there, built predominantly by Chinese tech giants like Alibaba and DeepSeq.
Speaker: So from a machine learning security standpoint, if the entire global economy builds its software stack on top of a few closed API gateways on one side, and uninspectable open weight binaries from strategic adversaries on the other. What happens when the ecosystem, what happens to resilience? Like, and how does open, fully auditable infrastructure, like say the kind that you're working on, rescue us from that dual trap?
Speaker: So there's a common thread between both of those scenarios that people need to choose through. And it is the lack of lack of visibility, lack of transparency, lack of understanding. And we're we're kind of stuck between two situations. And actually one of them, in my opinion, is is actually more dangerous than the other.
Speaker: the when when When somebody is, let's make the comparison back to open source again. Open source, source so sorry, open source infrastructure. Yeah. If a company wishes to go download kernel from kernel.org, glibc from GNU and GCC from GNU and start compiling their own operating system from scratch or their own tooling or their own applications and running that themselves, yeah they are taking on the onus of responsibility and accountability themselves. If something breaks,
Speaker: It is on them. They went and downloaded an open, freely available thing, and they went and built it themselves. right The difference between open source and open weight models is they can actually look at the source. they can they Most people don't, to be clear. Yeah, I know. I mean, in principle, well you could. Yeah.
Speaker: And you could audit it to see, is there any malicious code in here? Or are other people auditing it? And you're trusting the community, the crowdsource audited um source code, which is what most people end up doing. And, um but so you take on that onus, but you have a level of trust in what it is you're running. When you're leveraging a SaaS interface for a AI model from a company,
Speaker: Well, yes, there are free accounts, but you're you're agreeing to a EULA when you get that free account. But more importantly, companies don't use free accounts. no They are going to build a commercial relationship, which now means you have a commercial throat to grab.
Speaker: And you have some account of now responsibility and accountability through that commercial relationship. When we're again, when we're going back to open weight models, when a company goes and downloads an open weight model that has no company behind it, no accountability behind it, and it is a black box, it is literally the worst of all situations. And because It reminds me of just getting software globs in the 80s. I remember just going and getting globs. And then we were like, damn, that's a bad idea. i wonder what's in that glob.
Speaker: And and and it there's one aspect that even makes it worse, at least for American companies, is there's not even a throat to hold in this country.
Speaker: Like all of our laws don't apply. so it's like. it's it's It literally is the worst and most dangerous situation that you can come up with. I'm going to go download a free, you know, open weight model. I'm going to go run it on my core infrastructure. I've got no idea what's inside of it. It's a black box. I don't know what it's going to do. I can't possibly ever blame anyone else because they're not even in this country. So I can't even sue them if they if they really screw it up.
Speaker: And I'm going to give it access not only to my corporate network, but my commercial data. And I'm going to base my future on it.
Speaker: ah So anything wrong here? Oh, nothing to see here. Move on. move And, and, and it's, ah it's actually, it's also funny because they take, they, they're these binary blobs of data, which are the weights. And then they went and open source them under an open source license. Yeah. Right. So you can go download these open weights under an MIT or BSD or Apache two license. And it's like, wait a second, that's an open source license. So everybody says, Oh, it's open source. I know that I'm so pleased to do this episode because I've felt that way the whole time. And it was like, thank God somebody is doing the right thing. Uh, so thank you for your work on that front.
Speaker: Um, Whenever corporate interests attempt to lock down systems or enclose the commons, the open source community steps up to rebuild open alternatives. But as automated code generation accelerates and agentic tools begin opening pull requests and maintaining repositories all by themselves, we risk confusing automated synthetic churn with actual software engineering.
Speaker: So automated tools can pattern match across existing code, but they lack human insight and inspiration and intuition and all of those things that make a real maintainer incredibly valuable. What we call the three I's at BIML. As machine learning systems take over more of the build and maintain cycle, how do we preserve human stewardship at the heart of foundational software? Ensuring our infrastructure reflects genuine architectural wisdom rather than just automated noise.
Speaker: So I just i just released a research paper. And this research paper describes a process that I used to build an operating system kernel from scratch in Rust.
Speaker: It is a um from first line of code to a usable operating system that is Linux compatible.
Speaker: So I can go and spin up Rocky Linux, Alpine, Debian, whatever user space you want on top of a non-Linux kernel. And I wrote, again, the whole thing from scratch. And to say I wrote it is a misnomer. um I came up with a method, and I'm not, I'm sure I'm not the only person to do this, but I think I might be the first person to actually write it down and publish it, um where I call it Human Directed Agentic Engineering, HDAE.
Speaker: And it describes a process in which the The human is in charge of various roles and the AI is responsible for various roles. sure And with this process, I was able to create in 11 days, an entire operating system kernel that that is an alternative to the Linux kernel.
Speaker: I think the only reason that worked is because you are you. Yes. So this goes directly to- I mean, you could probably do it with a Swiss army knife also.
Speaker: it This is the human directed part. It has to be. So I have this hate relationship with the term vibe coding. oh Because to me, vibe coding is somebody who knows nothing about the discipline, nothing about the code, and goes and asks an AI to go solve a problem, build an app or do something. And the AI goes and does it and there's no oversight, none.
Speaker: right And that to me is vibe coding. That to me might end up with a proof of concept. It might end up with an MVP, but you are not going to get the same quality of um output than if you had a expert driving it.
Speaker: of course And so human um humans that are experts driving agentic engineering You can get 10 to 100x efficiency, capability, code output, and even quality if you do it right. And I outlined how I did it in this research paper that I've published. We will provide a link to that on the on the when we put the episode out.
Speaker: Very cool. um But I was also, just as an example, um the process in introduces a notion of gates where um you you define a particular milestone in your plan and architecture and you have the AI sit there and iterate on this gate until that gate is solved. Passes.
Speaker: Right. Yeah, of course. and And it would literally run all night. I'd wake up in the morning and be like, oh, cool. It's actually solved these three gates and it's on to the fourth one. and And then I come in and I'll be like, okay, let's do a let's do a checkpoint here.
Speaker: I need to you know validate where we are. I need to see where it's at. And sometimes I'll say after each gate, I want to actually look if it's ah if it's a big gate. and um And then I'll go and validate, make sure it works. But this resulted in, in total, it was about 180,000 code.
Speaker: There's no way. And I did this whole thing in 11 days. There is no way I'm reviewing hundred plus thousand lines of code. Right. So at some point, how we think about engineering, how we think about development has to shift.
Speaker: yeah And we have, we already talked about AI turning on the light and all of the bugs running away. But we can see those bugs now and we're bringing them to light. And like, that's going to happen faster and faster. Humans alone will not be able to keep up. And I'll just put that out there. We will not be able to keep up. Even the Linux kernel at this point is leveraging AI to solve a lot of the bugs that AI is finding. I know.
Speaker: We have to be smart about how are we going to move into the future now. And human directed agentic engineering, which is the concept that I wrote up in this this article, um I think is a really good possible way of doing it. And I'm sure there's many more, but...
Speaker: This is a, um there are multiple solutions to solving this problem. yeah yeah And we just need to have not only expert engineers in the room, but expert engineers who are willing to use AI.
Speaker: Yeah, and I totally agree with you. and that And the human part is not going to disappear. yeah On the contrary, it is essential to the success of the project.
Speaker: and it's and And the human role is going to change. It's going to evolve. And right now we are right in the midst of of evolution of technology and disruptive technology and disruptive disruptive innovation in general.
Speaker: is always going to cause evolution. And evolution is not a pain, excuse me, it's a very painful process. It's not a kind process, right? It causes it causes individuals to like die before before having offspring. You need a t-shirt that says, don't be cold. ah ah So we are in the process of evolution right now.
Speaker: We all have to change how we are thinking about our jobs, our work, how we operate, how we think. um i wrote I wrote a ah simple ah post just just recently that it said, um knowledge is no longer the moat, wisdom is the moat.
Speaker: Like you need to be smart. Like we are inundated with information and knowledge at this point. Everything is out there. If you look hard enough, you will find an explanation for every conspiracy theory and every silly idea you've ever had.
Speaker: yeah And you will get justified in your own mind. Now it's about having the wisdom to know, given all the information is available, what are you going to listen to? Oh boy, are we in trouble as a bunch of monkeys.
Speaker: So let's let's wrap this up. So we're hurtling towards some sort of stark fork in the road, I think. On one path we got, we surrender our digital future to some closed proprietary tech monopolies and black box models, where security kind of an illusion and users are certainly captured tenants.
Speaker: On the other path, we build sovereign, open, and rigorously auditable foundations, though doing so means fighting entrenched commercial interests in an ocean of unvetted synthetic noise every single day.
Speaker: So as we look at the next five years, which is a long damn time, this isn't a cooperative ecosystem dance, I don't think. It's an architectural war for control of compute.
Speaker: What is your battle cry for engineers and open source builders who refuse to bend the knee to the black box? The first thing that I'm going to say is between open source and the commercial industry, the commercial industry always seems to kind of drive a lot of the innovation. Not all always the wrong word. Often seems to drive a lot of the innovation. So we see things come out of commercial industry first and then open source kind of fills the gap.
Speaker: Yep. um I mentioned also earlier that there's two populations. My call to action. is I am really trying to pull those two populations together.
Speaker: Right now, there's there's too much separation between them. ah The AI world right now needs open source. It needs experts who have been doing this for ages to come in and be, um ah pardon the expression, the adult in the room on how to do- The wisdom, let's just say. That's a yes, yes. And the, the open source side needs to get a heck of a lot more open and curious about AI. We've got open source project leaders and there was actually just a ban in, not that I'm a Reddit follower or user, but um on the Reddit open source, you know, subreddit that if you use AI, you
Speaker: like you are not allowed to be part of this community or post to this community or something along those lines. And it's like- It's to be like one guy and his dog. produce I mean, but it will be, i think, but for now it's not. And the amount of people in open source that are making an active stance against progress is going to end up biting them and all of their users in the foot um or shooting them in the foot. So we got to bring them together.
Speaker: We've got to bring these two communities together. And um right now the overlap, OpenWALDO sits perfectly in that overlap. um It is a fantastic place for people on the open source side who want to get more involved with AI, understand how AI infrastructure works, want to know how to train. We have a composed ladder where every ladder, every rung on this ladder is a different training level, both for training a model get as well as training a person.
Speaker: Yeah. Get it and do it. Yeah. That's fantastic. And so anyway, bringing those two communities together is my, my personal like goal and call to action.
Speaker: Love it. Thanks so much for participating. It's been a pleasure. Thank you. This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security.
Speaker: You can find a permanent archive of all our episodes dating back to 2006 at garymcgraw.com slash technology slash silver bullet podcast. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at barryvilleiml.com slash podcast.
Speaker: This is Gary McGraw.
