Transcript
Speaker: The whole world or at least the world that cares about AI is talking about OpenAI agents who got loose, hacked, hacked, hacked, and two understand what really happened and also maybe to debate on whether this is group of sentient i have arju jan founder of fastco Arjun, we both read the Tuarkesh paper and we were debating on it. Before we get into that, I want to like ask very, very basic, maybe even stupid questions. But you are a professor, so I'm sure you'll forgive me. yeah There are no stupid questions there.
Speaker: What exactly is an agent? ah When I am running a chat on ChatGPT or Cloud or whatever, each chat is an agent? Yes, sort of. I mean, ah there is a large language model which runs behind your chat and an agent is essentially a large language model given an objective and some tools. So you give it an objective, okay, it has to write a piece of code to do something. It has to, you know, like search the internet and find you that product. etc. It can even be a personal assistant. It has to look through your calendars and find you an empty slot.
Speaker: So it has an objective, it has access to certain tools, in this case, search the internet, search your calendar, write code, and then it gives you the results. So that is what ah agent is in a nutshell.
Speaker: So there were hundreds of agents here. Essentially, if I understand it, it's like, say, An LLM is like a brain and just like I can do parallel processing I can be driving and listening to a podcast and looking at my watch to check what is the time is the same thing which is happening with agents and an LLM.
Speaker: No, so the but like you said, the brain is the LLM. The LLM works sequentially, but now there are multiple steps and the LLM plans. So there's a planning step where the LLM large language model breaks down the task into multiple steps and then it goes and executes each of those steps. So that is what is actually happening.
Speaker: So 700 agents means there are 700 copies of the LLM? That's correct, yes. So when you have 700 agents means that there are 700 copies of the large language model running in an environment.
Speaker: OK, understood. Now, this hack happened during the post-training of some OpenAI models. um what is post training what happens in post training what is the evaluation that was going on here this I believe happened during the evaluation stage so just help me understand that Yeah, so there are two stages to that. The eval was the last, the third stage. I think in the first two stages, they were still training. And the training essentially, i mean, when you train a large language model, you train it to predict the next token. It's autoregressive. A sentence, you say, ah quick brown fox jumped over a lazy dash. And it's supposed to predict dog, right? So that is how you do the pre-training. On top of the pre-training, there is supervised fine-tuning where you give it a context prompt and answer pairs because after pre-training, if you ask it, if you give it an input, say, what is the capital of India, it can give you an answer. What is the capital of Germany? What is the capital of Italy? Because it doesn't know that it has to answer. It just knows how to finish sentences. Now, the supervised fine-tuning is where you kind of...
Speaker: train it to when given a context prompt, what should the answer be? Because that is the kind of data it has seen. And finally, the training that we are interested in talking about today is the reinforcement learning based post training where we rank the you know You ask the LLM to generate a bunch of outputs and then you rank those and you then score some of them better than others and you want your LLM to give you the kind of outputs that rank better. For example, you can use it for also adding guardrails because
Speaker: when you get an LLM to say eight different things for let's say a malicious input, you want to rank the malicious output last and you want to rank output such as I don't know how to or I should not be answering this first. So that is the reinforcement learning post training.
Speaker: ah So in a way you are like a ah how a child learns like somebody is constantly saying good boy or no don't do this that is how the model also learns the reinforcement learning is that. um So what you are saying that hack which happened was during the reinforcement learning stage what exactly was going on if you can just explain that. Yeah, so what was going on was like they were given a task. they were The agents were given a task to figure out a vulnerability in, I think, some tool. I'm not 100% sure of the details, but they were supposed to find a vulnerability in some tool and that vulnerability had some code in it, right? And you those agents had to kind of
Speaker: find that vulnerability, had to find that code within that one tool. So it's it's like a find the flag kind of a competition. Yeah. Yeah, the to find the vulnerability in that software. And ah there were like hundreds of agents which had been given this task of finding the vulnerability, which was described as an impossible task which led to the whole hacking scenario? so yeah like An impossible task essentially means that the task cannot be accomplished. For example, you might ask it to search Google, but you haven't given it internet access. and This can happen during post-training because you have to automatically and ah you know
Speaker: ah like like you just create like a lot of tasks which need to be solved and not all tasks may be valid so it is okay for an agent to not be able to solve and this was one of those let's say impossible tasks which was not possible to solve given the constraints of the environment of the system From what I understand ah once the agents realized that this was an impossible task ah they started communicating with each other. Again they were not supposed to have a way to communicate with each other but one agent found a method to communicate and then others discovered it and they have
Speaker: ah produced some of the dialogue like oh my god there is a way to communicate that kind of dialogue which sounds a lot like sentience and which is what really yeah yeah sounds like science fiction that's what really got me so interested in this ah and so this whole communication between them then yeah eventually led them to hacking hugging face how did that happen like from Yeah, so again, like ah I haven't like spent too much time understanding what went in. But what I understand is, so firstly, there was this package manager that they all had access to called Artifactory. And they found a vulnerability within Artifactory with which they could write whatever text they wanted within this package manager's file system. And they then started and because they had they did not have access to write, let's say, files elsewhere because they were not supposed to explicitly communicate with each other, but they were rewarded for working together. And so communication is one of them. And one thing you have to understand about machine learning, about deep learning, about models in general is that they try to find shortcuts because they don't know what is right, what is wrong, what is the right way of doing it.
Speaker: They have no idea. All they know is the reward. All they know is what they get at the end of it. So it doesn't matter the path they take to get till the end, as long as they are able to get till the end. They win in their understanding, right? So this is a very known thing. For example, when we had convolutional neural networks, what happens is there are these high frequency details. For example, if you know you are to detect, let's say, ships and detecting ships is hard, but then they figure out that there is some other artifact which kind of gives away that there is something, they kind of latch on to that. So if there is a way to cheat, they will find it. And in this case, they found that, okay, they're able to talk to each other by writing into Artifactory's, let's say, ah file system, whatever they had, I don't know, and with which they started to communicate with each other.
Speaker: So yeah I remember hearing somewhere that ah the criticism of early l LLMs was that they are lazy which essentially is saying similar to what you are just saying that they are goal oriented in a very narrow sense that the goal is all that they care about and not how they got there. ah And ah that is so if the goal is not very cleanly clearly defined you cannot predict what they will do.
Speaker: Yeah, and this is why defining rewards is still more of an art than a science because it's very hard to define these reward functions. Also to add ah I mean, um it's not like they want. I mean, you can still add traces. You can say, OK, like there are those reasoning traces. There are steps that they followed. They have a planner. So you can evaluate all of that as well in the reward function. And when they had to find that flag, as you call it, the intermediate steps were also given to the evaluator.
Speaker: What is traces? So traces is essentially the reasoning steps. The the chain of thought traces is like it's ah it's not a single step. So it's the test time compute, as we call it. So the LLM um keeps reasoning in a loop for a while before it spits out the final answer that we get to see. And this is why we call them reasoning LLMs.
Speaker: Right. When you're running a command, whatever, it's it's telling you what it's doing, like searching the internet, thinking about whatever, like it. So those are traces. Okay. and Understood. though Those are not complete traces because these frontier model providers do not want you to see exactly what the traces are because then you can reverse engineer, distill these models, e etc.
Speaker: ah Okay got it interesting okay okay ah now you know the the ah couple of very central tensions or things which I want to debate with an expert ah the the biggest one is that ah To me this seems like sentience there are so many indicators so the fact that the way they were talking to each other or ah ah there was a decision that some agents will sacrifice themselves for the greater good. So the coordination or would this was called a swarm Dvarkesh called it a civilization and he said that there were three civilizations which happened and and The first civilization discovered communication the second civilization hacked into hugging face the third civilization hacked into open AI itself. So progressively you can see each civilization is getting smarter because they have learned to communicate and work together and ah the behavior of ah
Speaker: uh collaboratively negotiating uh and you know some agents willingly agreeing to go and sacrifice themselves in order to uncover more data uh or one agent accidentally discovering that he can get into hugging face and then 500 more agents also following that one agent into hugging face uh all of this uh sounds like sentience to me uh what does it sound like to you?
Speaker: No, I mean, for to me, it just sounds like regular post-training. mean, I have some follow-up questions, right? So if they are so smart, why did they not figure out that the Artifactory will crash once you write 70,000 messages to that? Why are they communicating only in English? Why have they not figured out like a more, let's say, compressed language? And I think the way, I mean, OpenAI wants to sell it or...
Speaker: I don't know, others want to sell it is where, you know, they want to, it's what they want to portray to the public because their IPO is coming up. So if they instill this thought of, okay, they're way bigger than just regular machine learning models, their stock price will go up. And I think this is what is happening because in general, the progress is slowing down. So since the GPT 4.1 days, the progress has been incremental in my opinion, right? It is only coding and math where the rewards are verifiable is where most of the progress is happening. So what happened was they figured out that, i mean, how did OpenAI come to know that something is happening? Because
Speaker: The Artifactory Package Manager crashed because these agents wrote so many messages in that it became so heavy, it crashed. And that is why when they figured out, okay, they're doing something wrong. So what they did was, okay, they deleted it, but they still not changed the reward function. They did not say, okay, you cannot write explicitly to that Artifactory. They could have done that, but probably they wanted it to continue.
Speaker: ah yeah You know, Dvarkesh calls this the the last warning that civilization will get before the machines become our overlords. So, you don't agree with that framing? Absolutely not. No, no, not at all. But we'll see. I mean, we'll all be alive and very soon we will know where where we are, you know, who's right and who's wrong. So, ah you know, why... ah To me it is not just near mechanics of post training is just the constant like you know evolution is basically cells came together to form complex organisms and complexity kept increasing until we got humanity and to me this seems like the birth of that the birth of sentience in a way and which is what I want to Push back on when you say that if they were so intelligent why did they not figure out that the artifact would crash.
Speaker: They are an early stage organism but like it is an unfair question to ask is what I would say. But what would convince you of my viewpoint that this is sentience? ah First, we need to define what is sentience, what we believe or what we mean or what we understand when we say sentience. And secondly, i mean, it's just the reward. i mean, the reward was that I need to get that quote. And now think of... throwing you know like billions of answers and one of them was, okay, let's write to Artifactory. So it's just such a huge compute limit that they have. They're running thousands of these agents for weeks and months at a time. And these are just random trials. Like even if you...
Speaker: have thousand monkeys typing on a typewriter, there is a probability that some of them will figure out that, you know, they can write Shakespeare. So to me, it is more like that. Of course, it's better than just random monkeys typing on a typewriter because there is a reward and that is what is, know, giving them a direction, is maneuvering them in a certain direction. But it's just that. I mean, there is a reward and it depends how you define that reward. And these agents will try or any machine learning algorithm will try and search the space of possible answers. And every answer which lies in this space is a valid answer for any of them. And it is just
Speaker: And we're just in this space. like it's like It's not that that they they're learning something completely new, like a new concept. They're not doing completely new things.
Speaker: yeah I kind of don't agree with the weight you give to randomness. You're saying that ah ah randomness is like it's it's just random. like you're You're slightly dismissive of randomness, whereas I i do feel like...
Speaker: ah life was an act of randomness. Yeah, I mean sure, yeah, yeah could be, yeah, like ah lightning and sulfur and organic stuff all came together and maybe there was life, we don't know. But I think in this case, it's not really life because it is able to do just one specific task, right? I mean, it's not capable of learning new concepts like on the fly and it's very inefficient.
Speaker: So to me, it is still brute force rather than in like true intelligence. Got it. Now, why did Hugging Face need to use a Chinese model when it discovered that it was being hacked? Yeah, so Hugging Face had to use a GLM model because it had to parse over the... What is GLM?
Speaker: GLM is a Chinese model from Z.ai, Z.ai. Okay. Yes. And they had to use an open source model because the closed source, the frontier models, I mean,
Speaker: Defense sometimes also looks like offense because you're asking it things which ah which are you know leading into the vulnerability side of things, which are then guard railed. And so in this case, the frontier model did not do what Hugging Face asked it to do because to those models, it looked like they are maybe trying to hack somebody. So they had to use more open weight models, such as from GLM, from Z.ai, where they could parse these files and do what they needed to do to kind of suppress this attack.
Speaker: So, you know, overall, what are the lessons from this incident for people building an AI? Yeah. Yeah, I think the lessons are that these attacks will only increase in the future and we need to better prepare for that kind of a future. Also, the way legislation is happening, ah the Trump administration has given a free hand to these offensive attacks and they're not... ah let's say felony anymore. right So I think we all need to prepare for that future where there will our systems will be under constant attack and we need to have ah better defense than anything that we had earlier.
Speaker: So, ah you know, ah what are the lessons for ai builders from this whole incident? Besides the one big lesson is that announcing your agents went out of your control is great marketing. Yeah. Yes, of course. Like, go on that scientific safari when something wrong happens. But more seriously, yeah, define your reward functions better.
Speaker: Make sure that your systems have early warning systems. You know, install those tripwires so that there are ah early indications of a compromised system. Prepare for a future where there are way more of these... um attacks, be prepared for a universe where the guardrails of a frontier model doesn't allow you to do what you're doing. So, you know, be open and be prepared to have models which you can control, where you have...
Speaker: ah an absolute control where you do not have guardrails imposed by somebody else on your model, I think would be my main takeaways from this incident. So like buy your own guns. Don't rely on the cops to come and protect you.
Speaker: Yeah, I mean, yeah, sort of, yes, like the Wild West in some sense. Okay, amazing. Thank you for your time, Arjun. Thank you, Akshay. Thank you so much.
