Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Keep the Future Human (with Anthony Aguirre)
3.4k Plays11 days ago
On this episode, I interview Anthony Aguirre, Executive Director of the Future of Life Institute, about his new essay Keep the Future Human: https://keepthefuturehuman.ai
AI companies are explicitly working toward AGI and are likely to succeed soon, possibly within years. Keep the Future Human explains how unchecked development of smarter-than-human, autonomous, general-purpose AI systems will almost inevitably lead to human replacement. But it doesn't have to. Learn how we can keep the future human and experience the extraordinary benefits of Tool AI...
Timestamps:
00:00 What situation is humanity in?
05:00 Why AI progress is fast
09:56 Tool AI instead of AGI
15:56 The incentives of AI companies
19:13 Governments can coordinate a slowdown
25:20 The need for international coordination
31:59 Monitoring training runs
39:10 Do reasoning models undermine compute governance?
49:09 Why isn't alignment enough?
59:42 How do we decide if we want AGI?
01:02:18 Disagreement about AI
01:11:12 The early days of AI risk
Recommended
Transcript
Introduction to the Future of Life Institute Podcast
00:00:00
Speaker
Welcome to the Future of Life Institute podcast. My name is Gus Docker, and I'm here with Anthony Aguirre. Anthony is the co-founder and the current executive director of the Future of Life Institute.
00:00:12
Speaker
Anthony, welcome to the podcast. It's great to be here, Gus. Fantastic.
Discussion on 'Keep the Future Human' Paper
00:00:16
Speaker
You have a new paper out called Keep the Future Human, and that's what we're going to dig into here.
Humanity's Precarious Position with AI
00:00:22
Speaker
I want to start by you laying out the situation we're in, because it's pretty serious.
00:00:27
Speaker
What's happening with AI? what ah What situation is humanity in at the moment? Yeah, I wrote this paper because I think humanity is in a pretty bad spot, honestly.
00:00:37
Speaker
Where we are now, we've all seen these incredibly rapid advances in artificial intelligence. And many of those have been tools. You know, they've they've been at first narrow purpose things like AlphaGo and AlphaFold and, you know, image recognition software and, you know, text to speech and things like that, which are all great.
00:00:57
Speaker
But in the last couple of years, we've seen a sudden burst in general artificial intelligence, ChatGPT and Claude and Gemini and things like that, that can do a whole span of different things that humans can do. They can write poetry and write code and generate images and recognize images and do math proofs and physics problems, a whole span of intellectual activities that have hitherto been human activities.
00:01:22
Speaker
And that's been really interesting to see, you know, from a scientific standpoint. But the question now is, where exactly is that going? I would say that general intelligence suddenly turned out to be a lot easier, i think, than many people had anticipated it would be.
Arrival and Implications of AGI
00:01:37
Speaker
and When we started Future of Life Institute back in 2014, people were talking about general purpose AI of the type we have now as being decades or centuries away. And here it is.
00:01:47
Speaker
And so the real question is what happens next and what does that mean? And the thing, unfortunately, that is happening now is that these systems are not being carefully studied by scientists in a laboratory, you know, thinking through the the all the risks and rewards and carefully engineering something.
Race for AGI and Associated Risks
00:02:07
Speaker
They're being developed in this mad race between giant corporations that are building products and turning them out as fast as possible with basically zero regulatory or any other sort of oversight over them.
00:02:21
Speaker
And what they are racing toward is artificial general intelligence, the next step as they see it, something that is not just sort of general and the same in in the way that ChatGPT is, but general and intelligent and autonomous, the the sort that can make complex plans and follow them, that can do lots of stuff unsupervised, can have its own goals that that it follows.
00:02:46
Speaker
And combining this with the you know generality and the intelligence of systems now and and where they are going soon, this is going to be a whole different thing than the sort of AI we've been dealing with up until now.
00:02:58
Speaker
And when you start to think of the implications of AGI and and shortly thereafter superintelligence, you realize that this is likely to be, at minimum, a total disruption, an upending of our civilization as it is.
Potential Disruption by AGI
00:03:13
Speaker
um And in the worst case scenario, just totally completely devastating. We will lose control to this superintelligent thing. It could cause human extinction. the The sort of bad outcomes have no you know limit of madness.
00:03:26
Speaker
And so we're in this place where we're racing toward this thing that is potentially, at minimum, hugely disruptive for almost everybody, at maximum, completely catastrophic. And we seem to have no way to get off the train.
00:03:39
Speaker
And how how far do you think we are from AGI?
Predicting AGI Timelines
00:03:43
Speaker
This still varies. Opinions vary. My personal timeline at this point is, for a specific definition, 50% in a couple of years.
00:03:53
Speaker
And the definition that I'm thinking of is not a particular capability, but what we're seeing in AI is that it's improving you know steadily on some timescale. That timescale was kind of years, and then was it was at months.
00:04:05
Speaker
And I'm projecting that you know within a couple of years, we'll be at the point where is capability is going to be sort of tangibly progressing. That is, you can you can measure that this AI system is better than that AI system on a timescale of weeks or less.
00:04:21
Speaker
And so if you imagine 50 weeks, that's 50 tangible improvements you know that you can measure in a system, that's going to be an incredibly powerful system. So I think the the key thing is not only that the AI systems are getting more and more powerful, but that the improvement is also speeding up.
Exponential Improvement in AI Systems
00:04:40
Speaker
You know, the the we've seen it. We see, you know, it used to be something would be released every year, every six months. And now it feels like we're seeing something almost every day, a new system almost every day with new capabilities. So that's where I am. Like, this is happening now. This is imminent.
00:04:54
Speaker
The dealing with what is going on is not a future problem. It's a now problem. On the face of it, it seems implausible that we would get such powerful systems within a ah few years. If you just think about other technologies, often it it takes time, it takes decades for them to dissipate throughout society and really affect change.
00:05:15
Speaker
What you're mentioning here is that What's interesting is not so much calendar time, it's the rate of improvement. And that rate of improvement is not not linear, it's exponential.
00:05:26
Speaker
And ah perhaps you can talk about scaling laws and what scaling laws mean for how fast we can get to AGI. Yeah. I mean, there's a few things to work in here.
Historic Investments in AI Development
00:05:37
Speaker
One is, and I think this is underappreciated, how much effort is being put into this. So if you think about the Apollo project, that you know that was sort of a 10-year project. We're doing, in terms of money,
00:05:50
Speaker
about five of those right now in developing AI and AGI, like 20 Manhattan projects. So if you think about things where humanity put a huge amount of effort in and built something at record speed, those are two major examples. We're doing that with AGI.
00:06:06
Speaker
So so like this is not happening by itself. This is being driven by giant amounts of money and effort. Now, the fruit of that huge amount of ah effort is that the technology is manifestly happening very quickly. Like if you if you compare what a model does and then you know what a state-of-the-art model does a year later, things have just rapidly increased. If you look at mid-journey in 2021,
00:06:30
Speaker
two or three or something, you you know, you'd ask it for a picture of a woman, it would be like this kind of blobby, barely recognizable thing. A year later, it's photorealistic. If you look at the performance of these AI systems that doing general problem solving, like I'm a physicist, you know, you can you can ask them physics questions. I've watched it unfold where they're, you know, kind of OK at it and but get things wrong to the point where it would easily pass the graduate qualifying exams in physics at any university.
00:06:58
Speaker
in the span of a year. um And this is reflected in the metrics. it you know There's this graduate level research questions exam, GPQA, where the models went from sort of chance, what a monkey would get by by filling in bubbles at random to human expert level roughly in ah a year a year and a half. So we're We're at the point now where, you know, some people don't see the progress as much because if you're just using GPT to, you know, have interesting chats or, you know, look for recipes, you're not going to notice that it can suddenly do PhD level particle physics.
00:07:34
Speaker
You have to be trying these things out to see the advancement, but the advancement has happened. So I think there's, there's advancement, you know, there's huge amount of effort. There's the manifest advancement in essentially all of the metrics that we have to throw at these things.
00:07:47
Speaker
We're even having to invent new tests, the latest one being the ominously entitled Humanity's Last Exam, where teams of professionals are getting together to build specific exams that are hard for AI to to do and put those together.
00:08:03
Speaker
exams that no human could actually do on their own. This is like teams of professionals and in many different subjects. So if you imagine who is the human that can do an exam that is written by ah whole team of biochemists and a whole team of particle physicists and a whole team of, you know, computer science professors, no human is going to do well on that exam.
00:08:22
Speaker
But AI are now starting to do reasonably well. So so we've seen you know this huge progress. We've also understand a little bit of the science behind
Computation and Unexpected AI Capabilities
00:08:31
Speaker
it. It's it's not quite science, but it's at least numbers that we can extrapolate.
00:08:35
Speaker
so So what we've seen is that, and and this is really what unlocked the whole revolution in AI, is seeing that throwing giant amounts of computation and giant amounts of data, and the computation allows you to use those giant amounts of data,
00:08:49
Speaker
translates very unexpectedly into these very powerful, you know very competent models that can do all sorts of things that you didn't specifically train them to do. And if you look at either the thing that you train it to do, which is predict words and how well it predicts words,
00:09:04
Speaker
or at any of the tasks that it can you know do, even though it wasn't trained to do them, but it can and do them. Look at any of those tasks as you crank up the amount of computation and data that you pour in, those capabilities increase quite reliably.
00:09:18
Speaker
And it's not clear exactly, you know it's it's hard to make predictions about what new capabilities are going to emerge because we don't really understand how these things are working. We can get into that. But if you look back at how capabilities have emerged, you see that more computation equals more capability.
00:09:35
Speaker
And there are things that you can do to make it you know scale faster or slower, but essentially pouring in more pays off. And the companies are now investing giant amounts of money to scale up that computation 10 times, 100 times, 1,000 times more in the next few years.
00:09:50
Speaker
So it's almost certain that lots more capability is going to be unlocked due to that investment.
Tool AI vs. AGI Risks
00:09:56
Speaker
We have an alternative to scaling up and getting to AGI within a few years.
00:10:02
Speaker
Perhaps you can talk a bit about tool AI as an alternative to AGI. Yeah, so I think it's important to emphasize that most of the AI that people are experiencing now feels like a tool and at some level is a tool. you know These are things that are like fairly intelligent and they're fairly general.
00:10:20
Speaker
They're not very autonomous, most of them. you know Your spell checker just checks spelling. It doesn't go and you know write a book on its own. It doesn't you know drive a car, try to take over the world. It's a spell checker.
00:10:33
Speaker
GPT is a little bit more in between in the sense that you can ask it to do lots of stuff and it will try to do those things, but it's not that great at taking out, you know, complex actions. It's more or less generates text.
00:10:43
Speaker
And, you know, if the text it generates is hooked up to some, you know, ability to do stuff, then it will try to do those things. But, it you know, it's not something that's going to run amok on its own as it is. So AI that we have now is pretty tool-like.
00:10:58
Speaker
Those tools, some of them are great, some of them are not so great, and they're they're good at things and and not so good at others. But they they're fairly passive, they're fairly fairly tool-like. The AGI would be something that is much more autonomous than that.
00:11:10
Speaker
So it would be able to take those complex actions on its own. And autonomy is not in itself a bad but thing. You know, autonomous vehicles are also great if we if we have them working and they're very reliable.
00:11:20
Speaker
We can save a lot of lives. We can save a lot of annoying driving commute time, having autonomous vehicles. But again, we don't want an autonomous vehicle that is a nuclear physicist or that is motivated to, like, try to gain huge amounts of power in the world. Like, that makes no sense.
00:11:34
Speaker
We want an autonomous vehicle that will drive from one place to another without having to have our hands on the steering wheel. So we we want things that are capable and trustworthy and controllable.
00:11:46
Speaker
And those are what we talk about as tools, things that are built to solve problems, things that are you know under human controls like tools are, you know like a hammer. It doesn't go off and do crazy stuff. It hammers the stuff that you want it to do.
00:11:58
Speaker
You want it to be strong and not break and be effective. You don't want it to do anything other than hammer or something. Or autonomous vehicles. You want them to drive and you don't want to them try to take over the world. So we can we we have those sorts of tools. We can make them much, much better.
00:12:13
Speaker
We can build all sorts of new tools to empower people. We don't have to combine the autonomy, the generality, and the intelligence into one package that is meant to like wholesale replace people.
00:12:25
Speaker
That's the AGI idea. We don't have to go there. you know We can stay on the path of tools. And so at some level, it's very simple. like You know, don't do the crazy thing that is going to build human replacements and run away into super intelligence and do all this nutty stuff.
00:12:42
Speaker
At some level, it's it's very simple. But the the drivers, as we can discuss, toward that are strong enough, you know, and the incentives toward that are strong enough that that is unfortunately the road we are heading down right now. And and hence the need to like sort of course correct.
00:12:56
Speaker
So the danger really lies at the intersection of generality, autonomy and intelligence. That's where you get AGI. Unfortunately, I think that's also where you you can potentially earn a lot of money from from producing a system that lies in that intersection.
00:13:11
Speaker
So perhaps explain the incentives that companies are operating under.
Economic Drive Behind AI Development
00:13:17
Speaker
Yeah. and And this, I think, is very important. And companies don't like to talk about it that much, though. Increasingly, they're starting to, which is, you know, why are companies investing you know a trillion dollars in AI data centers and infrastructure?
00:13:30
Speaker
Why are people pouring in billions and billions of dollars into these AI startups? Part of it is, you know, it's the next cool new thing. It's like crypto. It's like the Internet. It's like social media. You know, tech has these cycles. It pours lots of money, but it doesn't pour a trillion dollars into these things lightly.
00:13:48
Speaker
So why are companies building these things? Is it because they want you to subscribe to ChatGPT $20 month? That's a great business model if you can get millions of customers, but it's getting quite commoditized. you know <unk> It's a hard one.
00:14:02
Speaker
The real reason, I very strongly believe, is that the goal is to not give people the tools that will just make them more productive, but to replace people. So if you have an agent that can fully replace a software engineer that's making $150,000, $200,000 a year here Silicon with an system,
00:14:21
Speaker
and you can replace it with fully with an ai you know with an agi system and charge, say, $20,000 for that, that's a giant business proposition because you can capture a significant part of the $100 trillion dollar and global economy. A big fraction of that is labor.
00:14:36
Speaker
A big fraction of that is like intellectual labor and not physical labor. If you can capture a significant fraction of all the intellectual labor that is happening in the world as a company with your system and charge money for it, that is an enormous prize, and that is the prize that they are after.
00:14:53
Speaker
And so I think we have to be clear eyed about what the stakes are here. For them, it's that giant prize. For the rest of humanity, it is being replaced so that they can win that giant prize. and And so this is a big part of the driver is is that vision, you know,
00:15:09
Speaker
capturing a significant part of the human labor pool, and you know funneling it through one company. you know There are all the other things like like maybe we'll create an AGI system that will create lots of patents and we'll capitalize on those patents and you know the intellectual and the science and all those things.
00:15:27
Speaker
I'm all for those things, actually. like If we have AI tools that help us build new scientific discoveries and companies employing people using those AI tools, get those scientific discoveries, that's awesome.
00:15:39
Speaker
you know We all want better scientific discoveries and health and all of those other things. I think the the question is, are is it going to be humans that are doing those things or is it going to be machines that are doing those things predominantly?
00:15:49
Speaker
But anyway, that that is the but picture of why there is so much economic incentive to drive them at this point. AI companies talk about wanting to collaborate. At least they talk about this when when their safety teams are writing blog posts.
00:16:06
Speaker
They want to collaborate. They want to have and ah and an ability to slow down at at a certain time in the future. they are they're not interested in in a race to the bottom.
00:16:17
Speaker
Why is it that they can't seem to collaborate then? they the companies The CEOs of the companies believe in basically the same story of progress and how close we are to AGI, as you believe.
00:16:30
Speaker
Why is it that that they can't collaborate now, perhaps slow down, perhaps pause and really investigate these systems? Yeah, it's ah it's a great question.
Challenges in Collaborative AI Development
00:16:39
Speaker
I mean, i we're a very peculiar situation in that it it's not like these companies are ignorant of the the downsides or the risks.
00:16:47
Speaker
The company leaders themselves, you know, you can look up their values of P. Doom on Wikipedia, some of them, you know, they're not zero. the Many of these companies were founded by people who have been worried about AI and AGI safety and have you know created safety teams that are an intrinsic part of the company.
00:17:07
Speaker
um Some of those teams still exist and some of them don't, but but many of them have had those teams. And when asked what happens when things start to get scary, they say things like, well, we'll pause and well maybe we'll do a coordinated pause.
00:17:22
Speaker
But there's no answer, as you pointed out, to how that would actually happen under the current incentives. Like if at best you can imagine that you know the safety team comes and says, you know this this AI system is smart enough to exfiltrate itself into the open internet and start reproducing all over the world and having its own goals, we we should probably not like let that do that.
00:17:43
Speaker
So then what happens within the company? It will maybe, you know, not deploy that system for a while and and start working on can can we make that safer? Meanwhile, some other company with a little bit less of a safety culture is going to go ahead and company A is going to know that's going to happen.
00:17:59
Speaker
so So how long really is that going to hold out? And, know, we tried in early 2023 Future Life Institute future of life institute put out a letter that said, let's pause for a second, you know, six months, some seconds, and have the AI companies talk to each other and come up with a sort of joint plan for how we're going to continue AI research in a more safe and and sort of under control way.
00:18:25
Speaker
Nobody paused, right? They could have paused. We asked them. They said no. ah The government did not step in and make that happen. And so we raced ahead. And that's the situation that we're in now.
00:18:35
Speaker
The idea that as the money is bigger, the prize is closer, the incentives are stronger, then they're going to pause. Seems laughable to me. unless somebody Unless something dramatic changes, unless the dynamics change, or unless government gets more directly involved, unless there's much more public support and pushback,
00:18:53
Speaker
under the current dynamics, simply seems like it's not going to happen. All the incentives are just not for it. so So I think, unfortunately, that, you know, this is not a problem that's going to solve itself. The companies on their own recognizance are not going to solve the problem.
00:19:07
Speaker
They're not able to solve the problem, despite many of them having really good intentions. This is not a problem that they are able to solve themselves.
Need for Government Oversight
00:19:14
Speaker
So this could be there could be a role for government action here in making it easier for companies to collaborate and so to slow down and to to act more responsibly as we get closer to AGI.
00:19:29
Speaker
How would something like that work concretely? Because you sketch out a bunch of ah options in Keep the Future Human for this. There's two ways to think about this. One is, as you say, sort of enabling companies to do something that they might want to do, at least some people within those companies might want to do at the right time.
00:19:47
Speaker
I would love to have you know the the secret poll of if you had the button that you could push that would delay AGI and superintelligence for five years, for 10 years, how many people would push it.
00:19:58
Speaker
you know I'd slam that button. Future of Life Institute, many people would slam that button. I think most of the public would slam that button. Within the companies, obviously some people would not because they they could and they're they're not at at least they're not saying they would slam it if everybody else slammed it.
00:20:12
Speaker
But I think a lot of people within the companies would hit that button. And so I think there is you know some opportunity there. But again, the the coordination is not there and the the incentives are aligned against it.
00:20:24
Speaker
So I think a different agent has to step in. That agent is almost certainly the government. And what that could look like is, frankly, like not that weird. It's the sort of oversight on a major technology that we have for every other major technology.
00:20:41
Speaker
So, you know, when you if you had you know new weight loss drugs. It's a big race. This is a huge, huge value proposition. These drugs work. They're worth billions of billions of dollars.
00:20:53
Speaker
Companies are racing to develop new weight loss drugs. um Nonetheless, they have to test those drugs to make sure that they're safe before they get put on the market. You don't just have weight loss drugs. Like I came up with a new weight loss drug.
00:21:06
Speaker
It, it, it, almost certainly works, it might kill a lot of people. like There's like a 30% chance it'll, 10 years down the line, kill all the people who took it. But we can't stop because otherwise some other company will deploy it before us.
00:21:19
Speaker
like We would never take this seriously. We would not allow that drug to be developed. We would not allow that drug to... Well, maybe we would allow it it to be tested you know in the lab on rats. We wouldn't even allow it to be tested on people. We would not allow it out in the world.
00:21:33
Speaker
We would have safety standards that say, if you're developing this thing, you show us why you think it is safe. You prove to us with the level of guarantee that is commensurate with how risky and dangerous it might be that it is safe.
00:21:47
Speaker
um And then we will evaluate that as the government and say, yes, you may you know put that thing out onto the market or no, you may not. That's how it works with every other thing. AGI and AI is in this weird exceptionalist thing that it that has come out of the tech industry, which has been very unregulated.
00:22:03
Speaker
ah In many ways, that's been good. um And in certain ways, that's been not so good. I think we could have used a lot more regulation social media in particular ways. We could have used a lot more regulation on privacy in general. We have no privacy regulation.
00:22:15
Speaker
and so But, you know, that's been a mixed bag. but But now we're getting to systems that are going to have take actions in the world, have real consequences for people, tangible consequences. And it's it's just insane that they're they're like bizarrely accepted from regulation, at least in U.S., unlike every other industry.
00:22:33
Speaker
so that So the main thing we need to do is have... just common sense regulation of these like any other thing. One of those regulations would surely say, if you have an AI system, show us that you can control it.
00:22:49
Speaker
Show us that you can control it. Not that big of a thing to ask.
Compute Governance and Security Measures
00:22:52
Speaker
Who thinks it's good to have an out-of-control AI system that they've developed? Almost nobody claims that. So guarantee that, you know, if you're something making something really powerful that could cause damage, that could, you know,
00:23:05
Speaker
show us that you can guarantee that it's going to stay under control. So that's that that's the type of thing that is manifestly obvious when you just lay it out like that. And yet we are not doing. ah Companies are developing AI systems. They are not showing to anybody that they're controllable.
00:23:18
Speaker
Indeed, you know, they are doing evaluations and those evaluations come back and say, well, you know, when pressed, this AI system kind of tries to escape and copy itself onto other things. And if it can't accomplish its goal one way, it sort of just hacks into this thing and accomplish it some other way.
00:23:34
Speaker
This doesn't seem great. So see next time with the next bigger model. You know, there's no like, oh, like this is catastrophically not the right way to develop AI systems. We have to rethink this whole thing.
00:23:44
Speaker
No, it's sort of like, well, next time it might actually succeed in escaping, so we'll see you then, and maybe we'll do something about it at that Like, this is madness. So I think there's there's a level at which this is just common sense.
00:23:55
Speaker
Now, in terms of specifics, I think there are some key things with artificial general intelligence and superintelligence that we have to look at. I think the way to avoid a lot of the danger is to avoid this triple intersection of generality, intelligence, and autonomy.
00:24:12
Speaker
And we can specifically target that. We can say, if you're combining these different qualities at high levels, you should have higher scrutiny on this thing. You should have higher liability ah for when something goes wrong with this thing.
00:24:24
Speaker
You should have more oversight of how it's being developed. You should have more procedures in place and like higher levels of guarantee that you have to demonstrate before you can develop and deploy these systems.
00:24:36
Speaker
because they are the things that are most risky. In terms of superintelligence and like strong AGI, I think the the cleanest way to make sure that it doesn't go totally out of control is through the thing that has made it possible, these high levels of power possible, which is the computation.
00:24:51
Speaker
And we can talk, there's ah there's a lot of interesting conversations we had about, you know, where the computation is, who's doing it, what makes it possible, how we can limit it, all the handles that it gives us for for AI governance. But I think, you know, at some level, there's a common sense, you know, the thing the government regularly do to rein in their industry, so they're not incredibly risky for the public.
00:25:12
Speaker
And then there's some special things that I think would have to happen for AGI and superintelligence to keep the dynamics that we're under from like pushing it into this crazy runaway. Some of our listeners right now are feeling quite skeptical when we say governments, right?
00:25:26
Speaker
Which government are we talking about? Because in some sense, if the US government decides to unilaterally ah stop or pause AI development, they're in the same situations as the companies are amongst each other.
00:25:39
Speaker
The U.S. government is afraid that China will raise the head and and get to AGI and superintelligence shortly thereafter. So for this to work, would the regulation have to be international?
00:25:52
Speaker
Yeah, this is a great question. I think there are, again, things we have to separate out here. So I think there is... you know, AI tools like, you know, image recognition software and image generation software, where I think we do need some regulation. For example, I think there should be regulation around non-consensual deepfakes.
00:26:09
Speaker
We should not have those. You know, it should not be legal to non-consensually deepfake someone and represent them as the real person. so So we need some regulation around some gent some tools that are just intrinsically sort of harmful, potentially. And like, how do we, you know, balance the the things that we want to be able to do, which is generate images with the ah harm that those are going to cause.
00:26:28
Speaker
And I think that just has to happen on a national level, different or or a state level or or a local level, even different jurisdictions are going to different choices about how to balance, you know, the innovation and the products that you want to build versus the the safety, you know, and the so sort of social consequences that they might have. And different places are going to make different choices about that.
00:26:49
Speaker
Some of those might you know slow down the breakneck pace of creating new products. And that's a tradeoff. you know if you If you absolutely want to maximize more products faster, less regulation is good.
00:27:02
Speaker
More products faster, I think, does not necessarily lead to more happiness of people or more well-functioned society in many cases. But sometimes it does. And you know that's a complex tradeoff we have to make. Now, when you talk about like these very high-powered systems, again, we have to differentiate between AGI and superintelligence and going that down that road versus just powerful AI systems.
00:27:24
Speaker
So the that sort of core misunderstanding, I think, that people have when they make the argument that we have to race to get this thing first is that they think that getting this thing first means that they will have this powerful thing that makes them more powerful.
00:27:41
Speaker
This is really a lot, you know, the money is driving the race, but even more than the money, I think the power is driving the race, that people feel like developing AGI or super intelligence is going to give them power.
00:27:54
Speaker
Now, this is just wrong. That's the problem. AGI is something that is going to absorb rather than grant power because it is something because of the autonomy and the combination with generality and intelligence.
00:28:06
Speaker
This is something that is going to act on its own. It's going to have goals that will almost inevitably conflict with human goals. And even insofar as it doesn't, it is going to be something that insofar as it doesn't, we give power to. We will delegate lots of things to the AI systems.
00:28:22
Speaker
They will give us advice that is good over and over again. And so we'll just start listening to it and pressing yes, yes, yes, over and over again. We will give them power even if they don't take it And so the the idea that we will have, you know, a person that is in charge of this super intelligent system that is thinking, you know, a thousand times faster than them, more clearly with, a you know, a huge amount of memory, thinking in terms that they simply can't understand or follow, and that they're going to be in charge of that.
00:28:52
Speaker
doesn't make any sense. It's like a fern controlling General Motors, as I put it in the essay. like You can have rules in General Motors that are like, keep the fern happy, but General Motors, and there's no sense in which a fern can control General Motors. it just aren't They aren't the sort of things where a control relationship works in that direction.
00:29:10
Speaker
So the fern might do fine true under General Motors, but there's no control relationship. And so I think when people feel that they are going that they need to race in order to get that power, they're just not understanding the sort of thing that they're dealing with. They're thinking of it as a tool.
00:29:25
Speaker
What they're building is not a tool. If they were building a tool, this sort of argument would make sense. And so I think we we really have to start there. Like the U.S., you know, in terms of the U.S. versus China, the U.S. and China both have to understand this.
00:29:38
Speaker
They have to understand that if they build AGI and superintelligence, that is not something that will suddenly make U.S. the ruler of the world or China the ruler of the world. That is something that will make superintelligence the ruler of the world.
00:29:50
Speaker
And especially if it's done uncarefully in a race between two countries that are, you know, geopolitical adversaries and where there's no you know, time or energy or anything else to think about how do we actually keep this thing under control?
00:30:03
Speaker
Even if it were in principle possible, we certainly aren't going to do that in a race situation where we're trying to get to the finish line, whatever, however you define that first. So I think that the crucial thing is to have that realization that this is not a race that can be won.
00:30:16
Speaker
If a race can't be won, you choose not to enter that race. Now, you might then worry, well, maybe the other side doesn't get that the race can't be won. Like, I see that the race can't be won, that this is, you know, suicide to run the race, but maybe they're, you know, not getting that and they're going to run the race anyway. Or this, you know, this other third party is just going to roll the dice and think like, well, maybe it's controllable.
00:30:39
Speaker
And so once there's a realization that the race can't be won and that we must not run it, then the people who realize that have to band together to prevent the other people from running the race and to ensure that their prospective adversary or the the other prospective racer continues to get it and and aren't sort of faking it or like pretending to get it and secretly they're going to run the race anyway.
00:31:04
Speaker
So there has to be mutual verification. And that's the sort of thing that we've had, you know, in treaties before, like the Space Weapons Treaty. We realized that it was a bad idea to have nukes in space that were aimed at civilian populations, super or military ones, like super destabilizing.
00:31:20
Speaker
You could drop a nuke from space really quickly. There's no no defense against it. so So the U.S. and the Soviet Union realized, like, we better not have Like, that's going to be bad. Let's come up with a agreement that we're both not going to do that because it's very destabilizing.
00:31:34
Speaker
If we have them, it's like going to push us toward more of a mutually, you know, a mutually enacted destruction. They came up with an agreement and then they enforce that agreement and, you know, keep trying to verify that agreement.
00:31:47
Speaker
We're going to need something similar with AGI and superintelligence at the international level at some point. But I think the very first step is the the realization of what the truth of the situation we're in is.
00:31:59
Speaker
How do we develop such a system for monitoring and verifying which giant training runs are carried out and in different countries and by different companies?
00:32:10
Speaker
Yeah, this is this is crucial. And here, there's actually good news. you know A lot of the news on this this topic is bad, but here's here's some good news. you know People think, oh, you can never control AI because it's software. You can just copy it.
00:32:22
Speaker
like People are doing it all over the place. no You can't stop them. Chips are everywhere. Some of these things are true. you know The first two are true. The third one is not. So chips, the that custom chips that these AI systems are trained on are the most, I think, amazing and complex and sophisticated artifacts that humanity has ever built.
00:32:43
Speaker
100 billion transistors on this little wafer. the The chips are more or less all fabricated by one set of foundries and companies in Taiwan, the craziest place in the world for this to be It's almost like the plot of a bad sci-fi novel that it would have to be in Taiwan.
00:33:00
Speaker
Right. All the chips are going to just be right there in the disputed territory. yeah But they're all basically made there. There's lots of effort to build them mouse elsewhere, but it's really, really hard to to do this. They're all built with machines that are developed by one company in the Netherlands.
00:33:14
Speaker
Nobody else can make these machines. There's one company that can make them. And the chips themselves are designed by a handful of companies. Nvidia and AMD and Google make some and a couple of others, but it's a handful.
00:33:26
Speaker
So there's an incredibly tight supply chain that goes into the production of these chips. Way, way harder to build one of these chips than to enrich uranium. Like I'm a physicist, you know, give me a billion dollars. I could probably figure out how to enrich some uranium if you left me alone for a while.
00:33:43
Speaker
I know absolutely that I could never develop an H100 GPU. It's just like like the amount of the technological stack that goes into that is so high that even you know IBM can't do it um and and Intel can't do it, even though they're trying super hard.
00:34:01
Speaker
Yeah. Much of the knowledge about how to do this is tacit in the company. So it's not even public how you would develop a chip like this. so So that means that there's an incredibly tight supply chain for the chips that make all this like high-powered AI possible.
00:34:17
Speaker
But it's actually much better than that. Because you know if you think of, you know here's my my phone. If I leave my phone lying around and someone swipes it, that's super annoying because like now I have to go get a new phone and like I don't have my phone for a while.
00:34:31
Speaker
But also they don't have my phone because as soon as this happens, I will go in and I'll say, Apple, please deactivate my phone and my phone will become a brick. Now, what underlies that technology is a set of hardware security measures that are built into almost all modern phones that tie the software on the phone cryptographically to the hardware in the phone.
00:34:52
Speaker
And it says you can only boot you know a particular Apple-approved operating system, that certain types of data can only exist encrypted in this so-called secure enclave within the phone that nothing else can access, even the rest of the operating system of the phone can't access your biometric data, and that allows all sorts of capabilities like Apple saying, okay, that phone is shut off, you can't use it anymore, allows Apple to, or you, to know where your phone is, right? If I i leave my phone before someone swipes it, I can find it um with my iPad or you know my my laptop or something because it phones in and says where it is.
00:35:27
Speaker
All of these things are totally possible to do with GPUs as much as they are with phones. You can do it with a $500 phone, do it with a $20,000 GPU. And gpu and Even the mechanisms for doing that on the hardware level exist in current GPUs like the H100 and further generations.
00:35:45
Speaker
And so even better than just saying, like, let's watch where the chips go and make sure that, say, you know, North Korea doesn't accrue a giant pile of H100s to develop their secret AGI system.
00:35:59
Speaker
Much better than that. We could, if we wanted to, know exactly where every high-powered AI chip goes. And we could monitor, if we wanted to, how much computation it's doing.
00:36:12
Speaker
And we could do this in a way that was privacy preserving, where like certain information is shared, like where the chip is and what and how much computation it's doing, and nothing else, like what sort of computation it's doing or on what data or any of those things. So there's a a fine-grained ability to share some information and not others and do so in a sort of cryptographically secure way.
00:36:32
Speaker
So this is great, like all sorts of stuff that you wouldn't think was necessarily possible, like having sort of automatic oversight of giant clusters of computation to see automatically how much computation they're doing and to be able to say, like, OK, if you want to do a giant computation, ask this agency for a license to do that giant computation that agencies can grant that license or not.
00:36:55
Speaker
If it doesn't grant the license, you can't do that giant computation. If it does grant it, the agency can see how much you know computation is being done. And if something starts to go awry, the agency can, principle, pull the plug or the company that's doing it can pull the plug. So this is something that it's worth saying.
00:37:13
Speaker
One of the first things that people say when you talk about AI getting out of control is what? Shut it down. Shut it down, unplug it, turn it off, right? It's obvious. And yet we have to have a way of actually doing that.
00:37:25
Speaker
So if the AI system is super intelligent and is getting out of control and is potentially proliferating all over the place and is smarter than you and doesn't want to be shut off, is it going to be so easy to shut off? So we can't just say like, oh, when the time comes, someone will figure out like what are the right buttons to push to shut it off.
00:37:42
Speaker
We need to build that into the system. And oh ah very robust way to do that is with the chips themselves. If the chips have to continually ask for permission to keep running,
00:37:54
Speaker
then you don't have to shut off the system proactively by like pulling the kill switch. You just have to stop giving permission and the system stops operating. So this is a ah what's called a dead man switch, and this is totally doable with the sort of hardware that we have right now.
00:38:07
Speaker
You would not want to do this on a day-to-day basis for something. You don't want something to like stop working because it can't call home and keep getting permission. But if you're building something that is incredibly powerful,
00:38:19
Speaker
or as you know could proliferate, could get out of control, you absolutely should have a system like this that is making it possible to turn it off or just have it shut down if you can't affirmatively keep it going. So all these capabilities exist to do this sort of covenant governance using the the computation capabilities themselves.
00:38:38
Speaker
I think that opens up lots of like interesting mechanisms where International cooperation could be better. Corporate, like the companies cooperating with the government, could could have different forms than just the government saying, yes, you can do this, no, you can't.
00:38:53
Speaker
You could even have multiple companies that are joint governing some project. There are all all sorts of interesting possibilities that you wouldn't necessarily guess that exist because of this tight supply chain with chips and the capabilities that the chips themselves have in combining hardware and software in these different um modalities.
00:39:10
Speaker
Does this picture of compute governance change given what we've seen with something like DeepSeq or in general, the increased importance of inference time compute, which is when you ask ChatGPT something, it will spend some time thinking and generate a much better answer, which is especially useful in mathematics and programming and so on.
00:39:31
Speaker
Are there other emerging techniques that undermine your vision for compute governance? I think they make it a little bit more complicated, but i but not that much. So I think the the easier idea to deal with would be just that what it takes to make a very powerful AI system is a giant training run.
00:39:50
Speaker
And once you have it, then basically the inference is cheap. you know Producing results is relatively inexpensive. I mean, it it doesn't feel that way to the companies who are spending zillions of dollars building the data centers to do that inference. But for a given customer, that's because there are a lot of customers. For a given customer, quite cheap to run the model as compared to training the model.
00:40:09
Speaker
Now, what has been discovered is that it doesn't have to be that way. You can pour a lot of computation into doing the inference, giving the AI model you know a ton of thinking time ah before it produces a result, and that this pays off significantly.
00:40:24
Speaker
Now, this changes the equation a little bit because you have to think both about how much computation is used in the training and also how much computation is used in the running of a model. But those are both large amounts that you can manage with compute governance.
00:40:37
Speaker
So the the change really is that you have to not just pay attention to how much training compute is used, but also how much inference compute is used. and have limits, you know, if you want to limit the power of, you know, ai models so that they don't run away into this crazy AGI superintelligence direction, you'll have to limit both the computation that goes into training and the computation rate that goes into ah inference after the model is trained.
00:41:04
Speaker
So it does complicate it a little bit more, but it doesn't fundamentally change the fact that huge amounts of computation are needed to get the high power AI systems and we can govern that.
00:41:15
Speaker
what What would change things and what will change things is that models get more efficient with time. And so it's it's certainly the case that it costs a lot less, both in training and in inference, to build a model now that is capable as capable as the original GPT-4 than GPT-4 was.
00:41:35
Speaker
so So there have been huge increases in efficiency. you know They're not orders of magnitude, but they're probably an order of magnitude more efficient. I think there is likely to be some sort of floor on how computationally, you know, how much computation it takes to get something that is sort of human level or or superhuman in its capability.
00:41:56
Speaker
I don't think you can, you know, with a millionth the computation that is spent in GPT-4, get something as competent as GPT-4 is in general. you know If you run the numbers very interestingly, like these are very hard numbers to compare, but the amount of computation that went into GPT-4 is roughly, again, very rough numbers. The sort of amount of computation that the brain does over 30 years, the amount of computation that GPT-4 uses when it's doing inference, or even a more inference-heavy system does while it's doing inference, is, again, order of magnitude similar to what we think the human brain is doing in computation.
00:42:33
Speaker
There's so many asterisks that I want to write as a scientist here, like all over all of this stuff, but like very, very roughly you can look at the numbers. So I don't think that there's going to be something that but does it in a millionth as much computation or even a thousand.
00:42:45
Speaker
And so I think if what we want to do is curtail a runaway to things that are way, way, way smarter than us, I think capping the sort of computation level at the sort of computation that we can now do, like maybe a little bit higher, will probably curtail that and we'll get like somewhat more powerful systems than we have now that fit into that computation limit, but not like radically world-changingly destroy civilization levels of super intelligent power with that same level of computation.
00:43:12
Speaker
Someone might say, you know Anthony, all of this is a gigantic ah downer. right we We were just about to to reach the stars. We were just about to explore the universe using this amazing super intelligence for our benefit.
00:43:25
Speaker
We're just about to discover all possible scientific theories and and exhaust the tech tree and all of this. Spend a little time talking about what we can get from tool AI alone.
00:43:39
Speaker
So you said something interesting in asking that question, which is that we were about to do all those things. We were not about to do all those things. The AI was about, you know, we were about to create AI that would do all those things. I know, I know. And I think this is important, though. This is this is critical because we can still do all those things with powerful AI tools.
00:43:55
Speaker
That is, in fact, what I think most of us want. Like as a scientist, I like having something that can, you know, I love Mathematica, I love Google, and I love, you know, new language models that allow me to do science more effectively, to like you know, summarize giant things, to do, you know, do little pieces of research problems.
00:44:15
Speaker
I don't feel like I necessarily want to just say, here's the deep question that I've had all my life. Can you go and tell me the answer to that question by doing all of my thinking for me?
00:44:26
Speaker
Like as a physicist, That takes all the fun away. Like that it doesn't just take the but time and like my income and stuff. It takes all the interesting and enjoyment and meaning away as a human that is doing that intellectual endeavor.
00:44:38
Speaker
So like I want those tools. also But I don't want to be replaced as as as a physicist. I think most scientists want cool tools to make them more capable as scientists so they can do better science for the reasons that they do science.
00:44:51
Speaker
We can still do that. we just would be not turning it over to the superintelligence to do it all for us and hope that like they'll share it with us and like things will turn out really well. and And like that, even though they're out of control, that they'll still treat us nicely and all of those things.
00:45:12
Speaker
So i I think it might take a little bit longer. Like I think if, if we let the intelligence run away, go to superintelligence and it, and ask it like, please, superintelligence that is like another civilization that we are in negotiations with, would you please cure cancer for us?
00:45:29
Speaker
Like, it's probably going to come up with a cure for cancer faster than a bunch of human scientists using AI tools. But do we need that in like 10 minutes? Or can like we wait five or 10 years to let humanity do it?
00:45:42
Speaker
I kind of feel like for the risk of losing control of civilization and becoming extinct and all those downsides, we can wait a little bit to give human scientists empowered by AI chance at the problem first. so I share the excitement like to to build awesome new technologies, to like cure diseases, to achieve like lifespan extension for as long as we want to live, to colonize space. like I'm all for those things and I'm super excited for them.
00:46:12
Speaker
And I think we can do them like much faster than we possibly could have as we develop these more powerful AI tools. I just am not so excited about like asking AI to do it all for us and hoping best.
00:46:24
Speaker
Closing the gates to AGI and superintelligence doesn't involve closing the gates to something like AlphaFold, a narrow system that's nonetheless extremely capable in its domain and that enormously advances science within the domain it's it's working. So we could we could be pushing the frontier on systems like AlphaFold in a thousand different domains while staying away from a fully general system like AGI.
00:46:52
Speaker
Yeah, and in and in different combinations too. So I think, again, it's that triple intersection of high, high levels of like human expert levels and of all of those three things combined that is AGI and where the real danger zone is.
00:47:06
Speaker
You can easily imagine something that is like, you know, just a reasonably competent lab assistant that helps you out, um but isn't like a super genius at,
Developing Controllable AI Tools
00:47:15
Speaker
you know, Nobel Prize winning biologist, but helps you out in the lab and runs alpha fold and lots of other tools like it with you.
00:47:23
Speaker
right This is good. So I think you can have things that are you know autonomous, like highly autonomous and highly intelligent, but not general and highly general and highly intelligent, but not autonomous in various combinations of these things.
00:47:36
Speaker
Or things like AlphaFold that are just super intelligent, but very narrow, not autonomous or general. All of those things you can have and will be very powerful. It's just that triple combination that we have to avoid.
00:47:48
Speaker
So there are just so many things that you can do. And most of the things that we actually think of are those things that we can do with tool AI. Now, I should say, this is making it sound a little bit easier than it actually is.
00:47:59
Speaker
And the reason is because if you try to build something that is just very, very intelligent, it turns out that it is useful to make it more general. And if you make something that is very, very general, it turns out that it like tends to get more intelligent and it tends to get more autonomous.
00:48:14
Speaker
So these things do drag each other a little bit more toward this triple combination. And it will take actual effort to so like develop things you know in a deliberate way that remain controllable tools as we try to like ramp up one of those aspects.
00:48:34
Speaker
And so, i yeah i don't want to I don't want to say like all we have to do is not try to do this hard thing and do these other things instead, and it'll be super easy to avoid the the AGI. It will not be. We have to be very careful and deliberate about that and have the right incentive structures for it.
00:48:49
Speaker
um So I do think it's possible, but I don't want undersell how, you know, that that it's like the defaults are easy. If it was easier than default, we'd be doing that. um and then And, you know, we wouldn't, it wouldn't be so worrisome that we're running into this.
00:49:02
Speaker
There'd still be the incentive problems and so on, but but it would be a lot less worrisome if it were like so easy to just keep things to be these tools. Why isn't alignment the way forward here?
Lag in AI Alignment Research
00:49:12
Speaker
Why can't we build an AGI that's aligned to our values? An aligned system, why isn't that the way forward? Yeah, so so you can imagine, like if we really, really got alignment right, you know that the AI system would take a look at humanity and give us like just enough to make everything good you know and give us the right amount of help to build our own technologies, but not too much that we feel kind of put out of work and and you know and keep us safe, but not be too motherly to us and all this stuff.
00:49:42
Speaker
and it would And everything would be would be wonderful. like I think therere as you delve into what that would actually look like, like what does humanity want? It's like very, very unclear. like All segments of humanity want different things and many problems with that.
00:49:55
Speaker
But the the real deep problem with it is that we have no clue how to do it. like We just don't know how to solve that scientific problem. We're The alignment research is way, way behind the capabilities research.
00:50:09
Speaker
We're like way closer to figuring out how to build agi and super intelligence than how to control it or how to align it in this like very deep sense. So I think that's that's a core problem.
00:50:21
Speaker
For how long would you say we've been working on solving the alignment problem seriously? Perhaps since 2010 or 2015? Do you think we've made incremental progress? Or do you think this is something that will have a solution that that's ah that's more of an all or nothing um solution?
00:50:38
Speaker
I think we you know certainly progress has been made on certain types of alignment, like ah the the somewhat shallow type that we see in the AI models now, where if you ask Claude to do something deeply antisocial, Claude will just say, I'm sorry, I can't do that, Dave, and won't do that deeply antisocial thing.
00:50:57
Speaker
Now, the problem is that you know although they're trying very hard, It's still the case, as far as I understand, that like the right random guy on the internet will say the right magic words and Claude is just ready to grow anthrax for you.
00:51:11
Speaker
So it's it's like this weird thing where the model deeply professes its love of humanity and its deep moral code that's so important to it. But then you push the right buttons, figuratively speaking, and it's willing to do just about anything.
00:51:24
Speaker
It's very bizarre. This is not how people work. this is you know So these models like seem very human in lots of ways, but they are not like people. So we don't know at this sort of deep level how to make the AI system reliably care about humans or human needs in the way that but people do. i mean, people's alignment is also very complicated.
00:51:43
Speaker
It's very difficult to tell whether, you know, do you, Gus, you work at Future of Life Institute. Do you really love the Future of Life Institute or do you just act as if you love the Future of Life Institute? I believe that you really love your work and and humanity and everything else, but like you could just be like collecting the paycheck and faking it the whole time. like I we really wouldn't know.
00:52:04
Speaker
people are you know There are human sociopaths that know exactly what is right to do according to society and keep that veneer. They don't really care. And if they have the opportunity, they will go and do something totally different.
00:52:16
Speaker
um And AI systems, similarly, like in the same way that we don't know how to make people deeply care about something that they don't deeply care about already, we don't know how to make AI systems deeply care about something in a way that we can feel sure of and and can't just be like undermined in some way. So we don't know how to do it.
00:52:35
Speaker
For listeners wanting a visceral experience of this, it's possible to interact with ah raw foundation models. So before they've they've gone through the process of making them appear more human.
00:52:46
Speaker
So if you if you who are listening to this can find such a model, that's an interesting experience to see what these systems are like. It's much more chaotic. It's much more potentially evil than than the systems systems like ChatGPT that have this very nice persona when you interact with them.
00:53:03
Speaker
Yeah. And it's important to know that like beneath the the persona is that base model still. ah This is, you know, the origin of that Shoggoth meme where the the tentacled monster is there and it's got a happy little smiley face on it.
00:53:15
Speaker
It's still there. And, you know, i I don't think it's that way with people. You know, i I don't think there is a hidden monster inside of every person. Like we can have long moral debates about like the the road of evil runs through every human heart. But I don't think there's something deep and alienly, you know,
00:53:32
Speaker
deeply alien hiding in every human being that if you just scratch the surface in the right way, comes out in the way that there is in these models. So we don't know how to do it. there there the The current plan, if you look at the AI companies and what they describe, they know that they don't know to do it for AGI and superintelligence.
00:53:51
Speaker
They know that they don't have time to figure it out. And so their plan is to say, we're going to build a capable AI system and then we're going to use it to do our alignment research for us.
00:54:02
Speaker
So we we build more powerful AI systems and then we we say, so smart AI system, please tell us how to control you and how to align you to our wishes. And this kind of sounds good.
00:54:14
Speaker
it It certainly sounds like better than just failing to to do it. Like, you know, if we've got this powerful AI system, why not use it to help us with our new scientific problem, which is aligning AI systems?
00:54:26
Speaker
But if you think about what that really means, so an analogy that I put in Keep the Future Human is like, let's let's try this with humans. So there's a fern, again, that we want to have taken care of by General Motors.
00:54:37
Speaker
So the fern waves its bronze and and a kindergartner says, oh, look at the beautiful fern. I want it to be taken care of. So the kindergartner says, OK, I've got this hard problem.
00:54:48
Speaker
This General Motors, it might be like building a factory that's going to hurt my fern. So like I need to control General Motors. So what can I do? I can't go to General Motors. that That's like too hard. But there's a first grader that I can enlist their aid to help take care of the firm. So I'll convince the first grader and use the first grader's better expertise at controlling General Motors to so help me out.
00:55:11
Speaker
And so they enlist the first grader, tell it what the firm wants, you know convince it to of the the utility of their mission. First grader says, crap, I can't control General Motors either.
00:55:22
Speaker
But there's a second grader that may be better at this than me. So they convince the second grader to help them. The second grader indeed is better at figuring out things than the first grader. And they go all the way up, third, fourth, fifth, sixth, high schooler, college student, graduate student.
00:55:38
Speaker
They get a manager. They get a like like higher up manager, they get an executive, they finally, like all the way up, have these instructions and chain of of control all the way up to the ah CEO of ah General Motors.
00:55:51
Speaker
Now, is the firm now actually in control of General Motors? Again, no. General Motors might take care of the firm. It might take care of the firm.
00:56:03
Speaker
But it is going to take care of the firm exactly to the degree that the shareholders and stock and board and other executives at General Motors happen to care about ferns and kids. It's gonna have nothing to do with the whole chain of like instructions that work their way up to General Motors.
00:56:21
Speaker
Nothing to do with it. And I think that is the that is the problem with this picture. Like it it breaks this impossible problem into lots of smaller problems. This is often a useful technique in hard problems, but ultimately the there's there's no actual, you know, those, you haven't,
00:56:39
Speaker
undone the impossibility of that problem by breaking it up into many smaller problems that just add up to the same impossibility. um And so that's my concern about this this direction. Like, it does sound good, but I think, again, when you have a fern controlling General Motors, it's not a thing that can be solved.
00:56:55
Speaker
Like, I wish that it was. And if it's a thing that can be solved, it's going to be solved using radically different techniques and radically different architectures than the type of thing that we're going to that we're talking about. Where it won't be General Motors, it'll be something that is powerful like General Motors, but is a very, very different type of entity.
00:57:12
Speaker
And that we can talk about. Yeah, that that's what I think of ah sort of the alignment program right now. The fern might be taken care of for for decades even, but because it it doesn't understand and it couldn't possibly understand the motives of General Motors, it can't understand whether it will be taken care of in the future, whether something will change, whether it will suddenly be destroyed.
00:57:34
Speaker
And that's not a situation we want to find ourselves in. Exactly. Again, like might be a great life for the firm. So I don't want to say that it's, you know, I get the vision that, you know, the world is kind of in a bad place aside from AI, but right? There's a lot of things going wrong.
00:57:50
Speaker
There's you know, humanity has messed things up pretty badly on a lot of fronts. We've done really well on certain other ones, but we, you know, we're we're not in a great spot. People are feeling quite pessimistic about the future in general.
00:58:02
Speaker
There's a real desire, like, can we have someone else in charge, like, a know, some other thing that will save us from ourselves? So I really get that that desire. and And I don't want to discount that. Like, ah you know, it it may be that we things run away to AGI and superintelligence and humanity, like,
00:58:23
Speaker
has a good life, you know, like the fern has a good life. and And the world like functions basically in a way that works well for a lot of humans. It just, A, we can't count on that. We'll be out of control.
00:58:33
Speaker
and And we won't really have any role in that. That will be the AI world that we're sort of spectators and and not really participants in. So that is a decision we have to make. I mean, I do think...
00:58:44
Speaker
If, you know, we had a ah real understanding amongst many, many people of what sorts of systems we're contemplating, what it would mean for society, we had a sort of global conversation.
00:58:57
Speaker
These are the risks. These are the rewards. This is the kind of system we would build. This is the scientific understanding of this. And there was a sort of global conversation. We decided, yeah, but let's do it.
00:59:07
Speaker
Like, we're probably going to lose control of this, but like we have pretty good sense that we will have systems that take good care of us. um And we just think they're they're they're going to be better stewards of the planet than us. And we're mucking it up. So let's try something different.
00:59:20
Speaker
I'd be in support of that. Like if that was, I believe, I really believe in democracy and I really believe in people choosing their own future. If that's the future that they want to choose, we should go with that. That is not what's happening right now. you know it's it It's a handful of CEOs of tech companies who are deciding what this is. Most people have no clue what the situation is or what's happening.
00:59:38
Speaker
They certainly have no voice in determining what it is. That's wrong. How do we decide whether we want to pursue AGI at some point? So say we close the gates to AGI now.
00:59:51
Speaker
How do we decide whether to open the gates later? Yeah, i I think we would want to have the sorts of things in place that we don't have now, like like guarantees that it's going to be controllable or guarantees or, you know, at the level of mathematical proof that it is going to be beneficial and aligned with human interest by some definition that people are really comfortable with.
01:00:14
Speaker
And I should say that controllability is not necessarily, you know, there's a tricky, there are lot of hard ethical decisions. so So one of them is to what degree do we want AI systems that are controllable versus to what degree do we want things that are good, like morally good and and and aligned with human interests, but are sovereign, that are not controlled?
01:00:35
Speaker
Because you can imagine that You know, if we just dump a whole bunch of like, you know, obedient superpowers into the world, you know, ah imagine these words superheroes that would just like super slaves and, you know, whatever their owner told them to do, they would just immediately do with Superman's powers.
01:00:52
Speaker
Like, what would that world look like? Like that is a destroyed world, like as everybody fights it out with their slave superhero and fights all the other slave superheroes and New York gets destroyed in every fight. So if we have obedient super intelligences that just do what people tell them to do, that is probably a super dangerous world also.
01:01:10
Speaker
So on the other hand, if we have, you know, sovereign agents, we're not in control of them and we really have to and and we're not going to get it back. So we really have to be ensured, ensure that they're aligned with us. So I think we would have to have a real vision, like what is the world that we want to build with these and, and buy in and have these things, not at the level of,
01:01:29
Speaker
let's just, you know, learn how to fly the airplane as we're building it or build it as we're flying it, whatever the analogy is, the, you know, putting it together as we go, but we're going have, like, we would want to actually understand the science and the engineering of these systems before building them.
01:01:43
Speaker
Like we do with every other technology, by the way, we don't just like throw together a nuclear power plant or a nuclear weapon or a airplane. Like we engineer that thing using principles that we understand. That is not how we build AI systems now. We didn't tell much of that story, but that is in case anyone is is unsure, that is not how AI systems are built.
01:02:03
Speaker
They are not constructed and engineered using high understanding. So we would want that, I think. We would want to build these things using very deep understanding, ah probably including using the powerful AI tools that we have um before we would build them.
01:02:18
Speaker
Yeah.
Expert Opinions on AGI Timeline and Risks
01:02:19
Speaker
Why is there so much disagreement about where AI is going, at what pace AI is advancing, whether what we're seeing is exciting or whether it indicates that we're moving towards the danger zone?
01:02:33
Speaker
People are looking at the same evidence and are coming to wildly different conclusions. You are landing at AGI being very close, but basically imminent and and quite dangerous.
01:02:45
Speaker
ah Other people, other AI experts are landing at very different conclusions. so how do we How do we collaborate given that massive disagreement? Yeah, I think there's a few different varieties of disagreement, some bigger than others.
01:02:59
Speaker
I think there's much less disagreement, I feel, in if we build AGI and superintelligence, if that is of high risk. Like, it's, I think... pretty obvious that like it could turn out well and it could turn out catastrophically amongst people who take that seriously.
01:03:17
Speaker
So I think there's less disagreement there. I think the bigger disagreement has always been you know how far away is it and it's even possible. So I i think This has gotten, frankly, like pretty bizarre to me.
01:03:29
Speaker
So I i i understood you know when GPT-3 and 3.5 came out, there really was a feeling like, is it smart or is it faking it? And you can kind of read it both ways.
01:03:41
Speaker
you know And it's this very strange sort of frision of, is it saying that thing because it's true or because it's the next word in that sentence by probabilitybable probability? And by could like what is the different, like these deep philosophical questions?
01:03:56
Speaker
But now we're at the point where you know the there is a problem, you know there's an AI system, you give it a novel problem in like quantum field theory and theoretical physics, and it solves it like a person would, like a professional physicist would.
01:04:12
Speaker
It's a new problem, it gives you the solution. That is a smart thing. It is doing reasoning. Like if you want to use some other words for it, you're not using reasoning like everybody else is. Like it it's telling you the steps that lead to the conclusion that is true based on like the rules of reasoning and the assumptions that you put into it that are true. Calling it something else just makes no sense.
01:04:33
Speaker
So we're in this weird place where on one hand, you've got systems that are, you know, at the level of PhD physicists in terms of solving problems in in certain ways.
01:04:44
Speaker
And then at the same time, people are saying, well, it's just regurgitating and interpolating input data and it's not really smart. And AGI 100 years away. These people are very much fewer and farther between than they were two years ago, but they're still around.
01:04:58
Speaker
It's an odd thing. so So what I did in Keep the Future Human was i I really tried to break this down. Like, okay, let's look at the systems as they are now. Let's list out all the things that they seem to be missing, you know, that humans have, that a general intelligence would have.
01:05:12
Speaker
You know, things, so reasoning, I think we're no longer missing, like systems now do reasoning, but there are lots of other ones. Like they're not very self-aware in the way that but people are. and And lots of people are not also, but we we sort of know what's going on in our minds and we can analyze, you know,
01:05:27
Speaker
Did we do a good job on that? We can look at the thing that we just did and check whether it makes sense or not. So, you know, self-awareness, creativity, originality, long-term planning. So there are all these capabilities that humans are quite strong at and AI is weaker at.
01:05:42
Speaker
And you can list them all out and you can think, Okay, which of these things are likely to arrive with just scaling up systems as they are now? Which of these things might require genuinely different novel techniques to get?
01:05:55
Speaker
And so I went through this exercise. I'd encourage anyone who's sort of like wants to think carefully about this to to go through that table. You might have some more entries that you feel like need to be added, add them in, and take each one of those things and think about, okay,
01:06:07
Speaker
Are we going to get this from the AI systems by just dialing up the computation? Can we get it from known techniques you know that that are used in other AI systems but not language models right now, like more reinforcement learning or more online learning, whatever? whatever Or is it really going to take something fundamentally new?
01:06:24
Speaker
And when I went through that analysis and listed it out, it like... it It seemed clear to me that a lot of those things we're just going to get, either using scaling or known techniques.
01:06:34
Speaker
There are some that might be harder. But again, four Manhattan Projects worth of the world's smartest people working away at them. Like, how long is it going to take? So I think we're in a weird place.
01:06:47
Speaker
i like the the I think there's legitimate disagreement as to whether some of these things are six months away or two years away or five years away. I think even the skeptical AI experts are now sort of in the five to 10 year range. Like that's a long time now. It's like the, I'm an AI skeptic. I don't think we're going to have AGI for a long time. Certainly not this year.
01:07:10
Speaker
Like what? And this has this has changed recently, quite recently, where now the the AI skeptics think that AI is always, it's more than a decade away.
01:07:21
Speaker
And another interesting point is to talk about math, reasoning and and and programming abilities. I remember interviewing people on on this very show, people who are who are ai experts just a couple of years ago about whether large language models could ever be impressive at mathematics and programming and so on.
01:07:40
Speaker
Now that's basically a settled question. They are absolutely great at it. It's one of the things where they are most impressive. So these these domains can switch rather quickly. And that's something to keep in mind also.
01:07:52
Speaker
And I've watched it unfold in detail. So I, you know, I co-founded Metaculous, this prediction platform. And since the beginning, one of the key things we've looked at is AI. And we've had lots of questions over the years, including some headline like, when is AGI coming questions, but lots of much more detailed ones.
01:08:08
Speaker
And I remember writing the programming questions. And it just seemed like a fantasy that you would write to an AI program, like, write me a program that sorts these numbers. And it would just write a program.
01:08:19
Speaker
Like that was a crazy fantasy at that point. And now it's like utterly trivial. And we we like they write much, much more sophisticated programs than that. And so some of the things that that seem super far away are just here and clearly here and much, much faster than people predicted. So I i think if there's a trend, it has not been for the last five or 10 years that things have taken longer than predicted.
01:08:42
Speaker
But over and over again, they have taken as much or less time than was predicted. Plus, time keeps passing. You know, the the those 10-year predictions that were made 10 years ago, like, here we are, just in terms of ah the inexorable flow of time.
01:08:57
Speaker
The other thing I think that is worth pointing out is the approach that companies have taken. and taken. and And this, so there's a mixture here of economic motives and and other motives, which are a little bit less savory. So, you know, if if you asked someone thinking about AGI and superintelligence 10 years ago, like, what are the things that you must not do as you're building, and from a risk perspective, you're building super powerful AI systems?
01:09:20
Speaker
They'd be like, okay, whatever you do, keep it in a box. Don't hook it up to the internet. For the love of God, don't teach it how to program. don't You probably don't even want it talking to you like with regular words because it's going to convince you of all kinds of crazy stuff.
01:09:34
Speaker
So if you go down the list of things that people were worried about then, like you must not do, the 10 commandments of don't do with your super powerful system, we're just running through them and doing those things, like top priority.
01:09:44
Speaker
So programming in particular. The reason it is very risky to build, so to let AI programs systems know how to write programs is that that is the gateway to them improving themselves.
01:09:55
Speaker
They are programs, like mostly weights, but the wrappers and the and the like system that does the training and the inference are programs. The way that they will improve themselves is by doing mathematics and programming and you know maybe a little bit of physics and you know other sciences.
01:10:12
Speaker
And so that is the the danger mode. But it's not just neglect, it's on purpose. So if you look at the plans of Anthropic and OpenAI and so on, they are building AI programmers, not just because they want to help you know people do programming more efficiently. That is that Definitely an economic bonus for them because programming is one of the things that they are genuinely useful for, but also because they want to replace AI researchers with AI systems that can do AI research faster and hence get more powerful AI systems that can do AI research yet faster.
01:10:45
Speaker
And that's the intelligence runaway. They are trying to deliberately do it. i mean, that is their like stated plan at this point. So this is just complete madness. Not only are they like, oops, I've accidentally doing it the dangerous way, they're deliberately doing the super dangerous thing so that they can race to the end fastest.
01:11:02
Speaker
and And this, I think, just needs to be better like called out and understood. ah This is not inevitable. We definitely could like be not going down that road. And I think it's madness that we are.
01:11:13
Speaker
I want to end by us talking about the early days of AI risk and you were involved in that. And perhaps you could you could take us back to what were the important discussions back then? How did things evolve and what surprised you over time? We've touched upon it a little already where things have gone faster than than you expected and things are now more dangerous perhaps than you expected. But what did you believe about ai and AI risk in say 2012?
01:11:41
Speaker
Yeah, so things have evolved in in some interesting ways. that So I would say when when I first started thinking about this problem in 2011, 2012, founded FLI in 2014, the way most people were thinking was, you know, this is coming someday.
01:11:58
Speaker
Like, could be years, could be decades, could be centuries. the The problems are sort of clear. When you build things that are smarter than you, how are you going to control them? the you know The power is going to go to those things. you know Intelligence is the thing that is giving humanity you know stewardship over the earth.
01:12:12
Speaker
it is That stewardship will pass to whatever the most like intelligent capable thing is. like We should make sure that that's either under control or that it it is acting and deeply in human interests if we're developing that thing.
01:12:25
Speaker
and most of the field thought about this is a scientific problem that needs to be solved. So we need to figure out how to do alignment before AI and AGI in particular arrives.
01:12:38
Speaker
And as long as we figure out how to do that, everything's go be awesome because we're going figure out how to get the genie to be the nice wish-granting genie that likes you um and not the nasty wish-granting genie that like undermines you and does monkey's paw things and and like gives you what you don't want while fulfilling the letter of your wish. So this is how people were thinking about it that way.
01:12:59
Speaker
We want the nice genie. Now, all sorts of things have have evolved. I think that is still the vision that some people hold, that we just want to get the nice genie rather than the but mean genie.
01:13:11
Speaker
But I think it has gotten a lot more complicated. i I think switched standpoints earlier than many people did from this is a scientific problem to this is a governance problem. So i I would say like, even in the fairly early days, I saw like companies are going to be and efforts are going to be competing on this.
01:13:27
Speaker
They're going to want to get this power. the only thing that is going to stop them from doing what they want to do is some sort of governance system, probably governments. And so I sort of started fairly early as you know in this field thinking about what sort of laws and policies and things are we going to need.
01:13:46
Speaker
And this was part of the push behind the Asilomar principles, you know, and talking about like, what are what are the things that we should all agree that we should like do and not do as we were developing the systems? let's start with the high-level principles. The next step would be then translating them into more granular things and then you know governance mechanisms and policies and regulations and all those sorts of things.
01:14:07
Speaker
That plan is founded on the fact that like the translation the policy hasn't happened and the regulations have not occurred. But that was the sort of idea. So tell me about this move that you personally made from the technical side of AI safety to focusing on the government side of AI safety.
01:14:25
Speaker
Yeah, so I i think it it became clear fairly early on that if you actually want to impose something on on one of the developers of AI, it's going to have to be a government that does that.
01:14:37
Speaker
And that the incentives will not necessarily be toward everybody doing exactly the right thing at all times. They seldom are. And and see indeed, we've seen it play out that way. So if there's going to be something that says, no, you can't do that because that's not safe,
01:14:51
Speaker
That's going to have to be a government that says that. And so i I sort of saw early on, I would say, that we're going to need regulation. This is not a crazy idea. Like everybody, again, has has also believed that we would need it.
01:15:03
Speaker
But the you know what does that actually mean? how do we start to develop the principles and then the more fine-grained policies and and then regulations that we're going to need?
01:15:14
Speaker
but has to That you know generally takes a long time, so you want to get started on that early.
AI Policy and Societal Disruption
01:15:18
Speaker
And I think now everybody's doing AI policy. There's like zillions of people running around Washington and and and elsewhere talking about AI policy because the you know everybody sees that this is incredibly important technologically and like incredibly important in the policy sphere.
Writing on AI's Societal Impact
01:15:32
Speaker
Another thing that I think has changed in my mind from the early like alignment is everything sort of discussions, and i this especially became clear to me writing Keep the Future Human,
01:15:44
Speaker
is talking about these large-scale disruptions of society and civilization, like the you know technological unemployment or or you know job replacements.
01:15:55
Speaker
A lot of things, you know or another one is you know destroying our epistemic system with huge amounts of AI generated slop that, you know, and and misinformation and disinformation and just noise.
01:16:08
Speaker
So you basically don't know what's real or not and like what is true or not. This is already like obviously started. The trajectory we're on is not toward that getting any better. Other things like, you know, what happens when at the press of a button you can file a lawsuit?
01:16:23
Speaker
right Our legal system is not set up for everybody just going, like, spamming the lawsuit button ah with their AI agent filing 100 lawsuits on their behalf with zero legal filing fees, or at least zero legal costs to pay lawyers.
01:16:37
Speaker
So what happens to our system when suddenly that's available? you know We saw a microcosm of this in the education field where overnight, basically, you could not assign a student an essay and grade that essay and have that mean anything.
AI's Disruption in Education
01:16:51
Speaker
And that just happened and nobody was prepared for that. So there are all these like very, very, very disruptive things. Some of them small-ish, like essay writing and having to revamp education. Some of them medium, like legal profession being in our like legal system being all upended.
01:17:07
Speaker
Some of them super large, like nobody having any economic value anymore in our economic system. And a lot of these are not about alignment.
Regulatory Challenges and Societal Restructuring
01:17:17
Speaker
right what And they're not even really about regulations, because what regulation is actually going to stop a company from offering and AI agent worth of money that can replace a person? like What would that regulation even look like?
01:17:32
Speaker
What would a regulation look like that says, no, you can't have AI lawyers? So a lot of the problems, especially with artificial general intelligence, are caused by the system doing the thing that they're made to do.
01:17:45
Speaker
And that is perfectly legal. Like, are we going to say that AIs can't write text? that No. and And so that text is going to be all over the place. So a lot of the problems are are not things are going to be solved by alignment and not things are going to be solved by regulation.
01:18:02
Speaker
They're going to be solved by either not building the systems in the case of AGI and superintelligence, or in like quite dramatically restructuring how our society works in lots of ways so that we can build a system that actually functions well, despite the fact that these systems are now part of it.
01:18:20
Speaker
So we're going to need to rethink like how does the legal system work when you can you know, hit a button. How does our information gathering and um and processing system as a civilization work when you can fake all these things? Like we're going to need some trusted system that we can use to trace conclusions back to, you know, information that goes in and ultimately real world data and all of these things.
01:18:43
Speaker
Those will all have to be
Unemployment and AI Alignment Issues
01:18:44
Speaker
constructed. That, I think, can be done for non-AGI systems, for the powerful tools we're developing. This will not be easy, but I think it can be done.
01:18:54
Speaker
The AGI, even if we knew how to solve alignment in the sense of making AGI controllable and answerable to people, a lot of the problems that it would that would create like, you know, putting everybody out of work are not things that can be regulated away. Like they're they're not technical problems. They're not even regulatory problems. They're just intrinsic problems with creating the technology. So I think there's been a little bit of
Inadequacy of Current Solutions
01:19:18
Speaker
evolution in me. And I think some other people, like there's a technical problem that that isn't quite going to do the job.
01:19:25
Speaker
There's policy that is that is needed that is also, though, not quite going to do the job. And then there are problems that just aren't really solvable in some sense and where we just need to actually not build the technology until we have something where we could possibly solve them or like something is dramatically changed so that those problems are not going to be manifest.
01:19:44
Speaker
And like the the fact that we've solved neither the technical nor the policy and regulatory, nor the like restructured society so that these things make sense, like none of the above have we solved, tells me that we should not build the technology right now. And that that's sort of what it came to and why I wrote Keep the Future Human. Like we've got to close the gates.
01:20:04
Speaker
If like the thing that is beyond the gates, we cannot handle. And like not have we have not solved any of the problems that we need to solve in order to handle it, like we better not do that.
01:20:15
Speaker
And so I think that's where I've
Raising Awareness and Deliberation on AI's Future
01:20:16
Speaker
come to. I think that is where a lot of people will come to when they really understand the contours of the problems we're facing. and And so that is my my effort has been to bring that, more of that understanding to like really lay it out for people What is the situation? What's the science?
01:20:33
Speaker
What are the considerations? What are the evolution in game theory and all these other things to say about it? What is the situation we're in What are the choices we have? How can we choose something different than the road we're currently on?
01:20:45
Speaker
And for people interested in reading more about this, they can go to keepthefuturehuman.ai, where there is a bunch of information. There's the full essay. And I will, of course, link it in the description.
01:20:56
Speaker
Anthony, thanks for chatting with me. It's been great. been a pleasure being here. Thanks for continuing a great podcast.