Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Could Powerful AI Break Our Fragile World? (with Michael Nielsen)
3.9k Plays21 days ago
On this episode, Michael Nielsen joins me to discuss how humanity's growing understanding of nature poses dual-use challenges, whether existing institutions and governance frameworks can adapt to handle advanced AI safely, and how we might recognize signs of dangerous AI. We explore the distinction between AI as agents and tools, how power is latent in the world, implications of widespread powerful hardware, and finally touch upon the philosophical perspectives of deep atheism and optimistic cosmism.
Timestamps:
00:00:00 Preview and intro
00:01:05 Understanding is dual-use
00:05:17 Can we handle AI like other tech?
00:12:08 Can institutions adapt to AI?
00:16:50 Recognizing signs of dangerous AI
00:22:45 Agents versus tools
00:25:43 Power is latent in the world
00:35:45 Widespread powerful hardware
00:42:09 Governance mechanisms for AI
00:53:55 Deep atheism and optimistic cosmism
Recommended
Transcript
Neutron Discovery and Destruction
00:00:00
Speaker
People who were working on the discovery of the neutron, I doubt they were sort of thinking about the incineration of Hiroshima at that time, even though it occurred only 13 years later. Just a tiny, tiny sort of gap from these people just playing with systems in their lab to this absolutely immense destruction.
AI and AGI: Advancements and Risks
00:00:17
Speaker
AI and and AGI are going to result in a large number of just extraordinary things. I'm the beneficiary of one of the very first drugs that was done using protein science.
00:00:28
Speaker
and very sympathetic to people who are just excited for that sort of positive upside. But there's the possibility of an enormous negative downside. I think at first I rolled my eyes a bit, you know, i would see the Pope commenting or the the United Nations commenting.
00:00:44
Speaker
And that was just completely wrong. It's important that all those people are thinking about this and solutions will come from very different and maybe unexpected directions. Michael, welcome to the Future of Life Institute podcast.
00:00:57
Speaker
Thanks for having me, Gus. You've written a series of wonderful, insightful essays on on AI risk. And one point you make here is that...
Understanding Reality: Opportunities and Threats
00:01:09
Speaker
Deeply understanding reality is intrinsically dual use. What does that mean? And what are some examples of that? I suppose I just mean that if certainly, mean, to some extent, I'm speaking of ah an empirical fact.
00:01:24
Speaker
If you look back through millennia, whenever we've had sufficiently deep understanding of reality, it's always created opportunities to change reality in ways that give us power.
00:01:36
Speaker
And that can have both constructive and destructive uses. I mean, a relatively recent example in the scheme of human history is something like the discovery of subatomic ah physics and quantum physics in the early part of the 20th century.
00:01:52
Speaker
And this led to many, many wonderful things, a lot of modern material science, the semiconductor industry, And I mean, much of the modern economy, it was also led in part to nuclear weapons and thema nuclear weapons.
00:02:05
Speaker
And it's very difficult to see how you could get one without also getting the other.
AI Risk Beyond Alignment
00:02:11
Speaker
I can't prove that this is always the case, and but certainly looking historically at many examples, if you go back further and you think about Newtonian mechanics and things like that, and sort of the way that plugs into artillery calculations and and things like that, it's kind of another example to some extent.
00:02:29
Speaker
And this is the core of the problem of AI risk, in in your opinion, right? it's It's about the fact that when we understand or when we create systems that can understand reality at a deeper level, that then confers power to these systems in ways that can be both good or bad.
00:02:48
Speaker
Yeah, in fact, it's not even AI risk, it's a broader thing, which is risk created by science and technology. There's this wonderful paper from, I think it's 1955 by John von Neumann, which really mirrors a lot of discussion of AI risk.
00:03:02
Speaker
Other people like Oppenheimer wrote similar things, although not quite as broad. And it's just, you as human beings attain this very deep understanding, our ability to do things like modify the weather starts to increase or modify the climate starts to increase.
00:03:18
Speaker
So, yeah.
Adapting to AI Risks
00:03:19
Speaker
This is a different framing than thinking of of AI risk in terms of alignment. You are quite concerned that it will be easier to create a truth-seeking system than an aligned system.
00:03:31
Speaker
Why would that be easier? I don't mean, well, easier. not sure that's quite where I would focus. Certainly the point is that, you know, whenever you have yeah the the the notion of alignment, you famously people who were like aligned what?
00:03:46
Speaker
And the historic end. So, I mean, we've been solving the alignment problem as a ah society and as a civilization for ah millennia. um You go back to the Code of Hamiya Rabbi and and people like that, and we're trying to attain this kind of you know social consensus, and then the institutions necessary to implement that social consensus, however, imperfectly.
00:04:04
Speaker
Obviously, civilization is is not perfect. Bad things still happen. But we've achieved pretty good degree of alignment with some consensus notion. the The issue becomes a very significant issue is when it is possible for people to unilaterally defect locally, if tremendous power is available to them, that they they can violate it, this social consensus.
00:04:31
Speaker
It's also just sort of the sheer fact that different groups will converge on different consensuses. So the United States military, the Russian military, the Chinese military,
00:04:44
Speaker
are unlikely to have the same standards of alignment as, say, consumer apps in the United States. And consumer apps in the United States are going to have very different notions than, say, applications in China when DeepSeek released their model.
00:04:58
Speaker
Quite a number of people, at least informally, reported, I don't know whether... I didn't see any sort of formal evaluation, but they informally reported that a lot of things that the CCP doesn't particularly like appeared to be censored in the model. So, know, there's a significant difference in the alignment just along that axis.
00:05:17
Speaker
Why is it that we can't solve AI risk by simply doing what we've been doing with other risky technologies, which is adapt as we as we go?
Historical Tech Adaptation and Regulation
00:05:26
Speaker
we Something happens, there's a catastrophe, we implement new regulation, and when and we can co-evolve with this technology.
00:05:35
Speaker
Why would advanced AI be different? Yeah. I mean, it's a great question. It's kind of the the the quadrillion dollar question. And and yeah maybe it turns out that we can't.
00:05:47
Speaker
It remains to be seen. There's a certain amount of sort of optimism there. If you think historically about sort previous technologies, particularly previous sort of major platform technologies.
00:06:00
Speaker
Very often this works incredibly well. An example I love is the introduction of jet engines and actually one of the very first companies to work on this comet had, I think it was three, maybe four fatal crashes in their first year of operation. And of course,
00:06:13
Speaker
i they They shut down. But at the same, by the same token, while that's a terrible human loss, it created an enormous incentive for the other companies to get it right and yeah really improve the technology a lot.
00:06:24
Speaker
So that sort of, I mean, not just capitalism, but in fact, just plain consumer sentiment and whatnot often does a really good job ah of doing this kind of alignment. A very large number of technologies that works remarkably Well, just sort of word of mouth, it's kind of, you know, it's sufficient to exert a lot of pressure on the people doing it.
00:06:46
Speaker
It's not perfect. You still get snake oil being sold. You sometimes get terrible things being sold. um Underground economies often have big problems with this, they things like new drugs, which actually cause deaths. There's no legal recourse there, and it's very difficult to communicate.
00:07:01
Speaker
you're sort of in a public way, you can't say this supplier is bad. you know On Twitter, you're opening yourself up to kind of all kinds of problems. So sort of the signaling doesn't work there. Now, if you think about problems like nuclear weapons or climate change or fluorocarbons back in the 70s and 80s, those kinds of things, all instances where the traditional institutions are have at best struggled for a variety of reasons.
AI Risk versus Democratization of Nukes
00:07:29
Speaker
Climate is, i think, a very interesting one in particular.
00:07:32
Speaker
You know, you've just got this situation where you've got this enormous kind of engine or capital, ah which has strong interest in continuing to perpetuate this situation. And they don't particularly want to switch away. They're not experts on renewable energy or or things like that.
00:07:48
Speaker
yeah. you the the the question becomes, what does AI risk or ASI X risk, what character does it have? How do you how do you classify it there?
00:08:00
Speaker
And I suppose at least my intuition, my belief is that this ability to put enormously dangerous capabilities in the hands of individuals makes it much more similar to the case of something like climate or or sort of democratized nukes than it is to a conventional technology.
00:08:20
Speaker
There is also the framing of AI or ASI as a generator of new technologies that could potentially be dangerous.
AI in Protein and Virus Design
00:08:30
Speaker
And then perhaps talk about this this framing of of AI as ah as a generator of new technologies in a world that is potentially vulnerable.
00:08:40
Speaker
Yeah, I mean, the obvious example that's sort of a, you know it's a project or, you know, it's a prototype of what we're we're going to see. AlphaFold, you solve sort of the protein structure prediction. If you can predict structures pretty well, that's actually enormously helpful for doing design work and say, you know, here's the design I want.
00:08:57
Speaker
You know, sort of the way the... the ah generative models tend to work is they just, they try lots of slight variations, they keep running the predictor and seeing how ah well it works. So we've got to add a big iteration in the last few years on air ability to do antibody design and things like that.
00:09:14
Speaker
Now, at this point, my understanding, and I'm not an expert, is that we're we're kind of in an intermediate stayed there. There's not a whole lot of new drugs have hit the market. There's not a whole lot of ah these kinds of things which have hit the market, but it's getting close.
00:09:28
Speaker
Like, mean, it's really interesting just to look at the literature and see these people who are interested in protein design. They just go absolutely kind of bananas after alpha fold. I can't remember who was. Somebody pointed me ah pointed out to me, like the job listings at one of the companies that does this.
00:09:43
Speaker
It was all for Python programmers and people have experts PyTorch and things like this. but that That kind of thing. But it's still early
Data Impact on Virus Design
00:09:52
Speaker
days. But then you start to think about things like prion design, which is two words that ought to strike terror into anybody's hearts.
00:09:59
Speaker
And then yeah the question of why it was protein design and not virus design that was kind of done first is an interesting one. My understanding is just there wasn't as much of data collected about about viruses. You have the Protein Data Bank, which was this enormous, very well curated service and and you had just less good data, partially due to historic accident, is my impression as a non-expert.
00:10:26
Speaker
But again, sort of thinking, you know, probably not that far away from being to do high quality virus design. And that's going to have yeah know many, many, many wonderful impacts.
00:10:37
Speaker
But it's also something that you don't necessarily want in the hands of individuals unstable. i think in the early days, it is going to be a specialist. It's clearly going to be a specialist technology, which will require a considerable amount of background to to use and deploy.
00:10:51
Speaker
Over time, you know access to that will become increasingly democratized. Maybe take a step back. You asked about the vulnerable world. and that is just the question of whether or not there are so yeah relatively simple, easy to deploy easy to make and deploy technologies, which can cause catastrophic or existential risk to humanity.
00:11:11
Speaker
And this sounds, I think, implausible to most people, sort of a priori. But then when you start to look at actual kind of examples from even the history of life one on Earth, and it's maybe not...
00:11:23
Speaker
not not so ridiculous. People, yeah very, very well-informed people are extremely skeptical that nuclear weapons would work. You go back much, much, billions billions of years in time, there's things like the great oxygenation event where the ability to metabolize into oxygen and actually cooking know wiped out probably most species on Earth.
00:11:45
Speaker
And many theyre they there are many examples one can give along the way which show that very simple, but completely transformative technologies are possible. And you have to ask the question, are we close to being able to do that again?
Vulnerable World and Catastrophic Risks
00:12:00
Speaker
yeah Any tool which is capable of significantly advancing science and technology can potentially yeah plug plug in and and and help enable that. And then that, I think, naturally leads to the question of whether our institutions can keep up with the pace of development of these new new ways to affect the world that could be potentially dangerous.
00:12:22
Speaker
How do we know whether we are on the right path here? Well, I mean, to some extent, yeah of course, the but yeah the ah funny and ironic thing about this is that small scale disasters are very helpful for improving institutions.
00:12:35
Speaker
Certainly, I think things like if you look at things like, say, the nuclear nonproliferation treaty and the various test ban treaties, The infrastructure, and when I say infrastructure, I mean sort of almost the social infrastructure that was necessary to create those, that all gets reused to some extent. Even an example I love is the Vienna and the Montreal Protocols, which resulted in the banning of CFCs.
00:12:57
Speaker
One of the things which is interesting about those is is that every state in the world signed it and that had never been done before. And so you have this sort of interesting situation where it's like, oh, it's possible everybody to agree and just to blanket replace ah technology. So I mean, I think those kinds of things represent really significant institutional progress.
00:13:18
Speaker
Unfortunately, they're not always, you know, they're quite bespoke. They're not, you know, it's not that we've got something that it's kind of, you know, helping those things directly scale up. Something that I think is valuable about podcasts like this in terms like existential risk and and things like that.
00:13:32
Speaker
They do start to create kind of common norms and just a common language which people can use to communicate about this. um And that's obviously incredibly valuable at the, you know, it just, it starts to make it easier to coordinate.
Control and AI Risks
00:13:47
Speaker
How concerned should we be about losing control to advanced AI? Is that perhaps the the right framing here? i mean, I i think yeah that that's a very significant issue. It's one of many possible risks.
00:14:01
Speaker
ah type I mean, your listeners will have heard many people talk about this before, and it's probably the most prominent of all the the ASI Express scenarios. So, Mike,
00:14:12
Speaker
Personal opinion, yes, I'm very concerned about that. um I don't think it's kind of the the fundamental issue. ah I'm just more concerned in general with this possibility for destructive technologies. it's It would be an example of such a destructive technology. It is not necessarily the only one.
00:14:29
Speaker
Which direction are the the incentives here, the market incentives?
AI Development Race
00:14:34
Speaker
Which directions are they pushing in? Because on the one hand, you you can you can see that it's not useful to put a product out there that's dangerous, right? that's not It's not profitable in the long run to to deploy a dangerous products.
00:14:48
Speaker
On the other hand, it seems like we are raising the companies, countries and so on towards developing advanced AI systems at ah very fast pace, perhaps without thinking long and hard about the the consequences.
00:15:05
Speaker
It's not even just the the question of ah profitable. It's also just the fact that you it's people running these companies and they don't want to die. They don't want their children to die. So that's kind of... um For many of those people, that's an even more significant thing than concerns about, you know, will I get some nice stock options?
00:15:22
Speaker
toent To return to the quantum mechanics and the nuclear physics example, though, you know, the people who are working on the discovery of the neutron, and not, I believe, anyway, I doubt they were thinking about the incineration of Hiroshima at that at that time, even though it occurred only 13 years later.
00:15:39
Speaker
you just a tiny tiny sort of gap from these people just playing with systems in their lab to this absolutely immense destruction and yet from the point of view of the people who were discovering things like the neutron and the proton and the structure of the nucleus and so on you know that that just seemed like a wonderful game with enormous benefits to humanity in fact it does did have enormous benefits to humanity.
00:16:03
Speaker
That's maybe, I mean, that's part of what is really difficult, of course, about the situation. AI and and AGI are going to result in a large number of just extraordinary things.
00:16:14
Speaker
I suffer from a mild, chronic, lifelong condition. I am the beneficiary of one of the very first drugs that was done using protein science. It's a previous iteration. It wasn't done with what we would today call an AI system.
00:16:27
Speaker
But boy, do am i like do I ever think this is a great thing at some level? and yeah And I'm so excited for other people ah kind of get access to those kinds of things as well.
00:16:40
Speaker
And very sympathetic to people who are just excited for that sort of positive upside. But there's the possibility of yeah a a very hard to defend an enormous negative downside.
00:16:50
Speaker
How do we recognize which world we're in then? If the bad world in which there's a huge downside looks like the good world in which we get a bunch of upside from AI and we avoid the risk.
00:17:02
Speaker
this is This is one of the things that makes this debate quite difficult or makes this arguing about AI risk difficult because how do we distinguish between these worlds?
Historical Technological Risks versus AI
00:17:13
Speaker
it it It is difficult. Actually, and I'm speaking a little through my hat here. I'm not quite sure. but yeah if you think about, as I say, sort circa 1930, people had started to consider the possibility of of ah nuclear-like weapons at a about that time.
00:17:28
Speaker
time, it was still very speculative. But I think it was probably very unclear just how, are we going to get things which can wipe out a city block, a city, a country?
00:17:40
Speaker
And so there's kind of some unknown level of of destruction. And so if you're sort of thinking, oh, maybe some of this work that I'm doing on understanding subatopic physics might one day be used to ah in this destructive way, it's just this sort of enormous range of possible negative outcomes that I think they probably needed to wait. ah Sort of the E equals MC squared gives you some bounds on on the the ah size of it. It turns out think thermonuclear bonds, they're something like 0.7% efficient or something.
00:18:11
Speaker
um And at some level, the question of sort of cost per megaton becomes the ah relevant one there. And I don't think it was at all apparent what that cost was going to be.
00:18:21
Speaker
for a long time. So that's the analog, i suppose, of your question. I want to put it in that sort concrete context to just make it clear that actually a very, you know, it wouldn't surprise me at all if at that time in history, it could have ranged over five, six, seven orders of magnitude, sort of a plausible estimate.
00:18:39
Speaker
So that's a that's a problem. I'm not sure I've addressed part of your question. and I don't think I've addressed all of it, unfortunately. Perhaps one thing to ask here is alignment and the question of whether focusing on alignment is the right approach.
The AI Alignment Dilemma
00:18:54
Speaker
you You describe what you call the alignment dilemma.
00:18:57
Speaker
Could you describe what that but better ah yeah An analogy ah I've come to life recently is imagining you're an Olympic swimmer. you You directly identify that if you want to be an outstanding sprinter, if you want to win yourself a gold medal, you need to work on your strength.
00:19:14
Speaker
In fact, you need to work very, very hard on your strength. But if all you do is work on your strength, that's actually going to be quite counterproductive. You are not going to be a good ah good swimmer. So you need to sort of work on that, but in the sort of the larger context.
00:19:28
Speaker
I think it's very clear that a major impact of the work on alignment, which has been done so far, has just been to speed up the creation of these systems.
00:19:38
Speaker
Certainly, i think in particular, the invention of well invention and successful use of RLHF really turned chatbots to some extent, not just RLHF, but other things as well, but that were that was very important, turned chatbots from being sort of interesting systems into things which are very friendly, but yeah you can make them much more friendly to consumers.
00:20:02
Speaker
You can make them much more friendly to governments. And so that's sort of a way of really speeding up but the capitalist race that you referenced before. So, you know, is that yeah it helps you with one set of important goals, which is indeed yeah'm making your AI system relatively controllable and sustainable.
00:20:23
Speaker
and so on, but at the expense of yeah causing tens of billions of dollars, hundreds of billions of dollars to come pouring into space and really accelerating everything.
00:20:34
Speaker
um So it's not clear to me. To me, that seems maybe a little bit like having gone absolutely nuts on a certain type of strength training program while trying to, and to you know, basically slowing, you know, really damaging your ability to swim.
00:20:50
Speaker
You're not actually solving the problem. What would actually solve the problem then? What is the alternative that a person that could be working on alignment should be working on? Yeah, well, i mean, I suppose I'm just pretty negative actually on on most of the work on ah
Governance and Environmental Focus in AI
00:21:06
Speaker
alignment.
00:21:06
Speaker
Anything which is focused purely on the properties of the individual systems, I think it yeah the the tendency is just for that to align with the interests of the companies. The things that I am more interested in and think are much more promising It's all external. It's all sort of governance in the rest of the world.
00:21:24
Speaker
So people are interested in questions like how to do real-time sort of monitoring of biological threats and and sort of yeah know response models there, computer security, these kinds of things. safety yeah know Safety isn't a property of a system.
00:21:38
Speaker
Safety is a property of the system and its complete environment. And in fact, very often that yeah it's really working on the environment, which matters ah much more. At the moment, the people who can work on safety of the systems, those systems are controlled by the companies.
00:21:54
Speaker
yeah The natural out yeah effect is, All of that work will be aligned with the interests of the companies. It's not perfect, of course, because they're large entities and it's very hard to get that alignment perfect.
00:22:07
Speaker
But I think generally speaking, yeah it's basically it most work on the alignment of individual systems. is serving the needs of capital. if you're thinking about sort of the alignment problem society in general, if you're ensuring for flourishing human civilization, that kind of thing FLI is dedicated to, yeah that yeah know, you you're talking about people who are yeah working on these, I think, sort of much broader problems. even I mentioned things like ah the nuclear non-proliferation treaty and things like that.
00:22:39
Speaker
You know, that's sort of a long line of work that is aimed at creating this kind of collective safety. Do you think it matters whether the companies are aiming at creating agentic systems or whether they're aiming to create better and better AIs that remain tools?
Market Interest in Agentic AI
00:22:57
Speaker
Yeah, I mean, I think you can flip the question around a bit and sort of ask what's in capital's interest. And I think there it's quite clear yeah there's an enormous interest in in making them agentic, even just for very simple kind of reasons. yeah People are going to love sort of your friendship and romance spots, and it's better.
00:23:15
Speaker
you know ah They're going to be more interesting and more attractive if they're somewhat more agentic. Yeah. They can buy you flowers. They can you do these kinds of things. Even that sort of very simple kind of thing.
00:23:26
Speaker
Over time, there's something of a slippery slide. I mean, famously, there have been quite a number of flash crashes. They're often a little bit obscure, what's going on, but some kind of AI-like sort of early machine learning type or data data science type system, you know, which is hooked up.
00:23:44
Speaker
to the markets, in many cases, seems to be causing some kind of massive change in the market, sometimes sort of a trillion dollar kind of a fluctuation. Is that agentic in...
00:23:56
Speaker
you know, the full human sense? No, but it's a system that, you know, has to some degree kind of goals and rewards and is capable of taking not just a little bit of action in the world, but potentially, know, being responsible for the movement of hundreds of billions or trillions of dollars.
00:24:15
Speaker
And, you know, those examples go back, let's see, I mean, certainly 15 years, there was a flash crash in 2010. So I think I've think it's very hard to see a situation where they remain merely tools.
00:24:30
Speaker
Whatever that is, I mean, it's it's not even clear, like, how do you, what's the, what's the criterion by which something is just a tool versus something which is, you know, it's the it difference between an amanuensis and a composer at some level.
00:24:44
Speaker
Actually, those two things are not so distinct. How did you come to be convinced that there's actually something to be concerned about with AI risk?
Historical Awareness of Risk
00:24:52
Speaker
Yeah, I think mostly. So in the late 1980s, I read a bunch of writing by Carl Sagan and some of his friends about nuclear ah risk.
00:25:00
Speaker
A lot of people now view some of that as being naive or wrong. I also read Eric Drexler's book, Entries of Creation, where he talked about great goo and sort of scenarios like that.
00:25:10
Speaker
There's a lot of technical problems with some of that work, but it got me... sort interested and aware. It also seemed, he's quite upfront about sort of the technical problems at some level there.
00:25:22
Speaker
And then later on, know, I became a theoretical physicist. I worked for a very long time on quantum computing and it just became clear, oh yeah, the world actually has.
00:25:34
Speaker
yeah this enormous latent power inside ah our current technology seemed to me just to be merely scratching the the surface. You've got to explain that, I think. What what ah what does latent power mean?
00:25:48
Speaker
Is this something that you that you understand when you understand the discipline deeply, when you become a physicist or a biologist or something? It's funny. you know I'm not super expert on something like nuclear fission, but I kind of understand it well enough It's a little bit little bit shocking.
00:26:05
Speaker
yeah You kind of go through and you realise, wow, this is actually really quite simple. And it's just hidden. you know It is not obvious at all that it's out there in the world.
00:26:16
Speaker
And it's a good thing for us. and It's sort of rather fortunate for us that it's not a little bit easier to make these these bombs and it's not a little bit easier to make them much larger. Those are due to somewhat contingent random facts about the world.
00:26:29
Speaker
Is that experience, I guess... Over and over, I worked on things like quantum teleportation and quantum algorithms and things like that. And again, just have this experience of, you know, people write down, know, there's very simple sort of models of the world and there's all these things hidden inside them that you don't realize. that Science is just so full of that.
00:26:48
Speaker
Actually, public key cryptography I think, a wonderful example. It's miracle. yeah It could have been invented, I think. Yeah. I mean, certainly many hundreds of years ago, just these incredible ideas get in essentially inside number theory.
00:27:03
Speaker
A recent example of latent power hidden in biology might be something like mirror life. Again, of course, i'm I'm not an expert here, but it seems to me plausible that mirror life is an actual threat. And this is something that that you wouldn't be this is something that you wouldn't have been able to guess if you're not a bio biologist kind of ah with deep understanding here.
00:27:25
Speaker
i think that's I think that's right. It is a good example. I mean, it's by no means clear that it is actually an enormous threat, but I'm not particularly keen to find out either. You know, it's just that's that's playing with fire.
00:27:38
Speaker
Actually, but fire is, I mean, yeah, the primordial example is... is is it's such a good example. It is, you know, sometimes people will say, oh, you know these sort of vulnerable world recipes for ruin, that seems so implausible.
00:27:52
Speaker
And then you say, well, you know, can you take ah a simple technology which you can buy for well under a dollar, anybody can operate, anybody can get it anywhere in the world, and with no extra input have it cause a billion dollars of damage and yeah a thousand deaths?
00:28:06
Speaker
And of course the answer is yes, you just need it much. And it's a good thing. yeah Imagine yeah if you increase the yeah oxygen content of the air just a little, fires would become much more fierce, be much easier to establish a firestorm.
00:28:18
Speaker
There's anthropic reasons why this doesn't happen. We live in a pretty friendly, we know we live in a pretty friendly world, Because if we didn't live in a pretty friendly world, we wouldn't have survived to this point.
00:28:30
Speaker
Unfortunately, now we have these enormous brains and you know all these wonderful ways of deepening our understanding of of of the world. And it's by no means clear that we're going to remain in that relatively friendly regime.
00:28:45
Speaker
It's a pretty deep question, actually, whether we live in a friendly world or not, whether the world is vulnerable. What evidence do do you think we have so far
Transparency in AI Development
00:28:53
Speaker
about this? is just Is the way that the world works as we understand it scientifically favourable to our existence is one way of asking the question.
00:29:02
Speaker
I don't know had to attack the broad question, except in so in a very negative way, where you could, in fact, demonstrate, oh, here's a yeah yeah here's a particular lethal technology.
00:29:15
Speaker
Actually, this is problem with discussions of ASI X-Risk. I've had many conversations with sincere, thoughtful leaders in technology who are like, you know, I kind of see why people are concerned, but like, you these AI doomers, they always get quite vague when I ask them to ah yeah connect the dots in the scenarios.
00:29:36
Speaker
So like, well, I'm not sure i actually want yeah how much we should want the dots connected. I get the feeling some of them are not going to be satisfied with anything other than a detailed description of a recipe, which might be intellectually satisfying, but also seems like a dreadful mistake to make.
00:29:53
Speaker
This is one of your persuasion paradoxes, where if you make a highly persuasive case case, a detailed case of how AI could be risky, you're then putting dangerous information out there. And this is something that hinders the debate from moving forward.
00:30:08
Speaker
Yeah, I think it i mean it hinders it somewhat. it ah Honestly, at this point, i don't think it hinders it that much. Although it is by its very nature quite difficult to know. you know I hear dark whispers from people that you know they yeah there are rumours that a model was used to do such and such.
00:30:26
Speaker
It's maybe a fun thing to to hint at. I suspect very often the reality it's less is less interesting than the gossip for now. Perhaps that's a key point here when you say for now, because if this, if skepticism about AI risk is is is skepticism about how intelligent current systems are, then you can, experience tells us that you can probably wait a year or five years, and then you will have more advanced systems and you will at some point face the issues.
00:30:58
Speaker
Well, there's that. also, I mean, you just have to wonder as well, like that there's some gap between what is being done publicly and what is being done privately. You know, i mean, one of two things is going to happen. Either militaries are going to start working in intelligence agencies and whatnot on systems themselves, or quite likely, know, there's going to be contracts with kind of the big ah current labs where there's some...
00:31:21
Speaker
you sort of separation. And it's not necessarily going to be completely obvious, actually, what the capabilities are in the public eye anymore. You think about things like, you know, there are significant parts of the the budget for the intelligence agencies, which are ah not even, they're not even just that that basic kind of information.
Open Source AI Models: Risks and Challenges
00:31:39
Speaker
It's not public, information not always public information, much less details the program. And so you you you wonder what what's going to take place there and in particular, what but capabilities will be developed.
00:31:51
Speaker
i I agree. And I would actually expect that as models become more capable in the domain of coding, for example, some of the labs might or some of the companies might be tempted to to deploy these models internally in order to have them help on improving AI ah research further so that they can perhaps get a further lead and in the race.
00:32:13
Speaker
and And that that would would be the maximally profitable or interesting thing from their perspective to do with the models as opposed to deploying them broadly in society. that that That situation could become quite dangerous because then you don't have this public information about AI models.
00:32:32
Speaker
You don't have this this kind of back and forth as we talked about where institutions can of can adapt to the risks and so on. That's actually something that I've changed my mind a about a bit.
00:32:43
Speaker
I mean, just a few years ago, i was not wild about open source models at all. And I think conditional on there being major private efforts, then I actually, I'm very in favor of there being comparable open source models.
00:32:58
Speaker
Even though they create a threat vector, they also create sort of a surface area of being able to say, oh, you know, these are some threats that have been created. It also means that the organizations who are doing, know, computer security audits and who are doing all these kind of evals, it means that they don't have to, ideally they would be in a very adversarial relationship with the major labs, but in fact, they need to work with them as partners.
00:33:26
Speaker
And so to some extent, they they don't really have quite the right relationship, but if they're using, know, Lama or one of the other open source models, it doesn't matter at all.
00:33:37
Speaker
paper I really like, it's a silver small paper was Kevin Esfeld's group did this fine tuning of Lama to see how much it could help with creating pandemic agents. yeah It's kind of telling that, I mean, it just it's just good to be able to use Lama and in that kind of ah kind of a way.
00:33:55
Speaker
And they don't need to get permission from OpenAI or Anthropic or whoever. That's the correct kind of, ah yeah yeah that that's the situation you want. So it's a way to get information into the world about the current capabilities of models.
00:34:09
Speaker
and And I think that's a good point. But it's not just information, but you actually want to be able to do sort of the most adversarial things possible with it. Things with maybe the, you know, the labs might not be so keen on having discussed publicly.
00:34:22
Speaker
You can just sort of say, well, you know, I don't care. And in particular, not having the organizations doing the evaluations need to be aligned with the companies creating them is, I think that's very helpful.
00:34:35
Speaker
One thing that makes me nervous about open source is that you can't recall these models. So whenever you have ah an open source model released, you can never recall that product, no matter what it's capable of. And it's kind of proliferated into the world.
00:34:50
Speaker
as we As we've been talking about this, if it falls into the wrong hands, that could be ah real problem.
Future Accessibility of AI Models
00:34:57
Speaker
Yeah, I agree. I mean, that's certainly a significant and downside.
00:35:01
Speaker
It's interesting to think about the question, just how large are the models eventually going to be? What do you mean, how large are the models eventually going to be? It's very much sort of a stopgap intermediate kind of an argument I'm i'm trying to make.
00:35:14
Speaker
If a model actually can only do inference, you're running on a significant cluster, if it can only... you know, those kinds of things. I think it's implausible as a long-term, you know, compute is going to continue to get cheaper.
00:35:27
Speaker
Yeah, maybe, sure, maybe it eventually turns out that yeah you need whatever beyond a quadrillion, you know parameter model is. And so it's actually quite complicated to do this.
00:35:38
Speaker
But yeah, that's all. ah it's It's a massive reach to push back on your argument at all. i mean i mean, so far we can kind of depend on ah consumer hardware not being advanced enough to run frontier models, you know, using a lot of inference or thinking reasoning models, thinking models using a lot of inference.
00:35:59
Speaker
Right at the moment. Yeah. yeah But but but as you're as you mentioned yourself, that's that's probably going to and to change over time unless we then find a way to, but most advanced models to require even more inference in a way that keeps up with with computer hardware. I don't think that's that's going to happen. i I think we're going to enter into a world where where you can run very advanced models on ah basically smartphones.
00:36:24
Speaker
it I think that's quite likely. hey i mean, the way in which that you know, there's sort of an economic reason why that's going to fail to be true in its strongest sense, which is just, you if you believe that more scale ah in the model helps, then you should expect whatever sort of the the most powerful current chip ah is, is probably going to be required ah to run the best model. And so if that's costing $30,000 or whatever,
00:36:52
Speaker
you know, it's going to have an advantage over smartphone. I don't actually know whether or not it's true. It's probably not public information whether or not current frontier models, when they're doing inference, are they simply running on a single chip?
00:37:08
Speaker
I'm actually just shamefully ignorant of that. i assume the answer is probably yes. And so they've got it deployed across the cluster, but yeah you yeah run your particular chat GPT query and it's essentially a single chip, which is responsible for the output.
00:37:24
Speaker
I don't actually know whether that's the case, but yeah that's sort of interesting. If so, it means that for a few sort of tens of thousands of dollars, you can do inference with the chip. Actually, I suppose, so they are doing a lot of things in parallel.
00:37:39
Speaker
Okay. So I just don't think about the reasoning models. Okay, I take that back actually. There probably is quite likely it's something ah more expensive being done, which may may not be so easy to do then or no.
00:37:53
Speaker
Okay, so it it's ah it's a question of sort of long term what algorithms are done and how easily can they be sort of scaled up beyond a single, beyond a, how easily can inference be scaled up beyond a single chip?
00:38:03
Speaker
And that's kind of a, that's an interesting, I suppose that's a, it's an interesting economic constraint on your assertion about eventually the cell phone version, ah not being so, not being so different from the frontier
Societal Governance in AI Risk
00:38:15
Speaker
model.
00:38:15
Speaker
Good. Great, great, great thought. Yeah. There are a bunch of open questions when, when we have a conversation like this and, and some of them can be answered by some people in the world, but there are also generally open questions where no one has the answer.
00:38:31
Speaker
When we're thinking about AI risk or highly advanced AI in general, we are we are so much kind of fumbling in the dark without having an established science.
00:38:42
Speaker
And it feels like perhaps we don't have the time have an established science because that can take... maybe maybe decades, maybe maybe a century. How do we act in that situation when when things are moving incredibly quickly and we don't, you know, we don't, it seems to me that we don't really have enough time to deeply understand the models that we are creating before so mo before the models get a very advanced.
00:39:11
Speaker
No, well, I agree with all of that. I wish I had a magic yeah solution. What I'm asking here is like, how do we pragmatically approach that situation? I know you have a, you know, you've been kind of involved and in open science and so on.
00:39:25
Speaker
And, and you know, you know something about how science works. How do we how do we get a scientific grasp on on the situation we're in? Well, okay, first all, I don't think it's primarily a scientific problem, unfortunately.
00:39:38
Speaker
ah you know To a very large extent, yeah it involves everything. That's part of the problem. And it's also part of why suppose I'm interested and excited, excited's maybe be the wrong term, encouraged at least, to see people from so many different areas get involved.
00:39:56
Speaker
I think about people like Vitalik Buterin, actually, or a bunch of economists who yeah They have different ways of thinking about how do you deal with the cost of externalities in the world.
00:40:08
Speaker
And so sort of seeing that kind of person, even, i think at first I rolled my eyes a bit, you know, I would see the Pope commenting or the the United Nations commenting or, you know, people saying,
00:40:20
Speaker
yeah the representative from yeah whatever, the ambassador from whatever country would make an announcement at the UN. And ah initially, i think i I did the sort of the very arrogant physicist thing of sort of rolling my eyes yeah and that was just completely wrong.
00:40:36
Speaker
It's important that all those people are thinking about this and and worrying about it and further I hope and believe that it's quite likely that solutions will come from very different and maybe unexpected directions.
00:40:50
Speaker
but There's a think there's ah kind of really interesting hubris in pessimism. To be pessimistic about something, you have to believe that your so glad you and your friends are so clever and so all-seeing that if there was a solution, you would know about it.
00:41:06
Speaker
And yeah when I grew up, I was told things about the... that CFCs were going to kill us all or excuse me they are yeah know damage from the ozone layer was going to cause terrible, terrible problems or that acid rain or many, many other concerns.
00:41:22
Speaker
And in each case, what was going on was the people sort of diagnosing the problem who were very pessimistic about ah didn't realize that always that like just how much cleverness was being brought to bear by people of goodwill, sometimes with very different backgrounds to themselves.
00:41:40
Speaker
So that that ah at least find encouraging. I also like the thought, you know, people with very diverse expertise and very diverse interests working on it seems very important to me.
00:41:53
Speaker
It's why having having the Pope comment is actually, yeah it's actually quite helpful. sort of, if Rajat's done thoughtfully in an engaged way, because it does bring more people into into the conversation.
00:42:05
Speaker
And in particular, it starts to bring more expertise into the conversation. Do you think that over the long term, this problem is is something that can be solved with governance mechanisms or cultural norms or so on?
00:42:19
Speaker
Can we have a world in which we we haven't solved the the foundational technical problems of how to, say, align or how to control AI systems, but then we have good enough governance that things somehow work out?
00:42:35
Speaker
I mean, okay, so yeah know governance tends to get used in two separate ways in these kinds of conversations. One is just a very practical kind of a one. You have all these institutions that you can point to. So, the Hague, United Nations, the US government, et cetera, et cetera, the judiciary, those are all governance institutions. And then it's also sometimes used in this vague way, which is how human beings control human actions and outcomes, generally speaking.
00:43:02
Speaker
And ah yeah it's always the case that with difficult new technologies, we have to expand the the the former. The question is, how much do we need to expand? I think there is a yeah ah very large expansion necessary. It may be one but that that is essentially insoluble.
00:43:19
Speaker
One of the terrible things is that yeah disasters are one of the main things the that cause an expansion. You get the Union Cobalt carbide, or you get what's her name, Rachel Carson and Silent Spring, these kinds of things where they can point to a very large problem that leads to improved governance mechanisms. It is very unfortunate.
00:43:38
Speaker
There's kind of this epistemic problem where if you can point to a very immediate, very legible problem, threat, it is much easier they get an expansion in sort of the governance mechanisms we have available.
AI Risks versus Climate Change
00:43:53
Speaker
Whereas if you're putting to a relatively illegible threat, it's so difficult. mean, that's, you climate has been such a problem. It took 60 years really to make almost any progress at all on the the climate models and then yeah many decades of wrangling and wrangling and and wrangling this very long-term problem.
00:44:11
Speaker
So you don't get the same kind of immediate turnaround as to some extent you get with something like the threat from the ozone hole that we had, where people could just go and look, oh my goodness, there is this very rapidly expanding hole.
00:44:24
Speaker
One thing that that makes AI risk different from climate change is that the risks are going to be more apparent in people's lives, I think. I think people are going to be able to interact with models that are impressively advanced, and that that will make them probably that will convince them that there's something to this issue.
00:44:42
Speaker
Whereas climate change, you you can... yeah without you know Without scientific grounding, but but you can kind of look out your window and be a climate skeptic and and think, okay, nothing is really happening here.
00:44:55
Speaker
Yeah. I mean, even just the timescales, we don't notice yeah changes over 20 years and also just random luck. One question I'm interested in is over the course of history, you have a a lot of people coming with doomsday scenarios, predictions of of of danger, right?
Skepticism of Doomsday Predictions
00:45:12
Speaker
And a good heuristic has been throughout history to be quite skeptical about about this. Yeah, but basically in nor ignore them. Y2K, great example. Exactly, yeah. Or, yeah, pick pick your your doomsday scenario.
00:45:27
Speaker
When do we know whether that heuristic no longer applies, right? Certainly, I mean, in this particular case, insofar as you can actually point to significant problems which are averted or deal or dealt with, that's going to be very helpful.
00:45:42
Speaker
The problem with Y2K, of course, is that after the fact, it's very difficult to say, know, there are still sort of two schools of thought. One says, oh, there was never really a problem. ah we spent goodness knows how much money and and whatnot on ah on a non-problem and then there's another school of thought that will say oh we spent so much money and that's why nothing happened you know i love ah examples like if you just look at the number of of ah nuclear states you know it's going up very rapidly then the non-proliferation treaty comes into force and it doesn't quite flat line but it's like it it
00:46:15
Speaker
yeah you can see and you can see the impact of the intervention there. It's certainly, I think, interesting and potentially a ah really good thing about some of avalice organisations that they may be able to start to make legible certain impacts.
00:46:28
Speaker
That would be i mean very confidence-inspiring. People do this in in things like computer security, you know, where there is just a certain amount of kind of public awareness. and There's not enough by all accounts, you know, ransomware and and whatnot is most of the damage there is hidden from public site.
00:46:46
Speaker
So it's a little bit difficult to see the impact of institutions and tools. hopefully people will be able to make some of the impact of any interventions much more legible, um simply as a way of, you then being able to see what works and what doesn't work.
00:47:00
Speaker
I would love a sort of a nice compact example like that, you the tailing off and the increase in the number of nuclear states. So we don't have, you we might have had 100 or 150 now, and instead we've just had a small increase because of the nuclear cartel.
00:47:17
Speaker
I don't have a ready example to mind. how we're not We're not in the situation quite yet. One paradoxical situation you've pointed out is that if humanity reacts to a threat and thereby decreases the risk of that threat becoming a a reality, well then the skeptics will be able to say, okay,
00:47:38
Speaker
there was never a threat to begin with, right? This, of course, i would love for this to be the the situation with AI risk, that we actually react. and And then it's fine that the skeptics can say there was no risk here to begin with.
00:47:51
Speaker
But this this is such a good example to point out how how kind of difficult it is to navigate this terrain epistemically. you you We probably won't ever have a clear answer about You know, how big was the risk at a certain time?
00:48:08
Speaker
we we probably never will have a settled science here. So yeah've I've asked you this. I asked you this
Scientific Understanding of AI Risk
00:48:14
Speaker
before. But what is what are the good heuristics here? What are the good kind of rules of thumb for navigating a situation like that?
00:48:25
Speaker
ah I'm flattered that you think I might have something intelligent to say about that. I'm not sure, well, I'm sure there are people that have more intelligent things to say than I do. I'm not sure anybody's solved it. Actually, I just want lean a little. You asked me before, and you've sort of mentioned it again, ah about this science of AI risk.
00:48:41
Speaker
And there's certainly, mean, very clearly, there are some local things that that that that can be said. you know There are particular threats and and that we can reason about in a scientific manner.
00:48:51
Speaker
Thinking about it sort of more broadly, is... peculiarly difficult. I mentioned the example of climate before. There is this kind of just a astonishing fact that people started making arguments about climate change in the 19th century. And there was kind of a famous debate between Engstrom and Horinius at the start of the 20th, I think, where they kind of came to opposite conclusions, both based on really quite plausible conclusions arguments and quite plausible physical experimentation.
00:49:20
Speaker
And ah from a policy point of view, it left us in a really strange situation. We kind of had a good argument that climate change might happen. We had a good argument that climate change would not. And it wasn't really until the 1950s or 1960s that, in fact, that anything like a modern understanding started are to develop.
00:49:38
Speaker
And then by the nineteen ninety s it was really becoming the consensus and quite clear sort of what but was goingnna but was going to happen. That's a period of, what, 80 odd years for what is fundamentally a very simple physical problem.
00:49:53
Speaker
And in the case of AI, it's so much harder because fundamentally you're talking about the structure of knowledge itself. yeah how are our systems going to change our ability to navigate that but structure?
00:50:09
Speaker
it's ah It's a much, much deeper and much less accessible sort of part of the universe. um I don't see... It's like there is no science of science you know in some sense. in in serving the there are some i mean people I have made contributions to the field known as the science of science, but it's not a predictive model of discovery.
00:50:33
Speaker
By definition, but that you you you don't have such such a thing. You don't know what what hasn't been discovered. That seems much, much harder to ever imagine having a sort of a detailed science of.
00:50:46
Speaker
It maybe seems a lot more like like aircraft safety engineering engineering, where at least you can have sort of yeah reasonable models of the kinds of things that but that can happen.
00:50:57
Speaker
I don't know. i mean, you're talking about changes in epistemics. Gosh, think about things like, if you think about if you think about sort of a science of science, yeah its it's so hard to do because the tools any tools that you would be using are actually themselves subject to change.
00:51:11
Speaker
yeah know At some point, probability theory yeah was invented and then people like Kolmogorov really massively improved it. So if you had a science, ah a predictive science of science, sort of back around, whatever, 1920 or so, 1930, when he was getting ready to do that, I think it was 1933, his famous paper,
00:51:29
Speaker
The tools that you would be using were actually intertwined with the tools which he was discovering for people like Perle and others with the modern theory of causal inference. So there is this sort of funny intertwining. Our very epistemic tools are going to be changed. So i
AI Monitoring AI
00:51:45
Speaker
don't know.
00:51:45
Speaker
All right. That seems very, very, very hard. Do you think that's a limit that that we're reaching because we're human? Or do you think that's more of a fundamental limit?
00:51:57
Speaker
I'm thinking of whether we could use AI to to understand AI, whether we could, in a hopeful scenario, dedicate a lot of, say, AI inference or AI thinking to trying to help us navigate or understand this terrain.
00:52:14
Speaker
Sure, sure, sure. I mean, we we actually, I mean, we do this kind of thing. So one estimate I've seen is that the US spends about $300 billion dollars a year on fire safety. And so at at some level, that's actually, it's actually quite a similar sort of a situation. A lot of that is monitoring, trying to understand what are the threat factors, you what kinds of technological modification should we do locally to make things more fire safe and so on.
00:52:36
Speaker
And it's truly remarkable the number is that large. And you can imagine sort of yeah dedicating a substantial fraction of all the world's AI resources to yeah monitoring and attempting to understand and make legible what the other AIs are doing. It's kind of like a justice system for AI. Yeah.
00:52:54
Speaker
the problem is it's difficult to know where to begin there, right? What is it exactly you have your overseeing AI system do? Because you're you're trying you're you're you're kind of trying to have the AI solve the problem for you. But before you can begin solving the problem, you need to know to kind of stake out the territory that the where the work is even done.
00:53:15
Speaker
It just seems, so I mean, so hard for sort of, I mean, ah several several separate reasons. One is just the possibility to do stuff sort of encrypted away from yeah where it becomes illegible to outside eyes.
00:53:26
Speaker
And the other is the intrinsic illegibility of intellectual work anyway, you know it can just be sort of very difficult to understand the implications of what is going on.
00:53:37
Speaker
So you're monitoring your systems and it you yeah it it may be very difficult to tell that something is actually something that that has very negative implications.
00:53:47
Speaker
It's really tricky problem, great problem for a philosopher like yourself. If it was just intellectual, it'd be fascinating.
Deep Atheism and AI Risk
00:53:55
Speaker
Yeah, I mean, on that note, as ah as a final topic here, I would love for us to talk about your recent essay on deep atheism and optimistic cosmism.
00:54:06
Speaker
these These are two different worldviews or yeah approaches to to seeing the world. how how do they How do they differ? Oh, yeah, I mean, this was just... So joke Joe Kassman wrote this great essay series, Otherness and Control in the a ah Age of Agile.
00:54:23
Speaker
really just about how ah human beings think about the control of technology, the control of the universe in general, and how they were relate to the universe. And Deep Atheism is Joe's term for having a very fundamental stance of distrust towards the universe.
00:54:38
Speaker
People often object, they hear this term, and they say, that's not atheism. yeah It has nothing really to do with whether or not God exists, which I'm sympathetic to as an objection, but like let's stick with the term anyway.
00:54:50
Speaker
Yeah, and I suppose, i mean, I just, I read i read this essay of Joe's a year or so ago. i thought that was interesting. And then I was surprised by just how often the concept has come to mind ever since.
00:55:04
Speaker
And so I wanted to understand just a little bit more for myself what I think about about this. I think in particular, I've been reading William James's, fact, I've just finished just before this,
00:55:15
Speaker
William James's wonderful book, The Varieties of Religious Experience. And James does this really what wonderful thing, which is instead of sort of asking question, know, is this true or whatever, he just asks, what are individuals' religious and mystical experience?
00:55:31
Speaker
And in quite a non-judgmental way, he's just interested in kind of, you know, essentially going and looking for accounts of, oh, this is what it's like to, you know, go into a trance. Oh, this is what it's like to have a sudden conversion experience and so on in a very open way. He's not completely nonjudgmental. He's not a credulous person, but he is a person who is quite open to just hearing ah a lot of different things.
00:55:54
Speaker
And I find that but very beautiful. and And I think of this this idea of deep atheism as it's a particular version of that. Some people really believe that as sort of a psychological stance that the universe is kind of out ah to get them while also much more more trusting. And it's interesting to to to try and understand what causes that orientation. I think it's related surprisingly much to your views on on AI Express.
00:56:21
Speaker
You know, some things, like i said i I said before, understanding things like the great oxygenation event or the origin of nuclear weapons and whatnot probably changed my stance towards the universe ah to starting to think, oh, wow, there's like these enormously powerful things that are sometimes hidden in plain sight.
00:56:41
Speaker
And the only barrier is understanding. We learn a little bit more and we realize, oh, wow, we can change the world a lot, sometimes in really negative ways.
00:56:52
Speaker
So that's kind of ah you know a change in my own stance, which was caused by these, mean, apparently quite innocuous types of understanding. But I'm very interested, actually, in the question.
00:57:03
Speaker
I mean, I happen to be in New York today and... um I lived here for just a sort of a couple of months last year, and I was fascinated moving from neighborhood to neighborhood at the types of feeling in each neighborhood and the types of institutions which were available.
00:57:22
Speaker
In some sense, yeah what you see walking down the street, typically as a kid, yeah that's kind of the set of actions which are available to you. Oh, yeah I can go to the park. Oh, I can play on the swing.
00:57:34
Speaker
Well, that's only true if there are parks in your neighborhood. Otherwise, you sort of you don't internalize that. yeah If you're Stanford student, you internalize, if I want to do something, I can i can raise venture capital and yeah start a company.
00:57:46
Speaker
If you go to certain other high schools or universities, you're not going to have that experience. It's not of a verb that's in your lexicon. but so So also sort of thinking about the way in which that works.
00:57:58
Speaker
kind of experience conditions people like what sets their level of optimism what sets their level of agency what sets sort to what extent do they feel ah very win-win orientation versus something else what experiences in their past it's not just experiences it's There's genetics, there's environmental ah determinants, there's myths which condition it.
00:58:22
Speaker
And these are all, I suppose, sort of closely related to this orientation towards deep atheism, and which I got interested in because of the, the to to some extent, just because of the connection to AI Express.
00:58:35
Speaker
Very, very random connections that... These factors probably play a large role in in in how people in what people end up believing about AI risk in the end.
00:58:47
Speaker
We're not always as as rational as we might that we were. that we were Actually, one of the things that I love in Joe Cusman's essay is, you know, he takes a sort of, kind of his prototypical example in many ways is Eliezer Yudkowsky, who course is one of the people who has done a great deal to develop and popularize these concerns.
00:59:09
Speaker
And he finds quite a few examples to suggest that Yudkowsky, like really, you know, he does have this attitude of fundamental distrust towards the universe.
Personality Traits and AI Risk Perception
00:59:18
Speaker
Yeah. which I thought, i mean, just is fascinating. It's a very concrete example of the connection that ah you mentioned.
00:59:25
Speaker
The psychologist Danny Kahneman, I heard him talk on a podcast once, and he said, his opinion, like how optimistic or pessimistic a person is, tends to be just a personality feature, which is not particularly rationally granted.
00:59:38
Speaker
And there's not much you can do to change it, but it does determine an awful lot about what you believe about the world. And this is also clearly, you know, very closely, very closely. at the At the same time, though, there is this, you know, there is a reality out there that something is going to happen once we develop a superhuman AI systems. In some sense, there there might be a test of whether the world is friendly to us or whether the world is out to get us.
01:00:06
Speaker
Yeah, I mean, it's going to be quite a test. I don't know. I'm laughing better than crying. I must admit, I do find the semi-paradox quite concerning in this context.
01:00:18
Speaker
You know, somebody else out there should have, if it was possible to navigate ASI, it it should have been developed and and they should have colonized the stars and we should see them everywhere.
01:00:30
Speaker
It worries me that that's not the case. that That I think is part of the reason why worry a lot about the Vulnerable World Hypothesis. You know, the Fermi Paradox seems like a considerable piece of evidence in favor of the Vulnerable World Hypothesis.
01:00:45
Speaker
I should say, by the way, actually, people sort of worry about being killed by ASI. I, at some level, the alignment problem still persists, even for, you know, if had a society that was just AGI's, you know, the problem in many ways for them would be even worse.
01:01:02
Speaker
They would have more power. They would have more capacity to wreak destruction. unless you kind of get to this singleton situation where there's just sort of, know, essentially a single force running, running everything.
01:01:13
Speaker
know, we sometimes think of the alignment problem as being and just sort a problem for humanity, but it's actually a problem in general. Yeah, I agree. Michael, thanks for chatting with me. It's been, it's been a pleasure.
01:01:26
Speaker
Thank you so much, guys. This was really fun.