Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
What Happens After Superintelligence? (with Anders Sandberg)
2.6k Plays16 hours ago
Anders Sandberg joins me to discuss superintelligence and its profound implications for human psychology, markets, and governance. We talk about physical bottlenecks, tensions between the technosphere and the biosphere, and the long-term cultural and physical forces shaping civilization. We conclude with Sandberg explaining the difficulties of designing reliable AI systems amidst rapid change and coordination risks.
Learn more about Anders's work here: https://mimircenter.org/anders-sandberg
Timestamps:
00:00:00 Preview and intro
00:04:20 2030 superintelligence scenario
00:11:55 Status, post-scarcity, and reshaping human psychology
00:16:00 Physical limits: energy, datacenter, and waste-heat bottlenecks
00:23:48 Technosphere vs biosphere
00:28:42 Culture and physics as long-run drivers of civilization
00:40:38 How superintelligence could upend markets and governments
00:50:01 State inertia: why governments lag behind companies
00:59:06 Value lock-in, censorship, and model alignment
01:08:32 Emergent AI ecosystems and coordination-failure risks
01:19:34 Predictability vs reliability: designing safe systems
01:30:32 Crossing the reliability threshold
01:38:25 Personal reflections on accelerating change
Recommended
Transcript
Introduction and Integration of Human Preferences with AI
00:00:00
Speaker
I think we could take a lot of ethical advice from smarter entities, but we might also want to have a debate with them about it and actually share understandings. You actually want to weave our preferences and our discourses into this system in the right way.
00:00:15
Speaker
Ideally, we should become a kind of cyborg civilization where we both have super intelligence guiding and coordinating us. If you're below a certain error threshold, you can combine error-prone processes in such a way that you get a new process that has a much lower error rate.
00:00:30
Speaker
I believe that something like this might happen with AI. There is a kind of transition in reliability, but once it's reliable enough, you could make this redundant system, make the reliability go up enormously.
Anders Sandberg's Background and Manuscripts
00:00:43
Speaker
Welcome to the Future of Life Institute podcast. My name is Gus Docker, and I'm here with Anders Sandberg. Anders, welcome to the podcast. Thank you for having me. Could you say a little bit about your background?
00:00:56
Speaker
So I'm usually presenting myself as an academic jack of all trades. I started out studying computer science and mathematics. Then I took a course about neural networks and kind of fell in love with the brain.
00:01:07
Speaker
So I took neuroscience courses, psychology courses, a bit of medical engineering. And then I ended up in the philosophy department of Oxford University at the Future Humanity Institute.
00:01:18
Speaker
So these days, when people ask what I am I say some kind of futurist philosopher, something, something. You have this wonderful manuscript called Grand Futures, which is, last I checked, 1400 pages where you dig into the physics, the economics of all the kind of different paths that humanity could take and become a much larger presence in the universe.
00:01:48
Speaker
could you Could you say as a what's the status of of that manuscript right now? and So for a while, the manuscript had been resting because I needed to finish another book, Law, Liberty and Leviathan, Human Autonomy in an Era of Artificial Intelligence and Existential Risk, which feels a bit urgent. We kind of need to figure out some of those things out.
00:02:08
Speaker
So the manuscript had been resting on my sofa, kind of waiting yeah for me to finish with that. lightweight 600 page volume. But the nice part is, of course, now I learned how to write better.
00:02:22
Speaker
And science has kept on advancing. So I've been piling up references and things to add. So it's not like grand futures is going to be how the world was back in 2023 when I started on something else. It's rather now I'm rebooting it, which is also very useful because now I have many more people who can help me actually check that what I'm writing is correct or at least plausible.
AI as a Research Tool and its Evolution
00:02:45
Speaker
And perhaps you have assistance from ai at this point? Yes, that was fascinating. When I started, ChatGPT was not a thing. I've been working on this for quite some time.
00:02:57
Speaker
And during the process, I tried the sometimes, okay, can AI help me ah develop this paragraph? And the first few attempts were very hopeless. Okay, that's just nonsense.
00:03:09
Speaker
And then it became annoying because, yes, that's an answer. I don't trust it at all. Indeed, it turned out to be totally wrong, full of hallucination. It took longer to figure out what reality was than actually making use of it.
00:03:22
Speaker
Indeed, I discovered that quite a lot of apparently straightforward questions were surprisingly confusing, both to AI and to me. But over the past few months, it's gone from, okay, it's not useful at all, to it's actually helpful.
00:03:39
Speaker
I wouldn't say that I'm trusting all the results, but it's very good at digging up obscure literatures, saying, i have this question or about some vague domain.
Post-Superintelligence Paths and Transformative AI Potential
00:03:51
Speaker
Is there a name for this? it finds the name of a domain. And then of course, what are the main papers? Then I can start reading up on that. And once you have the terminology, suddenly you have a pretty useful assistant.
00:04:04
Speaker
So I'm looking forward to this brave new world where my me and my human assistants are having AI assistants to amplify the ability to both write, but also fact check and style check and develop these ideas. so I think one starting point here that could involve both Grand Futures and yourmo your the book that... Remind me again of the title of the of the other book?
00:04:31
Speaker
Law, Liberty and Leviathan. Yeah, exactly. And that book is to kind of ask a concrete question. so On this podcast and in in culture in general, we talk a lot about when we will reach AGI, when we will reach superintelligence. We talk try to forecast that question.
00:04:49
Speaker
And we talk less about what happens after that point. So for the purposes of of this discussion, we could assume that we reach superintelligence in 2030 and then see which paths we could take species from there.
00:05:04
Speaker
as a species from there so Of course, this involves physics and economics and sociology, I think, to to quite a large degree. But where would you start with trying to to to forecast what happens after superintelligence?
00:05:22
Speaker
Assuming we have ah enough alignment and that it goes well enough so it doesn't just doesn't break the world. And I think one should recognize that even if AI in itself is pretty safe, you might just shake the world apart at the seams if it just amplifies human ability to do things without amplifying human ability to coordinate about safe and sane things to do.
Material Wealth, AI, and Societal Shifts
00:05:46
Speaker
then you probably end up with a lot of and they're obvious things that are going to be pursued with great zeal and the intelligence. So right now we have an issue with energy in the world.
00:05:56
Speaker
We need better energy sources. We also need to avoid messing up the environment. And we have a lot of insecurities that are quite important, like food. ah And it's pretty clear that we're going to apply a lot of AI power to that.
00:06:11
Speaker
Maybe AI is self-directed agents. Maybe it is just tool AI or an ecosystem that humans are setting agenda for. But it's pretty clear that we're going to be pushing for material wealth and welfare.
00:06:24
Speaker
After all, peace and prosperity is kind of one of the key drivers for most human economic activity. And we should expect that an AI-empowered economy is also going to do a lot of that.
00:06:36
Speaker
So one of the first chapters in Grand Futures is so basically, okay, how much material wealth can we achieve? And when I started on that chapter, I believed that, okay, this is going to be relatively straightforward to write about because it's going to be about the physical limits to manufacturing and energy production.
00:06:55
Speaker
And to some extent, how much they are compatible with living on a finite planet. It turned out that there was quite a lot of weird economic sidetracks and indeed and even psychological sidetracks and that that made that chapter much more unwieldy and much more exciting than I had expected.
00:07:14
Speaker
And what were those side strengths? So when we think about the world of enormous ah wealth, at first people would imagine golden palaces and as much food as you can eat.
00:07:25
Speaker
That's kind of what our ancestors would have said, oh that's what to go for. They would, of course, also have said, oh, I want as much fat, sugar and salt as possible. That's the dream life.
00:07:37
Speaker
And we kind of know, actually, that's not very good for us. But certainly being able to make any material object that is useful is something we want to do. We want to solve the fundamental manufacturing problem. This is where I think Eric Drexler's vision about molecular manufacturing yeah and atomically precise manufacturing in the long run, which might not be terribly far away if we get super intelligence by 2030,
00:08:05
Speaker
is actually going to transform the world. Even if you say, okay, that's unrealistic, we're just going to have 3D printers and robots and biotechnology, that already produces a world where you can get most stuff relatively cheaply and easily.
00:08:21
Speaker
The really interesting part, however, is that when you think about wealth, if we think about the lifestyle of a rich and famous, it's not that much about that we eat enormous amount of food and that we have a lot of cars.
00:08:33
Speaker
We certainly have a lot of cars, but the food is fancy food. And although the villas might be large, it's more about that they're elegantly furnished and they have a lot of people waiting for them.
00:08:46
Speaker
That is, of course, what we actually mean by wealth these days. we're We're quite post-materialistic in the sense that we want services. So we want that massage. We want to have that personal coach. We want to have those assistants doing whatever they do to keep our wealth flowing.
00:09:02
Speaker
So the other part of wealth is, of course, actually services. Now, the good news is if we actually manage to get ah ai to work really well, we're going to get services very, very cheaply.
00:09:15
Speaker
So that would mean that we could all have that entourage of robot the software servants giving us the massages and managing our bank accounts and doing the legal stuff for us.
00:09:27
Speaker
That sounds great. Except, of course, you get both weird side effects. The bad guys are going to have endless lawyers in the cloud. They're going to sue everybody. You better have your own personal digital lawyer to resist getting a frivolous lawsuit.
00:09:42
Speaker
There's going to be a lot of stuff going on behind the scenes that is very complicated. But the really interesting part is, of course, in this world, that who gets to have that villa right there yeah and at the edge of the peninsula in Lake Como that has the best view?
00:09:59
Speaker
doesn't matter that we could manufacture a lot of such villas. There is only one spot that has that view. There is still going to be some things that are kind of a zero-sub game. Even worse, there are social zero-sub games.
00:10:12
Speaker
Who gets to be coolest at the parties? that Of course, we might have different views on that. Maybe the music star think that they look the best. And I think I have the most interesting academic publication. They're both very smug at the party.
00:10:27
Speaker
But there are things that even in a world of material and the service post-scarcity are going to be limited. And then we get to the final most interesting part. Why do we want all this stuff?
00:10:40
Speaker
Well, people would say, well, it makes me happy. And that happiness is
Human Psychology, Status Games, and AI Interventions
00:10:45
Speaker
actually the real resource we want. And of course, philosophy has been going on for thousands of years about that. That's what we should be aiming for, not getting as rich as the Athenians, but we should be and a content with what we got.
00:10:58
Speaker
How do you reach contentment? Maybe you can have your AI life coach give you really good advice, but there is probably also aspects of brain functioning and psychology, not just healing us so we don't have depression, but actually coming up with better ways of being very, very happy.
00:11:14
Speaker
Happy in an effective way rather than drooling in a corner in the the opiate enjoyment, but actually being able to have a really fulfilling life. So that gets to an interesting aspect this aspect of post-scarcity, actually bounding yeah our enjoyment of the world.
00:11:31
Speaker
And maybe that means that actually we don't want those gigantic villas and the endless entourages of robots. We're going to be ah instead living a simple but very, very happy life. I think we're going to see a mixture. We're going to want to have both a golden palace and good spiritual contentment.
00:11:48
Speaker
But this is going to be a tricky thing. It's not going to be that trivial to solve, even perhaps for super intelligences. there are you know There are quirks of human psychology that means that we are very concerned about hierarchies and status hierarchies. And these games are, as you mentioned, you know not everyone can be the coolest person at the party.
00:12:12
Speaker
And so is there any way to intervene in our psychology such that we can steer away from zero-sum status games? we can steer away from serosum status games I think there are ways. There are certainly some people around that are not in zero-sum status games. There are the nice people who don't care about these things.
00:12:36
Speaker
And presumably, since that is an aspect of how the brain works in them, we could replicate that. We could imagine that we studied these saintly people and figure out a way of becoming saintlier.
00:12:48
Speaker
There are ah interesting issues here. One is, of course, figuring out how how it works. But I think that is something more advanced psychology and neuroscience would be able to do. We know, for example, in psychology that there are two different kind of status. There is dominance, the kind of bullying yeah status you get. to prestige.
00:13:08
Speaker
You're really good at something and people admire you for it. And both of them are a bit rewarding. It feels very nice to know that you're very good at you do what you do, or for that matter, to bully people and feel that I'm on the top of the pack.
00:13:22
Speaker
However, How we react to that already when the dominant people fall from grace, when they lose some of the dominance, people typically kick them.
00:13:32
Speaker
Suddenly the underdogs strike back. It's not very fun to be a and a formerly popular bully. While high prestige people, people generally like. And even more importantly, it seems to be this rewarding thing that drives us to want You could imagine removing that altogether, maybe some form of gene therapy or and brain and manipulation.
00:13:54
Speaker
ah A world where people don't strive for social status a it would be a very humble world. And it would probably work extremely different from ours. It's not even obvious that we would want that. Because that gets to the other really tricky point.
00:14:08
Speaker
When thinking about enhancing humans, I worked a lot on the ethics, social impact of cognitive enhancement. People are generally very happy with thinking about, oh, learning language is better. Better memory, yes.
00:14:21
Speaker
Staying alert longer, yes. Really useful. Becoming kinder. In that one survey, only 9% wanted the to take a hypothetical pill that made them more kind to people.
00:14:32
Speaker
Perhaps because people are afraid of being screwed over by the world, you don't maybe you don't want to become kind if you feel like you thereby open yourself up to attacks from from a cruel world.
00:14:45
Speaker
I think that is one part of it. But this actually came through, or this view of people not wanting to enhance things that were fundamental to the sense of self came across for many other things, like empathy.
00:15:01
Speaker
And the I think in general what happens is we're afraid of not being ourselves. i I might on one level want to be a kinder person, but it would also change my personality a bit if I took that pill.
00:15:14
Speaker
So I'm a bit afraid of doing that. And it feels like my kindness might be closer to who I want to be than my ability or non-ability to speak French. So I think there is this interesting interplay that even if we could offer some enhancements, it's very likely that many people would be reluctant to take them.
00:15:32
Speaker
Of course, we might imagine a society where people say, actually, um it's so important given the ten tremendous power we have thanks to our technology. We need to be more sane and more kind. We can't allow unstable, crazy people to wield these technologies.
00:15:47
Speaker
So you might see some very interesting social struggles about exactly how much therapy and adjustment do you want your citizens to have? And this is going to be politics for the 21st century, I think.
00:16:01
Speaker
yeah Yeah. If we return to the question of the physics and economics of of a post-superintelligence society. So so we what you've been discussing here is a bit of of the economics and perhaps what we'll end up doing if we ah emerge into some kind of post-scarcity state.
00:16:18
Speaker
and Another question I'm interested in is in in Grand Futures, you investigate kind of what can happen on very long timescales. If you think about the hypothetical in which we have superintelligence in 2030, how far do you think we can get in into bit by 2040?
00:16:36
Speaker
Just because there's
Superintelligence Infrastructure and Energy Concerns
00:16:37
Speaker
a tendency, I think, to think that if we get superintelligence, we are... we are there are no limits anymore. But of course, even superintelligence is bound by natural laws, the laws of physics and so on.
00:16:50
Speaker
So how much can we expand and what becomes possible in in a decade with superintelligence? That is a very good question because the limitations are set by the material conditions of the world and the meta physics of computation.
00:17:06
Speaker
So first, it's a bit unclear how much energy, for example, it takes to run a superintelligence. Right now, people are going on endlessly about how much energy is consumed by data centers.
00:17:17
Speaker
I think it's a lot of energy, but it's not that key problem. Because the real question is, how much good the decisions do you get out of a given amount of intelligence?
00:17:29
Speaker
And that's probably not scaling very clearly with energy. energy And once you get super intelligence, you're probably going to get it to find ways of optimizing its energy consumption. We know a human brain runs in about 20 watts of power.
00:17:43
Speaker
So we know we cannot at least get that much intelligence out of one kilogram of matter and ah running at 20 watts. But the real problem is, of course, building a new data center today takes a few years.
00:17:56
Speaker
You need to bring ah the the funding in. You need to get approval for the plans. You need to set up the foundations. You need to build the building, install the service. They need to be shipped over from Taiwan or elsewhere.
00:18:09
Speaker
You need to test it out. How much of that can you speed up if you had really, really smart systems? And it seems like some of these processes are materially limited. If you need to ship something on a boat, that boat is not going to move faster just because the entity that ordered the ships happened to be very smart.
00:18:29
Speaker
Of course, the very smart entity might be a very impatient entity and might say, okay, I'm going to ship it over by air, or I'm going to invent that cargo zeppelin. But now again, you have a problem. Okay, I invented a cargo zeppelin.
00:18:42
Speaker
I need to get it approved. The Federal Aviation Authority might have opinions about that. It needs to be tested. That's going to slow things down. And even on the AI that is amazingly good at getting bureaucrats to do what they're supposed to do fast is still going to be limited by that.
00:18:58
Speaker
Even if it could just ignore bureaucrats and just sense the robots ah to do things, there are still limits on how much you can move around without messing up the environment. So Earth has this interesting property that right now we're consuming a fair bit of energy and then it turns into waste heat. And waste heat is a very minor problem for us right now.
00:19:18
Speaker
We're certainly concerned about climate change, but that's because we put carbon dioxide in the atmosphere and the changes the greenhouse effect. But the waste heat from all our servers and cars and the gadgets, that just is taken up by the atmosphere and radiated into space.
00:19:34
Speaker
If we were to increase our energy consumption by a factor of 100, we would get heating no matter what we did with the carbon dioxide, even without any carbon dioxide, even if it was powered ah by magical unicorn rainbow power with no other environmental effects.
00:19:50
Speaker
the waste heat for 100 times more energy-active humanity would actually start heating the world by about one degree. So there is a limit on how much you can do without starting to overheat the Earth.
00:20:03
Speaker
Similarly, if you move large amounts of mass around, you are going to have problems with the environment. so Keeping or Earth-like requires staying in some limits. These limits, if you have really advanced technology and are really and smart about how you use it, are, of course, much wider than we normally think about in the environmental discussion.
00:20:24
Speaker
The typical environmentalist discussion tends to assume that we need to do less things because that's the only way of saving the environment. If you actually look at efficiencies, etc., you realize actually more high-tech things can quite often be much more and impactful on the environment if you do them carefully. but The problem is, however, that once you start scaling up a civilization, you need to be more and more careful.
00:20:49
Speaker
So in the long run, I think you will want to to put the superintelligence data centers in orbit. ah People are already discussing that in Silicon Valley. And right now, I don't think it's going to be cost-effective because cooling in space is much harder than on Earth.
00:21:04
Speaker
We are kind of cheating by our waste heat turning into heat in the atmosphere, and then it's radiated away from the top of atmosphere. If you have your data center orbiting the Earth, Now you need to have radiators sending it out in the coldness of space, which sounds really good because space is really cold, except that you need to do this as radiation.
00:21:23
Speaker
The convection from a computer and it just moves heat away from it very efficiently. But if I need to radiate it that way, it would suddenly be hard to cool it well enough. And of course, the problem for that data center is subject get to a lot of sunlight from the sun.
00:21:38
Speaker
Great for the photovoltaics power it, but also heating things up. So building things that work well in space is
Technosphere vs. Biosphere and Evolutionary Comparisons
00:21:45
Speaker
tricky. and Yeah. Could you go over that again? Just so I understand, why is it that it's more difficult to cool down a a data so data center in space, given that space is is very cold?
00:21:57
Speaker
So on Earth, the the data center is going to probably connected to a cooling tower or ah maybe a lake of cold water. And the waste heat moves into the air and water and even from the water into the air and eventually ends up at the top of atmosphere and gets radiated away into space.
00:22:15
Speaker
So we can use the entire top of Earth's atmosphere as a gigantic radiator sending away this radiation. In space, if I have my data center there, I need to build my radiator.
00:22:26
Speaker
I can't just have the heat waft off into the vacuum of space because there is nothing there to conduct the heat. with And the big problem is... There is something called the Stefan Boltzmann law that tells you how much heat gets radiated away.
00:22:42
Speaker
And it says that it grows with the fourth power of the temperature of your radiator. So this means that a very hot radiator is amazingly effective at radiating away ah the energy. The problem is a cooler radiator is less effective.
00:22:57
Speaker
And this is quite often a problem. For example, the International Space Station has this issue. It ah has sections that are full of people and they need to be about 20 degrees centigrade.
00:23:08
Speaker
And now you need to keep them... at that temperature and radiate away the waste heat. So now you have a radiator that is about 20 degrees centigrade. That is a very wussy radiator. It doesn't get much energy out.
00:23:20
Speaker
There are other radiators on the space station that are actually cooling some motors and compressors that are much hotter. And they are much more effective. They can actually be smaller than the radiators you need for human section.
00:23:33
Speaker
So now we have to imagine this data center in orbit Its big problem is that you normally don't want your chips to be too hot. That's kind of bad. You want them to be really cold, actually.
00:23:44
Speaker
But the colder they are, the worse the radiators become. And so something that's quite concerning, I think, is that it will be tempting to build data centers on Earth, even though the more we build, the more these data centers take up of the Earth's surface and the more the...
00:24:03
Speaker
say solar panels cover the earth surface, the less compatible earth will be with with biological life, with with human life. So there are, in some sense, biological life is fragile and we require a quite specific environment in which to thrive.
00:24:20
Speaker
But ah server farms, computers can thrive in a wider range of conditions. Do do you think this is is a kind of fundamental issue rift between the interest, so to speak, of a superintelligence in expanding its own power and humanity's interest in staying alive.
00:24:42
Speaker
My friend Forrest Landry has made the argument that in the long run, a technosphere is always going to outcompete the biosphere because it has this wider range. And I think it's an interesting argument. I don't buy it straight that away.
00:24:55
Speaker
ah think it depends quite a lot on, well, who's in charge of that technosphere? How does it actually work? What is the economics and goals guiding what is getting built? The reason data centers are so general is that we design them right now to work in different environments.
00:25:11
Speaker
So this typical data center, well, that's a big building where people in and in shirt sleeves can go around and repair the service. It can't be too hot to ah because then it's going to be ineffective.
00:25:23
Speaker
But there are other people working on data centers that actually you're literally have in a container on the seafloor. They have different performance, but they are designed to do that. And we can imagine constructing data centers that are intended for the Arctic or tropical jungles or space.
00:25:41
Speaker
So there is a design issue here. But that design thing is, of course, a fast transition. the Evolution has taken millions and millions of years of coming up with organisms that can thrive in different ah environments on Earth.
00:25:55
Speaker
But in this case, you just plonk down a bunch of engineers in a meeting room and say, ladies and gentlemen, we want to build a data center that works very very well in the Taklamakan Desert.
00:26:05
Speaker
And they start designing way. but And within a few months, there is a bunch of servers standing in the desert. Now, that rapid ability to change is very different from natural evolution.
00:26:18
Speaker
I do think that we are going to intervene a biology, too. I don't think there is anything sacrosanct about biology. So we could probably speed up evolution by doing technological evolution. Indeed, synthetic biology is a good example.
00:26:30
Speaker
And in a world with superintelligence, it should be fairly easy to redesign biology. But that doesn't mean that you can just redesign ah our ecosystems to thrive if there is loads and loads of data centers heating everything up.
00:26:44
Speaker
That's not how we're going to solve it. In particular, because I think we don't want to transform things too much. We have this conservative view that actually the biosphere should probably be green and blue and it shouldn't look totally alien. um ah There's probably going to be some pretty interesting environmental discussion about repairing and our environmental damage we have done in the past if we get rich enough ah and flexible enough to do that.
00:27:08
Speaker
And now we're going to have a struggle. Should we restore a large part of the world to how it was before UMass arrived? Or maybe even earlier? Or should it be some kind of parkland? You have a lot of options.
00:27:19
Speaker
And these choices are going to be big, contentious political issues. But the deep issue that you pointed out is interesting. Technology and is more open-ended because it's guided by intelligent action.
00:27:32
Speaker
And intelligence can decide on jumping to somewhere else in state space. Biology works by evolution and learning. Organisms change very gradually.
00:27:44
Speaker
If there is no the easy path that is always better for your ah fitness, a species cannot evolve into another species. But of course, in technology, we might realize that actually we need to switch from a light of an air in a transport to heavy of an air transport and let's construct an airplane that doesn't work at all like a balloon.
00:28:04
Speaker
And maybe we want supersonic transport. Okay, let's mutate that in an airplane, not just in the sense of doing gradual changes, but actually a radical redesign. So this ability, I think, means that in the long run, the world is going to be guided by intelligent agents and what they do rather than this natural evolution.
00:28:25
Speaker
Except that a lot of these agents are, of course, competing and copying and cooperating with each other. So there is a form of evolution going on, but it's in the cultural space rather than in the biological space.
00:28:36
Speaker
And that means that, of course, the world becomes a cultural artifact for the civilization living in it. it's It's an interesting discussion to think about what is it that will actually determine the shape of the of the future here. is it Is it, say, the fundamental limits of of physics as you explore in Grand Future, or is it is it more determined by culture?
00:29:00
Speaker
This is... this is ah and I mean, my my kind of naive guess is that culture is what matters most in the short run, whereas perhaps physics is what matters most in the long run, such that culturally we will determine what we'll do, say from 2030 to 2040.
00:29:19
Speaker
But ultimately over hundreds of thousands of millions of years, we are probably, if we survive as a culture, we're going to explore various limits of of physics in in a bunch of different directions.
00:29:33
Speaker
do Do you think that view is correct, that that culture is more significant in the short run? I think it might be significant ah both in the short run and long run. We're certainly limited by the laws of physics, and they are much nicer to think about for me because I can say things in a more rigorous way.
00:29:51
Speaker
ah The light speed limit, there are kind of profound reasons why you can't move stuff faster than with the speed of light. ah The laws of thermodynamics are true for us mere humans and for future superintelligences.
00:30:04
Speaker
Even when you cheat about them, those cheats have their own costs. There are limitations and they place boundary conditions about what you can do in the future. So I can make a very confident prediction that in a million years...
00:30:18
Speaker
then The intelligence from Earth is still not going to be in the Andromeda galaxy because it's very, very unlikely that you can break the light speed limit. So even those super intelligences, even if they were racing to Andromeda, they're still on their way getting there.
00:30:34
Speaker
But why are we going there? That's a cultural question. It might be because it's a religious goal. It's a pilgrimage. It might be because it's an art project. It might be because we're competing fiercely for all the resources.
00:30:48
Speaker
And this is culture. And it's extremely open-ended because it has so many degrees of freedom. If we look at current society, a lot of the activities we're doing are not terribly bounded by the laws of physics.
00:31:01
Speaker
Certainly, I need to pay my energy bills But they're not enormous. I don't spend a lot of time working per week to just get fuel. I don't need to pay spend that much of my resource on that.
00:31:14
Speaker
The amount of resource I spend on getting food to survive are pretty minimal. If I had been a hunter-gatherer or if I'd been the kind of pre-human living in the forest, I would have spent essentially all my available time on that.
00:31:29
Speaker
Now, this has changed because as we get richer in a material sense, we are able to use the other resource for other things that
Cultural Influence, Historical Patterns, and Institutional Changes
00:31:38
Speaker
we care about. Social interactions, intellectual work, spiritual work.
00:31:42
Speaker
And the end result is, of course, that the activities we do becomes much more diverse. A colleague of mine, Karim Jabari, wrote a really interesting article about whether taking backups of civilization by making refuges ah in the case of a disaster, if that makes sense from a survival standpoint versus actually backing up the unique part of a civilization. And his point was, if you look anthropologically at societies, all human societies are full of their own cultures, but you can get much more culture and it's much more unique.
00:32:16
Speaker
the more and a a well off you are. Hunt and gatherer societies are certainly different because of the environment, but they're very constrained by the environment. You can't actually lug around the temple and the library if you're nomadic.
00:32:28
Speaker
If you're having an agricultural society, you can start building your temples and libraries And now you have more options for different styles. But typical Bronze Age civilization will make pyramids because they're fairly straightforward to make.
00:32:40
Speaker
Once you get to Iron Age civilization, the options become bigger and bigger. Your civilization becomes more contingent. So I think that a super civilization where you have super intelligence and reach the material limits...
00:32:53
Speaker
might choose extremely different things. It might be that it's all very green and it's aiming at preserving life or spreading life across the universe. It might be that it cares about welfare beings and it's working very hard to make sure all entities it can reach are happy.
00:33:08
Speaker
Or it might be doing science or competitive economics or all of above at the same time in some complicated political framework. There's a sense in which if you if you look at history over the very long term, or say say from from the emergence of humanity as a distinct species until today, you get this sense that we are expanding and that the pace of change for so for humanity is accelerating.
00:33:36
Speaker
And perhaps you you get this slightly... religious feeling that we are we are being pulled towards some destination where we will fulfill our potential. Now, that is that is in some sense teleological thinking, and I'm not i'm not sure how how rigorous ah that is.
00:33:52
Speaker
But Do you think in some sense we are being driven to expand by forces we don't fully understand? Do you think we have some kind of cultural evolution where those institutions and the people that drive us towards more growth, more expansion, tend to win out simply because they control more resources such that civilization or humanity, however you want to model this, is pushed towards is is pushed towards
00:34:24
Speaker
more growth and ah faster pace of change. I think it's a kind of ratchet effect. On the micro scale, people are running around doing all sorts of things for all sorts of reasons.
00:34:36
Speaker
But of course, people are people. Most of us have somewhat similar goals about survival, ah social recognition. You can fill in the muscle of hierarchy of needs, but they're also very different.
00:34:47
Speaker
But from that, of course, you get patterns emerging. There is a reason economics actually makes sense. Supply and demand is a pretty solid finding. It's not as trivial as most textbooks make it out, but you do get this effect.
00:35:02
Speaker
If you have somebody being more effective than somebody else, generally they can out-compete others in that niche. Except, of course, we humans are very good at coming up with ways of changing the niches.
00:35:14
Speaker
Just because your company is doing really well doesn't help you. If I got better lobbyists and I and then make sure that the the laws are written in my favor, That's true. But then you also have competition between countries where there's no there's no world government. And we have, a but we have ah a in some sense, anarchy between countries. So that such that if one country creates a more competitive environment for their companies, they'll tend to attract ah more companies. And so kind of resources and talent will shift into that country.
00:35:46
Speaker
Yeah. ah And the interesting part here is that you have these effects that actually do generate patterns. So saying that it's all totally up to human autonomous behavior is a mistake because human autonomous behavior generates institutions and patterns.
00:36:02
Speaker
We get markets. ah We get and various forms of complex institutions that we use to solve coordination problems. I think Friedrich Hayek was very right in that markets embody a lot of knowledge and our institutions in society are the result of a kind of cultural evolution that generates things.
00:36:22
Speaker
And then, as other and economists and sociologists have demonstrated, different institutions can help or hinder a society. And societies that are kind of burdened by really bad institutions They tend to have trouble and so they get behind.
00:36:37
Speaker
and then sometimes it's so badly that, okay, but you need to do a revolution and add new institutions, usually copying from more successful societies. So in the really large, I do think that there are these very big trends.
00:36:51
Speaker
I think they're not coming around is exactly because of some kind of strong law of nature. But it's a bit like how evolution ah tends to do optimization. It's haphazard and sometimes weird mutations do happen just out of sheer chance.
00:37:06
Speaker
It's not and and entirely deterministic. But on a large scale, you see a general move. If we look at the the economic growth in the world for the past 2000 years and probably far beyond that, it's been a pretty smooth exponential with slight wiggles because of the fall of the Roman Empire and the Black Death and the the World Wars.
00:37:26
Speaker
That is really impressive because we're talking here about billions and billions of people acting on their own, but generating these overall patterns. The problem with macro history is, of course, that both people tend to think that they can predict too much from it.
00:37:41
Speaker
And quite often it turns out that these predictions don't work that well. Predicting the future is surprisingly hard. I think, for example, most macro history totally misses the risk from existential risk.
00:37:52
Speaker
Actually, if we have a nuclear war, that economic exponential is actually going to have a pretty nasty break. if it If it's a survivable the nuclear war, it's probably going to keep on recurring and and and we get a new exponential, but it's getting delayed by a few centuries.
00:38:08
Speaker
But if we go extinct, well, that's just the end. But I also think that rules can change in interesting ways. so And this might, of course, really transform things. Right now, humans are the only intelligent actors. In order to do work, you need labor, and that is humans.
00:38:26
Speaker
But we're increasingly having machines. And right now, we're just amplifying it. But it might very soon be that if I need a lawyer, I just spin one up in the cloud. If I need 10 lawyers, well, I spin up 10 in the cloud. And the economy is so scary that why shouldn't I run 1,000 lawyers and have them give second opinions? And then 100 lawyers evaluating the second opinions and getting the best ones.
00:38:48
Speaker
Suddenly, the amount of labor being applied for maybe my frivolous lawsuit became much bigger. And that might change how this actually works. It might change the dynamics. It might change the fragilities.
00:39:01
Speaker
So I think there are big patterns in history. And I think if we zoom out even more and try to think about what's going on here, I do think that we see life in a very general sense, starting out as simple.
00:39:14
Speaker
It replicates, it uses its environment. It evolves towards being better at doing that. It gradually expands. It might change its environment. Then it gets better at adapting to environment and eventually brains emerge.
00:39:26
Speaker
And then in an instant, these brains take over. And now what is expanding is not so much biological life as intelligence, transforming matter and energy into forms that are suitable for it.
00:39:38
Speaker
And that might, of course, still go awry all sorts of ways. But I think we see a phase transition, actually, on the large scale of matter in the universe, from inert matter sitting there and just maximizing entropy, more or less, over to more dynamic matter, where are various complex processes going on, to life, to intelligence.
00:39:58
Speaker
And the intelligence is interesting because the goal of intelligence is typically getting desired outcomes, even if they're very low probability from the start. But I can make it likely that I have a bookshelf by buying it online and then assembling it using tools.
00:40:13
Speaker
but Finding a bookshelf out in my garden would be very unlikely, but I can make use of the fact that not just that humans can make bookshelves, but we set up an entire economy and make it very easy for me to get it.
00:40:26
Speaker
And if we need to solve a new problem, like curing a disease, suddenly a lot of low probability events start happening. And before you know it, vaccines are everywhere and the the poor virus didn't know what hit it.
00:40:40
Speaker
what So this actually leads me into my next question, which is, how do you think superintelligence would disrupt technology? our current institutions. So if we're thinking about markets and and governments in particular, and perhaps to constrain the question, we can we can think again of a hypothetical scenario in which we get superintelligence by 2030, and then think about what institutions would look like by 2040.
00:41:06
Speaker
So the problem with updating institutions is that they're full of people who are very much guarding their own jobs. So any change in how you run an institution, that means that your job description is going to change.
00:41:20
Speaker
It's going to be resisted fiercely. One of my favorite examples is that the pulse oximeter, the little clip-on thing you have on your fingertip measuring your blood oxygenation, it spread over a span of 10 years across intensive care units around the world.
00:41:33
Speaker
And it was unproblematic because yet another beeping device saving lives, and it didn't change the workflow. Meanwhile, laparoscopic surgery, and then where you and they don't have to make a big incision, took about a generation of surgeons because you need to work in a different way.
00:41:50
Speaker
And I think the same thing happens to economics and governance. So you get ai How do you use it? Well, the most obvious thing is, of course, right now that everybody having to write a boring report uses ChatGPT to write a report.
00:42:06
Speaker
And maybe they don't admit it, but most bureaucrats taking these reports are using LLM to read the report and summarize. Did they say anything important here? That creeping and modification is going to happen on a larger and larger scale.
00:42:21
Speaker
In many cases, this is a very good thing. You don't need super intelligence to improve ah governance. You could ah just have it systems approving things that should be obviously approved so you can get the governance running twenty four seven that You might say send the more contemptuous cases to people in a higher up.
00:42:40
Speaker
ah You then we end up with this interesting creeping cyborgization of our organizations. So Max Weber, in his famous um view of bureaucracy from the turn of the 20th century, was arguing that rationality means that bureaucracy expands into this iron cage, as he vividly described it.
00:42:59
Speaker
Because having this neutral organization is effective and implements the political will well. And he would, of course, say that AI is just continuing this. We're just going to eventually end up with an algocracy.
00:43:11
Speaker
Instead of having individual bureaucrats deciding things, you replace one with algorithms that can be totally reliable. And I think that is going to happen even though the administrators and bureaucrats are going to fight that tooth and nail.
00:43:25
Speaker
You're going to see ah the lawyers and doctors ah very clearly lobbying all the governments to make sure that you always need to have a lawyer and doctor rather than an AI to get legal or medical advice.
00:43:38
Speaker
But what you also get is, of course, that you can optimize things when starting new organizations. So I think the market is probably going to be a place where you see the most radical changes.
00:43:49
Speaker
Because traditional companies, they're basically bureaucracies, they might be slow in changing. There's going to be a lot of middle management that but doesn't want to be replaced. They might want to replace those unruly engineers and art directors and other annoying people, but they don't want to to change.
00:44:05
Speaker
And they are also good at retaining the position because they're management. But then you have a competing company, which might be one of those fired unruly engineers who just set up himself as CEO and then has a dozen or a thousand or a million very, very smart ah virtual employees implemented. it And I think those companies are in the long run going to win.
00:44:28
Speaker
And the long run might actually not be terribly long. That depends a little bit on a competitive market. But if you get super intelligence, that means that but I should be able to get super managers to handle the AI. I might get super marketers, I might get super engineers, and I might get super advisors.
00:44:45
Speaker
So the only thing and that I put in was the will of starting the company and maybe some initial capital. So that might mean that now you get an economy that's full of entities that are doing very smart things.
00:44:58
Speaker
That doesn't necessarily mean that the growth rate goes through the roof because quite often you're constrained by how much supplies you can get. but Those ships coming in from Taiwan are still slowly making its way over the Pacific.
00:45:10
Speaker
And until you can build the super fast ships, which is still going to take a few years, and you are going to have that limitation in the economy. and In Germany, there are still rules that certain contracts need to be signed and stamped in the proper way with seals.
00:45:26
Speaker
That rule might take years to overturn, even if you had the best lawyers so you could get in Germany. So I think by 2040, in a surprising way, the world might still look somewhat the same.
00:45:39
Speaker
But to borrow from a very insightful observation in Charles Strauss' novel, Halting State, there he describes but the future Edinburgh.
00:45:51
Speaker
The main character points out, this looks like it did 20 years ago, which is kind of our present. But everything is totally different because behind the scenes, the nervous system is running on a much more advanced internet. It's using various forms artificial intelligence and various weird goings on that wouldn't make sense to us in the present or now part of everyday life.
00:46:12
Speaker
And we already see this, of course, when we see people scanning QR codes and spending their time with phones. 20 years ago, we didn't have that underpinning.
AI in Decision-Making and Power Structures
00:46:21
Speaker
We have actually transformed how the society works without changing the material basis that much.
00:46:27
Speaker
I mean, this is this is ah an observation that I've had myself and that I've i've heard heard others make. Just the the fact that life has actually changed a lot since the year 2000, for example.
00:46:38
Speaker
But it it doesn't really feel like much much has happened in some sense, just because psychologically we are so quick to adapt to to a changing circumstance.
00:46:49
Speaker
Just the internet, for example, we have kind of adapted to that. do Do you think there is a chance that we will psychologically adapt to superintelligence in the same way such that we will feel like, okay, we we now have material abundance, say, or we have, so you know, the world has gotten strange and weird in various ways, but we don't feel like much has changed just because we might have this psychology that that kind of reverts to to a baseline quite quickly.
00:47:20
Speaker
I think at least partially that's definitely going to be true. You're going to go out in your garden and see that tree that has always been there and it changed a few a few in the branches and leaves in the last few in years, but you still recognize that that same tree, except that now it might very well be ah online and have its own blog.
00:47:41
Speaker
It's just that these transformations also affect us deeply because we feel deeply unsettled when the foundations of our existence do change.
00:47:52
Speaker
I'm starting to feel the AGI in an amusing way because my academic survival trick is that I know a little bit about almost any topic. I can riff on almost any subject, which is great.
00:48:06
Speaker
I can run around between different departments and try starting collaborations. This used to be something extremely unique, but now, of course, any LLM and they can riff on any topic too.
00:48:18
Speaker
My advantage over very many of LLMs might be smaller than some of my more specialized colleagues, which I find absolutely hilarious and also deeply unsettling. There is that pit in my stomach. Okay, am I actually going to be useful?
00:48:34
Speaker
Now, I think I can still be useful for a while, even if ah Claude and ChatGPT know more trivia about any topic. And that is, of course, because right now i can still kind of ask the right questions. I know what is important. I know how to connect things.
00:48:51
Speaker
But superintelligence might very well be able to do that. and might actually be better at figuring out what's important. Indeed, I'm very bad at keeping to the really important question. I get sidetracked by fun and curious questions instead.
00:49:05
Speaker
Now, the really interesting thing is, of of course, a world where you have super intelligence available, not listening to advice from it would be a very stupid thing. And it starts, of course, when ah the the manager at the company doesn't listen to the advice so of the AI.
00:49:21
Speaker
Well, that's going to be worse compared to the actual listening. So it's going to be worse for the stock price. So another piece of advice is, of course, fire those managers that don't listen to AI advice.
00:49:33
Speaker
The companies that do that are going to be more successful. You get a gradual takeover in some sense. The president that doesn't listen to a super intelligent advisor, somebody combining the diplomatic knowledge with Kissinger and the but technical astuteness of Feynman...
00:49:53
Speaker
Yeah, that is a precedent that is going to be at a disadvantage compared to that country where the president actually did listen to the very smart advisor. But do you think the feedback loop with governments is as as fast as it is in markets. So there are startups competing with existing companies that can outcompete existing companies.
00:50:12
Speaker
There are, in maybe in some sense, you could say there's ah there is an analogy, but there there aren't really startup countries. Of course, as I mentioned, there's competition between countries, but it seems rather slow and it seems like presidents and leaders of countries that aren't implementing AI would be able to survive for longer just because the competitive pressures wouldn't be as strong in governments.
00:50:42
Speaker
I think the inertia is higher. ah So I think there are, as you say, there are a few startup countries. You find a few interesting examples like Estonia, which in some sense is a startup country because it's a relatively new one. And it also hasn't got a two sclerotic government yet.
00:51:00
Speaker
It tends to happen in human institutions. So one interesting question is whether ai might entrench that tendency. So we might get super powerful institutions that defend themselves in a very clever way and in order to not change or whether instead this adaptation becomes faster.
00:51:16
Speaker
Because those super advisors, the first step is they just give advice. And then you get the feedback loop on how quickly do you turn about that advice into action. And for most political purposes, you don't need to to do that very quickly.
00:51:30
Speaker
Indeed, most governments take in information at an exceedingly small and slow rate. If we think about the America, for example... When people vote every four years, that means that they're sending information about who they want to run the the government and what values should it be.
00:51:48
Speaker
But if you think of it about it as bits per second, it's not that many bits per second. i did a calculation a while ago, and I think it was something like half a bit per second. That's not a very impressive input in terms bandwidth.
00:52:02
Speaker
Now in a market, of course, you get way more information from the prices. In practice, of course, the government actually listens to what people say, but you could imagine using AI to get much higher bandwidth information into government, into markets, into decision making.
00:52:17
Speaker
And this might be particularly important when there is a crisis. Most of the time, you want things to follow routine and you don't really care about making false decisions. But suddenly, when something hits the fan, you really want to make a quick decision.
00:52:33
Speaker
yeah So there are these wonderful systems in the earthquake prone zones that detect earthquakes and signal to trains to start slowing down. So by the time the earthquake hits them, the light speed signals have already passed by and ah told the train to reduce speed to reduce damage.
00:52:52
Speaker
You can imagine doing the same thing, of course, in a lot of other crisis situations. And at first, people will say, yeah, but we will always have a human in the loop. But gradually, of course, the problem is the human is going to be too slow.
00:53:04
Speaker
And this is, again, where you see the creeping automation. We see that in drone warfare. So right now, drones are mostly being flown by people actually maneuvering them directly. But of course, there is a lot of controllers you can just run for ah autopilot of flying.
00:53:20
Speaker
And the most the militaries would say, yeah, and we're definitely not going to want to kill orders to be done by the drone. But you have AI systems that might be finding the right angles and kind of requesting, it can you can I shoot now? Can I shoot now?
00:53:35
Speaker
And more and more of a poor drone pilot's job is basically being approving things and taking the blame if something goes wrong. And i ah certainly heard military people say, yeah, and of course, we don't want to to remove people from the loop. But if our adversaries do that, we are totally going to have to do that.
00:53:54
Speaker
And even the perception that maybe they're doing it means that now you're developing a system where you can take people out of the loop. And while the drones and warfare side is really sinister and scary part, I think this goes in a lot of other domains too. If there is a sudden stock market fluctuation, we already got switches that actually stops things if the stock market moves too fast.
00:54:18
Speaker
That's probably a fairly sensible reaction, but you can imagine economic policies that also happen. If suddenly in the middle of a night the the interest rate does something crazy, maybe you don't have ah shouldn't wait turn until you've woken up the head of the central bank, but maybe you want some piece of software dealing with that and then waking up the head of the bank.
00:54:41
Speaker
You raised a quite interesting ah conundrum earlier, which is just what effects will superintelligence have on on the change in institutions? So is it the case that if we get superintelligence, we will lock in existing power structures, and existing institutions,
00:55:02
Speaker
the companies and governments that are currently in the lead will continue to to be in the lead for the foreseeable future? Or is it the case that introducing a technology like superintelligence actually ah changes the pace at at which you see new institutions and and new power structures?
00:55:20
Speaker
What do what do you think is the is the answer there? So I don't know. I think certainly existing institutions are going to do their best to remain stable using AI. And there might be economies of scale that makes it very easy for them to do it.
00:55:34
Speaker
ah Military forces have great organizations and the enormous resources, and they are going to try to maintain their structure. and But at the same time, the underpinnings in society of how society works might also change radically.
00:55:50
Speaker
and We have seen how politics have changed the last few decades because of introduction of the Internet and social media. And that change was not something that politicians were asking for.
00:56:01
Speaker
It's not like yeah any institution has said, okay great. Social media is going to be good for us to maintain our grip. Of course, currently, you can imagine current politicians and the influencers saying, yeah, social media are great. We understand them. We want them to remain the useful.
00:56:19
Speaker
well Let's prevent other media from emerging.
Cultural Narratives and AI Influence
00:56:22
Speaker
The really interesting question is, is the world big enough and has enough openness that new things can emerge that unsettles existing systems?
00:56:32
Speaker
but What happened during the Industrial Revolution was that something out of left field and the improvement in economic productivity really upset the political institutions. At the start of Industrial Revolution, Europe was ah ruled by various kings and queens.
00:56:47
Speaker
At the end of it, it was mostly parliamentary democracies. the The kings and queens have lost power and but even the roles in society have been transformed. The kings and queens, if they had the a crystal ball, would have probably been very much against the industrial revolution, even though it was not obviously directed against them.
00:57:07
Speaker
The current kind of social media revolution, again, had had there been a crystal ball, I think the 1970s institutions and political parties would have said that no way, we definitely should not have that horrible internet thing.
00:57:20
Speaker
But at the same time, I think the world is actually better off with it, even though we love of blaming social media for all the world's ills, because it gets us off the hook. It's not. it's kind of ah It's Facebook's fault instead ah of our own credulity that we believe in conspiracy theories.
00:57:37
Speaker
it's kind of yeah It's a non-starter. But the problem might be that you could get surveillance systems, you could get ways of locking in things way more powerfully. Indeed, currently we have many platforms that control cultural productions very strongly.
00:57:53
Speaker
but And AI is in some sense also doing it. If try to use ah an LLM to write an novel, that works really well until I get to the sex scene. Suddenly, I can't get any help.
00:58:06
Speaker
And after that, of course, the LLMs I can get access to, at least the online, are it going to be saying, no, that's too steamy for me. I'm not going to help you others.
00:58:18
Speaker
Of course, I could run a local one on my local computer, but that's going to be slower and less effective, or I might actually leave out that sex scene. So in some sense, we're already getting this very interesting, soft entrenchment of certain things.
00:58:32
Speaker
Not because we're living in a society that's really against violence and sex. It's rather that corporate America is very afraid of getting sued and getting bad reviews by allowing that. So we're ending up build in restrictions here. and We might end up building in some restrictions that we later find that, okay, this was way too restricted, but now we have no way of getting out of it.
00:58:53
Speaker
In a world where you can't talk about sex because the smart software that is running everything is just gently censoring it or leading away the conversation from the naughty topics, suddenly it becomes very hard to change that restriction.
00:59:07
Speaker
that There are other ways in which this is happening. if you If you use large language models to brainstorm or draft ideas, you will be subtly influenced by the values that are incorporated in the models you're using.
00:59:20
Speaker
If you use it as your starting point, you will be pushed in ah in a certain direction. I think it's it's different when you use it to critique your own ideas and so on. But it's it's pretty important, I think, to be aware of the values that that you're being influenced by when you when you use these models. And that is...
00:59:37
Speaker
That is indeed something that is that is, in some sense, a power structure that is that is encroached in the world at this point. Just because you want to use the best models and say the best models are from open AI, you are then adopting the values of open AI in some way.
00:59:59
Speaker
And the the problem is, of course, OpenAI, if you ask them, why did you put in these values? They would probably hum and haw and say something about corporate liability and that they want neutrality.
01:00:11
Speaker
They have a set of values kind of given by kind of typical West Coast American sensibility. But why are they so against having sex? Well, as a European, would say, yeah, but that's because Americans are actually deep down puritanical about it.
01:00:28
Speaker
and And the corporate America in particular, because of various aspects of how American laws work and lawsuits work, but they are very afraid of getting into any nautilism.
01:00:39
Speaker
but While violence is why more way more taboo in Europe. If we had the the the AI companies mainly being based in Europe, you might see them be much more open about talking about sensuality and sex, but even more restrictive about violent stuff.
01:00:55
Speaker
And of course, any society does that. Back in the Victorian era, there were certain topics you couldn't write about. That was overtly or but quite often more subtly and privately censored. You simply didn't do that.
01:01:08
Speaker
The problem now is that we're putting in more more culture in autonomous systems. And the really scary part is, of course, these systems also, to some extent, distill this. So in a few years, if there are very few texts on the internet ever daring mentioning naughty stuff, that means that the new training data for ai it means that text is even less likely to contain references to that.
01:01:33
Speaker
There might be ways around this. ah Certainly when it comes to making naughty pictures, people have been extremely creative and running their own stable diffusion models to generate endless dirty pictures. And it might very well be that we see that because of scaling, it might be that ah smaller actors actually can make AI that can compete.
01:01:53
Speaker
It's not given that it's going to remain that the frontier models are the total dominant ones we are all going to use, but that is one possibility.
AI Alignment, Risks, and Disaster Scenarios
01:02:02
Speaker
That has some advantage in terms of safety because it might be easier to keep a few models safe so people don't make bioweapons and doomsday bombs too easily.
01:02:12
Speaker
But on the other hand, it might be better for ah and then the creativity and democratic process if people can make their own open source models. But then we might need other ways of handling that people are coming up with better doomsday weapons ah in the comfort of their own homes.
01:02:27
Speaker
Are you more optimistic and or or less optimistic now about AI risk than you were before we saw the current dominant paradigm of large language models? So should we should we become more optimistic when we see that large language models are the the paradigm that's most driving AI progress?
01:02:48
Speaker
I have this mixed feeling. On one hand, LLMs demonstrated that it seems to be surprisingly easy to get a a fair bit of alignment just out of human production of text and ideas.
01:03:01
Speaker
ah When we started thinking about ai safety back yeah in the late ninety s Our models were very much more based on logic programming and reinforcement learning. And it seemed very much that, okay, but AI might go off the rails and it might be extremely hard to even get it to understand that there is something called humans in the world. Okay, LLMs actually get that because they're already in some sense embedded in our social world to a very large degree, maybe even a too large degree.
01:03:29
Speaker
It's not given, of course, just because a language model happily tells ah us that it's friendly and it wants to follow the law and be ethical, but it actually would do that if it's doing an actual action in the world.
01:03:44
Speaker
But it certainly seems very easy to grip on. The problem, however, is that when we started thinking about AI safety at FHI, we were a bit worried about what if we end up with neuromorphic a built on big neural network, maybe based on scanned brains that are very hard to understand, very opaque and hard to align with human values.
01:04:05
Speaker
So for a long while, many of us felt like this is a reason to maybe stay away from scanning brains and doing brain emulation, which is one of my kind of pet ideas that it would be a great thing to do.
01:04:16
Speaker
After all, it's a good way for us, currently biologically humans, to become post-human eventually, assuming a long laundry list of philosophical and scientific assumptions. But we ended up in that world ah of neuromorphic AI, even though LLMs are not based so much on biological simulations as just enormous neural network.
01:04:37
Speaker
We have opaque systems where Now we're starting to figure out some ways of figuring out what's going on. Mechanical interpretability is an exciting field. We're learning so much interesting things about what's going on inside the systems.
01:04:50
Speaker
And on one hand, okay there are opportunities here for alignment. On the other hand, it's a very hard problem. But at the same time, I think yeah this mixture of systems might actually allow us to get some grips on it.
01:05:05
Speaker
I still think that most of the real risk of stuff going badly wrong comes from multipolar scenarios where you have fairly aligned ai but it's very useful and not terribly dangerous.
01:05:16
Speaker
individually, but you have a society where you have a lot of it, you might find that the society moves away in some crazy direction ah because we humans are formally in charge, but in practice, it's the software that is actually running the show.
01:05:31
Speaker
but How would that look like? Could you give an example there? or What are you imagining when you when you say that? So imagine my advisor example. So you have AI advisors in companies that are giving really good advice.
01:05:45
Speaker
And it's so good that that ah one pink piece of very good advice is fire and the the the the managers who don't listen to the advice. And the companies that do that do much better.
01:05:56
Speaker
And of course, everybody is having their own AI advisors that are really useful. And we use this for more and more tasks. So now a lot of stuff is going around each person.
01:06:07
Speaker
that Everybody has a little swarm of AIs helping them and doing things better. And you get emergent properties from these interactions. So my and calendar AI talks to your calendar AI and they realize that we should totally ah schedule a podcast. This would work really well.
01:06:23
Speaker
Let's suggest to our humans that they ought to meet. And once they say something positive, we're going to immediately find a good part of Callit. Gradually, things get done. So more and more of the agency resides on the AI side.
01:06:36
Speaker
And at first, this looks very good. but And each of the AIs are aligned. They are doing nice stuff for us. Of course, we might have be on no opposing sides, but but again, you might have ah arbitration AIs helping us.
01:06:48
Speaker
The problem is the collective system here. That actually acts as a big optimizer for something. That something was not set by humans. It's just an emergent property. and It's a little bit like how a market that optimizes for certain things.
01:07:02
Speaker
But as we know, markets can also have bubbles. Markets can have crashes. Markets can start optimizing for things that nobody in the market actually wants. And this happens on electronic timescales by systems that are much smarter than us and might also have various forms of agents we can't even understand.
01:07:20
Speaker
So we might notice that the world is getting weirder and weirder. At first, everything is nice. And when we ask for what's going on, we get good explanations for AI. It's just that it keeps on getting weirder.
01:07:31
Speaker
And gradually we realize, wait a minute, why is the world getting optimized the the for that? Why is the nickel price going off so much that AI companies are now colonizing asteroid belt to get more nickel?
01:07:44
Speaker
And nobody can really explain it. And eventually we end up in a world that is totally optimized for things that no human selected and is not valuable to any human or AI for that matter.
01:07:54
Speaker
ah Basically, you have this interaction matrix between things that has been randomly set. That one is not aligned, even though all the parts are aligned. This is a very different disaster scenario from the one where you get one super intelligence that has been told to make paper clips or optimize stock market value and take over the world and optimizing for that.
01:08:14
Speaker
Here we have an emergent system, which might not be intelligent or conscious or anything. And we might say, oh, this is horrible. We need to stop this. So we try to take actions, but it might be very hard to coordinate actions against the entire system from inside. ah So that is kind of my favorite disaster scenario.
01:08:32
Speaker
This is a kind of gradual disempowerment or... loss of loss of control of of the future that happens in a subtle way.
01:08:42
Speaker
But I'm just wondering if you go back to 1800 or 1900, and you you query the the people there about the world of today, might they say, well, things have gone totally off the rails, we are now optimizing for for things that we did not intend to optimize for the world is weird, the world is moving too fast and so on. is Is it is it the case that we can handle more change even in the things we're optimizing for than we might imagine?
01:09:12
Speaker
I think that is very true. but I'm a transhumanist. I think that that we should be upgrading ourselves as beings. I want to have a bigger brain. I want to be able to think deeper thoughts. I want to stop the aging.
01:09:24
Speaker
And many people say, wait a minute, that's weird, Anders. You're kind of a weirdo. And the future you're very enthusiastic about sounds scary to me. I don't want to live in that future of genetically upgraded the people.
01:09:37
Speaker
And we might have a disagreement about it, but it's a very human disagreement. but The people from before the Industrial Revolution, they might say, you're living in a really crazy, weird world, but the weirdness is about human stuff. We might say that many of the institutions we have and the values we have are really good. And then we might have a profound disagreement about, let's say, gender equality and gay rights and the the pre-industrial people would say, oh, that's horribly immoral.
01:10:05
Speaker
And we say, no, we actually arrived at that through a long intellectual and cultural discourse and this is totally valid. The problem with that AI off the rails future I'm describing is that that might not come about because of any sensible discourse.
01:10:22
Speaker
or at least not a human discourse. It's kind of derived from interactions between software, which itself might not even be conscious. It's and built originally by humans, but then of course, generation after generation upgraded by software.
01:10:36
Speaker
It has very little to do with what we would regard as valid and authentic human reasons. Now, it might be, of course, that you could say ah maybe it's still a successful sim civilization. There are some people like Jürgen Schmidhuber who just thinks that we should let the AI loose across the universe and have the best utility functions compete with each other. and then it's going to be all glorious and beautiful.
01:10:58
Speaker
I'm not so convinced about that. I wish I was as optimistic as he is that we get this natural convergence on the best possible future. I don't think that is natural. I think you actually need to work on it.
Dialogue with Superintelligence and Predictive Control
01:11:11
Speaker
And that means that you actually want to weave our preferences and our discourses into this system in the right way. You want the AI to be aligned, not just individually with us, but also that we gradually have an AI aligning itself with our civilization and that the civilization comes together in the right way.
01:11:29
Speaker
Ideally, we should become a kind of cyborg civilization where we both have super intelligence guiding and coordinating us. But we humans are also providing important input in setting the goals and values for this without necessarily that being just one way.
01:11:44
Speaker
Because I think we could take a lot of ethical advice from smarter entities, but we might also want to have a debate with them about it and actually shared understandings. The problem might happen that we are set up systems without getting objectives right, without getting the feedback loops right.
01:12:01
Speaker
And this is, of course, tremendously hard because already our normal human systems are mysterious and as they are. ah The way our politics is going wrong all the over the place is kind of a good demonstration that even when it's people doing autonomous stuff and talking to each other, it can already go wrong.
01:12:17
Speaker
So we are in trouble, but I think the trouble is more subtle than the paperclip maximizer. ah That one is still a physically possible risk. And I think we should be concerned about two powerful smart systems.
01:12:30
Speaker
But I think the real threat comes from this kind of coordination failure. Mm-hmm. so So say it's 2040, and i am I'm stressed out about the pace of change, and the world is is seems weird to me. And i ask a super intelligent researcher, say, to explain why the world is optimised, or is optimising in the direction that it's optimising.
01:12:53
Speaker
That super intelligent AI might be able to give me a fantastic answer, a very convincing answer. and An answer so so good that it's difficult for me to differentiate between whether I'm been giving the real explanation or whether it's optimizing for simply convincing me.
01:13:10
Speaker
If we want to dialogue with future super intelligent AIs, how are we how how are we keeping keeping up with them ah where we where we kind of maintain yeah maintain a dialogue about what we would want the world to look like?
01:13:29
Speaker
and So that that gets to that interesting question about what kind of explanations can we get from AI? So right now, looking at the chain of thought in the LLM that is solving a problem looks pretty illuminating, but I always have my doubts that that is the actual process going on behind the scenes.
01:13:48
Speaker
But then again, that also goes for talking to people. I'm married to a prosecutor ah who previously been a judge. And of course, in the legal world, you can't just decide things. You actually need to give reasons. so They need to be laid out in a clear manner, ah etc And that formal layout is in some sense the output.
01:14:10
Speaker
In practice, of course, a judge makes a judgment quite often based on a lot of more things that are never listed on that piece of paper. But that you might still hold them to, look, you gave that reason. And if that is not valid, then the judgment is not valid.
01:14:24
Speaker
And similarly, you can actually look at an explanation. You can ask things about it. You can look into it. So the Royal Society and ah here in England, their motto is nulla in verba.
01:14:35
Speaker
Don't take our word for it. You need to do experiments. And I think that is also the way maybe a superintelligence and me might have a dialogue in order to test out things.
01:14:46
Speaker
So it might say, look, the reason the world is crazy in this way is is this particular economic theorem. And then I might ah need the help walking through that theorem and seeing that it's ah true.
01:14:58
Speaker
I might also say, I'm going to run that through that proof checker. I can't really actually understand economics, but I can check that the mathematics of that proof at least works out.
01:15:08
Speaker
I might do different things to test it. And I think this is actually what is going on when we have a real authentic dialogue in society. That's rarely about one knockdown argument.
01:15:22
Speaker
What is the argument for gender equality? It's not a single one. It's a bookshelves of arguments. Some which are great, some of which are novels, some which are very formal, some which are crappy, but still compelling. Some of which own are jokes or ah songs...
01:15:39
Speaker
There are very many forms. And you can kind of use the multiplicity also to constrain things. Now, one risk is, of course, with enough superintelligence, you can get that bookshelf of fake arguments.
01:15:52
Speaker
ah Maybe superintelligence find it very easy to just blather on and make things that are not true. But I think generally the constraints of reality make it harder to actually just come up with the things, at least about material facts about the world.
01:16:09
Speaker
It might be trickier when we get into the cultural or philosophical realm. It might be easier to make up stories about emotions and what's right and wrong than it might be about talking about atoms in the physical world.
01:16:23
Speaker
But I think many of important social things are still linked to observable things. And I think that is the way of actually having authentic, testable discussions with entities that are smarter than us.
01:16:35
Speaker
And similarly to parents talking to kids, a good and ah the discussion there typically consists of parents showing the kids how things work. And I think that is actually an important thing we should ask AI.
01:16:49
Speaker
Show me. Show me this. Let me test this. Okay, you showed me a video of this. Okay, I'm running it through this other AI that doesn't know what the question was ah to check whether it's fake. Yeah, and and maybe that's the path forward.
01:17:02
Speaker
this is This is a tactic that that seems to work with human experts, where you are listening to different experts on a topic where you know less than they do, and you are trying to synthesize what they agree on.
01:17:15
Speaker
You're trying to so to map out the the uncertainties and the disagreements. So if we could be in dialogue with multiple different instances of superintelligent AIs that have...
01:17:28
Speaker
specific that have slightly different values or different starting points, and then perhaps see where there's some convergence where and and from that decide what we should believe or what we should do.
01:17:42
Speaker
Yeah. And one problem we have often when talking about superintelligence is that we kind of reify it. We imagine it like some being and sitting in the cloud, but you might actually have specialized systems that are in some sense superintelligent in more narrow domain.
01:17:58
Speaker
So my friend Eric Drexler ah has been arguing that actually building this agent-like big superintelligence, that's a very dangerous and stupid idea. And actually, we don't really want that. We want an ecosystem where we have modules.
01:18:11
Speaker
So we might want to have a planner module that comes up with good plans for a problem and then an evaluator module that reads plans. And but the only thing it cares about is, is this a good plan or not?
01:18:22
Speaker
And then a third one that takes the winning plan and implements it really well. Now, these three systems themselves or don't have any agency. They're unlikely to go off and invent ah bad things ah because it a fits their evil scheme.
01:18:39
Speaker
There's still risks with even this kind of system, so it's not a perfect solution. And I think a world with superintendents might actually have but systems that allow us to get super explanations. And then we might also have a super checking of explanations and other tools.
01:18:56
Speaker
We might want to think about what cognitive tools would we need in order to live in this world. For example, in a world that is changing very radically and fast, obviously you want to have some kind of guide that to help you.
01:19:09
Speaker
or probably several guides. You might actually want a bunch of the guides giving you slightly different forms of advice. You want your maybe your life life coach and your job coach and your ah intellectual coach.
01:19:21
Speaker
And we maybe even sometimes have them debate each other and and see, okay, what comes out of that? But we might want to invent these things now before they even can exist. So when they can exist, we get them as early as possible.
01:19:35
Speaker
do Do you think a world with super intelligence is a world where we are better at prediction and therefore the world is is more controllable in a sense? So over the past 500 years, say, we have made a lot of scientific progress. We have more knowledge. We understand the world better. But we are still quite bad at making predictions about the evolution of social systems, so what will happen in the economy, what's going to happen between countries, what is the world going to look like in 2050.
01:20:05
Speaker
Do you think superintelligence is enough for us to predict the future much better? Or do you think that that these... predictions about social systems are intractable such that superintelligence wouldn't be able to make inroads here.
01:20:23
Speaker
I think it's going to be a bit of mixture. So in my grand futures book, yeah I talk about extremely long-term futures. And one important question is, can we even make predictions?
01:20:34
Speaker
And the answer is, yes, in astrophysics, this is great. We can make very good predictions about the orbits of the planets. Assuming that we don't start changing the orbits of the planets, that's going to remain tractable.
01:20:47
Speaker
But I mean, what i mean isn't isn't maybe that's the entire point here, because if we imagine one of these grand futures, we will... we will begin to affect the the physical universe to ah to a larger and larger extent. And so in some sense, much ah much an increasing share of the universe becomes something that we are influencing and we are a part of a social system, right? So a world with super consultants also pushes in the direction of more on pri like unpredictability.
01:21:20
Speaker
Exactly. but So the interesting part here is that the astrophysics predictions, as long as nobody interferes, are very reliable because you don't get any actors. You have chaotic systems like climate or the atmosphere of the sun, and they are harder to predict.
01:21:40
Speaker
The weather is harder to predict than climate because in climate we're more interested in the kind of average long term. But when you get over to the human side, fashions and the stock market, they're anti-predictable.
01:21:51
Speaker
They or deliberately try to evade prediction. If I knew what the would be cool to wear next year, that would not be the optimal fashion. It needs to be a bit of a surprise.
01:22:02
Speaker
If there was a pattern in the stock market, I could make money by exploiting that, and that makes the pattern disappear. So what happens is basically on the human level, on the cultural level, we get these fundamentally unpredictable events.
01:22:15
Speaker
Now, you can sometimes go too far in claiming this is unpredictable. Karl Popper famously argued that macro history was totally bunk. He was mostly having a broadside here against socialism and fascism.
01:22:28
Speaker
But he he accidentally also kind of attacked the macro history kind of sunk that ship in academia for several decades. And his main argument was basically history is driven very much by ideas, but you can't know about an idea before you have it.
01:22:45
Speaker
So hence, it's logically impossible to predict these things. The interesting thing, however, is in his book about this, The Poverty of Historicism, he also talks about social engineering.
01:22:56
Speaker
He points out that actually, if you want to have a society that works in a different way in the future, you can make it. You can actually do engineering. It's possible probably to control society.
01:23:07
Speaker
He didn't think that was desirable, and he was very much kind of ignoring the it for for the rest of the text. But I think a world with superintensis might very well design things to be predictable.
01:23:19
Speaker
If you think about how our society works today, it's enormously complex, but many of the institutions we have are all about ensuring predictability. There is a reason why quality control and quality assessment is such a big business. and We are very upset when the internet is down a little bit.
01:23:38
Speaker
Even though in but the past, communications were much more rickety, but now we have created very reliable systems and we demand even more reliability. So we create reliability in many dimensions.
01:23:51
Speaker
And ideally, we do this in dimensions that matter, that are actually very good, that they're reliable. We want kind of safety and prosperity that reliably gets delivered. We don't want to get nasty surprises in those domains.
01:24:03
Speaker
Then we might say, okay, let the culture ah bloom. Let's that That might change in all sorts of different ways. We also want an open-ended future. And that's the tricky part.
01:24:14
Speaker
Because if we could perhaps lock in values, and that's a very scary prospect, because if and some past generation had locked in values, we might say, given our current values that would never have come into being, of course, say, that's horrible. If the Romans had decided what the life we would be living, that would be a horrible world, so because compassion to them was not a particularly important value.
01:24:37
Speaker
But in our culture, compassion is a really important one. We hold it very highly. So locking in value seems to be bad, except that there are some values we actually have good reasons to say that maybe we should never have a future civilization that is cruel.
01:24:52
Speaker
We should perhaps permanently close that door, at least shut it, lock it and make it rather hard to open that. We don't want to have the doors open to existential threats. So maybe let's close them and lock them very carefully.
01:25:06
Speaker
But then you probably want to have an open-ended future because we actually don't know the fundamentals of the universe. Now, it might be that in 2040, the superintendent says, actually, we solved it. We figured out the meaning of life and universe and everything.
01:25:20
Speaker
It's in this PDF file, if you're interested in reading it. It's 100,000 pages thick, but there is an executive summary at the start, and we should just implement that. At that point, maybe we should go with that, but I have my doubts that we're going to end up in that future.
01:25:37
Speaker
It's very possible that there is no clear answer, that it's actually just very different options, and then we might want to hedge our bets and have a diversity of options open.
AI Reliability, Predictability, and System Design
01:25:48
Speaker
But I do think the predictability of the future is a fascinating thing because it's becoming more and more of a matter of design.
01:25:54
Speaker
We could lock ourselves into a kind of eternal dystopia with surveillance and the AI, ensuring that we live our lives in a particular way forever. That sounds like something worth fighting against tooth and nail, given our current value and the values. And I think they know and the that is worth and they taking seriously because superintendents might lock in things.
01:26:15
Speaker
But I do think that with using superintendents, we can also get those useful locks on the bad doors, but keeping many other things open. Yeah. Do you think predictability is a good frame for how to think about AI safety and the system we are in the systems we are interacting with?
01:26:33
Speaker
So for example, I want my AI assistant to be predictable. i want to when i When I give it a task, I want it i want that task fulfilled. but I want basically all the technological systems I'm interacting with, I want to be predictable. And if they are sufficiently distributed in society and kind of segmented from each other.
01:26:56
Speaker
I'm not super worried about the future of society as a whole becoming predictable in a bad way. But I want predictability for specialized systems I'm interacting with.
01:27:09
Speaker
Do you think that frame is perhaps useful, is ah is a useful way to think about safety? I think it might be useful, but I think reliability might be an even better one.
01:27:21
Speaker
You want a reliable system. And that doesn't mean that it always acts in the same way. Sketch out the difference there for us, actually. What's the difference between predictability and reliability?
01:27:33
Speaker
So in predictability, you kind of know what it will do in a future situation. In a reliable system, you know that you can trust what it's going to do in a future situation, but you might not know exactly what it's doing.
01:27:47
Speaker
So that's a bit like having parent. You can't always predict as a child what your parent will do for you. But if they're nice parents, you know that they're going to try to do something good for you.
01:28:00
Speaker
Of course, so in practice, this is more complicated because parents are not perfect and the reliability and predictability are never absolute things. Generally, I found that when dealing with LLMs, they're very nice and useful for things where I don't quite know what result I want.
01:28:17
Speaker
But right now, when I know exactly what I want, I'm usually quite frustrated. They're quite often not doing exactly what they should. And that actually limits the usability. Yeah, I very much agree. It's super difficult to steer an LM towards exactly the the type of ah goal you want to achieve, I think.
01:28:36
Speaker
But if you have a a more open ended goal, it's it's you're often surprised in in in nice ways by what they can do. And the open-ended goals are also easier to steer, partially because we're not trying to steer as hard.
01:28:50
Speaker
If I try to solve a particular mathematical problem and I find that the solution is going off the rails, it's ah very hard for me to get LLM back on track. I might try ah rerunning it a number of times, but quite often I end up being frustrated and solving it myself or realizing that maybe I should have split it into smaller subtasks, but each of them are very doable for the LLM.
01:29:13
Speaker
But I think future AI is going to be better at this. The really tricky part is it needs to be reliable enough that you can leave a task to it. Right now, I would not book any and airline tickets using AI.
01:29:25
Speaker
I want to get to my destination with a very high probability. So the probability that I get there is much better if I do it myself or if I have a friend do it for me. But that's going to change in a few withinner months or years. I think that's very likely.
01:29:41
Speaker
The tricky part here is a predictable AI system. You already know what the result is going to be in some sense. That means that you actually have a bit of a limited range. And that's a little bit like Karl Popper's critique of new ideas.
01:29:55
Speaker
You can't really get a new idea out of a system if it's too predictable. You want a reliable system. If I'm brainstorming with it, it might come up with various ideas, including good ideas, but I don't know what they are going to be.
01:30:09
Speaker
So when thinking about society, a predictable society is quite often very bad when something that is outside predictions finally happen. While a reliable society, okay, now we can start improvising.
01:30:23
Speaker
That shock to the system, okay, we need to improvise and we need to do it quickly. But if it's too based on everything being predictable, it's brittle. Could you give an example of some system to today that's engineered to be reliable, perhaps in the realm of AI?
01:30:41
Speaker
Well, I think the best example of a reliable system is actually the internet. So back in the 1970s and 80s, there was an entire genre of emergency posts so on mailing lists about, oh no, we found a problem.
01:30:56
Speaker
The internet is going to be unusable by September. Basically, they have discovered some technical problem and people immediately started working on finding a way around it. the The kind of standard story that it was developed from ARPANET to be kind of resistant to nuclear war is a bit exaggerated. But the interesting thing is the Internet is a heterogeneous system where a lot of different systems have to work together.
01:31:21
Speaker
And there is no assumption that all the other servers work as they should. In fact, some of them are malicious. So you have to take that into account. This produces a fairly robust system.
01:31:32
Speaker
We still do get ah trouble when the some big cloud the provider makes the wrong update. Oh, dear. Chaos issues. And then everybody says, OK, now we're going to sue each other for that. And we're going to change the software so that problem doesn't happen.
01:31:48
Speaker
Not everybody changes the sha of software. It's still fairly heterogeneous. And that is adding a lot of reliability. Similarly, when we think about AI, if I want to solve a problem, I can, for example, run several instances of AI and take their outputs and then try to see which is the best one.
01:32:07
Speaker
That is likely to find a good one if i have a good evaluation metric. In some cases, of course, I might use another AI even to try to make a guess at which one looks and the best one, but it's still not a super reliable system.
01:32:21
Speaker
Many of the big advances more recently on solving hard math Olympiad problems consist of actually running several parallel instances in the parallel and then selecting the best results.
01:32:34
Speaker
That is a surprisingly good idea, and it's a bit similar to how our brains are sometimes working in parallel on the problem. Yeah, we we might worry that if we're pushing too hard in the direction of reliability, we also thereby make our AIs more agentic. And if we're worried about agents and worry about which actions they might take.
01:32:56
Speaker
It might be against our interest to make to make very reliable systems, just because it seems like today, ai agents are bottlenecked by them being too unreliable to carry out tasks that consists of many distinct steps.
01:33:11
Speaker
So is there a trade off here? Or am I thinking about reliability in a wrong way? No, I think you're right. But there is this interesting problem that if AI always remains unreliable, it's going to be a sideshow. It's going to be able to do some things in some domains, but it's not going to transform the world.
01:33:29
Speaker
In order to become transformative, it needs to be reliable enough to be able to do the tasks that transform the world. So this is, of course, where you might actually say that maybe we don't want reliability to increase too rapidly.
01:33:43
Speaker
But at the same time, in order to get alignment and safety, you want reliability. You actually want it to reliably be safe. That is something you need for. If it's unreliable about safety, we have not achieved anything.
01:33:57
Speaker
So typically the length of a task you can do and it also increases quite strongly as reliability increases. If you have a constant risk per unit of time of going off the rails, the length of a task is, of course, going to be set by that probability. And as that probability goes down, you can do longer and longer tasks.
01:34:17
Speaker
Of course, there it might even be a kind of threshold here. or So there is a whole family ah theorems in computer science, but started with John von Neumann when he wanted to build a sequel to ENIAC.
01:34:29
Speaker
And people told him, you can't build a bigger computer. Look, those radio valves you use, they're breaking all the time. If you build a bigger computer, it's going to be broken all the time. It's not going to work.
01:34:39
Speaker
And von Neumann that proved a simple theorem, but, well, if I replace one logic gate with three logic gates and then have a little majority thing, then it doesn't matter if one of them breaks.
01:34:52
Speaker
And that metagate actually has a lower and risk of failing than the original was. So then you could actually keep on recursively building up this abstract machine and make it larger, and you could get any the level of effectiveness.
01:35:08
Speaker
You could actually make the probability of failure very low. And this is similar to channel coding, theorems and stuff in quantum comp computing. If you're below a certain error threshold, you can combine error-prone processes in such a way that you get a new process that has a much lower error rate.
01:35:24
Speaker
And that way you can push it down arbitrarily far. I believe something like this might happen with AI. There is a kind of transition in reliability, but once it's reliable enough,
01:35:35
Speaker
You could make this redundant system, which is kind of inefficient, but you could just make the reliability go up enormously. And at that point, then then the improvement in reliability just makes it smaller and more effective.
01:35:48
Speaker
And that threshold, I don't know where it is. I don't think anybody knows where it is. But I think we might cross it in the next few years, which is both exhilarating and very scary.
01:35:59
Speaker
And what makes you say that we might cross it in the in the next few years? Well, people are working very hard on the reliability aspect here, because obviously that is limiting both agency, but also how well you can solve complex problems.
01:36:14
Speaker
So especially in programming tasks, which is of course what the big AI companies are really interested in because they want to automate programming. If a long programming task is too long, and then it's just going to make bad code.
01:36:27
Speaker
And even splitting a programming task into smaller pieces reliably is also hard. If you can reliably do that splitting of tasks and implement it, now you can do coding.
01:36:38
Speaker
Suddenly they don't need that many programmers anymore and they can start improving their software by having software improve on it and everything is going to be great. Before that threshold, instead, you you just get worse and worse software and and it's totally incomprehensible.
01:36:54
Speaker
So that's kind of useless to them. So they are going to be pushing for this. And similarly, I think customers are also going to want to have systems that are reliable and less likely to hallucinate or do something else that they don't like.
01:37:07
Speaker
So there is a strong push in this direction. The question is, of course, is this easy or hard? So many of the critics of AI say, oh, you can never avoid hallucinations. So I have a paper proving that or and that yeah it's always going to be missing things because of this. But generally, I don't think these predictions have held up very well.
01:37:27
Speaker
Quite often they're based on older versions of and the AI systems. Quite a lot of them are very motivated thinking. People are wishing that there is something that is impossible to do.
01:37:37
Speaker
There is a kind of, besides the hype bubble, there's a cope bubble where people are and grasping at straws for why AI is just normal stuff and it's not going to mess up the world in weird ways and especially not to threaten my
Learning from Past Predictions and Future Expectations
01:37:50
Speaker
job. but But I think what is going to happen is, of course, as reliability gets better, both AI gets applied to more areas, which is going to be a good source of money for the AI companies. and you're going to find better ways of ah using AI to control AI.
01:38:08
Speaker
which might actually be good news from an alignment and safety standpoint, because you want better monitoring systems. If you could have a little AI agent watching every single thing and actually raising the alarm when stuff goes off the rails with some reliability, bingo, that might they actually be really useful for safety.
01:38:26
Speaker
as ah As a final topic here, Anders, I want to ask you about your kind of perspective on life, just because you've been engaged in, you've been researching the future and the future of technology and AI and the very long-term future for decades at this point.
01:38:42
Speaker
I'm wondering if you feel like you you perhaps live in two different worlds, because when you then kind of leave your research and go out into the into your everyday life, it it must seem as if, you know, you're living in ah in a very old world in which things function in some sense, like they always have, you still have to do the dishes, the train isn't on time, and and so on.
01:39:05
Speaker
How do you think about this? Do you do you as it began, you know, you said earlier that you have begun feeling the AGI, at at least to some extent. So perhaps, yeah, do do you feel like you're living in in two different worlds? And do you think perhaps the worlds are colliding?
01:39:20
Speaker
I think the worlds are colliding. So ah my kind of formative decade where the late eight nineteen eighty s and the early 1990s, my home computers were getting better. The larger memory, 512 megabytes of ROM, color, wow, internet.
01:39:40
Speaker
And then I joined the Extropian's mailing list and we transhumanists were chatting excitedly about the technological singularity somewhere into the twenty forty s and the life extension and nanotechnology and space and ai But a lot of that still, we expected accelerating change.
01:39:59
Speaker
And we were were really wrong about the speed. So we expect the biotechnology to give us life extension much earlier. As a middle-aged transhumanist, I'm of course very interested in life extension, ah kind of every gray hair is a reminder about why are we so bad at handling this really complex problem?
01:40:18
Speaker
But of course, in the 2020s, life extension is actually getting to be big business. We're actually having companies and startups of that are actually working on things that work well in slowing aging in the lab, in the lab animals.
01:40:31
Speaker
These things are coming. It's just not as fast as I kind of wished to back in the ninety s ah Similarly, we were optimistic about ai but we didn't expect it to have any breakthroughs.
01:40:43
Speaker
We were were, like everybody else, surprised that by the 2010s when suddenly things started working for reasons that to this day remain kind of unclear why the large neural networks have the powers they do.
01:40:55
Speaker
space, we assume that, oh, it's going to arrive once we have nanotechnology, because right now, of course, nothing is happening because of NASA and the Russians. They're big, entrenched bureaucracies. They're not going to be building anything. But once the new materials arrive, and suddenly we get spaceships going off the made out of stainless steel instead. Okay, it was a matter of control technology for making them reusable and the entrepreneurship.
01:41:19
Speaker
Okay. Okay. This shows that being a futurist means that your predictions are not necessarily getting there in the right ordering. What you want to understand is rather the dynamics.
01:41:30
Speaker
And the interesting part is, of course, today I'm living a world that is actually a bit like the future envisioned in vision the 90s. I literally got a virtual reality headset lying around on a desk in this room, way better than the VR headset I was using in the 90s connected to a mainframe computer And this is a consumer product.
01:41:51
Speaker
And I'm honestly not certain what use I should have for it, except computer games, which is also fascinating. and um The life extension is happening. We're getting AI, we're getting space.
01:42:02
Speaker
Nonotechnology in the Drexlerian sense might have been dormant for a long while, but we're getting protein engineering empowered by AI that is getting there. And of course, this is mostly not part of my everyday life.
01:42:16
Speaker
When I go out and they get the bus, I'm not using any advanced technology. Except of course, but actually my ah credit card that I'm swiping is using a near field chip.
01:42:26
Speaker
And the bus actually has ah internet. And actually on the bus, I'm interacting with the rest of the world using a little device. But the the actually was a supercomputer back during my youth and also has a lot of the powers that my wearable computer system that I built in the 1990s and made me look like an extra from a low budget Star Trek ripoff.
01:42:52
Speaker
well, my smartphone is much better than that system. And it's not just because of Moore's law, it's because of better app design, being wirelessly connected, et cetera. So I think in many ways, we are living in the future and we're just not recognizing it because we are so quick at adapting.
01:43:11
Speaker
and And to some degree, I find my job as a middle-aged futurist is telling younger people that actually things were really different just a few decades ago. And part of this is, of course, the old man, grumpy. Oh, back in my day, my computer had one kilobyte of memory and you had to connect it to the television set.
01:43:30
Speaker
And I had to stop programming at five o'clock because then state television in Sweden turned on the radio transmitter and it was too much noise on the screen. That might be the grumpy old man, but it's also a useful thing to recognize how rapidly things change.
01:43:46
Speaker
as well as what didn't work in the 90s and why does it work now? Trying to figure that one out. and Some of these things, maybe they haven't changed and it still doesn't work. and Maybe some fundamentals have changed.
01:43:57
Speaker
We can actually learn from the past. One of the coolest things about a rapidly changing world is that you actually get to transmit the information to the future in the hope of affecting the future.
01:44:09
Speaker
We want to have foresight. And sometimes when you get that by looking at history and realize how weird the past was. Sometimes you need to think entirely new thoughts and just try entirely new things and get surprised by them.
01:44:21
Speaker
Nobody expected the deep neural networks to take off like they did. Nobody in the field. but People were certainly hoping that they might work, but nobody could get their expectations up. Nobody expected Transformers and LLMs to be that powerful.
01:44:35
Speaker
And we got to that surprise. We should expect more surprises. Indeed, I expect the super intelligent systems of the future to get a lot of surprises. Hopefully, mostly welcome surprises.
01:44:47
Speaker
Makes a lot of sense. Andrus, thanks for chatting with me. It's been great. Thank you. This was great.