Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Samuel Hammond on AGI and Institutional Disruption
291 Plays1 year ago
Samuel Hammond joins the podcast to discuss how AGI will transform economies, governments, institutions, and other power structures. You can read Samuel's blog at https://www.secondbest.ca
Timestamps:
00:00 Is AGI close?
06:56 Compute versus data
09:59 Information theory
20:36 Universality of learning
24:53 Hards steps in evolution
30:30 Governments and advanced AI
40:33 How will AI transform the economy?
55:26 How will AI change transaction costs?
1:00:31 Isolated thinking about AI
1:09:43 AI and Leviathan
1:13:01 Informational resolution
1:18:36 Open-source AI
1:21:24 AI will decrease state power
1:33:17 Timeline of a techno-feudalist future
1:40:28 Alignment difficulty and AI scale
1:45:19 Solving robotics
1:54:40 A constrained Leviathan
1:57:41 An Apollo Project for AI safety
2:04:29 Secure "gain-of-function" AI research
2:06:43 Is the market expecting AGI soon?
Recommended
Transcript
Introduction to Podcast and Guest
00:00:00
Speaker
Welcome to the Future of Life Institute podcast. My name is Gus Stocker, and I'm here with Samuel Hammond, who is a senior economist at the Foundation for American Innovation. Samuel, welcome to the podcast. Hi, Gus. Thanks for having me. Fantastic.
AGI Arrival Expectations
00:00:15
Speaker
All right. I have so much I want to talk to you about, but I think a natural place to start here would be with your timelines to AGI. Why is it that you expect AGI to get here before most people?
00:00:29
Speaker
I don't really know what most people think. I think the world divides into people who are paying attention and people who are basically normies.
Diverse AGI Timelines and Definitions
00:00:38
Speaker
In my day job, I work on Capitol Hill and in Washington, D.C. talking to folks about AI. If you think about what people's implicit timelines are, you can read out people's implicit timelines by their behavior.
00:00:51
Speaker
Right. You know, I know Paul Christiano has short timelines because he puts he's doubled up into the stock market. Right. He's sort of practicing what he preaches. But then when you have, you know, Sam Altman testifying to work to Congress, I like to say people are taking him seriously, but not literally. Right. He's saying we're going to develop something like AGI potentially this decade and super intelligence thereafter. And then, you know, you have folks like Sandra Marshall Blackburn being like, what will this mean for music royalties?
00:01:21
Speaker
And when the focus of policymakers is things like music royalties or impact on copyright, it's not that those are invalid issues, it's that they belie relatively longer timelines.
Human-Level Intelligence Benchmark
00:01:35
Speaker
And then we also have this definitional confusion where folks like John LeCun would say AGI is probably decades away because he is using AGI to mean something that learns like a human learns in the sense that it's born as a relative blank slate and sort of
00:01:50
Speaker
can acquire language with very few examples. So people have these moving goalposts of what they mean. For me, I think we can sort of avoid those definitional conflicts if we just talk about human level intelligence. And humans are quite general. We're generally intelligent. That's what separates us from animals in a
Training AI on Human Data
00:02:12
Speaker
lot of respects. And when you look at how machine learning models are being trained today, like large language models and non-multi-modal models,
00:02:20
Speaker
They're being trained on human data and they're being trained to reproduce the kinds of behaviors and tasks and outputs that humans output. And so they're kind of like an indirect way of emulating human intelligence. And so if you benchmark AI progress to that,
00:02:39
Speaker
then you can sort of put information theoretic bounds on what's the likely timeline to basically an ideal human emulator, something that can extract the sort of base representations, the internal representations of our brain through the indirect path of the data that our brain generates.
Architectural Gaps in AI Learning
00:02:59
Speaker
You have an interesting sentence where you write that AI can advance by emulating the generator of human generated data, which is simply the brain. Do you think this paradigm holds all the way to AGI? I think it holds this decade to systems that in principle can in context learn anything humans do.
00:03:18
Speaker
Again, this is a semantic question. Do you want to call that AGI or not? I think there are still outstanding issues around the limits of other aggressive models for autonomy and the question of real-time learning. The way we train these models, we are freezing a crystal in place and humans are continuously learning.
00:03:39
Speaker
So there's still our genuine potential architectural gaps. But from the practical point of view, from the economic point of view, we don't need to debate whether something is conscious or whether something learns strictly the way humans learn if it demonstrably can do the things humans do.
Turing Test and Human Task Competence
00:04:00
Speaker
And that goes to the original insight of the Turing test.
00:04:05
Speaker
It's sometimes presented as a thought experiment, but what Alan Turing was getting at was if you can't distinguish between the human and a computer, in some ways, indistinguishability implies competence. And we can broaden that from just language, because arguably we've surpassed the Turing test, at least a weaker version of it, to human performance on tasks in general. If we have a system that can output a scientific manuscript that experts in the field can't distinguish from a human,
00:04:34
Speaker
then debating whether this is real AGI or not is, I feel, academic.
00:04:43
Speaker
It is surprising in a sense that when you interact with GPT-4, for example, and it can do all kinds of amazing things and organize information, present information to you, or at least at some point it couldn't answer questions about the world after September 2021 or a date
Industry vs. Research in AI Progress
00:05:02
Speaker
like that. That would be surprising if you presented that fact to an AI scientist 20 years ago.
00:05:08
Speaker
How long do you think will remain in this paradigm of training and foundational model and then deploying that model? I mean, it's worse than that. I think it was surprised people five years ago. Progress is sort of moving along two tracks, the industry track and the pure research academic track.
00:05:23
Speaker
And they're obviously having feedback with one another. The pure industry track is just looking to create tools that are, you know, have practical value and can improve products and so forth. And so, you know, Meta has their own GPU cluster and their training models, so they can have fun chatbots in there.
00:05:43
Speaker
their messenger. And so those kinds of things are going to progress, I think, well within the current paradigm because we know the paradigm works, basically deep learning and transformers. And there's lots of marginality on the side, but that basic framework seems to be quite effective. And just scaling that up because we haven't hit the range of irreducible loss and what transformers can do.
Compute and Data Efficiency
00:06:10
Speaker
Meanwhile, there's also this parallel
00:06:12
Speaker
pure research track where people on seemingly a weekly basis are finding better ways of specifying the loss function, ways of improving upon power loss scaling and all these different
00:06:25
Speaker
Sometimes they're new architectures, but often they're just like bags of tricks. And those bags of tricks then, to the extent that they comport with the paradigm industries running with, they can be reintegrated and end up accelerating progress in industry as well. So do you think scale of compute is the main barrier to getting to human level AI?
00:06:50
Speaker
Yes, right. I mean, it's not all we need, but it's the main unlock, right? To what extent can more compute be used to trade off for lower quality data or for lower quality algorithms? Can you just throw more compute and solve the other factors in that equation?
00:07:12
Speaker
It depends on the thing you're trying to solve for. In principle, if we're talking about mapping inputs to outputs, then transformers are known to be universal function approximators. And so the answer is yes. That doesn't mean that they're necessarily efficient at approximating everything we want them to approximate.
00:07:31
Speaker
And sometimes universal function, you know, approximation theorems can be kind of trivial because they'll be like, okay, if your neural network is infinite width, then yes, we can approximate everything. The key fact is both that they're universal approximators and also that they're relatively sample efficient, at least relative to things we found in the past. And so that to me suggests that yes, they can compensate for things that they're bad at. On the other hand, the way research is trending is towards these mixed models.
00:08:00
Speaker
ensembles of different kinds of architectures, things like the recent Q transformer nets from Google DeepMind, which uses a combination of transformers and Q learning to have sort of the associational memory and sample efficiency of transformers with the ability to assign policies to do tasks that you get from reinforcement learning. So I imagine that there's going to be all kinds
AI and Economic Insights
00:08:23
Speaker
of mixing and matching. The key point is that
00:08:28
Speaker
in that space of architectures, it's a relatively finite search space. And as an economist, economists believe that supply is long run elastic. And so there's this famous bet between Paul Ehrlich and Julian Simon
00:08:43
Speaker
vis-a-vis the population bomb and whether population growth would lead to a sort of amathusian purge. And Julian Simon, being the economist, recognized that if prices rise for these core commodities, then that will spur research and development into extracting new resources. So he didn't have to know that fracking would be a technology. He understood that if oil prices went too high, people would find new oil reserves.
00:09:12
Speaker
I have an analogous instinct when it comes to progress in deep learning, meaning you can become too anchored to the current state of the literature, but over a 10-year horizon, you can say, well, there's a huge search on, a huge gold rush to find
00:09:31
Speaker
the right way of blending these architectures. And I don't need to know in advance, which is the right way to do that, to have high confidence that someone will find it. Yeah, we can sometimes if we're too deep in the literature, we might lose the focus on the forest for the trees in
Information Theory and AI Forecasting
00:09:49
Speaker
a sense. And if we zoom out, we can just see that there's more investment, there's more talent pouring into AI. And so we can predict that something is going to come out of that.
00:09:59
Speaker
You have lots of interesting insights about information theory and how this can help us predict AI. What's the most important lessons from information theory? The reason I start there is because within the conceptual realm, it's the most general thing that bounds everything else. And when you look back at the record of, say, Ray Kurzweil, I first read the age of spiritual machines when I was a kid.
00:10:27
Speaker
And, you know, in there, he makes a prediction that we'll have AIs that pass the Turing test by, you know, 2029 or so. And when was this book written? 1999. Yeah, that's pretty good. Right. And so, you know, and he got people complain that he got things wrong, because he said, we'll all be wearing AR glasses by 2019. You know, when in fact, like Google Glass came out
00:10:50
Speaker
in 2013 and now we have metaglasses five years later. So he was wrong on the exact timing but sort of right where the technology was, wrong where the minimal viable product was. But nonetheless, if you look at his track record, it's quite good for a methodology as relatively stupid as just staring at Moore's Law.
Human-Level AI Timelines
00:11:13
Speaker
and extrapolating it out. I think that reveals the power of these information theoretic methodologies to forecasting because they set bounds on what will be possible. The team at Epoch AI have a forecast called the direct approach where you can think of it sort of like a way of putting bounds on when we'll have
00:11:34
Speaker
A is that can emulate human performance through an information theoretic lens where they're looking at sort of how much entropy does the brain sort of process and how much compute will we have over time and what what's implied by a scaling laws you sort of.
00:11:50
Speaker
put those three things together and you can sort of set bounds on when we'll basically be able to brute force human level intelligence. And of course that's an upper bound because we're going to do better than brute force. We're going to also have insights from cognitive science and neuroscience and also, you know, ways of distilling neural networks and so forth and better ways of curating data. So their modal estimate for human level AI is 2029.
00:12:15
Speaker
And their meeting is like 2036. And I talked to the authors and they lean towards the 2029, 2030 for their own personal forecasts. And so going back to, you know, is this an outlier? Am I out on a limb here? I think among our circles, probably not, but among Congress.
00:12:34
Speaker
and among the broader public, I think people are seeing sort of, they think everything's an asymptote, right? They're imagining, okay, we have these chatbots and they're not seeing the
AI's Contextual Learning and Task Replacement
00:12:44
Speaker
next step. I see a very smooth path from here to systems that can basically in context learn any arbitrary human task. And so what does that look like? It looks like systems that can basically sit over your shoulder or can monitor your desktop, your operating system as you work and watch you for an hour or two and then take over.
00:13:03
Speaker
And that'll be key to overcoming lack of training data or why is it important that they can learn in context? Well, in-context learning is sort of the secret source of the power of transformer models. They learn these inductive biases and induction heads and so forth that let them a few-shot learn different tasks. So, you know, GPT-4 is very good at zero-shot learning on a variety of different things, but it's incredibly good at few-shot learning. If you give it a few examples, it can kind of pick up where you left off.
00:13:32
Speaker
When I think about myself, when I want to learn a new recipe, I can go read a recipe book, but often what I prefer to do is to go on YouTube and watch someone make that recipe.
00:13:41
Speaker
And just by watching that person put together the stir fry, I have enough of a world model and enough knowledge of how to cook in general that I can in context learn how to pick up from there and do that recipe myself. LLMs do that already, multimodal models are increasingly doing that. Some of the recent progress in robotics, like I mentioned, the Q-Transformer paper, it shows that you can basically build robots with a basic world model
00:14:09
Speaker
and then have it learn new tasks with fewer than 100 examples of the human demonstration. So the human sort of demonstrates the task and the robot can pick it up and take it from there. And why that's important is both for understanding the trajectory of AI, but also its economic implementation. Because we're sort of used to automation being this thing where you get a contract from IBM
00:14:33
Speaker
and you spend many millions of dollars within consultants and they build you some bespoke thing that doesn't really work very well and requires lots of maintenance. And so people have this prior that AI, even if it's near, will be rate limited by the real world because of all the complexity of implementation. But the point is, if you have things that can in context learn and perform as humans perform,
00:14:57
Speaker
you don't need to change the process. You can take human-designed processes and have the AI just fill in for the human. And so it leads to this paradox where we're probably going to have AGI before we get rid of the last fax machine. When we think of, say, old IT systems in large institutions, we might think of
00:15:20
Speaker
moving from analog storage of information to the cloud. That's still going on in some institutions. That transformation has taken over a decade now. And so what exactly is it that makes AI different
AI Integration in Human Processes
00:15:34
Speaker
here? It is that AI plugs in directly where the human worker would be? Yeah, precisely. You don't need to redesign an existing process to plug into the automation. And that applies both for the structure of tasks,
00:15:50
Speaker
Right? So much of a mechanical automation takes something like the sort of artisanal work of a shoemaker and has to translate it into something repetitive that a machine or an automatic seamstress can do over and over and over again. Our older school kind of automation requires sort of collapsing a task into a lower dimension so that simple automations can handle it. But when you have AGI, the whole point is generality.
00:16:20
Speaker
It's a flexible intelligence that can map to existing kinds of processes. So that's sort of why I think this will catch people by surprise because it's not just the AGI
00:16:32
Speaker
could be this decade, but that when it arrives and sort of crosses some thresholds of reliability, the implementation frictions could be very low. And do you expect, would AI have to get all the way there in order to substitute for a human worker? I mean, I would expect it to be a bit more gradual than that, taking over, say, 20% of tasks before 40% of tasks, 60% of tasks, and so on. But here we're imagining that the AI kind of plugs in for the human worker for all tasks, or what do you have in mind?
00:17:02
Speaker
These things are, yeah, you're right, much more continuous. It's not an on or off switch in part because the requisite threshold of reliability varies by the type of task.
Task Reliability and AI Adoption
00:17:13
Speaker
Arguably, self-driving cars like Waymo or Tesla have matched human performance, but regulators want them to be 100X better than human before they're loose on the road because of safety. Codex and coding models are
00:17:29
Speaker
arguably still much worse today than elite programmers, but everyone is using them because even if it generates 50% of your code and you have to go back in and debug, it's still a huge productivity boost. So I think it will vary by occupation, by task category, modulo the risks and stakes involved in those tasks.
00:17:52
Speaker
Yeah, I guess then the question is how many of our jobs fall into the, it's more like self-driving cars and how many of our jobs is more like programming. Right. I mean, I've been in a manager position before and I've had research assistants and interns, and I know that they're like a very lossy compression of the thing I want to
AI as Lossy Compression
00:18:13
Speaker
do. And so they require oversight and sort of co-piloting. We're sort of in that stage now with AIs in a variety of different times.
00:18:20
Speaker
I recently read a paper evaluating the use of GPT-4 for peer review in science, and it found that GPT-4 would write reviews of work that bore some striking correlations with the
00:18:36
Speaker
points raised by human reviewers, but also let some things out. And so it concluded by saying GPT-4 could be an invaluable tool for scientific review, but it's not about to replace people. And that's just a case of like, okay, give it five years. Yeah, this is a phenomenon you often see with some AI models out there, and it has some capabilities, but lacks other capabilities. And then
00:19:05
Speaker
people might over-anchor on the present capabilities and not foresee the way the development is going.
Exponential Progress and Human Biases
00:19:13
Speaker
I think people are continually surprised at the advancement of AI.
00:19:17
Speaker
Yeah, absolutely. Ramesh Naam, the sci-fi author and futurist and energy investor, he gives this talk on solar energy and other renewables. And he has this famous graph where he shows the International Energy Agency, the IEA, every year they put out this projection of solar build out. And every year it's like a flat line.
00:19:41
Speaker
But it's like a flat line on an exponential like the real curve is like going vertical and every year their projection is that it's just going to plateau. And I feel like people make that same mistake. And it sort of has this sort of ironic lesson to the extent that we're drawing sort of parallels with the way our brain works and the way these models work. It seems like humans have a very strong autoregressive bias.
00:20:03
Speaker
So what's going on there? Is it an institutional problem or is it a psychological problem? Why is it that we can't project correctly in many cases?
Neural Networks and Human Brain Similarities
00:20:15
Speaker
Well, to what I just said, I think it's probably both, but largely psychological, right? Our brains are evolved for hunter-gatherer societies that didn't really
00:20:23
Speaker
change over millennia. And even the last 40, 50 years have been a period of relative stagnation where we have a lot of pseudo innovation. And so I think people are just a bit disabused.
00:20:37
Speaker
Okay, you have some super interesting points about comparing the human brain, how the human brain works to how neural networks learn. What is universality in the context of brain learning and neural network learning? So universality is a term of art, it sort of refers to the fact that different neural networks independently trained, you know, even on different data, will often converge on very similar
00:21:05
Speaker
representations in their embedding space of that data. And you can extend that to striking parallels or isomorphisms between the representations that artificial neural networks learn and that our brain appears to learn. Probably the area of the brain that's been studied the most is the visual cortex. And it seems to me as like a layperson that the broad consensus in neuroscience is that the visual cortex is very similar to a deep convolutional neural network.
00:21:34
Speaker
is basically isomorphic to our artificial deep convolutional neural networks. And you train CCN on image data, and our brain is trained on our sensory data. And it turns out they end up learning strikingly similar representations.
Energy Constraints in Learning Systems
00:21:54
Speaker
And there are a few reasons for that, right? So one is sort of hierarchies of abstraction. It makes sense that early layers in a neural network will learn things like edges and simple shapes. And only later, and only deeper in the network will you learn more subtle features. So there's that sort of sequencing part of it.
00:22:13
Speaker
And then there's also just the energy constraint. Gradient descent isn't costless. It requires energy. It requires a lot of energy. These data centers suck up a lot of energy. The same is true of our brain. Our brain consumes a lot of energy, like 25% of our calories. And especially when we're young, there's a very strong metabolic cost associated with neuroplasticity.
00:22:36
Speaker
Our brain being something shaped by evolution was obviously very energy conscious. And so those energy constraints greatly shrink the landscape of possible representations
Evolutionary Steps and AI Development
00:22:46
Speaker
from sort of this infinite landscape of all the ways you could represent certain data to a much more manageable set of representations. And that doesn't guarantee that will converge on the same representations, at least suggestive of a weak universality.
00:23:02
Speaker
where even when we don't have the exact same representations, they're often a coordinate transformation away from each other. It's actually a bit surprising. As you mentioned, when we train neural networks, we don't have the same entity constraints as the brain had during our evolution. And I would expect, again, from evolution, the human brains have
00:23:23
Speaker
many more inbuilt biases and heuristics but if we then compare the representations in a neural network to those in a human brain, we found that they are quite similar. Isn't that the whole point of universality? Does the neural network have the same heuristics and biases that we have or what's going on here?
00:23:43
Speaker
Well, one of the primary biases in stochastic reading descent is sometimes called a simplicity preference, basically an inductive bias for more parsimonious representations. Parsimonious in the sense of Occam's razor.
00:24:00
Speaker
Right? And that's a byproduct of this information theoretic concept of Kolmogorov complexity, where Kolmogorov complexity means, is measured by, you know, is there a short program that can reproduce this longer sequence? And if you can find a short program, that's sort of a more compact or more compressed way of representing it. And when you're under energy constraints, you're looking for those more compressed representations.
00:24:25
Speaker
And so that simplicity bias seems to be also the origin of generalization, of our ability to go beyond merely memorizing data over fitting our parameters to finding a simpler way of representing those parameters, right? Where we go from sort of fitting a bunch of data points to recognizing, oh, these data points are being generated by a sine function. So I can replace all these data points by a simple circuit for that sine function or something like that.
00:24:53
Speaker
What can we learn about AI progress when we consider the hard steps that humans and our ancestors have gone through in evolution? It is beyond evolution. This often comes up in the discussion of the Fermi Paradox. Life on Earth to exist at all, let alone intelligent life, had to pass through many hard steps. We had to have a planet in a habitable zone. We had to have the right mix of organic
00:25:23
Speaker
chemicals in the Earth's crust and so forth, we had to have the conditions for abiogenesis, the emergence of the very earliest non-living replicators, probably some kind of polymer type of crystal structure. Then we had to have the transition from single-celled to multi-celled organisms to transition through the Cambrian explosion. Every one of these steps, you could think of as a very unlikely and probable thing.
00:25:51
Speaker
all the way up to the development of warm-blooded mammals and social animals that were heavily selected for, for brain size, to then the socio-cultural hard steps of moving from small primates to settled technological cultures, then technological hard steps like the discovery of the printing press or the discovery of the transistor.
00:26:19
Speaker
you put those all together and life seems just incredibly unlikely. And this often goes to the point of view that creationists or intelligent designers would look forward. But then you zoom out and then you recognize, oh, wait, there are trillions of galaxies, hundreds of billions of stars and hundreds of billions of trillions of planets.
Self-Reinforcing AI Development
00:26:42
Speaker
There's an awful lot of potential variation out there. And then meanwhile,
00:26:49
Speaker
Every one of these hard steps seems characterized by a search problem that is very hard. But then once you find the correct thing, like the earliest self-replicator, things kind of take off, right? So you imagine that before the earliest self-replicator, there were billions or billions of attempts to self-replicate, like that didn't succeed.
00:27:13
Speaker
Yeah, it's just a huge search problem, right? And, you know, maybe there are more gradual intermediate stages where you have sort of, you know, everything in biology ends up looking way more gradual the more you learn about it. But there are these phase transitions where you tip over and you get the Cambrian explosion or you get the printing press and the printing revolution. And so those hard steps end up looking relatively look, they look more easy in retrospect.
00:27:38
Speaker
Because even though the search was hard, once you've tripped over the correct solution, there's sort of an autocatalytic, self-reinforcing loop that pulls you into a new regime. And indeed, when you look at
00:27:51
Speaker
the emergence of life on Earth relative to the age of the universe. And Avi Loeb with some co-authors have done this. Life on Earth is incredibly early. The universe is 13.7 billion years old, but life couldn't emerge really much sooner. The reason being the universe started out as hot and dense and had to cool down.
00:28:16
Speaker
stars had to form, those stars had to supernovae so they could produce the heavy elements that are essential to life. And then those solar systems had to then take shape and then had to further cool so the solar system wasn't being irradiated constantly. And when you put all those factors together, human life emerged basically as soon as it was possible for life to emerge anywhere. And so this is one way to answer the Fermi paradox is that we're just in the first cohort.
00:28:44
Speaker
But it also should give you strong priors that passing through those hard steps isn't as hard as it looks. And what's the lesson for AI here? Developing AGI is sort of a hard step. We're doing this kind of gradient search for the right algorithms, for the right what have you. And we seem to be now in a slow takeoff where we've
00:29:07
Speaker
figured out the core ingredients, and there's now an autocatalytic process that's pulling us into a new phase. And what do you mean by autocatalytic? Self-reinforcing. Once it gets started, it sort of pulls itself. It sort of has an as-if teleology, right? You see this in nature, but you also see this in capitalism. And you would expect us to get to advanced AI basically as soon as it's computationally possible.
00:29:37
Speaker
It basically seemed that way, right? There was a kind of tacit collusion between Google and other players in the space. They had transformer models since 2017, but really, there's some of the precursors to transformers go back to the early 90s. But once you have this sort of profit opportunity that's in the background, it's hard in the competitive environment to stop an open AI from
00:30:02
Speaker
being like, oh, let's chase those profits. And then once that ball gets rolling, it's basically impossible to stop. This is why whatever the merits of the pause letter, it's virtually impossible to really have a pause in AI development because everything is structured by these game theoretic incentives to just keep going faster. Once you've stumbled on the gold reserve, it's hard to keep the prospectors from running there.
Government Adapting to AI
00:30:30
Speaker
Samuel, is the US government prepared for advanced AI? No. No, I mean, where do I start? I mean, the US government in
00:30:43
Speaker
If you think of it from a firmware level, you know, many countries have national IDs, the US doesn't have a national ID, we have social security numbers, there are these like nine digit numbers that date back to 1935. We have the core administrative laws date back to the early 40s, you know, much of our sort of technical infrastructure like the the system the IRS runs on.
00:31:08
Speaker
date back to the Kennedy administration and are written in assembly code. There's also been this general decline in what you could call state capacity, sort of the ability for the US government to execute on things. And you hear about this all the time. You hear about how the Golden Gate Bridge was built in four years or something like that. And now it takes like 10 years to build an access road. One of the reasons for that goes to what the legal scholar Nicholas Bagley has called the procedural fetish.
00:31:39
Speaker
really since the 70s, the machinery of the US government has shifted towards a reliance on explicit process. And proceduralism has pluses and minuses. If you have a clear process
00:31:56
Speaker
government can kind of run an autopilot to an extent. But it also means you limit the room for discretion and you limit the flexibility of government to move quickly. And moreover, in our adversarial legal system, you also open up avenues for sort of continuous judicial review and legal challenge.
00:32:13
Speaker
where famously New York has taken over three years to approve congestion pricing on one of their bridges because that's to undergo environmental review and people who don't want to pay the congestion price keep suing. Do you think having more procedures would make it easier for AI to interface with government? I would say having fewer procedures would make it easier for government to adapt.
00:32:37
Speaker
My assumption would be that having something written down, having a procedure for something would make it easier to plug AI into that procedure if it's less opaque and more almost like an algorithm step by step.
00:32:52
Speaker
Yes. But the analogy I would give is to the Manhattan Project, right? The original Manhattan Project was run like a startup. You had Oppenheimer in general, Leslie Groves sort of being the technical founder and the type A get things done founder.
00:33:11
Speaker
And they broke all the rules. They pushed as hard as they could. They're managing at its peak like 100,000 people in secret. And they built the nuclear bomb in three years. And so the way we would do that today under procedural fetish
00:33:28
Speaker
framework would be to put out a bunch of requests for proposals and have some kind of competitive bid and then we'd probably get the lowest cost bid and it would be Lockheed Martin and they would build half an atom bomb and it would take 20 years and
00:33:48
Speaker
five times the budget. And so that's sort of what I'm getting at. It's not about process versus discretion per se, it's about the way process hobbles and straight jackets our ability to adapt and sort of represents a kind of sclerosis, a kind of sort of like crystallized intelligence. We lay down the things that worked in the past as process
00:34:12
Speaker
and sort of freeze those processes in place, ossifying a particular modality. And when the motor production shifts and you need to completely tear up that process root and branch, it's very difficult because often there's no process for changing the process.
Lessons from Past Technological Transformations
00:34:30
Speaker
Yeah. I wonder if there are lessons for how government will respond to AI in thinking about how governments responded to, say, historical technical innovations of a similar magnitude, like the industrial revolution or the printing press or maybe the internet computer. Do you think we can draw general lessons or is it so specific that we can't really extract information about the future from them? I think they're very powerful general lessons.
00:35:00
Speaker
I think one of the first generalisms is that every major technological transformation in human history has preceded an institutional transformation, whether it's the shift from nomadic to settled city-states with the agricultural revolution or
00:35:16
Speaker
know, the rise of modern nation states or the end of feudalism with the printing press to, you know, in the New Deal era, the sort of transition with industrialization from the kind of laissez-fairer classical liberal phase of 18th century America to an America with a robust welfare state and administrative bureaucracies, and really in all new constitutional order, right? And so there's sort of
00:35:42
Speaker
better and worse ways for this transition to happen. There's sort of the internal regime change model. And you can think of, you know, Abraham Lincoln or FDR as inaugurating a new republic, a new American republic. Or there's a scenario where we don't change because we're too crystallized and sort of like an innovator's dilemma get displaced by some new upstart. And there are different countries have different abilities and different sort of capacities for that internal adaptation.
00:36:08
Speaker
As a Canadian, I'm a big fan of Westminster-style parliamentary systems. And one of the reasons is because it's very easy for parliamentary systems to shut down ministries, open up new ministries to reorganize the civil service because it's sort of vertically integrated under the prime minister's office or what have you. In the US, it's much worse because given the separation of powers, Congress and the executive are
00:36:36
Speaker
often not working well together just as an understatement. But then moreover, the different federal agencies have a sort of a life of their own. Often they're self-funded and all these other things that make it very difficult to reform. Do you think Canada responded better to the rise of the internet than the US, for example? Isn't there something wrong with the story because the US birthed the internet and Canada adopted the internet from the US?
00:37:04
Speaker
Let's compare, first of all, the impact of the internet on weaker states because, you know, Canada and the US are similar or sort of in one quadrant. They have differences, but the differences are small compared to other countries. If you think about like internet safety discussions that would have been taking place in the early 2000s, people would have been talking about, you know, identity theft, credit card theft, child exploitation, these kind of like direct first order potential harms from the internet.
00:37:32
Speaker
they didn't foresee that concurrent with the rise of mobile and social media, that the internet would enable tools for mass mobilization simultaneous with a kind of legitimacy crisis where the sort of new transparency and information access that the internet provided eroded trust in government and trust in other institutions. So you have these two forces interacting
00:37:57
Speaker
internet exposing government and exposing corruption and leading to a decline in trust while also creating a platform for people to rise up and mobilize against that corruption. And it's something that kind of rhymes with the printing press and the printing revolution.
00:38:11
Speaker
where you had these sort of dormant, suppressed minority groups like the Puritans or the Presbyterians, the nonconformists, and with the collapse of the censorship printing licensing regime, they actually had a licensing regime in the UK parliament back circa 1630. That licensing regime collapsed, I think 1634 or something around there, and that was like five years before the English Civil War.
00:38:40
Speaker
And you see something like this in the Arab Spring, where the internet quite directly led to mass mobilization in Cairo and Tunisia and elsewhere, and led to actual regime change, in some cases, sort of temporary state collapse. And that's because those were weaker states that hadn't democratized, that hadn't sort of had their own information revolution earlier in their history the way we did. Right. In some ways, like the American Republic is sort of a founder country built on
00:39:10
Speaker
the backbone of the printing revolution. So we were a little bit more robust to that because it's sort of part of our ethos to have this open, disagreeable society. But clearly, the internet has also affected the legitimacy of Western democracies. I think it's clearly one of the major inputs in sort of rising populism, the mass mobilizations that we see, whether in the US context, the 2020
00:39:40
Speaker
you know, racial awakening or the January 6th sort of peasant rebellion, right? These sort of look like the kind of color revolutions that we see abroad. And you know, some people want to ascribe conspiracy theories to that. I think there's a simpler explanation, which is that people will self-organize with the right tools. Our state hasn't collapsed yet, but there's clearly a lot of cracks in the foundation, if you will.
00:40:07
Speaker
Would it be fair to say that the main lesson for you from history is that technological change brings institutional change?
Economic and Labor Shifts Due to AI
00:40:14
Speaker
Yeah, not necessarily one for one. I'm not kind of a vulgar Marxist on this, but yes. And the reason for that is because institutions themselves exist due to a certain cost structure. And if you have general purpose technologies that dramatically change the nature of that cost structure, then institutional change will follow.
00:40:33
Speaker
Yeah. And I think we want to get to that. But before we do, I think we should discuss AI's impact on the broader economy. So not just the government, but the economy in general. Economists have this fallacy they point out of, the Lumber Labor Fallacy. Maybe you could explain that. The Lumber Labor Fallacy is essentially the idea that there's a fixed amount of work to be done. If you were thinking about the Industrial Revolution and what would happen to the 50% of people who are in agriculture,
00:41:01
Speaker
You couldn't imagine the new jobs that would be created, but new jobs were created. And the reason is because human wants are infinite and so demand will always fill supply. The second reason is because there's a kind of circular flow in the economy where one person's cost is another person's income.
00:41:21
Speaker
society would collapse if we had true technological unemployment because there would be things being produced, but no one to pay for them. And so that ends up bootstrapping new industries and new sources of production. There's still this open question, is this time different?
00:41:36
Speaker
Yeah, that's exactly what I want to know. In retrospect, let's say, it's easy to see how workers could move from fields to factories into offices. But if we have truly general AI, it's difficult for me to see where workers would move, especially if we have also functional robots and perhaps AIs that are
00:42:00
Speaker
better at taking care of people than other people are. I'm not asking you to predict specific jobs, but I'm asking you whether you think this historical trend will hold with the advent of advanced AI. The first thing to say is, when Keynes wrote Economic Possibilities for Our Grandchildren, a famous text where he predicted that technological progress would lead to
00:42:29
Speaker
the growth of a leisure society. And this was in the 1930s? Yeah. You know, people have dismissed him as being wrong. But actually, you look at time use data and employment data, and people are working less, you know, it's not it didn't match his the the optimism of his projection, right? Because it turns out, you know, maybe maybe if we fixed living standards that what he expected, people people want more and people will work more for more.
00:43:00
Speaker
But overall, people are working less. People do have more leisure. We've moved to a defective four-day work week. So there is one world where rapid technological progress continues that trend. And we all work less. It's a technological unemployment that's spread across people and is enabled in part because in a world of AGI, maybe you only have to work a few hours a day to make $100,000 a year.
00:43:27
Speaker
There's another possibility, which is that, well, AGI could, in principle, be a perfect emulation of humans on specific tasks. It can't emulate the historical formation of that person. So what I mean by that is, if you had a perfect Adam by Adam replication of the Mona Lisa, it wouldn't sell at auction. Because people aren't just buying
00:43:56
Speaker
the physical substrate. They're also buying the kind of world line of that thing. And that's clearly the case in humans as well. There are certain talking heads that I go and enjoy not because they are the smartest or what have you because I'm interested in what that person thinks on this because they have a particular personality, a particular world line. The third factor is
00:44:23
Speaker
sort of artificial scarcity. And so even in a world with abundance and supply in services and goods, there are still things that will be intrinsically scarce. Real estate being probably the canonical thing, but also energy and commodities and so forth. And the reason real estate is intrinsically scarce is because people
00:44:44
Speaker
want to live near other people. And people want to live in particular areas of a city. They want to live in the posh part of town. And those are positional goods. We can't all live in the trendy loft. So that builds in the kind of artificial scarcity. And so people will still be competing over those things. This is sort of related to artificial scarcity, but there's also sort of break it out into a fourth possibility, which are sort of tournaments and things that are structured as tournaments.
00:45:13
Speaker
having chess spots that are strictly better than humans at chess hasn't killed people playing chess. If anything, more people play chess today than they've had in human history. Yeah, it's more popular than ever.
00:45:25
Speaker
Yeah, and the reason is because people like to watch other humans playing, and also they're structured as sort of zero-sum tournaments where there can only be the best human. You look at other things that have been created just in the last 15, 20 years, like the X Games, right? I think people will still want to watch other people.
00:45:43
Speaker
the Olympics or do motocross and all these other things. And so maybe more of our life shifts into both maybe greater leisure on the one hand, more competition over positional goods, and more production that is structured as a tournament.
00:46:01
Speaker
Yeah, I can see many of those points. I'm just thinking, again, with fully general AI, you would be able to generate a much more interesting person playing chess, or at least a simulation of a very charismatic and interesting human chess player.
00:46:17
Speaker
Why wouldn't people watch that chess player as opposed to the best human? Maybe they will. It's hard to know. The question is who's producing that video stream because you still need the human behind it that had the idea, right? And
00:46:40
Speaker
You could imagine people being dishonest about the history of this chess player. This simulated chess player could be fully digital, fully fictional, so to speak, and just pretending to be human. Right. So they could fool people. That's the case too. No, I can't rule that out. But I would just say that however that person is monetizing, they're a deep fake chess player.
00:47:06
Speaker
they're making money, which they're then spending back into the economy. And so they'll produce jobs somewhere. Do you think more people will move into, say, people focused industries like nursing and teaching? Is that a possible way for us to maintain jobs?
AI's Impact on Education and Professions
00:47:22
Speaker
Maybe nursing, at least in the short run. I'm not very long on education being labor intensive for much longer. But you don't think education is at least, say, great school education. Is that really about teaching people or conveying knowledge? Or to what extent is it about conveying knowledge? And to what extent is it about the social interaction and specialising your teaching to the individual student?
00:47:51
Speaker
Well, AIs are very good at customization and sort of mastery to education is a bundle of things. And, you know, for younger ages, it's it's also daycare, it is socialization, like you said, at the very least, it means it's just a reorganization of the division of labor. Because the types of teachers that you would select hire for may differ if the education component of that bundle is being done by AI. And maybe you select for people who are
00:48:22
Speaker
maybe don't have any subject matter expertise, but are just highly conscientious and good around kids. Or maybe you unbundle from public education altogether and it re-bundles around a judicial school or a chess academy because you'll have the AI tutor that will teach you math, but you'll still want to grapple with the human
00:48:42
Speaker
What about industries with occupational licensing, like law or medicine? Will they be able to keep up their quite high wages in the face of AI being able to be a pretty good doctor and a pretty good lawyer? It's easy to solve for the long-term equilibrium. With the rise of the internet, you can do a comparison of the wage distribution for lawyers pre and post internet. And, you know, circa the early 90s,
00:49:10
Speaker
lawyer incomes were normally distributed around $60,000 a year. After in the 2000s, they become bimodal. And so you have one mode that's still around that $60,000 range. Those are like the family lawyers. And then you have this other mode that's into the six figures and those are like big law. It's the emergence of these law firms where you have a few partners on top and maybe hundreds of
00:49:34
Speaker
of associates who are doing kind of grunt work using Westlaw and Lexus Nexus and these other legal search engines to accelerate drafting and legal analysis. So if that pattern repeats, I could imagine these various high-skill knowledge sectors to also become bimodal, where in the short run, AI serves as a co-pilot, sort of like Westlaw or Lexus Nexus was for legal research.
00:50:03
Speaker
and enables the kind of 100x lawyer. And so there's a kind of averages over dynamic. Longer run,
00:50:13
Speaker
You start to see the possibility of doing an end run around existing accreditation and licensing monopolies where obviously the American Medical Association and medical boards will be highly resistant to an AI doctor. I tend to think that they'll probably end up self cannibalizing because
00:50:34
Speaker
The value prop is so great even for doctors to do simple things like automate insurance paperwork and stuff like that. But to the extent that there is a resistance, to the extent that in 10 years there's still a requirement that you must have the doctor prescribe the treatment or refer you to a specialist even though the AI is doing all the work and they're just sort of like the elevator person that's actually just pushing the button for you.
00:51:01
Speaker
it'll be very easy to end run that because AI is both transforming the task itself, but also transforming it's the means of distribution. And if you can go to GPT-4 and ask for, put in your blood work and get a diagnosis, that no regulator is going to stop that, right? And so GPT-4 becomes sort of the ultimate doctor of up orders. You write a lot about transaction costs and how changes in transaction costs
00:51:28
Speaker
change institutional structures? First of all, what are transaction costs and how do you think they'll be affected by AI? So transaction cost is sort of an umbrella term for different kinds of costs associated with market exchange. And this goes back to Ronald Coase's famous paper on the theory of the firm, where he asked the question, why do we have corporations in the first place?
AI and Transaction Costs
00:51:50
Speaker
If free markets are so great, why don't we just go and spot contract for everything?
00:51:56
Speaker
And the answer is, well, market exchange itself has a cost. There's the cost of monitoring. If you hire a contractor, you don't know exactly what they're doing. There's the cost of bargaining. Having the haggle with the taxi cab driver is a friction. And there's the cost of associated searching information.
00:52:14
Speaker
So taking those three things together, they're not all that companies do, but they structure the boundary of the corporation. They explain why some things are done in-house and some things are done through contracts. If there's high monitoring costs, you want to pull that part of the production into the company so that you can monitor and manage the people doing the production.
00:52:33
Speaker
And some of the same effects go for the existence of governments, right? Yes, because governments, you know, with a certain gestalt, governments and corporations aren't that different. There are kinds of institutional structures that pull certain things in house and certain things are left for contracting or outsourced. And you know, you can see sort of
00:52:53
Speaker
different kinds of governments having different parallels with different kinds of corporate governance, right? Relatively egalitarian democratic societies like Denmark are kind of like mutual insurers, whereas more hierarchical authoritarian countries are more like Singapore, say, is more of a joint stock corporation.
00:53:15
Speaker
And indeed, Singapore was founded as a entrepot for the East India Company. So there are very deep parallels. And it's also essential transaction costs are essential to understand why governments do certain things and other things. All Western developed governments guarantee some amount of basic health care. But outside of, say, the National Health Service in the UK, most of these countries
00:53:45
Speaker
guarantee the insurance, they don't necessarily nationalize the actual providers. And the reason goes to transaction costs and sort of an analysis of the market failure and insurance. Likewise with roads, it's possible to build roads through purely private means and indeed, countries like Sweden, a lot of the roads are run by private associations.
00:54:08
Speaker
But if you have lots of different boundaries, different micro jurisdictions and so forth, there can be huge transaction costs to negotiating up to an interstate highway system. And those transaction costs then necessitate public infrastructure projects. So the transaction cost in this case would be being a private road provider. You have to go negotiate with 500 different landowners about building a highway, whereas the government can
00:54:37
Speaker
do some expropriation and simply build the road much faster, or with less transaction costs at least. Yeah, precisely. And we're seeing this dynamic in the US with permitting for grid infrastructure and transmission. We're building all the solar and renewable energy, but to build the actual transmission infrastructure to get the electrons from where it's sunny to where it's cold requires building high voltage
00:55:07
Speaker
lines, across state lines, across different grid regions. And there are all kinds of NIMBYs and negotiation costs involved, holdouts and so forth. And so the more those kinds of costs exist, the more it militates towards a kind of larger scale intervention that federalizes that process. Yeah. The big question then is how will AI change these transaction costs? What will the effects be here? It's easy to say that they will be affected.
00:55:37
Speaker
And obviously the internet affected them to an extent. And we were talking, when we talk about sort of the ease of mobilizing protest movements or the kind of the sunlight that was put on government corruption, those are reflecting declines in the costs associated with information and coordination. I think AI takes us to another level. And I think it's important to think through in part because right now the AI safety debate
00:56:06
Speaker
at least in the United States is very polarized between people who are like, everything's going to be great. And people who are like, this is like a terminator scenario or an AI kill us all existential risk. You know, even if we accept the, you know, existential risk framing, there's still going to be many intermediate stages of AI before we flip on the super intelligence. And those intermediate stages have enormous implications for the structure of the very institutions that will need
00:56:35
Speaker
to respond to super intelligence or what have you. The ways we can see this is because all these information and monitoring and bargaining costs are directly implicated by commoditized intelligence. Start with the principle agent problem.
00:56:53
Speaker
There is no principal agent problem if your agent does exactly as you ask and works 24-7. It doesn't steal from the till. AI agents dramatically collapse agency cost monitoring. Now that we have multimodal models, in principle, we could have cameras in every house that are just being prompted to say, is someone committing a crime right now?
00:57:15
Speaker
whether we wanted to go that direction or not, it gives you a sense of how the cost of monitoring have basically plummeted over the last two years and are going to go way lower. And so you're starting to see this rolled out in the private sector with Activision has announced that they're going to be using language models for moderating voice chat and call of duty.
00:57:36
Speaker
Right and this is a more robust form of monitoring because in the past you would have to like ban certain words like certain swear words or things associated with sex or violence. But then people could always get around those by using euphemisms.
00:57:53
Speaker
Like on YouTube, the algorithm will ding you if you talk about coronavirus or if you talk about murder or suicide, these things that throw off red flags. So what people have taken doing is saying they were unalived rather than murdered. And that doesn't fool a language model. If you ask a language model, if you prompt it in a way to look for broad semantic categories, not just a narrow word, it's much more robust.
00:58:22
Speaker
And so what that means, you know, you already start to see it, like I said, with Activision and the use of LLMs and content moderation, you're going to start, you're going to see it in the use of multimodal models for productivity management and tracking. You know, Microsoft is unveiling their 365 copilot, where, you know, you're gonna have GPT-4 and Word and Excel and Teams and Outlook. But at the same time, you're also going to have a manager who's going to be able to say, you know, to prompt the model, tell me who was the most productive this week.
00:58:51
Speaker
Something as vague as that. And so you see this diffusion in the private sector. The question is, does it diffuse in the public sector? There's obvious ways that it would be a huge boon. Inspector General GPT could tell you exactly how the civil services is working, whether there's corruption, whether there's a deep state conspiracy or something like that. And at first blush, a lot of what government does is kind of a fleshy API.
00:59:19
Speaker
bureaucracies are nodes that apply a degree of context between printing out a PDF and scanning it back into the computer. It varies. There's degrees of human judgment that are required. But on first order, government bureaucracies seem incredibly exposed to this technology in a way that could diffuse really rapidly because going back to Microsoft 365 Copilot, Microsoft is the biggest IT vendor in the US government. And so you can imagine
00:59:48
Speaker
once everyone has this pre-installed on their computer that the person at the Bureau of Labor Statistics who's in charge of
Public vs. Private Sector AI Adoption
00:59:54
Speaker
doing the monthly employment situation report, the jobs report, you know, at some point he's going to be walking to work and hitting a button, right, that, you know, asking Excel to find the five most interesting trends and generate charts and the report is done.
01:00:10
Speaker
And in the private sector, that person would be reallocated and maybe doing things that the computer's not good at yet. But these positions are much stickier in government. To the extent that diffusion is inhibited on the public sector side, I worry about the kind of disruption and displacement of government services by a private sector that's adopting the technology really fast.
01:00:29
Speaker
This is something we'll talk about in a moment. Before that, I just want to get to your complaints about isolated thinking about AI. You've sketched out some complaint about people thinking about AI only applying to one domain and then not really seeing the bigger picture. What are some examples here? Why do you worry about isolated thinking?
01:00:52
Speaker
a few dimensions to this. One is what I've called the horse's carriage fallacy. The kind of view that what automobiles were was just a carriage with the horse. And so that anchors you to the older paradigm and it's like you're changing one thing and everything else stays the same. And you neglect all the second order ways that
01:01:12
Speaker
the development of the automobile enabled the build out of highway systems, the total reconfiguration of the economic geography. And then implications for institutions like the state where once you have road networks or telegraph networks or any of these kind of networks, it suddenly becomes easier to monitor
01:01:33
Speaker
agents of the state and other parts of the country. And so you can build out more of a federal bureaucracy. And so all these things were second order. And we're kind of neglected if you just were too focused on the first order effects of displacing the horses. And in a sense, the second order effects turned out to be much more consequential in the end. Yes, they seem to always be. And likewise with the internet and sort of the I think this comes up a lot in the in how to think about AI use and misuse.
01:02:03
Speaker
there's lots of valid discussions there, but they're always very first order. And when you think about the way the internet has disrupted legacy institutions, yes, there's disinformation, but often the thing that's disrupting is not fake news, it's real news that's being repeated with misleading frequency, right? That's like throwing off our availability heuristic or it's
01:02:26
Speaker
valid complaints whether the protest in Iran had this striking parallel to the protests following the George Floyd protest and protests seen in other countries where they even have a three-word chant or the case of the Arab Spring in Tunisia that started with the person self-immolating.
01:02:50
Speaker
There's the structure that repeats where you have a martyr or some shocking event. And because of the way social media is organized, it synchronizes people around the event in a way that's kind of stochastic. It's like lightning striking. You don't know what event it's going to strike on. But once we're synchronized, then we start moving back and forth in a way that causes the bridge to buckle.
01:03:14
Speaker
Nothing about that is a misuse. Those are all valid uses, but their use is under collective action. It's sort of solving not just for the partial equilibrium, but the general equilibrium when everyone is doing this.
01:03:29
Speaker
And I think the person who wrote the best on this conceptually was Thomas Schelling. And one of his little books, Micro Motors, Macro Behavior, has been influenced on me as a kid where he talks about all these toy models where you're at a hockey game or a basketball game.
01:03:45
Speaker
And something is happening, something exciting is happening in the arena. And so the people in front of you stand up to get a better view, and then you have to stand up to get a better view over them, and so on. And so it cascades, and suddenly everyone went from sitting to everyone went to standing, and no one's view has improved. And so these sort of general equilibria where you sort of solve for everyone's micro-incentives and the kind of new Nash equilibrium that emerges.
Institutional Adaptation to AI
01:04:10
Speaker
That ends up being the thing that drives a kind of multiple equilibrium shift from one regime to another. And throughout, there may be no actual examples of misuse involved. It may just be people following their individual incentives.
01:04:25
Speaker
I think it's worth stressing this point you make about the effects of earlier AI systems on our institutions. They might have effects that deteriorate our institutions such that we can't handle later and more advanced AI. Ignoring this would be an example of isolated thinking and ignoring the second order effects.
01:04:47
Speaker
Right? Yeah. And they also, it also changes the, the sort of agenda, right? The AI safety agenda shouldn't just be about the first order of things, rent alignment, you know, very important. But, you know, it's led to a discussion of, do we need a new federal agency? And if so, what kind of agency? Whereas it may be more appropriate to think
01:05:11
Speaker
not what new agency do we need, but how do all the agencies change? And how do we brace for impact and enable a degree of co-evolution rather than displacement?
01:05:25
Speaker
I don't know whether the question of how to get our institutions to respond appropriately is more difficult or less difficult than the problem of aligning AI. But it certainly seems very difficult to me. So are we making it harder on ourselves if we focus on the second order effects on institutions? I mean, it's unavoidable. I mean, we can't pick and choose what kind of problems.
01:05:54
Speaker
You know, the alignment problem, the hard version is yet to be solved. But we have many examples of governments buildings take capacity and having kind of, you know, shifting from very, very like clientelistic sticky.
01:06:08
Speaker
corrupt governments to modernized governments where state capacity is built and then that government can break out of the middle income trap and become rich. You mentioned Estonia as an example of a country that's pretty advanced on the IT front, on the technology side.
Estonia's Digital Government Example
01:06:26
Speaker
Maybe you could talk a bit about Estonia.
01:06:28
Speaker
Yeah, I would just say in general, it's hard for any organization to reform itself from within when there is path dependency, but I would say at least we have examples of it being done, where we don't have examples of alignment being solved yet. When it comes to Estonia, Estonia is an interesting case, it's sort of an exceptional case because after the fall of the Soviet Union and the breakup of the peripheral former Soviet states, they kind of had a blank slate.
01:06:56
Speaker
They also had a very young population and people who had a kind of hacker ethic within their civil service. And so with that blank slate and with that hacker ethic, they were very early to adopt and to foresee the way the internet was going to shape the government through a variety of e-government reforms.
01:07:14
Speaker
So early in the late 90s and into the 2000s, they were some of the earliest to digitize their banking system, like e-banking, to build this system called X-Road, which is kind of like a cryptographically secured data exchange layer. It resembles the blockchain, but it was about a decade before blockchain was invented.
01:07:35
Speaker
for exchanging information between different government entities. Your medical information could be uploaded to the system and then be available to all systems that have the right to see that information. Exactly, in a way that's cryptographically secured and distributed. So if a missile hit the Department of Education, you don't lose your education records because it's distributed. And that also enabled an enormous amount of automation
01:08:03
Speaker
where, for instance, this is my understanding, a child born in Estonia, once you file that birth record, it more or less initiates a clock in the system that will then enroll your child in school when they turn four or five, like automatically, because it knows that your child has aged, and then unless it had a death record to cancel that out. That also means you can do taxes and transfers much simpler, you get your benefit within a week,
01:08:32
Speaker
can integrate across different parts of public infrastructure, like use the same card to ride the bus as you do to launch a new business. It also serves as a kind of platform for the private sector to do government by API, to build
01:08:51
Speaker
new services on top of government as a platform and integrate with government databases. And so the point here for us is that institutional reform is possible, modernizing government is possible, at least under certain circumstances. We have proofs of concepts of this happening. The hard thing is the path dependency. There's always a strong instinct to want to start from scratch, and it's normally not advisable because it's too hard.
01:09:20
Speaker
And so this is why it's hard in the US. This is why you have African countries that leapfrog us in payment systems and so forth. The challenge of this decade or century is, how do we solve that path dependency problem? And how do we get to Estonia? It used to be, get to Denmark. Now, let's get to Estonia and find that pathway up Mount and probable.
01:09:43
Speaker
Great. Let's get to your wonderful series of blog posts on AI and
AI's Influence on Government Power Dynamics
01:09:49
Speaker
Leviathan. In this context, what do we mean by Leviathan? Well, this all interrelates. Leviathan was the book Thomas Hobbes wrote at the start of the interregnum after the English Civil War.
01:10:01
Speaker
And it was his basically his early political science, early defense of absolutist monarchy as a way to restore peace and order after after a decade of infighting. And Hobbes kind of hit on some basic sort of structural game theoretic properties of why we have governments at all.
01:10:26
Speaker
He talked about life being nasty British and short in the state of nature, war of all against all. And peace is only restored when people who don't trust each other offload enforcement and policing responsibilities to a higher power.
01:10:43
Speaker
that can then restore a degree of peace and order. AI and Leviathan is talking about, how does AI change the story? Does it reinforce the Leviathan? Does it lead to a digital police state a la China? Or is it something that we impose on ourselves? And we talked about how multimodal models could in principle be used to put a camera in everyone's house and have it just continuously monitoring for people doing any kind of crime. That's something that North Korea might do.
01:11:11
Speaker
In the US context, it's something that we're very liable to just voluntarily do to ourselves because we want to have ring cameras and Alexa assistants and so forth. And so that leads to a kind of bottom up Leviathan that is potentially no less oppressive and maybe even more oppressive because there's no one that we can appeal to to change the rules.
01:11:36
Speaker
So Leviathan is one way to respond to technological change, but you mentioned two other ways we could alternatively respond. Right. So basically, any time a technology greatly empowers the individual, it creates a potential negative externality. Hobbes called these are natural liberties. In the state of nature, I have a natural liberty to kill you.
01:11:58
Speaker
or to strong arm you. And governments exist to revoke those natural liberties, right? But for a higher form of freedom, right? And so there's sort of any time a technology greatly increases human capabilities vis-a-vis other humans. The three canonical ways we can adjust are seeding more authority to that higher power, the Leviathan option. And then the other two options are adaptation and mitigation.
01:12:25
Speaker
and normative evolution. So the example I give is if suddenly we all had x-ray glasses and you could see through walls and see through clothing. One option, we have a draconian totalitarian crackdown that tries to seize all those x-ray glasses. Another option is we
01:12:44
Speaker
adjust normatively, culturally, that our privacy norms wither away and we stop caring about nudity. And then the other option is adaptation mitigation, where we put in a mesh into our walls and wear leaded shirts and pants.
01:13:02
Speaker
Yeah, I guess continuing that analogy a bit between the smart glasses and AI, you have this amazing write up of ways in which AI can increase the informational resolution of the universe.
Data Analysis and Privacy Concerns
01:13:19
Speaker
So you give some examples that are, I think, specifically of AI identifying people by gate, for example. Right. So gate recognition is nothing new. China has had advanced forms of gate recognition for a while now.
01:13:33
Speaker
So even if you cover your face, it turns out we're constantly throwing off ambient information about ourselves, about everything. And the way you walk, the particular gate that you have, is a unique identifier. Another example is galaxy surveys. We've had, from Hubble telescopes now, the JWST, tons of astronomical surveys of distant galaxies and so forth.
01:14:01
Speaker
And all of a sudden, all that old data, it's like that same data set is now more useful because applying more modern deep learning techniques, we can extract entropy that was in that data set, but we didn't have the tools to extract yet and discover that there are new galaxies or other phenomena that we missed. Another example you give is
01:14:25
Speaker
listening for keystrokes on a keyboard and extracting information about a password being typed in, for example, which is something that, of course, humans can't do, but we can do with AI models. Yeah. So that was a paper showing that you can reconstruct keystrokes from an audio recording, including a Zoom conversation. So I hope you haven't typed in your password because people in the future
01:14:49
Speaker
And so this goes to the fact that it's sort of retroactive, that even if the technology wasn't diffused yet, any Zoom conversation, any recording where someone typed their password in the future will be like those Galaxy surveys where someone will go backwards in time and turn up the information resolution of that data.
01:15:05
Speaker
Yeah, this is pure speculation. But I wonder if I mean, imagine anonymized people in interviews, say 10 years ago, whether they will be able to stay anonymous, or whether AI will be able to extract the data about their their face or their voice that, that wasn't part wasn't technically possible when the interview aired. Yeah, exactly. There are already systems for like depixelating. You probably do do something similar for the voice modulation and
01:15:33
Speaker
And then also, again, going back to this ambient information we're always shedding, identifiers in the way we write, where we place a comma, the kinds of adverbs we like to use, and so forth.
01:15:47
Speaker
people just dramatically underrate how much information we're shedding, in part because we're blind to it. Some people who are taking great efforts to stay anonymous online, people in the cryptography space, for example, will put their writings through Google Translate to French and then back to English to erase subtle clues that could identify them personally. Why is AI so much better at tasks like the ones we just mentioned compared to humans?
01:16:17
Speaker
Well, it goes back to what we were talking about with sort of putting information theoretic bounds on AGI. When you minimize the loss function in a machine learning model, you're trying to minimize the cross entropy loss. A cross entropy is how many bits does it take to distinguish between two data streams? And if it takes a lot of bits to distinguish between the two, that means they're relatively indistinguishable. So that's going again to the Turing test.
01:16:40
Speaker
If we have a Turing test where I can tell right away that the AI is different from the human, that suggests a high cross-entropy. But if I could talk to it for days and do all kinds of adversarial questioning, I might still be able to, in the end, tell the difference between the two. But we've minimized that cross-entropy loss. And so when you have any arbitrary data distribution that you're trying to predict, whether it's trying to predict
01:17:06
Speaker
galaxies and astronomical data or passwords from fingerprint data on a phone screen. All these things embed a kind of physical memory of the thing in question and can often be reconstructed through this kind of loss minimization where you have a system that asymptotically extracts the entropy that was latent in the data.
01:17:30
Speaker
And this can be done in a way that is often quite striking where we can, with stable diffusion, make fairly accurate predictions of what people are imagining in their mind using fMRI data. And fMRI data is blood flow data in the brain. It's a very lossy representation of whatever's happening in the brain. But there's still enough latent entropy in there that we can reverse engineer or
01:17:55
Speaker
decompress it into a folder picture. And this could turn into a form of lie detection. Yeah, I think it already basically has. If you have fMRI data, or EEGs or other kinds of direct brain data, it's probably a lot easier. But we already have systems that are over 95% accurate at detecting deception from just visual video recordings.
Open-Sourcing AI Models
01:18:24
Speaker
We can see how all of this information that we are continually shedding gives rise to the possibility of a Leviathan, either of the private or of the government's kind. I wonder what role do you see open sourcing AI models playing here? What are the trade-offs and risks in open sourcing AI?
01:18:46
Speaker
Among the people who are most bullish to open source, there's often a kind of libertarian ethic undergirding it. Regardless of whether that's a good idea or not, one of the things I'm trying to communicate to that group is to say that be careful what you wish for because of these kind of paradoxical Hobbesian dynamics. The fact that in America, you never know if someone has a gun or not. On the one hand,
01:19:13
Speaker
the Second Amendment enhances our freedom. On another hand, you don't get the sort of like everyone's doors unlocked and people are, like the police in England don't even have guns. There's a certain freedom that derives from us not all being heavily armed. And likewise with open sourcing powerful AI capabilities, it empowers you as an individual
01:19:41
Speaker
But in general equilibrium, once we all have the capabilities, the world could look much more oppressive, either because we're all spying on each other all the time and we can all see through each other's walls, or because there's a backlash and the introduction of sort of Leviathan type solutions to restrict our ability to spy on each other all the time. And my general sense is that we can only delay
01:20:03
Speaker
we can't really prevent things from being open source over the long run because there's a sort of trickle down of compute requirements. But in the interim,
01:20:12
Speaker
There are definitely things that are valuable to open source. Having 70 billion parameter language models is not a threat. In fact, I think it's probably useful for alignment research for something like that to be open source. But if you are a researcher and you've developed a emotional recognition model that can tell with 99% accuracy whether someone is lying or not lying and whether your girlfriend loves you or not, these things
01:20:43
Speaker
or the ability to see through walls using, like I talk about the use of Wi-Fi displacement. There are people who have built pose recognition models using the displacement of the electromagnetic frequency of your Wi-Fi, and it was wall penetrating so that you can see through walls. What's the rush to put that on hugging face and to make it as democratized as quickly as possible? I would say that
01:21:13
Speaker
If we value the adaptation and mitigation pathway as opposed to the Leviathan pathway, then there's a value in slow rolling some of these things. How do you think a relative government power will be affected by AI?
Private Regulation of Public Spheres
01:21:32
Speaker
You write somewhere in this long series of blog posts that AI will cause a net weakening of governments relative to the private sector. Why is that?
01:21:42
Speaker
Yeah, specifically Western liberal governments under constitutional constraints. So if you imagine society being on this kind of knife edge, and I talk about this in the context of Farron Essamogle's book, The Narrow Corridor, where he describes liberal democracy as sort of being in this corridor between despotism on the one hand and anarchy on the other. And we sort of have to stay in this saddle path where society and the state are kept in balance.
01:22:11
Speaker
If you veer off that path, you can, on the one hand, you know, the state could become all powerful. And that's the sort of China model or authoritarian digital surveillance state. And indeed, you know, China built up their digital surveillance state and their internet firewalls and so forth.
01:22:27
Speaker
after watching the Arab Spring and seeing how the internet was destabilizing to we hear governments. And so I fully expect that AI will be very empowering and self-reinforcing of the power of the Chinese government. And indeed, their draft regulations for large language models stipulate that you can't use the model to undermine national unity or challenge the government. And so they're baking that in. In liberal democracies, we think of ourselves as open societies.
01:22:55
Speaker
And the issue is that we're only open at the meta level. There's a public sphere, right? There's freedom of information laws. We have freedom of speech. I don't have freedom of speech if I walk into a Walmart. Wait, right? The Walmart is private property. In open societies, it's not that we don't have social credit scores and thicker forms of social regulation. It's just that we offload those functions onto competing private actors, whether it's a church that has very strict
01:23:25
Speaker
doctrines to be a member or other kinds of social clubs. The fact that these days, if you want to go to a comedy club, they'll often confiscate your phone at the door because they don't want you recording the comedian's set and putting it online.
01:23:39
Speaker
my anticipation is that because of those constitutional constraints that limit the ability of liberal democracies to go the China route, because of our civil laws or bills of rights and so forth, and also because of a lot of procedural constraints, this will naturally shift into the private sector. And
01:24:01
Speaker
We see that already with the use of AI for monitoring and employment, for policing speech in ways that would be illegal if done by the state, but are fine if done by Facebook. To the extent that the AI continues to increase these negative externalities and therefore puts more value on having a vertically integrated experience, a walled garden that can strip out
01:24:24
Speaker
the negative forms of AI and reinstate the degree of harmony between people that more and more of our social life will be mediated through these sort of private organizations rather than through a kind of open public sphere.
01:24:41
Speaker
You're imagining that government services will be gradually replaced by private services that are better able to respond. Won't governments fight to uphold individual rights? In Walmart or on Facebook, you are regulated in ways that the government couldn't regulate you, but you still have the choice to go to Target instead of Walmart or X instead of Facebook.
01:25:08
Speaker
isn't that the fundamental thing? So the fundamental thing is the choice between services and won't governments uphold citizens' rights to make those kinds of choices? Yeah, no, I agree. And so this would be the defense of the liberal model is that we allow thicker forms of social regulation because it's moderated by choice and competition. And the issue with Chinese Confucian integralism
01:25:36
Speaker
isn't the fact that it's super oppressive. It's the fact that you only have one choice and you don't have voice or exit. So yeah, it's obviously a matter of degree, right? When ride hailing first arose, I remember back in 2013, 2014, it wasn't that long ago. I think Uber was founded in 2009, but it really only started taking off in the early 2010s.
01:26:06
Speaker
No, people thought it was crazy to ride a car with a stranger.
01:26:11
Speaker
And then within five years, it was the dominant mode of ride hailing. And in that five year period, essentially we saw a kind of regime change in micro, where taxis went from being something that was regulated by the state through these commissions that were granted legal monopolies and used licensing and exams and other sort of brute force ways of ensuring quality.
01:26:39
Speaker
to competing private platforms where you have Lyft or Uber to choose from. And they replace the explicit governance of legal mandates with the competing governance of reputation mechanisms, of dispute resolution systems, of structured marketplaces that collapse the bargaining
AI and Societal Divides
01:27:00
Speaker
frictions. You never have to haggle with an Uber driver. You just get in.
01:27:03
Speaker
And that was obviously a much better way of doing ride hailing. So even though there was a violent resistance early on, literally like in France, they were throwing rocks off of bridges and cab drivers in New York were killing themselves. So for the people affected, it was a very dramatic regime change, but for everyone else, it was a huge positive improvement. And yet, it's only made possible because Uber has a social credit score. If your Uber rating goes too low, you'll get kicked off the platform.
01:27:31
Speaker
And so we're fine with social credit scores. It's when you only have one and don't have an option. And it can follow you across all these different verticals that becomes a problem. Do you imagine that because of rising danger in the world, you talk about the externalities from the widespread implementation of AI all across society.
01:27:55
Speaker
Because of those dangers, those externalities, you will either use Uber or whatever service, or you can't participate in society. Do you imagine increased pressure in that direction?
01:28:07
Speaker
It does seem to be a longer term trend. I don't know if AI will accelerate it. I have another series of essays that I call Separation Anxiety. And it's a reference to the fact that in insurance markets, there's kind of two equilibria. There's the pooling equilibria, where we're pooled together into one risk pool.
01:28:28
Speaker
and there's a separating equilibrium where the insurance pool unravels and we break up into the great power insurance for senior citizens who never had an accident and stuff like that. And it turns out that insurance markets are competitively unstable, that without government regulation or social insurance, that insurance markets will naturally tend to unravel because of adverse selection into the high-risk people being in one pool and the low-risk people being in another pool.
01:28:57
Speaker
And it turns out you can use that as a mental model to look at other kinds of implicit pooling equilibria. So within company wage distributions, often there is 20% of the workers who are doing 80% of the work, but they're pooled together under one wage structure. And that was the dominant structure of the period of wage compression in the United States in the 50s and 60s.
01:29:24
Speaker
And once we had better monitoring technologies and were able to tell who were the 20% that were doing 80% of the work, it suddenly became possible to differentiate pay structure. And a lot of the rise in inequality in the United States is actually between firm. So what happens is Ezra Klein is like the most productive whiz kid at the Washington Post and he realizes, why don't I just go start my own website, right?
01:29:48
Speaker
And so that dynamics are played out across a variety of domains leads to a world that, to the extent that these features are correlated, that does separate, right? Where you have, you know,
01:30:01
Speaker
one-star Uber riders driving the one-star Uber drivers, the drivers driving the riders. And people who have the five-star Uber ratings and the perfect credit scores self-sort into communities with other people with perfect driving records and perfect credit scores. And we see that to an extent already with the enclaves of rich zip codes with private schools and everyone is sort of self-selected.
01:30:26
Speaker
AI could, it seems to me that AI would exacerbate that. I mean, at first blush, just because going back to the point about signal extraction, it can find all these different ways. You're a high risk type and I'm a low risk type and so forth that are probably latent in all kinds of data that we don't even need to give permission to the insurance company. They'll just like the same way that they use like smoking or going to a gym as a proxy. There's all kinds of proxies they could use and likewise for employers and how they pay people.
Social Norms and AI Integration
01:30:54
Speaker
society kind of runs on us not being entirely open and entirely honest all the time. Otherwise, you wouldn't be able to have kind of smooth social interactions and so on. Won't these norms be inherited by the way we use AI? Yeah, I think this is a really big issue. I'm a big fan of Robin Hanson and a lot of his writing on social status and signaling is sort of
01:31:22
Speaker
presenting humans as basically hypocrites, like we're constantly deceiving other people. And we often deceive ourselves, so it's better to deceive others, as the evolutionary biologist Robert Trivers once pointed out. So, you know, all the kinds of polite lies that we tell are, I think, critical lubricants to social interaction, and actually, like,
01:31:52
Speaker
It's good that there's a gap between our stated and revealed preference. I think a world where we all lived or stated preference could be hellish because we don't actually mean it. And AI has a direct implication on that because if I can have a pair of AR glasses on that will tell me if you're interested, if you're bored, if you're over on a date and are you really attracted to me, all that sort of polite
01:32:19
Speaker
veneer, that social veil could be lifted in a way that we'll probably want to coordinate to not do. But again, it's this Nash equilibrium where it's in my interest to know whether you're interested or bored. And so I'll want to have the glasses on and my ideal world is where only I have the glasses and you don't. And the other way that our hypocrisy is being exposed and challenged is the need to
01:32:47
Speaker
you know, explicates the utility function that we want these models to work under. You know, we need to formalize human values if we want to align these models. And so then we have to be honest and open about the fact that our stated preferences probably aren't our true preferences. And that's a very challenging thing because it cuts right to the nature of the human condition and involves topics that are intrinsically things that we lie to ourselves about.
Techno-Feudalist Future and AI
01:33:17
Speaker
You have what you call a timeline of a techno-feudalist future, which I found quite interesting. It's great writing and it's very detailed. We don't have to go through all of its detail, but maybe you could tell the story of what happens in what you call the default scenario. This is the scenario in which Western liberal democracies are too slow to adapt to AI, and so we get something like a replacement of
01:33:41
Speaker
government services with more private services. What happens in the techno-feudalist future? Right. And this sort of piggybacks everything you've just been discussing, right? And I don't want techno-feudalist to carry too much of a pejorative. I'm sort of using it descriptively.
01:33:58
Speaker
And certainly some people would prefer this world. So the example of Uber and Lyft displacing taxicabs is sort of a version of this in micro, where we go from this regulated taxi commission to competing private platforms that use various forms of artificial intelligence and information technology to replace the thing that was being done by explicit regulation.
01:34:25
Speaker
And as AI progresses and both creates a variety of new negative externalities, whether it's like suicide drones or the ability to spy on each other, there's going to be a demand for new forms of security and also kinds of opt-in
01:34:46
Speaker
jurisdictions that like tie our hands in the same way that we give up our phone before we go into the comedy club. And so I think this leads to a kind of development of clubs, the kind of club structure maybe at the city level as the vertically integrated walled garden that will police and build defensive technologies around the misuse of AI.
01:35:13
Speaker
and at the same time provide a variety of new AI-native public goods that are only possible once AI unlocks them. And it's easy to see how this could very quickly displace and eat away at formal government services, both because we saw it already with Uber, but also if you map that model to other areas of regulatory life,
01:35:38
Speaker
It doesn't make sense to have a USDA farm inspector. A human person has to go to a commercial farm and maybe only goes to that farm once every few years because there's so many farms and only so many people. It does a little checklist and says, oh, you're not abusing the animals and you got all the process in place and you get the USDA stamp of approval.
01:35:59
Speaker
Does it make more sense to have multimodal cameras on in the firearm 24-7 that are continuously generating reports that throw up a red flag anytime someone sneezes on the conveyor belt? To the extent that government is going to be slow at adopting that, will there be
01:36:18
Speaker
a push for the kind of Uber model of governance as a platform where you have the kind of AI underwriter, the consumer reports that sells these firms, the camera technology and the monitoring technology and builds their own set of compliance standards. And then you want to go to those firms or what have you that have the stamp of approval of that underwriter because it's much higher trust. It's sort of like the end of asymmetric information. And you can map that from, you know,
01:36:49
Speaker
food safety to product safety to OSHA and workplace safety. There's other parts of government that maybe just rendered completely obsolete. Once we have self-driving cars that are a thousand nicks more safe than humans, do we need a National Highway Traffic Safety Administration?
01:37:08
Speaker
Once we have sensors that are privately owned everywhere and can model weather patterns better than the National Oceanic Administration, do we need a national weather service or could we bootstrap that ourselves?
AI in Drug Approval and Internet Security
01:37:21
Speaker
And then once we have AI accelerated drug discovery,
01:37:27
Speaker
Do we want to rely on the FDA to be a kind of choke point to do these sort of frequentist clinical trials that are inherently slow and don't capture the kind of idiosyncrasies and heterogeneity that could be unlocked by personalized medicine? Or do we move to an alternative drug approval process that is maybe non-governmental but much more rapid and much more personalized?
01:37:55
Speaker
That's the overall picture. I'll just run through the timeline here, picking up on some of your comments that I thought were especially interesting. This is in 2024 to 2027. You write that the internet will become balkanized and it will become more secure and more private in a sense. Why does that happen?
01:38:19
Speaker
We're already starting to see this a little bit, right? Once people realize that the data that's being generated on Stack Overflow or Reddit or whatever is valuable for training these models, suddenly everyone's closing their API. And consequently, Google Search and the Google Index have sort of started to degrade already. So I think that will continue for the kind of privatization of data reasons. Then we also think about how websites are going to handle
01:38:50
Speaker
sort of the growth of bots and catfishes and catfish attacks and cyber attacks and so forth. It makes sense that we're going to move from a sort of open, you know, everything goes kind of Twitter-esque platform to things that are much more closed because they require human verification and identity verification to sort of build the trust that you're talking to other people and not deepfakes.
01:39:18
Speaker
And then medium term, again, over this sort of 2024, 2027 horizon, you could also start to see the emergence of intelligent malware, sort of modern AI native cyber attacks that could be devastating to legacy cybersecurity infrastructure in a way that I talked about could harken back to the famous Morris worm that in the late 80s basically shut down the early internet. Like they literally had to partition the internet
01:39:48
Speaker
and turn it off so they could rid the network of the worm. So for all those reasons, I think you start to see the internet vulcanize and then particularly at the international level, we're already starting to see sort of the semiconductor supply chain become a critical part of national security. The growth of the Chinese firewall, the European Union is going to have to have their own quasi-firewall and they kind of already do with GDPR.
01:40:13
Speaker
the EU-AI Act, and so the kind of nationalization of compute and telecommunications infrastructure that will take off once people understand both the security risks and the value prop of owning the infrastructure for the AI revolution.
AI Model Alignment Challenges
01:40:28
Speaker
In 2028 to 2031, you write about alignment turning out to be easier than we thought with the increasing scale of the model. That was somewhat surprising to me. Why does alignment turn out to be easier? Part of this is imagining a scenario where alignment is easy, so we can talk about what happens if alignment is easy.
01:40:53
Speaker
But I think there are reasons to think that the classic alignment problem will be easier than people think. I think that some of the early intuitions about the hardness of the alignment problem were rooted in a view of maybe AI turns out to be a very simple algorithm rather than like a deep neural network that achieves its generality because of its depth. Clearly, the kind of value, I forget what Eliezer Joukowsky used to call it,
01:41:23
Speaker
There's like a value alignment problem where how do we teach the model our values? That part of the alignment problem seems trivial now because our large language models aren't like autistic savants. They're actually incredibly sensitive to soft human concepts of value and context. They're not going to have a... The paperclip maximizer sort of monkey paw kind of threat models don't really make sense in that world.
01:41:52
Speaker
But there's a difference between the output of the model and the weights or what the model has learned. Just because a model can say the right words that we wanted to say, but what has it actually learned? We are not entirely sure. It has learned to satisfy human values to some extent, but has it learned to want to comply with human value?
01:42:17
Speaker
It out of distribution sort of yeah in other domains and in a deep sense. I'm not I'm not sure about that. I agree. So so I'm sort of just laying some of my groundwork for to explain my priors on this. No, I agree. Like, you know, reinforcement learning from human feedback is
01:42:34
Speaker
not alignment in the same way that you could argue that the co-evolution of cats and dogs with humans led to a kind of reinforcement learning from human feedback in their short run evolution that made them appear as if they experienced guilt and shame and these human emotions when in fact they're just sort of a semi-lacker of those emotions because it means that we'll give them a treat.
01:43:01
Speaker
But I've done plenty of episodes on deceptions in these models and so on. We don't have to go through that. But I just wanted to point out that, yeah, maybe there's some complexities there. So my first prior is that these models aren't autistic savant the way they might have been. The second is going back to universality. Well, it is true that
01:43:24
Speaker
that is possible through reinforcement learning from human feedback, for example, that you're not selecting for honesty, you're selecting for a deep pick of honesty. But in the bigger picture,
01:43:38
Speaker
The intuition that these models are converging or convergent with human representations should give you some confidence that they're not going to be as alien as we think they will be. It's also useful input for thinking about interpretability. There's some recent work showing discussing representation interpretability where instead of trying to interpret individual neurons, you interpret collections of neurons and
01:44:05
Speaker
and circuitry through human interpretable representations. And one of the lessons of universality is that some of these high level human concepts like happiness or anxiety, these seem like vague psychological abstractions that there's no way they can correspond to the micro foundations of the way our brain works. But in fact, they may actually be very efficient, low dimensional ways of talking about what's happening in our brain.
01:44:32
Speaker
And then the third thing is I think that I just have seen, you know, my sense is that the work on interoperability is actually making some good progress. You know, whether it can scale is another question, but I think we'll get there. In my timeline, I talk about sort of AGI level models within the human emulator plus domain. I do later on talk about like super intelligence emerging maybe in the 2040s. And that's another story, right? And so I think some of the stuff
01:45:00
Speaker
maybe goes out the window if we have models that are bigger than all the brains combined and have strong situational awareness. But I don't think that happens this decade. Certainly not with the current way we're building these models. The way we're currently building these models I think comes much closer to a simulacra of the human brain.
Future of Robotics and AI
01:45:20
Speaker
Got it.
01:45:20
Speaker
In twenty thirty six to twenty thirty nine you talk about and robotics being solved to the same extent or maybe even in the same way as as we are now solving a language that would i found that super interesting explain to me why.
01:45:37
Speaker
Why would robotics suddenly become much easier? Roboticists have been fighting for decades to get these models to walk relatively unencumbered. It's been an uphill battle. Why can we solve robotics in the 2030s? This may end up happening sooner than I'd project. But
01:46:02
Speaker
If you look at LLMs, one of the stylized trends with large language models is that natural language processing went from being this study of how to make machines understand language, went from being a dozen different sub-disciplines, people working on parsing, people working on syntax, people working on semantics, people working on summarization and classification. These are all different directions, research directions.
01:46:29
Speaker
And then along comes transformer models and it just supplants everything and LLMs can do it all. And I think robotics is sort of still in that ancient regime where a lot of what Boston Dynamics does is ad hoc control models, analytically solvable differential equations, different kinds of object recognition modules and
01:46:53
Speaker
and control action loops and so forth. And so it's still in that early NLP phase where they have 12 different sub-disciplines and they're sort of mashing them together. And of course, you get something that's not very robust. I think we're already starting to see that paradigm shift to end-to-end neural network train models like Tesla, for instance. I think one of the reasons why Tesla cars had a sort of
01:47:21
Speaker
a temporary decline in performance was because they were undergoing the transition from these ad hoc lane detectors and stop sign detectors and stuff like that to a fully end-to-end neural network transformer-based model. And that turned out to be a much more robust way to train the model because
01:47:39
Speaker
you know, stop signs look different in different countries and like maybe stop sign isn't the thing you care about really, so on and so forth. And so I think the transformer sort of scale deep learning revolution is only now coming to robotics and people in that field have are a little bit cynical because they're used to
01:47:58
Speaker
relatively small RL models, thinking that the fit with actuators and some of the hardware is a really challenging problem, and also believing that we don't have the data sets for it. But then you look at this recent RoboDog that you may have seen on Twitter, a fully open source robot model for a Boston Dynamics-style dog. It was trained on H100s, 10,000
01:48:28
Speaker
human years of training and simulation, and then some fine tuning on real world data. And they have a very robust robot control model that you could plug into all kinds of different form factors and have something that can, you know, hop gaps and climb stairs and do all the things that Boston Dynamics robots don't do very well outside of their distribution.
01:48:52
Speaker
Do you think we'll have a general purpose algorithm that we can plug into basically arbitrarily shaped robots that can then navigate our apartments or our construction sites or maybe our highways? That's an interesting vision. Why is it that we achieve this level of generality?
01:49:11
Speaker
If you look at humans, humans are very good at if we've suffered amputation or you have to go through physical therapy and it's not easy necessarily, but humans are able to adapt to different kinds of physical
01:49:27
Speaker
layouts of her body. And I think there will be a trend towards unified robotic control models that aren't super tailored to two legs and two arms and so on and so forth. Once you've installed it through a little bit of in-context learning or fine tuning or reinforcement learning, adapt to that particular form factor. And this will parallel the pre-trained foundation model
01:49:56
Speaker
paradigm that is currently taking place in LLMs where you have like the really big foundation model that can sort of do everything reasonably well, and then you can fine tune it beyond that. If we get to the 2040s in your timeline, you talk about massive amounts of compute being available. You talk about post-scarcity in everything except for land and capital. And then you also talk about the development potentially of superintelligence at that point.
Superintelligence and Governance
01:50:25
Speaker
What happens there? Who is in control of the superintelligence, if anyone? Yeah, this is sort of where I start to get a little bit tongue in cheek. But first of all, I talk about how I tend to think that once we have exascale computing, and I think DOE just
01:50:46
Speaker
built their first exascale computer, maybe it was a private company, but we have one exascale computer in the world. By the 2040s, they'll be commonplace, and if we're ever worried about controlling the supply of GPUs, I don't know exactly how much compute will be on our smartphones, but it will definitely be possible to train a GP5 model from your home computer.
01:51:12
Speaker
And so any kind of AI safety regime that we build today that doesn't take into account that falling costs of compute will probably break down. And therefore, you know, amid this broader sort of fragmentation of the machinery of government, the state, I expect more and more government functions to be offloaded into, you know, basically private cities, HOAs, gated communities,
01:51:38
Speaker
And likewise with the internet, I expect more and more of our sort of permissioning regime for new AI models and deployment to shift to the infrastructure layer where telecommunication providers will be monitoring network traffic for unvetted AI models and so forth. And we'll have like Chinese-style firewalls that are specific to a particular local area network. And at that point, the world looks
01:52:05
Speaker
The United States, where this takes place, looks more like an archipelago of micro jurisdictions. I tend to think that a post-scarcity political economy looks a lot like the Gulf States, Gulf State monarchies, because Gulf State monarchies are basically living post-scarcity. They have a spigot of oil they can turn on, and then they can go build mega projects in the desert, and they have infinite labor because they can just import guest workers. And so you end up with this
01:52:35
Speaker
But if we can't have a Gulf state monarchy in the United States, instead we have a bunch of micro monarchies dotting the country. So I jokingly say, no, who's going to stop the free city of California that's home to all the trillionaire ML engineers and tech founders from the decade prior from plugging in their humanity-sized supercomputer into a fusion reactor and turning it on?
01:53:03
Speaker
Yeah, and this is really your kind of end point of the discussion or your main point of institutions being eroded and then afterwards being unable to respond to strong AI. Yeah. And leading up to this, it sounds like a scary dystopian type of thing. It doesn't have to be, right? Uber is not dystopian. Airbnb is not dystopian. Private airports in other countries are way better than the public
01:53:33
Speaker
airports in the United States. So privatization and the sort of techno-feudalist paradigm doesn't have to be bad, but what it is is more adversarial, right? And, you know, people have sometimes speculated, you know, what do we have? Did the crumbling of the Roman Empire was a kind of prerequisite to a renaissance?
01:53:52
Speaker
Because it allowed for these principalities to compete and to get the Florentine creativity and so forth. I think the next couple decades could similarly be a renaissance for science and technology and for understanding the world. But it's partly a renaissance because we'll be moving into a much more competitive adversarial world where these city-states and so forth will be hard to coordinate.
01:54:22
Speaker
And so to the extent that there are still these meta risks where we would value some large scale intra and international coordination, like peace treaties and so forth, the disintegration of the United States where this revolution is occurring would be bad for that. You talk about or you hint at an alternative path. What we've been talking about, your timeline here is the default path.
01:54:49
Speaker
You had to add a path where we have something you call constraint leviathan. What is constraint leviathan? It's limited government, right? So this is, this is, um, uh, Jaron also moguls refer from the narrow corridor. And if you, if you trace the, the rise of sort of, uh,
01:55:08
Speaker
of what we associate with liberal democracy. It is part of a particular technological equilibrium, in particular an equilibrium that favored centralized governments with impersonal rule of law and impersonal tax administration and so on and so forth.
01:55:25
Speaker
So we associate today with libertarians with being anti-government, but the basic idea of liberalism is actually associated with a strong government, a strong and personal government that can impose the rule of law. And so if we want to maintain that kind of equilibrium in a world where AI is diffusing on the society level faster than it is on the state and elite level,
01:55:48
Speaker
then we want to accelerate the diffusion of AI within government. And there's obviously lots of low hanging fruit. We talked about how bureaucracies are basically fleshy APIs. Even today, I have a friend at the FTC, the Federal Trade Commission,
01:56:04
Speaker
They have a 30-person team that is part of the healthcare division, and they're in charge of policing the entire pharmaceutical industry in the United States for competition. His day job right now looks like manually reading through 40,000 emails that they subpoenaed from a pharma CEO. And today, you could take those emails and put them into Clod2 or something like it with a big context window and ask, find me the five most egregious examples of misconduct.
01:56:34
Speaker
And it would do that. It might not be perfect, but it's a hell of a lot more efficient than reading through them manually. And obviously, big law is going to be doing that. And the pharma CEO and his personal attorneys will be doing that conversely. To maintain our state capacity in the face of AI is to run in this arms race. And you can kind of liken it to an evolutionary biology they called the Red Queen dynamic.
01:57:02
Speaker
which comes from Alice in Wonderland, where the Red Queen tells Alice that sometimes you need to run just to stay in place. And so I think our government needs to be adopting this technology as rapidly as possible so that they can basically tread water. And that means both diffusing it in existing institutions, but also being open to radical reconfigurations that the machinery of government and addressing some of those firmware level
01:57:27
Speaker
constraints that we talked about, whether it's the lack of a national identification system or the outdated information technology infrastructure or the accumulation of old procedural kinds of methods of governance.
Government-Led AI Initiatives
01:57:42
Speaker
A focused way of doing this is what you've called for in a Politico article, which is a Manhattan project for AI safety. First question here, would it be better to call it an Apollo project as opposed to a Manhattan project? I mean, the Manhattan project created some pretty dangerous weapons, whereas the Apollo project might have been more benign. I mean, what the Apollo project and the Manhattan project had in common is that they came from an era of US government where we still build things.
01:58:10
Speaker
where we still had competent state capacity, where we still had a lot of in-house expertise and we weren't saddled with all these constraints. So today, we couldn't go to the moon in 10 years. NASA couldn't. SpaceX can. And so our modern
01:58:27
Speaker
Apollo projects are being done by the private sector through competitive contracts. And so one of the messages of my piece on the Manhattan Project is to say, the reason I make this analogy is not just because AI is a Oppenheimer-like technology, but also because responding to it will require a throwback to those kind of institutional forms where we gave the people at the top a lot of discretion
01:58:51
Speaker
and gave them an outcome and let them solve for that outcome without having much of prescriptive rules about how to solve for that outcome. The second reason to make the analogy is OpenAI and Nithropic, they both have contingency plans
01:59:09
Speaker
for developing AGI and having a runaway market power. In the case of OpenAI, it's their nonprofit structure. In the case of Anthropic, it's their public benefit trust, where they both are envisioning a world where they could potentially be the first to build AGI and become basically trillionaires. And so at that point, they need to become basically governed by a nonprofit board.
01:59:32
Speaker
At that point, that's not where progress ends, obviously. There's going to be continued research. It would make sense for the US government to step in and say, let's do this as a joint venture. We're no longer competing. In fact, the basic structures of capitalism and market competition are starting to break down. Let's just pull this together into a joint venture, study the things that require huge amounts of capital that the private sector doesn't have, but the government can.
02:00:00
Speaker
The US government spent $26 billion on the Manhattan Project and today's dollars. When you think about the financial resources of nation state actors to put behind scaling, it's nothing like what Microsoft or Google have. When's our first $200 billion training run? What kind of things can come out of that? I think that's something that you want to do with the Defense Department's involvement and working with these companies in a joint way through secured data centers
02:00:30
Speaker
and doing kind of like gain of function style research that really is dangerous and more like, you know, more Manhattan Project than Apollo Project.
02:00:40
Speaker
What would be the advantages here? We would be able to slow down capabilities research and spend more of the resources on, say, mechanistic interpretability or evaluations or alignment in general, because now the top AI corporations have kind of combined their efforts under one government roof. Yeah. And in my vision, they're still allowed to pursue their
02:01:10
Speaker
commercial verticals, and I have an extended version of the proposal where I talk about needing biosafety-style categories for high-risk, medium-risk, and low-risk styles of AI that closely parallels what Anthropic recently put out with their recommendations for BSL, Categorization of AI Research. I'm really talking about that BSL-4 lab and beyond style stuff, and some of that stuff
02:01:36
Speaker
Some of it will be to accelerate alignment and durability research to do versions of the OpenAI super alignment project where they're dedicating 20% of their compute to study alignment. Another part of it will be to forestall competitive race to the bottom dynamics so that they can coordinate and not violate antitrust laws.
02:02:01
Speaker
And then the third thing is sort of the gain of function stuff that we really only want to be doing with very strict oversight compartmentalization, kind of pooling of talent and resources so we can share knowledge on alignment and safety. But then also because government has this huge spending power relative to the private sector, anytime you build a supercomputer, you're basically borrowing
02:02:28
Speaker
the future. You're trying to see what the smartphones 20 years from now will be capable of. And so if we want to get ahead of the curve and see where scaling is leading, then I think governments are really the only actor that can waste a bunch of money basically scaling up a system and seeing what comes out of it.
02:02:48
Speaker
Yeah when we talk about gain of function research in in a it's it's an analogy to the gain of function research that's done on viruses in in bio labs but done for a models and this could this could be. Experimenting with creating more agent like models or.
02:03:06
Speaker
inducing deception in a model and planting it in a simulated environment, seeing what it does, or enticing it to acquire more resources, but again, perhaps if this is even possible, in a safely constrained simulated environment.
Safe AI Experiments
02:03:24
Speaker
And this is the type of research that we could do in this Manhattan Project, this government lab, because we would have excellent cybersecurity and secure data centers and the combined efforts of the most capable people in AI research.
02:03:40
Speaker
If you've watched Oppenheimer, the movie, a lot of that revolved around suspicions of communist spies and so on. And we really don't have great insight into the operational security of the major AGI labs. And that's something that bringing in-house of the defense department, they would necessarily have to disclose everything they're doing, but also hopefully beef up their operational security.
02:04:08
Speaker
Yeah, they're kind of stuck with a startup mindset, but they're not developing a startup product. They're developing something that, in my opinion, it could be more dangerous than the average startup. Yeah, and Daryamade has said as much that we should just assume that there are Chinese spies at all the major AI companies and at Microsoft and Google.
02:04:29
Speaker
When we think about gain of function research in AI, how do you think about the value of gaining information about what the models can do and what the models can't do versus the risk we're running? It would be a tragic and ironic death for humanity if we experimented with dangerous AI models to see whether they would
02:04:52
Speaker
Destroyers and then we hadn't constrained them probably and they they actually destroyed us. So how do you how do you think of that trade-off between gaining information and avoiding lab leaks? Yeah, hopefully lab leaks are less likely than in the biology context where you know getting a little bit of blood or urine on your on your shoes as you walk at the door and
02:05:17
Speaker
It's a difficult thing to talk about in part because we just went through a pandemic that very probably was caused by a BSL-4 lab leak. One saving grace is that AI models don't get caught in your respiratory system.
02:05:35
Speaker
Um, uh, and so hopefully there's forms of compartmentalization that are much, much more robust than in the biology context. And to the extent that this research is going to be done anyway, you know, it would be much better to move it offsite and hopefully in a way that, you know, facilities are air gapped and so forth rather than, you know, what, you know, Microsoft is doing right now that Microsoft just recently announced their auto gen AI, which are sort of agent based models. Um,
02:06:03
Speaker
very similar to like auto GPT, but like that work. And they're doing this through a creative commons, totally open source framework. All this, all this capabilities work as gain of function research where we draw the line between doing things that are intentionally dangerous or doing things that are dangerous, but we were kind of pretending that they're not, um, is, is hard. Uh, I do think there's, and Paul Christiano is also agreed with this sort of threat models that we would be valuable to be running.
02:06:33
Speaker
in virtual machines to see if the AI develops awareness, situational awareness, and tries to escape if it escapes into a simulated world that we built for it.
02:06:43
Speaker
Okay, let's end by talking about a recent critique of expecting ADI to arrive pretty in a short
Market Responses to AI Developments
02:06:53
Speaker
time. This revolves around interest rates. And I guess the basic argument is, or the basic question is, if ADI is imminent, why are real interest rates low? I can explain it, but you're the economist, so maybe you can explain the reasoning here.
02:07:10
Speaker
So it's really a question of how efficient are markets and how much foresight do markets have. We're coming out of a world of very low interest rates, of ultra low interest rates, near zero interest rates. And one way to think about that is there's a surplus of savings relative to investment.
02:07:26
Speaker
And so one of the reasons interest rates have been in secular decline is because populations are aging. And so old people have a huge amount of savings built up. And meanwhile, we're going through this technological stagnation. So the amount of savings relative to the amount of profitable investments was out of whack. And so that pushes interest rates down. In a world where AI takes off, it's a world where we have enormous investment opportunities, where we'll be building data centers left and right, and we can't do it fast enough, where there's new products,
02:07:56
Speaker
new commercial opportunities left and right. And so you would expect in that world where the singularity is near, so to speak, to be one where the markets begin forecasting rapidly rising interest rates because the savings to investment balance is starting to shift.
02:08:12
Speaker
In addition, there's a long run stylized fact that real interest rates track growth rates. If GDP growth takes off, you'd also expect at least nominal rates to also take off. Some have argued that looking at current interest rate data, like the 5-year, 10-year, 30-year treasury bonds, that the markets are not predicting
02:08:36
Speaker
AGI. The two responses to that are, one, first of all, interest rates are up quite a bit. Nothing's model causal. There's lots of confounding factors. Is this, to some extent, the markets anticipating an investment boom? Maybe they're not anticipating full AGI, but they're seeing the way LLMs are going to impact enterprise and baking some of that in.
02:09:02
Speaker
And then the second piece would be, okay, to the extent that they're not pricing in AGI, how much foresight to markets have anyway? Before we discuss market efficiency, I just want to just give a couple of intuitions here. If AGI was imminent and it was on a lion's say and it would destroy the world in five years, well, then it doesn't make a lot of sense to save money.
02:09:27
Speaker
Similarly, if AGI is about to explode growth rates, well, then a lot of money will be available in the future. You're about to become very rich, so it doesn't make sense to save a lot now. And the pool of available savings determines what's available for lending, which determines interest rates. But let's discuss whether markets then are efficient on this issue or to what extent they're efficient.
02:09:57
Speaker
This is the efficient market hypothesis, which comes in strong and weak forms. The strong form of the efficient market hypothesis would say that markets aggregate all the available information and are our best point estimate of anything we care about. The weaker form, which I think is more defensible, is that markets can be wrong, but they can be wrong longer than you can be solvent.
02:10:22
Speaker
So you can try to short a company like Herbalife. Famously, there was a big short position on that because Herbalife looks like it's a multi-level marketing Ponzi scheme, but yet the hedge fund that did that lost several billions of dollars before they ended their position because the markets stayed irrational longer than they could stay solvent. The second factor is the weaker versions of the Efficient Market Hypothesis are based on a no arbitrage condition.
02:10:48
Speaker
They say markets are efficient only insofar as you can arbitrage an inefficiency. You look at some prediction markets, for example, they predict it.
02:11:02
Speaker
they'll often have very clear inconsistencies across markets that look like they're irrational. But then you realize, oh, I can only make like $7,000 total on the website and there are transaction fees and there's work involved. And so if the market isn't very deep or liquid, there may be inefficiencies that exist, not because the market's inefficient, but as efficient as it can be under the circumstances. And when it comes to AI,
02:11:33
Speaker
How do you arbitrage? I've been thinking for a while now that Shutterstock, their market cap should be collapsing, right? Because we have image generation that is proliferating. And yes, people will make the argument that Shutterstock has all this image data. They could build a better image model. Maybe it seems like it's cannibalizing their business. It's sort of turning a moat into a commodity. And yet Shutterstock's market cap has basically held constant throughout this recent
02:12:04
Speaker
rebirth of image generation models. What if you borrow a lot of money cheaply and then put it into an index of semiconductor stocks or just IT companies in general, even just the general S&P 500 today? Would that be a way of arbitraging this AGI forecast?
02:12:23
Speaker
Yeah, I would say if you have short timelines, you should be putting a lot of money into equities. This is not financial advice. Right. And I mentioned earlier that Paul Christiana has said in interviews that he's twice levered into the stock market. He basically owns a bunch of AI exposed companies and he's borrowed enough money to double his investments. So that's putting your money where your mouth is. When you look at market behavior,
02:12:51
Speaker
over the long stretch of time. Markets didn't anticipate the internet very well. There was a short run bubble that led to a boom and bust of .com stocks. But in terms of the real economy, the internet just kept chugging along and kept being built out. And eventually, a lot of those investments ended up paying off even if you rode through the bubble. Markets are made of people. Some of the biggest capital holders in the markets are institutional investors.
02:13:20
Speaker
pension funds, life insurance companies, governments like the Saudi Arabia or the Norwegian pension fund. And often these are making safe bets. They're not taking very heterodox views on markets. And so as a result, markets can be a little bit autoregressive. They're a little bit biased to the past, past this prologue.
02:13:49
Speaker
and prone to kind of multiple equilibria where there's two prices that Shutterstock can be. Shutterstock could be a $50 stock or it could be a $0 stock and at some point the market will update and will wander go through like the great repricing and all these asset prices will flip in relatively short order. The efficient market hypothesis has to be false or else we wouldn't have Silicon Valley.
02:14:12
Speaker
We wouldn't have founders. We wouldn't have Elon Musk. So I would just say the markets are wrong. And partly they're wrong because to be right would require having a bunch of relatively bespoke and esoteric priors about the direction of the technology that are only now just sort of percolating into the mainstream. And that the big capital allocators can't really respond to because they're risk averse.
02:14:41
Speaker
Exactly. No, that doesn't mean Renaissance technologies won't respond to it, but they're not going to move the market. Samuel, thanks for this conversation and I've learned a lot. Thank you.