Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Samuel Hammond on why AI Progress is Accelerating - and how Governments Should Respond
4.9k Plays3 months ago
Samuel Hammond joins the podcast to discuss whether AI progress is slowing down or speeding up, AI agents and reasoning, why superintelligence is an ideological goal, open source AI, how technical change leads to regime change, the economics of advanced AI, and much more.
Our conversation often references this essay by Samuel: https://www.secondbest.ca/p/ninety-five-theses-on-ai
Timestamps:
00:00 Is AI plateauing or accelerating?
06:55 How do we get AI agents?
16:12 Do agency and reasoning emerge?
23:57 Compute thresholds in regulation
28:59 Superintelligence as an ideological goal
37:09 General progress vs superintelligence
44:22 Meta and open source AI
49:09 Technological change and regime change
01:03:06 How will governments react to AI?
01:07:50 Will the US nationalize AGI corporations?
01:17:05 Economics of an intelligence explosion
01:31:38 AI cognition vs human cognition
01:48:03 AI and future religions
01:56:40 Is consciousness functional?
02:05:30 AI and children
Recommended
Transcript
Introduction to Guests and Podcast
00:00:00
Speaker
Welcome to the Future of Life Institute podcast. My name is Gus Docker, and I'm here with Samuel Hammond. Sam, welcome to the podcast. Thanks for having me back. Great. All right. Maybe for the listeners who don't already know you, you could introduce yourself. Sure. My name is Samuel Hammond. I'm an economist, a think tank in Washington DC called the Foundation for American Innovation. I have a boutique tech policy shop. I mean, prior to that, I used to run social policy within within the Scanlon Center. Great.
Is AI Progress Plateauing?
00:00:28
Speaker
All right.
00:00:28
Speaker
Let's start with a very important question. Is AI progress plateauing? I certainly don't see any signs of that. You know, I think people have a bit of impatience, right? There's a lot of worries about, you know, somehow GPT-4 is leading to some kind of hard takeoff and we had six months to prepare.
Hardware Bottlenecks in AI
00:00:48
Speaker
Maybe that was a case of the boy who cried wolf. But the fact of the matter is that these, that scaling continues. We've had a hardware bottleneck.
00:00:57
Speaker
Last year Nvidia produced maybe half a million H100s. This year they're producing on the order of 2 million H100s and starting to roll out their next line of chips. So that bottleneck is just starting to alleviate. And as Kevin Scott from Microsoft said recently on ah an interview, you know, all signs are that we're still on some kind of exponential curve, but it's one that we only get to sample every few years as new hardware. And we've been stuck on a temporary plateau because GPD4 of course was the kind of, you know,
00:01:26
Speaker
2022 technology. And what we're seeing instead is a kind of catch up on smaller models that are now as performant as GPT-2 with a half or quarter of the size. But those algorithmic improvements are also being fed into bigger models. And when GPT-5 comes out later this year or maybe early next year, we'll we'll actually see for the first time yeah what looks what what what
Scaling and Release of AI Models
00:01:47
Speaker
looks like a sample from that from that higher level of scale. So this might just be a question of us being in between models and people getting impatient. Yeah, exactly. you know that The way neural scaling laws work is, on some level, there's always diminishing returns. like they're They're log normal. So to get a linear improvement in model performance, you do have to grow the data and compute in parameters exponentially. But we still have Moore's law and and Moore's law like trends running in the background.
00:02:18
Speaker
with algorithm improvements, hardware improvements. And so the the exponential computing needs that we need are in some sense more linear than meets the eye. And so we're not we're not on a true exponential where we're on some hyperbolic curve where every model is is ah going to be you know as big a leap as GPT two to three or three to four was, but we are adding more nines to the decimal place. right and that And that actually may contain a lot of really valuable information from the perspective of you know getting these models to work better as agents, to be more reliable, ah to cross certain thresholds for you know real life use cases. And that is something that all we only get to do every few years because of these hardware constraints.
00:02:59
Speaker
Yeah. And so the question is also how far will this take us? You write about how LLM scaling will converge to human level intelligence. Is that because simply that is that a a limit of the training data? Because we only have human produced data?
Human-level Intelligence and AI Self-play
00:03:15
Speaker
Well, I think current methods based on training on human data will converge to human level intelligence, of course, not like, you know, in some ways, the models will still be superhuman because they have know, vast knowledge that no one human has and are able to sort of be a superposition of different personas and so on and so forth and and think much faster. Like no no human can just ah sit down and regurgitate a thousand word essay within seconds. But in terms of the core competencies of the model, you know, the models are being trained on human data. So on some level, they're being bound by what
00:03:50
Speaker
by the abilities and intelligence and reasoning capacity that is latent in that human data. That being said, there's obviously a potential for techniques to bootstrap ourselves out of that bound, right? In the same way that AlphaZero, the chess playing out AI, and through unsupervised learning and soft play, was able to not just beat humans, but become several standard deviations better than any human could ever be. And so how would self-play take us beyond the current training paradigm?
00:04:20
Speaker
Well, we're seeing early signs of this already,
AI in Math Competitions and Reasoning
00:04:23
Speaker
right? So Alpha geometry ah recently came out to a paper that in DeepMind that scored near the top of the distribution of the international math Olympiad made a silver medal, you know, and the way that that model worked was sort of pairing, you sort of a neuro symbolic approach, right? Pairing the kind of a common sense understanding of Gemini and our language model with search and the kind of Alpha zero style ah reinforcement learning ah model over ah formal language to actually verify the mathematical proofs. And this this is sort of similar, I think, to how humans tend to... How humans are inventive, right? Humans don't just sit down and you know discover general relativity just by thinking harder, right? We have, for intuition, certain analogies. In the case of Einstein, his moment was the equivalence principle that you know being in an elevator that's falling is sort of the same as gravity. And so how does that
00:05:17
Speaker
hash out mathematically in terms of space-time. So, you know, large language models have that kind of analogical reasoning ability, that kind of semantic understanding, and that they're therefore able to make sort of really good guesses and generate ideas. And then those ideas have to then be verified, right, and checked against some ah ground truth. And so the models we have today are sort of pairing that in tacit understanding with the ability to then search down different branches,
00:05:43
Speaker
check and verify, and then recycle recycle that knowledge back and loop iteratively. And then you know what what has happened is you know some of these models have already broken new ground, right?
Verifying AI Ideas and Structuring Environments
00:05:54
Speaker
Fun search from last year made progress on, you know, four mathematical open questions. And so these models are seem capable of generating a degree of novelty when they're paired with reinforcement learning techniques and search techniques. Now, the question is then sort of the Holy Grail is how do you set that up in a way that could be fully unsupervised and automated and also applied to a more general reward function, right? And this is sort sort of the
00:06:21
Speaker
the The problem with general intelligence, we know how to make a ah Dota ah playing AI that can beat pros at a very complex real-time strategy game. um But that's because there's well-defined rules. Real life is much more complicated. There's no one win condition. you know ah Evolutionarily, it was you know reproduce and survive. um But we can't really use that as a reward condition for these models. And so you know the the question is, how do you structure play environment such that the reward signals hold the model towards a more general form of intelligence rather than a more narrow. And is this related to agents that you mentioned earlier also? Do you think do you think to go beyond LLMs, we need to to move towards agency somehow? Yes. And and partly, this is also prerequisite to unlocking a lot of economic value. And so there's a a natural a tendency towards this area of research. you know The way LLMs are trained today is off of large corporative texts,
00:07:18
Speaker
a lot of internet text, and a lot of that is sort of final outputs, right? You see the essay, you see the Wikipedia article, you don't see all the intermediate steps that went into writing that essay or the Wikipedia article. um And so the kind of data we have are great for sort of zero-shot writing an essay, but not for the kind of incremental chain of thought that goes into producing something new. I mean, on that note, how difficult would it be to screen record office workers for many thousands of hours and then training on that data? Or is that is that, I mean, maybe that's too naive, right? But something where you're gathering data about the processes leading up to a final output product and then using that as training data.
00:08:00
Speaker
Yeah, that's why Microsoft released their co-product laptop. you know all the All these new products come with optional sort of recall features that will record your screen and pass it through a multimodal model to see what you're doing and maybe you know sort of like a supercharged paper clippy.
Data Collection and AI Advantage
00:08:17
Speaker
But by the same token, that screen recording could be put on every employee at Microsoft's computer or or Google and build an enormous data set of sort of multimodal data that includes recordings of employee screens, keyboard inputs, mouse inputs, with you know the semantic understanding that comes pre-trained into a language model. And that presumably has much higher signal data for how to be a code monkey, um among other things, right? and And so that's where I think in the near run, ah a lot of the energy is going towards just finding new kinds of data sources that encapsulate some of that planning and multitask
00:08:57
Speaker
understanding. what Where are people looking for these new kinds of data? Well, like like like I said, like within you know large corporations, you know therere I think in general, there's a ah realization that all data could be potentially useful and so yeah know everything should be recorded and then we'll figure it out later. and But the same the same could be true of like yeah manufacturing processes. and you know One of the slightly non-consensus takes I have is that the US could have a substantial lead in AI vis-a-vis China, but the fact that China has is the factory to the world and has so much industrial
00:09:34
Speaker
acid knowledge embedded within their factories and manufacturing communities, that that represents a latent source of data for training general purpose robotics. We may develop the general purpose humanoid robotic platform, but that general purpose robot robot doesn't just you automatically a priori know how to assemble complex machine tooling or something like that. it but The point of generality is it's able to then learn that on the fly with some demonstrations or some supplemental data.
00:10:03
Speaker
and So our innovation in bits could in the long run become China's innovation in atoms if they're able to then transfer that into other factories because all the all those workers that they already have could be monitored either through you know GoPro cameras or cameras and on the and the ceiling and absorb all that tacit knowledge and then imbue it into ah into a robot.
00:10:24
Speaker
If you take a a smart person, you you mentioned Einstein, for example, that person will be able to look at all all data created or not look at all data, of course, but look at as some small set of the the output of humanity up until that point.
00:10:38
Speaker
and then make progress beyond beyond the limits of that data. So when should we expect AIs to get to that point where the kind of quality of the training data is no longer the limit?
AI Self-improvement and Meta-learning
00:10:50
Speaker
Here, I think we're beginning to talk about self-improvement, AIs doing AI research, that kind of thing. It's a trillion dollar question. So one of the critiques of large language models, large multiples and transformers in general is that they will always be sort of within distribution.
00:11:06
Speaker
and i think there's but think you know The question is, is that also true of humans? Because you know science progresses, i we stand on the shoulders of giants, right and things progress one funeral at a time. And it's a very infraent incremental sort of gradient descent through knowledge. this you know There have been some leap forward, right Aristotle, perhaps, or or or Isaac Newton.
00:11:29
Speaker
but we tend to be within distribution and then pop po it poke at the edges of the distribution based on the ability to receive sort of exogenous feedback from um reality and feedback from other people in our environment. And so I've long thought that getting to that point where we can bootstrap something that can generally get a a distribution will require some kind of multi-agent framework because those different agents will provide ah exogenous sources of data and you know being able to sort of check and balance the biases of a particular model and push it not get out of distribution in the same way that you know ah a band in the jam session could be thought of as the different ah musicians prompting each other and inspiring new ideas that build on each other and collectively move outside of their distribution because all they' all their distributions combined sort of
00:12:20
Speaker
aren't perfect overlapping Venn diagrams, they have areas where they diverge. Are you imagining a setup in which you have, say, one agent conjecturing a new scientific theory and then and a bunch of other agents critiquing that theory and then they're making kind of incremental progress by criticism and then updating their models? Could we replicate the scientific procedure that humans are engaged in in AIs like that?
00:12:45
Speaker
Yeah, absolutely. I think there's already work being done in this direction, automated hypothesis testing and idea generation. And you know one another way to think about this is you know any scientific question is this enormous search problem. Maybe it's an MP-hard problem where you know the time span of the universe wouldn't give us enough time to just brute force it. And the human mind seems able to do things that seem NP-hard, that somehow we're able to just leap forward in the computation and get to the correct answer. And it seems like the reason that is is because we have a series of inductive priors built into how we perceive the world that reduces the search space dramatically and makes it easier to find that needle in the haystack. And you you see see the same thing with some of these more prosaic applications of ah AI where like
00:13:38
Speaker
NVIDIA is using generative AI to beat the Shannon Entropy bounds on information transmission because we're not transmitting the bits fully faithfully. Instead, we're using the Dr. Priors of what a face looks like and what you know you know whatever every Zoom call has in common to reconstruct the data the data in a way that's vastly more efficient than if we sent it in a sort of lossless way.
00:14:03
Speaker
Yep. Is this meta-learning when you're talking about biases and heuristics and priors and so on? Are we talking about meta-learning there and and what is meta-learning? So yeah meta-learning broadly is this idea that rather than prejudice ourselves into one particular learning framework, we could have a meta-architecture that is able to dynamically discover which learning algorithm is is the best for a particular use case and search the space of learning algorithms one layer up before going down. There's probably something there there, but I still tend to be of the opinion that scenarios where we stumble into like some new architecture that breaks free of within distribution
00:14:46
Speaker
ah limits of current models than just like rapidly takes us to some new heights is is is unlikely. And and that's probably because that doesn't seem to be how humans work. You know, we have some degree of learning how to learn. Maybe a better analogy would be this debate over symbolic versus ah deep learning or AI.
00:15:03
Speaker
you know I think it's a bit of a false dichotomy because when you look at how humans process things symbolically, we don't seem to do it innately. right our our Our mind is a kind of deep learning architecture. You know you can represent if and then statements within neural network, not the most efficient way to do that, ah but you know humans struggle with four digit multiplication and and a variety of other things.
00:15:25
Speaker
What we instead do is we use language to supplement and extend our cognition because language is a sort of serial symbolic mode of processing. It's the the classic Danny Kahneman thinking fast, thinking slow. And is is the is the slow part of our brain really a ah module in our brain or is it actually just language, right? And our ability to externalize thought and use that as a kind of external scaffolding, a kind of extended mind to then give our heuristic ways of thinking a formal structure. And and and that's we see hints of that with the existing AI. There are approaches where we are going to explicitly put in some Python interpreter or some symbolic engine. But maybe with sufficient scale, we don't actually even need that. We just need the ability to serialize our thought and turn it into symbols.
00:16:13
Speaker
Yeah, I guess there's what we're talking about now is whether, say, agency or or reasoning is something that's fundamentally different from from what we're doing when we're training an LAM. And yeah, what what do you think is the answer there? Is agency and reasoning something that emerges out of the training processes we' are we're doing now? Or ah do these things require something something very different? So we don't have make agents, right? Reinforcement learning. ah We have many, many RL agents.
00:16:41
Speaker
And then we we know how to create sort of common sense reasoners, LLMs. And the the open question is how do you combine those two competencies? Agency seems driven by, you know, classic sort of Carl Fristen free energy principle where we are organisms that are trying to model the world and a sufficient degree of complexity and sophistication. You know, we go we move beyond being a mere thermostat that is sort of, you know,
00:17:07
Speaker
thermostatically adjusting the temperature based on the next time step of what the temperature is, to being able to model the world, model the future states of the world, plan for those states, and and then and then we can start speaking of those kind of systems of having sort of beliefs and preferences and being able to take choices. The the challenge, so so you know one one hypothesis would be that agency has been sort of slow to develop within these large language models because we're, first of all, we're not using RL.
00:17:36
Speaker
very much at all, we're using it sort in the post-training phase. And secondly, the data itself is sort of fitting a function for how to generate human text, but not necessarily how to model future states of the world and make decisions within that world.
00:17:50
Speaker
Let's switch gears a little bit and talk about US national security. You write that it is in the US national interest to monitor frontier AI
National Interest and Regulation of AI
00:18:00
Speaker
development. Why is that? Because there's a lot of uncertainty, right? And you know I have my own views. There are people who are much more um certain to either direction. And think I think a good epistemic process is to say, well, when there's as much disagreement, we should be we can't really discount anyone's views.
00:18:17
Speaker
There could be a 10% chance that within the next five years we develop some AGI-like system that rapidly bootstraps itself to super intelligence and is scary, powerful, and potentially decisive for the nation that is the first to get there or or not. Or maybe we do hit some kind of, maybe not a plateau, but more linear rate of progress. And you know within that uncertainty, it's good to have option value.
00:18:43
Speaker
So within the existing regulatory approach that the US s has taken, we haven't passed any kind of comprehensive AI law, but with the White House Executive Order from last year, and that the of Defense has the authority to request disclosures and and safety testing and red teaming from Frontier Model Labs that train models above the 10 to the 26th flop threshold that they set. I mean, those thresholds are viable to be updated over time.
00:19:09
Speaker
And you this is a very de minimis kind of oversight mechanism. It's just saying, you know if you're going to do a massive training run, and if you are one of the three or four labs that have building AGI as an explicit mission, you know maybe just let us know. And you know let us have a sneak peek, because we don't actually know exactly what we're building and what its implications are. And having some kind of break glass in case of emergency option could be released and And I think this is also something that should be acceptable to both the accelerationists, AI skeptics, and AI doomers. It's basically saying, we don't know who's right. Let's at least monitor and ah have some capacity to intervene if things go haywire.
00:19:54
Speaker
Yeah, you're more positive when it comes to monitoring with using compute thresholds than other types of regulation. Why is why is that? Why is using compute thresholds from for monitoring and and then requiring certain disclosures? but Why is that a kind of a minimal form of regulation?
00:20:14
Speaker
Well, there's it's really more practical than metaphysical. And there's been plenty of pushback against compute thresholds per se. And I and i wouldn't recommend putting compute thresholds into statute right because things can change really quickly. And it's actually good to have a degree of discretion in how we define these criteria. yeah But the you know the primary reason for the thresholds was to sort of triage oversight to the biggest players. you know there There may be three or four companies on Earth, at least within our sphere of influence, that could train models ah that pass that 10 to the 26th threshold, at least today. And all those companies, when they're training those models, they're not training the 10 to the 26th music generator, there they're training Frontier
00:20:54
Speaker
generalistic eyes. And so yeah maybe there is some better sort of criteria rooted in scaling, the scaling law literature, you know, this recent paper on ah observational scaling laws that shows through a principal component analysis that there are a handful of parameters and variables within a neural network that predict, reliably predict performance across a variety of benchmarks and and perform better than just a naive sort of flop measure. So you know maybe we should move towards that and in in the longer term. But you know this is this is about moving quickly and having some something that is clear, simple, understand, and achieves what the purpose is, which in this case is not to say that 10 to the 26 is some magic line, but that this is just a a useful short run way to proxy for the Frontier Labs.
00:21:40
Speaker
Yeah, these these numbers are are definitely temporary, as you also mentioned in in your writing yourself. At some point, it will get, what we really care about is model capability. And we can't really measure that directly, so we're measuring training compute instead. But at some point, you can train a very capable model for not that much money or using not that much compute. What do we do then? If compute thresholds are temporary, what replaces these thresholds?
00:22:08
Speaker
Yeah, that's the other side of the ah practical consideration is if it is possible to train a dangerous dual use kind of model with a couple of consumer GPUs, then you know oversight and enforcement is kind of but you know out of the question. right And and there there's really no point in having an oversight regime, much less like a licensing regime for capabilities that people can just build themselves off off the shelf. So in that world, you know yeah One of the intuitions and is that new capabilities will first be detected by Frontier Labs at scale. Carpathi had a recent Twitter thread where he discussed how you know the performance of these smaller models is catching up in part because you have to build the bigger model first to then generate synthetic data that is sort of disentangling reasoning from extraneous knowledge to compress it into a smaller model. We have good reason to believe that the warning shot from new capabilities will first come from the biggest model.
00:23:03
Speaker
And then, you know, a clock turns on that is the countdown to when the similar capabilities become possible through open source or we're much smaller with much smaller compute. And then we move into a totally different world. one what One where the kinds of adaptation and mitigation strategies we'll need are are very different from the world today, where we have this kind of bottleneck, this kind of ah choke point with large compute clusters and a handful of labs. It may mean, you know,
00:23:31
Speaker
having better network security infrastructure, right? you Maybe in the future we'll have the telecom infrastructure that monitors for what kind of AI agents are passing through the network and things like that. But that's that's a far off world, you know maybe by far off I mean yeah know five, 10 years, but it's a world where the kinds of mechanisms will not be you know from on high. It will have to be more decentralized by nature because the technology will be decentralized.
00:23:57
Speaker
yeah As things stand now with the compute thresholds we've mentioned, should we be worried that these will be used for ah these that these thresholds are some form of regulatory capture and that they will they all they will stand as a barrier of of entry to startups in trying to compete with AGI corporations?
00:24:18
Speaker
There are potential versions of regulations that use a few thresholds as part of the regulation that could be regulatory capture. Nothing, nothing like that has happened yet. What has executive order is merely requiring disclosures. It's hard to, you know, the classic theory of regulatory capture is you create a regulatory burden, a kind of fixed cost that everyone must comply with, but the biggest companies, because they have large legal departments that can absorb that cost better and then it shuts out their competition.
00:24:48
Speaker
In this case, the main barrier to entry is can you afford a billion dollar compute cluster and training run? And do you have you know the handful of maybe a few thousand the top ML engineers in the world. That is an inherent barrier to entry. And and so the the notion that a compute threshold is a kind of Trojan horse to regulatory capture seems a little farcical, and especially when you look at the actual views of the companies, right? You know, meta doesn't lock the threshold. There's a lot of pushback from OpenAI and Tropic over SBA 1047 in California, which includes the thresholds. um So as a matter of fact, the
00:25:23
Speaker
companies themselves don't like don't seem to want these. And so it's hard to make that these regulatory capture. That said, there are various forms of a regulatory capture that are more possible. right if If your model has to abide by a variety of ethical safeguards and have really strong refusal systems and so on and so forth, then that is something that Microsoft is much more capable role of of implementing.
00:25:47
Speaker
and close models are much more capable of implementing. And so I would know i would direct my IR there rather than at a mere monitoring capacity. What we're discussing now is something that that's been discussed by effective accelerationists and ah people who want to decelerate AI progress and so on. You you write about how the supposed dichotomy between acceleration and deceleration is actually a false dichotomy. Why is that the case? It's the case and in two ways. First, on a sociological level, effective accelerationists and effective altruists are... It's a narcissism that's all different. you know Five years ago, these are the same people.
Technological Acceleration vs Deceleration
00:26:25
Speaker
And it's a schism within a certain demographic, but they're both kind of rationalists and they both think really hard about these issues. The true believing yaks, you know, believe in super intelligence and think it's, and they just sort of want to know, have they have a different strategy about how to, how to get there and how to handle it. So that's the first point. The second point is that, you know, acceleration versus deceleration sort of imagines technology as this linear process where, you know, there's just,
00:26:55
Speaker
worse technology and better technology. And we should just be racing forward towards better technology. I think this neglects the ways in which technological development and economic development broadly is is know deeply path dependent. You know, why do we have a QWERTY keyboard and rather than some other configuration of keys in our keyboard? Well, this is just a total accident of history. And there are many examples of that through technology. And so, you know, the better mental model for technological development is more more search down branching paths. There are different kind of technology we've developed, they could be developed at different rates or differential investment. And and so this this is where you the DX, the defensive acceleration has sort of come in where they say, you know we are being chased by, you know this is the Vitalik-Peterin metaphor, we're being chased by a bear. And so we have to be running forward. ah We can't can't actually just stop or pause, but there are genuinely two forks in the in the road. there are There are paths that lead to doom and there are paths that lead to utopia. And we have to actually be very deliberate
00:27:54
Speaker
about which technologies are developed first, at which rate, either deployed to make sure that we navigate down that but better path. Yeah. How urgent are these decisions? How important is the next decade, for example? My, you know, and I don't sign a high credence to this, but I think it's probably the pivotal decade. Our ability to steer technology is relatively limited, especially once it has a kind of forward momentum. And so if there is any hope of sort of picking a better path, you have to do it early.
00:28:23
Speaker
point number one. And then point number two is that technological discovery and innovation tends to occur in kind of punctuated equilibria, where we are sort of muddling along or in some kind of technological winter and then suddenly we discover the the one trick that makes it work and there's this flurry of activity and investment and we pick all the hang low hanging fruit. And that's that's sort of what's happening right now. And if there are you know a handful of new model architectures or new tricks or or techniques that will get us to the next stage of AI, that they will probably be discovered within the next 10 years or so, the next 100 years. The AGI corporations have this goal of developing superintelligence, and sometimes they state it explicitly. Maybe talk a bit about how that... We should notice that this is a goal, and we should notice that this is an ideological goal.
00:29:15
Speaker
Yeah, I think it I said this on the last ah time I was on where you know the the the weird juxtaposition of Sam Altman testifying before Congress that they're planning to build a superintelligence that could be here within 10 years and then you know Senator Blackburn from NSC is like, what does that mean for musical royalties? And so there's this ah delta between the conversations for sort taking the companies ah seriously, but not literally. And we should take them literally.
00:29:39
Speaker
you know I was just at a dinner hosted by Anthropic with a bunch of DC policy folks and the representative from Anthropic was basically saying, you know look, how do how do we commute communicate to you guys that you know we're trying to build AGI and this is the goal and we think we're able to do it.
00:29:55
Speaker
and you gotta take that seriously. And then the conversation veered off into you know how to how to think about patents and copyright. So it's it's ah it's a challenge and it's also sort of a, seeing is believing, right? And it's easy because we're sort of in this temporary lull to talk yourself into the view that things are kind of stagnating or we're hitting some kind of ceiling. And that's ah that's a kind of luxury belief that we we don't have for much longer.
Human-like Traits in AI
00:30:21
Speaker
And the ability to sort of see through that requires some kind of prior about how technology works, how machine learning works, how deep learning works in the same way that you know being able to, being concerned about climate change requires having an understanding of, you know, atmospheric physics and some kind of modeling ability to say, well, actually, you know, a few centigrade warming on the on the planet could be the catastrophic um for all these various reasons. And so that is that is really but the the Delta right now where you have the people who are most concerned have some deep intuitions that
00:30:53
Speaker
why this technology will get there eventually and potentially quite soon. and Whereas others who are maybe a little more new to the field are just looking at the latest Shiny product and saying, well, you know this can this ah hallucinated a citation. So how how is AGI actually so so close?
00:31:09
Speaker
Is this a problem that will solve itself? Say we get to the next generation of models and, you know, we have another chat GPT moment where people become aware. If, if progress is, is fast, like we might imagine it to be, it seems to me that, that people will notice along the way and perhaps get more interested in, in the actual and deep problems of this transformation. Yeah. I've, I've kind of likened it to the kind of Tom Hanks got COVID moment. Right. But when, when we'll be.
00:31:39
Speaker
the moment where you know we go from saying, oh, you know COVID is interesting. There's a few cases in Italy. We shouldn't really worry about it yet. And by the way, we have to reauthorize the FAA and pass the NDAA. So Congress has no space to actually deal with the pandemic, to you know but within a matter of weeks passing a multi-trillion dollar relief package and shutting down the country. I don't think things will be quite that dramatic. So it will be more of a trickle.
00:32:06
Speaker
But seeing real demonstrations of agents within particular economic domains where no actual work is being performed and major automation is taking place, then it becomes a lot easier to extrapolate to say, well, we have these agent-based systems that are, you know,
00:32:24
Speaker
performing a human level within a particular domain becomes much easier to imagine why but they could break out into all domains. Makes sense. i I think that if you could email an AI and get some decent work done, or FaceTime an AI, talk to to a generated video and maybe have an AI assistant or something, that those are the innovations that people take note of. And those are some of the things that might change minds, I think. I mean, human psychology is so weird, right? Like,
00:32:51
Speaker
GPD 4.0, the voice capabilities, I think were an aha moment for some people, even though it was a very superficial capability. right and we ah you know Sound waves are you know a four-year transform, so like it should should have been obvious that we could have multimodal models that can speak with annotation and and sarcasm and all all the all the things that are being demonstrated. But because it sort of cuts to things that feel deeply human, it's much more evocative than something that is just like getting a gold medal on ah on some math test.
00:33:21
Speaker
It matters a lot to people when ai take when they pause in a sentence, when they stumble over their words, when they take a ah ah breath, for example. And these are these are small tricks that you can do when you output audio. But it matters a lot, and it and it it matters kind of intuitively, and it matters to me too.
00:33:38
Speaker
um so Yeah, if if we go back to to talking about the the ideological goal of building superintelligence, I think it seems to me that we have an alternative there. And perhaps this is something that that DeepMind is doing, where they're building narrowly very smart models for folding protein folding and for math proofs and for ah using energy in smart ways.
Narrow vs Superintelligent AI
00:34:02
Speaker
um Is that the alternative path you think? is Is that viable? How much can we get from these narrow but very smart models?
00:34:11
Speaker
I think you we can get a lot of what we want, right? We want better medicines. We want, you know, robotics that can automate and drive productivity growth. ah We want ah personal assistance and so on and so forth. So there's a lot we can do ah through the application of NARA models and we can even build NARA forms of ah agency. The example I think of is the AI orchestrators for Amazon warehouses that are kind of agents. They're sort of the collective form brain for all the robots running around the warehouse.
00:34:36
Speaker
the The question is, what does super intelligence actually solve? you know who who was Who is asking for this? know but you know AGI, human level, ah you can see lots of economic value being unlocked. um you know Maybe super intelligence is needed to start colonizing the galaxy, but is there a need to actually rush for that like not technological no milestone?
00:34:59
Speaker
i think you know You can make an argument either way, but it's just ah it's just a observation. ah The race to build a superintelligence is inherently an ideological goal. Even Sam Altman has said that he's ah somewhat tongue-in-cheek credited Eliezer Jukowski for or accelerating superintelligence timelines, because he put the idea up there. He got people thinking about it. and it's all It's all through science fiction, and he kind of gives credence to the ah Nick Land point. the These are arch accelerations that we're, this is yeah hyperstition. We're kind of we're building
00:35:30
Speaker
The sci-fi prophecies were kind of self-fulfilling. The idea in our brain, and now we have to go build the thing. But do we really? And we we do, but do we have to do it as fast as possible? And part of the challenge here is that when you debate our techno optimists, they will equivocate on what AI actually is. You know, they'll point out that we're now able to detect breast cancer from your breath and you know do all these incredible things. And it does seem like pure upside.
00:35:58
Speaker
And I would agree with with all that, you know, having a drug discovery platform that could, you know, and in a matter of minutes or days, generate novel treatments for new diseases in the same way that MRI technology was able to get a vaccine within within a weekend. Yeah, I agree. It's fantastic. And we should have more of that. Yeah, we should have a lot more of that. You know, if you know, as a thought experiment, actually, these companies were saying we we're we're trying to build the ultimate cyber weapon. And and and the mean In the meanwhile, we're also going to make a better search engine. And they just said said that. you know i think I think people will look at it and treat it very differently. It would sound a little bit more reckless than to say, we're just trying to build super intelligence. It's like intelligence, but super. Or or or as trump Trump said, super duper intelligence. It sounds a little more prosaic, but a super intelligence is a powerful cyber weapon.
00:36:52
Speaker
by definition. And I think this is also one of the eventualities here is that when the national security community wakes up to this, it will become militarized. Because at the end of the day, that is the real only use case for superintelligence above and beyond AGI or these narrow platforms.
00:37:09
Speaker
So how inevitable is the progress towards superintelligence? When we are making general progress in hardware, general progress in algorithms, that this is even beyond AI. we're we'rere We're just making progress in the computer industry in a very in various domains. Are we then inevitably making progress towards superintelligence? We're expanding the conditions of possibility right you know what within whatever Compute bounds are required to build a super intelligence. If we have more compute, it becomes more feasible. It still remains, I think, a ideological quest because obviously we can can't build models for everything, right? And we've built better weather prediction models because weather forecasting is useful. We've built audio generation models because we like music. We haven't built, you know, amazing consumer level you know sonar models.
00:38:02
Speaker
and I'm sure there are companies building that for enterprises and so on, but it's not a major area of ah private investment because we're not bats. We use vision visual like but spectrum and and audio within a certain frequency range. And so you know the the the landscape of ah ways we can apply deep learning is is vast. The notion that there's an inevitability to building a autonomous agentic system that is vastly superhuman is some kind of natural outcome or inevitable outcome. I think it's question begging. I see no reason for that other than because it's become a kind of a mind virus that people are now acting on because it would seem super cool. Should we expect the gap between open source models and proprietary models to grow over time?
00:38:55
Speaker
but my my my My bias is to think yes with the proviso that smaller and open source models will still be very capable. There is a kind of trickle down phenomenon, but because of the nature of this log, normal nature of scaling laws, that closed models will, the delta between what you can do with a closed model and an open model will grow over time. And maybe maybe flesh that out a bit. Why is that? um Because, you know, there are many actors in the world that can ah do a million dollar training run. There are many but somewhat fewer actors that can do a $50 million dollars training run, maybe dozens of actors that can do a $100 million dollar training run, very few that can do a billion, fewer still that can do a 10 billion. And you know beyond that, we're talking about you know multilateral projects like CERN that require hundreds of billions of dollars in investment, where not even private companies could compete in that.
00:39:45
Speaker
Yeah, Meta just released the Llama 3.1, which is state of the art in on some benchmarks. Is that a counter example or does it just mean that we because we are in between the release of proprietary frontier models, open source has time to catch up? Yes, so I think it's the latter, right? You know, there's there's very little that goes into training these models that isn't public knowledge, that isn't published in some archive paper somewhere. And and so the knowledge is out there.
00:40:15
Speaker
The main missing ingredient is access to compute and Meta is one of the biggest, if not the biggest owner of H1-100s. So they have this fast computing capacity and really they're one of the the only actors in the world that could open source a model of that that scale. And partly they're doing it because theirre their near run incentives are say, you know, we're a little bit late to the game on AI, we're not an AI company, we're a social media company. um And so we want to be the ones developing an open ecosystem in part because it undercuts our company. Is it actually true that the knowledge of how to train these models optimally is is all out there, is all published? I would expect companies and in front of the AI development to not disclose anything that that that could give them an in an edge in in training.
00:41:00
Speaker
i've I've never, I don't work at an AI lab. I've talked to people, I've talked to people, I've talked to members of the technical staff, so to speak, and you know, they're kind of restricted in what they can tell me. But that that should tell you something in itself, right? That should tell you that they probably have, or maybe they have some, some secrets that they can't disclose. And maybe some of those secrets are relevant to how to train models in the best possible way.
00:41:21
Speaker
Right, but the key he word there is best possible way versus a good enough way. And it seems like many of these are so-called bags of tricks that are being used to you optimize these models on the margin are kind of trade secrets, right? It relates to you know the right way to tokenize the data and yeah maybe the the the the yeah training procedure. like You train on code first and how much code and these these kind of things are not explicitly written down anywhere. there There's no real theory behind them is sort of a trial and error. And so it's more art than science in that sense, right? You know, a lot of people say that they love cloud 3.5 sort of general personality and you know there's questions about, you know, what what was the secret sauce that got them that better personality? And maybe that would be a competitive advantage that they don't want to have a leak.
00:42:08
Speaker
In terms of being able to you know have GPT-4 level capabilities, most of the main benchmarks, none of that is is is secret knowledge. so and And you look at the way diffusion has taken place, you know but're we're now getting open models like a 3.3 and Mistral has a new model as well that are now encouraging on GPT-4 level capabilities that are open source. But again, GPT-4 was a 2022 era model. And so we're sort of in this world where we maybe we can expect to see open models come out that parallel frontier models on it with a two plus year lag. But you know the the classic example of this recent history was the the famous like mystery around Q-Star. Reuters had this report saying that you know some some whistleblowers and within the company were worried about this Q-Star project and and maybe this was AGI achieved internally and all this speculation.
00:42:59
Speaker
and It turned out that it was just an application of chain of thought, prompting and and reinforcement learning that was first published in twenty twenty and a 2022 paper and you know and nothing more. But that doesn't mean that it's still not a very powerful technique. and you know One of the ways that the knowledge is hidden is the fact that there are 500 odd papers being uploaded to the archive every month that are making various claims. and It's very hard to tell what what are the ones worth pursuing, which ones are ah dead ends or or redundant in some way. And so that's that's where the the knowledge really is hidden. Knowledge is all out there, but it's hidden within just the vastness of all the noise around it. And so the fact that OpenAI seems to be using these chain of thought style techniques and so on is not telling us new knowledge, but is it is in a sense sending a signal about what existing knowledge was actually worth prioritizing.
00:43:53
Speaker
And maybe you're also making the point that if you have enough compute, you can kind of just steamroll all of these tricks and hacks and get to the same level of of performance without perhaps having the task of knowledge necessary to implement these these new optimizations. Yeah, precisely. You know, I'd i'd imagine, you know, GPT-5 scale model will be able to emulate the Cloud 3.5 without having, none it but you know, through through a few in-context examples, you know, without having to have their particular back tricks.
00:44:23
Speaker
Metta and Mark Zuckerberg recently came out strongly in favor of open source AI. Do you think that as we scale up the models, they will become more worried, more concerned, perhaps scared about what they're releasing and maybe not release future generations? Yeah, on a purely economic or financial basis, it's hard to imagine Metta releasing a further generation beyond you the $405 billion per model they already have. They go up to $4 trillion.
00:44:52
Speaker
you know then as a shareholder, I start to question their fiduciary duty. right there they're They're burning a lot of capital in doing this and it is a kind of you know the burning of giant pile of money. right And you can only do that for so long, ah just on a purely financial basis. And additionally, I think public companies like Meta, you know it it is not in their shareholders' interest for them to release a model that causes tons of havoc and maybe exposes them to liability and ah much less destroys the world. So I think there are actually quite strong built incentives, especially in a public corporation that has disclosures and variety of other ah transparency duties to being a ah good actor, broadly speaking. And Zuckerberg himself has said, you know, if they
00:45:37
Speaker
develop the model that had dangerous capabilities, they wouldn't release it. This is really not the point he's making. The point he's making is these general purpose foundation models are incredibly useful. ah They're also easily commoditized. And so let's just commoditize them because it will lead to more research, more experimentation, and allow smaller actors that don't have access to large big clusters to but actually have a stake in and AI.
00:46:06
Speaker
Perhaps you could talk about how open source models can be used defensively and how it it could be a could be in the interest of of the public ah say to have open source models that are below a certain capability level out there, perhaps ah for cyber defense. Yes, you don't want to be beholden to a closed model provider if you're building AI into critical infrastructure.
00:46:33
Speaker
Right. You know, it reminds me of the Ring computer interface company that went under and they had a number of patients that had chips in their head and were like, but you know, what what do I do with this chip in my head? The company's out of business. Did that actually happen? Or or is that some sci-fi story? It sounds like a sci-fi story. That actually happened. Okay. I think they had some kind of long term health insurance.
00:46:52
Speaker
yeah know but But on the flip side, you know open source models have to be actually open source. Semantically, there's this conflation going on between open source and open weights, where llama is an open weight model. ah The weights are public. You can go download them. But they haven't disclosed the training sets nor the exact training procedure, though their technical paper does so have give more detail than most. And so it's not genuinely open source. And we're at we have to take it on on faith. And you know more than faith, we know from their reputation and And we have reason to believe they wouldn't intentionally embed a sleeper agent into in LAMA. But we couldn't say the same thing about a model released from China or the UAE or or or Russia.
00:47:33
Speaker
right And so open models, open weights ah can be incredibly valuable, but they also are a bit of a black box. And so critical infrastructure needs to both have access to open source models, open weight models, but also genuinely open source models so that can they can have the assurance that there's no sort of back door. Who could release a truly open source model? What entity would would would have interests in in in doing something like that?
00:47:59
Speaker
I mean, there are there are open source models, genuinely open source models that are of smaller scale, but you know the the biggest players like Meta, they run the risk of you know releasing training sets and then everyone realizing, oh oh my God, this is listen contains you know half of the New York Times and a bunch of YouTube transcripts and so on and so forth, and just you know opening up the can of worms of copyright and related licensing issues. um So I understand why there they're not doing that, and I think no company would really do that unless required by law. And you can make the case that maybe publishing the training sets that include a ton of operating information is too much to ask, you know but maybe they should at least disclose it to some kind of intermediary that can audit it. And I think that that would be increasingly important going forward, especially as these models proliferate to variety different modalities and domains that have some kind of
00:48:54
Speaker
could be a formal regulator or or a nonprofit entity, but some somewhere where you can provide the training data and the training procedures in a confidential, secure way for auditing and transparency.
Technological Changes and Regime Shifts
00:49:05
Speaker
Do you worry about technological changes causing regime changes? if we If we look at history, if we think about the agricultural revolution, the industrial revolution, what can we learn from history about how technological change leads to regime change?
00:49:21
Speaker
I think about this in terms of sort of the second order effects of technology. We tend to tunnel vision on the first order effects. The horseless carriage fallacy that the the automobile was just replacing the horse pulling the carriage, but but actually what the automobile did at second order was you know allow governments to project power far beyond their capitals and change the economic geography of the world. um You could have longer supply chains and so on so on and so forth. ah you know We ended up building the interstate highway system. like all these All these second order implications of the automobile.
00:49:50
Speaker
And second order effects are are usually driven by collective action. And so the classic book on this that that i that I love i recommending to people is is Thomas Schelling's Macromoto's Macra Behavior. great Great little book where Schelling introduces the concept of collective action through a story where he walked into a lecture hall one day and everyone in the lecture hall was sitting at the very back. And he surmised that what had happened was people had an individual preference against sitting in the front most row but they wanted to still sit near the front. But as each person entered the room, they each had to sit further back than the person that came before them. And and this ended up being self-defeating because there still had to be a frontmost row. So there were still people sitting in the front row that just were way further back than any anyone preferred. And the way you solve this kind of collective action problems, and and by the way, they're pervasive, right is through some kind of coordinating mechanism. So you could have had a person at the door that was instructing them and posing a rule to say, you you must fill in the seating from front to back.
00:50:47
Speaker
that would have solved the problem and everyone would have been better off. The reason technology drives regime change is in part because it drives the possibilities of coordination. It shifts those coordinations it renders some forms of coordination untenable and creates opportunities for new forms of coordination. Institutions have to then reemerge around those new coordinating mechanisms. A very simple example would be Uber and Lyft. The smartphone and mobile created a new possibility of coordinating riders and drivers and matching them together in a way that was a totally different way of ah solving the problem than traditional taxi commissions. And you could say, well, you know, maybe, maybe the taxi commissions could have come up with ride sharing on their own, you know, if they had enough VC capital and, you know, maybe their, their local incentives weren't do that. And so what tends to happen is the creation of these new, new forms of coordination don't get
00:51:42
Speaker
captured by incumbents, but instead are are built up by new entrants that did then drive a platform shift. And that happens both at the company level and at the level of government's institutions. And how do you foresee AI doing this? How yeah how does AI fit into this pattern?
00:52:01
Speaker
zooming out a bit I think you can also think about this in terms of a broader digital transformation, right? So, you know, there's skeptics of AI will often point out that, you know, the industrial revolution didn't sort of happen overnight, right? There was this period in the late 1800s, early 1900s called the long depression, where America and most of the West was in a period of, you know, boom and busts, credit cycles, and were in recession for for many decades. and So per capita living standards weren't rising.
00:52:30
Speaker
But at the same time, there was this enormous build out going on in manufacturing, early telegraph networks, rail infrastructure. And so by the end of the long depression, there was this sudden sort of catch up where all of a sudden the world looked much more productive and living standard for much higher. And yeah know someone like Larry Summers has made this point with AI to say, well, you know, AI could happen really quickly, but we have reason to think it will be more slow because we have to build all this sort of complimentary infrastructure before we get to the payoff. Now, the counter to that is that that that was the last 40 years, right? we yeah Tyler Cowan called this the great stagnation. Peter Thiel said that we had innovation in bits, but not innovation in atoms. For the last four years, we've been undergoing this broader digital transformation, and AI should be thought of as just a continuation of that trend, where now yeah we have the the data infrastructure, he the cloud infrastructure, the telecommunications infrastructure for an intelligent explosion to to happen relatively quickly.
00:53:24
Speaker
So maybe we should we should think about this as what is available to a person that is sitting in front of a computer with internet access? What has been digitized in governments, in companies? What can be accessed online and so on? And and that ah that also tells us something about what can be accessed by AI models or AI agents perhaps in the future.
00:53:48
Speaker
Yes, yeah, precisely. and But okay, so going back to the regime change question, if you look at the history of the United States, just because it's the history in the best, there was a major transformation in our system of government from, say, 1850 to 1950. Dramatic change, right? It went from sort of a laissez-faire economic system, sort of a very agrarian to within 100 years having you know all the three letter agencies, a NASA space program, no the atomic bomb, totally different economic system. And you know we did have continuity. We still had the US constitutions throughout that, but there was a kind of internal regime change that took place. And how did that regime change change take place? Well, in the late 1800s, the early progressive era, the rise of these you know sort of gilded age corporations were the first examples of the kind of new management structure.
00:54:42
Speaker
right These were the first large corporate bureaucracies really that that innovated and and developed new science and management and understood how you could structure supervisors and managers and so on and so forth. And that learning was eventually brought into government but through the influence of like industrialists like Carnegie and Rockefeller, but also just because it was in the air.
00:55:02
Speaker
And so the early administrative state, beginning with, say, wood Woodrow wall street Wilson, and then you know really accelerating with with FDR's New Deal, was the application of this new science management to government itself. And you saw early into this in the 1800s, there's some recent research showing that the expansion of the U.S. administrative state tracked the development of rail and telegraph networks because it enabled Washington, D.C. to have agents that are far away in different parts of the country and still have the capacity to monitor and and solve for principal agent problems.
00:55:31
Speaker
And I think we're going to see something very similar with AI, where, you know say for the last 80 years, the two major power centers in the United States have been Wall Street and Texas oil. So it's a very like schematic way of thinking about it. And so we end up having either you know Rex Tillerson as Secretary of State or maybe maybe you know Jamie Dimon as being floated as a potential presidential candidate. right so So our current elite in the United States is really balanced between these two major centers of gravity, the energy economy and and New England. And since this digital transformation that has been going on for 40 years, there is now a new
00:56:07
Speaker
power center, a new source of wealth is being created on the West Coast. And you see this now being reflected in politics where Trump has GD Vance as a running mate who has a lot of backing from kind of right wing tech ecosystem in Silicon Valley. And these these people have different experiences, different ideological priors, but importantly for the analogy, they have different ways of structuring organization. Patrick Carlson and Stripe is a very different company from Visa or MasterCard.
00:56:38
Speaker
Right. Or JP Morgan. Right. You know, in some ways, Carlson is the JP Morgan of the Internet era. Right. In the same way that in some ways, Elon Musk is the Henry Ford of the Internet era. And they are bringing to bear there the expertise they've learned in building hyperscale companies to a critique of government. Right. So the the example I can give here is a story. So in 2017,
00:57:03
Speaker
When Trump was just elected, the inauguration hadn't happened yet. therere There was a a frenzy of ah speculation about who he would appoint to various positions because he didn't expect to win. And so he didn't really have a transition team in place. Blake Masters was running his transition part of it and you comes from Peter Thiel's world. And so there was this early early sort of signs of what it would look like to have this new tech elite have influence in in government. And a name got floated for FDA commissioner.
00:57:34
Speaker
No, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, manic paranoid ah personality very libertarian he talks a lot about crypto he's a very polarizing figure no,
00:57:56
Speaker
and so he was being floated for fba commissioner And I thought this was just astounding. So I tweeted this talk he gave at the start of university at YC about Silicon Valley's that ultimate exit, which was all about like how the US government is like ah IBM. It's a laggard, incumbent company. It's got all this technical debt. Silicon Valley needs to build but parallel systems that then will replace the US government. I think the the exact the exact question was something like, is the US government the Microsoft of nations? Which is kind of ironic and now, but but yeah.
00:58:27
Speaker
Yeah, exactly. Microsoft is now leading the AI revolution. It's the biggest IT vendor in the US government in some ways. Microsoft is the US government in terms of the rails, the bureaucracy. So all that being said, so I tweeted this link to this talk and I described it as infamous. And then seconds later, my phone rang and it was Balaji. I don't know how he had my number, but before I could even ask, he said, infamous? I think he meant famous. So I ended up offering to write a profile of him and what his plans would be for FDA.
00:58:55
Speaker
And he ended up pulling out because of, you know, just disagreeing ah with Trump's actions on immigration policy and and other things. But, you know, in talking with him. there was at the time a variety of these articles from Vox and Recode saying that Bellagie would be a disaster because he's this naive Silicon Valley tech bro that once proposed having a Yelp for drugs. now Maybe we should just legalize all drugs and have five-star ratings for the good drug. And I thought, well, this has to be a bit of a straw man, but let's see what he really wants to do. And the the plan he described to me was but as follows, extrapolating from the current trend in personalized medicine and and genomics, the falling cost of gene sequencing and so forth.
00:59:34
Speaker
he He said that the FDA's pre-market approval regime, right, where you have to do clinical trials before the drug can be approved and put on the market, was ill-suited for an era of AI and personalized medicine. So what we needed to do instead was build a parallel system based on post-market surveillance that would use adverse ah effect reporting systems. So there is one of these that exists. Any time a new drug hits the market that has some one in a million interaction with but something, it gets reported. We can build on that those systems and other big data techniques, you know, maybe bit bit and so forth to correlate using Bayesian machine learning techniques, the efficacy of a drug for a particular person with a particular idiosyncratic genetic profile and then have this have a Bayesian prior to whether the drug is effective that updates over time. You know, whether this would actually, you know, work or she had any hope of implementing this, it is, it is the way that we should be thinking about government going forward, you know,
01:00:26
Speaker
What is the community notes for the, for everything, right? What is the, what is the scalable solution that harnesses decentralized knowledge and you know, AI local at the edge to be able to manage this transition because those new platforms, those new kinds of institutions will be necessary. You know, going back to what we were talking about earlier about what happens when you can train a power from all of this well below.
01:00:52
Speaker
a threshold where anyone can do it. ah You're going to need different kinds of systems. It's not going to come through a moribund bureaucracy that takes two years to approve a new regulatory. so So I guess summarizing what you're saying here is that in reaction to AI, existing power structures like the the US government will probably not adapt, but it will instead be replaced by new technological alternatives that will then crowd out for the existing governmental ah structures or power structures.
01:01:24
Speaker
Precisely, yeah you sometimes called the long 20th century. The core administrative law that structures how the U.S. Ministry of State works, called the Ministry of Procedures Act, I think it's from 1945. Social security numbers are from 1935. Most of the tech stack from the IRS on down was developed in the 1960s. So we're we're living on capital that was accumulated during the New Deal and Great Society eras and we're spending it down, right? And there are pockets of where this is' regime change, so to speak, is already taking place. So take take the defense procure you You have all these upstarts like Palantir or NDRIL that are applying software at AI to develop new technological solutions for you know intercepting drones and iron dome-like technologies or or operational intelligence platforms and so on and so forth that would never have been developed by Lockheed Martin or Raytheon because of their local incentives. But historically, Raytheon Boeing
01:02:19
Speaker
Blocky get all the contracts because a legacy system that is oriented around these two power networks that I discussed before. Within the Department of Defense, there's a program called the Defense Innovation Unit that is meant to be a parallel, sort of fast track for these startups to actually have skin in the game and be able to win contracts. You could imagine a world where the scales start to tilt and the old procurement system shrinks in relevance and the new procurement system grows.
01:02:46
Speaker
and And rarely is it the case where you just abolish something and have the new thing. It's more it's more where you have an ah and an explosion of growth that means the thing that is staying still shrinks in relative terms and the new the new platform, the new system, the new institution grows and becomes the new the new norm after a transition.
01:03:06
Speaker
Yeah, I guess a key question is whether existing power structures will let this happen. So will the US government name accept being disrupted by, say, private tech companies? Do you expect the US government to allow, say, OpenAI to create superintelligence, for example? Because, I mean, we've all heard stories of government inefficiency, and perhaps it's it's widespread in some domains. But there are also parts of the US s government that are extremely capable.
01:03:36
Speaker
So, yeah what what do you foresee here? Will the government simply let itself be replaced? yeah So the government is not in a mile with. We only have a launch capacity because of SpaceX. And that was only possible because some enterprising you know ah civil servants realized that the procurement system is broken. And if we had these milestone payments, we could actually bootstrap our launch capacity to the private sector. And that learning is is now is still taking place, but it has to be extended through other areas of government as well. And there's always pushback. right There is, in the case of SpaceX, you know they've been targeted by the DOJ. you know Elon had to
01:04:12
Speaker
pay indulgences to the ah Florida Save the ocila the Ocelot Foundation because apparently there's ocelots in Florida. and right So there's this extractive ecosystem around it that is is this sort of nibbling at the edges. But at the end of the day, the the the benefit of having SpaceX is unambiguous and everyone can see it. And so you know and in moments of crisis, we don't worry about this. right And during the pandemic, for for instance,
01:04:37
Speaker
paycheck protection program, which was offering, you know, forgivable loans, small businesses had to close during lockdown, ah was being channeled to the small business administration, right? Relatively old and, you know, as an institution that normally makes loans to the gas stations and nail salons, not every ah very small business in the country. Those payments ended up being channeled through banking system. And the SBA was sort of the wrapper around pipes that were they were built by the financial sector.
01:05:05
Speaker
and on the day of the program launching, the entire system crashed on the SBA side. And it was only saved because Amazon Web Services stepped in and rebuilt the portal and used their scalable ah infrastructure to ensure that it didn't crash. So in periods where things start to break,
01:05:24
Speaker
you don't start worrying about you know who's whose power structure is this. you know It's like, who can actually come fix the problem? And you know maybe we move into a world of you know AI agents that are transacting with each other but every 100 milliseconds or something like that. And that's a world where you know I don't think Bank of America is going to be managing those payment reels. Maybe it'll be done through the blockchain or maybe it'll be done through Stripe or or some other newer company that is optimized for that. But those are the kind of things that happen on their own, not because anyone planned it, not because of the outcome of an election, even, but because the second order effects of the technology drive the shift and the ways we need to coordinate society.
Nationalization and Power in AI
01:06:03
Speaker
Do you think that the US government could nationalize corporations like DeepMind, OpenAI, i Anthropic, those that are in front of AI development? ah We certainly could. Yeah, the question is, what how likely is it? Yeah, that's a better question.
01:06:19
Speaker
So let me use another pandemic era example. In September of 2019, Trump signed an executive order titled something like accelerating vaccine approvals for national pandemic preparedness. Very fortuitous timing. And this came out of a CEA report that was written by a couple professors from the University of Chicago, a Chicago school libertarian, economics professors who who had been critiquing the FDA for years and said, well, the FDA is too slow. It has all this hidden, you know, the hidden loss of drugs that don't make it to market. And in particular, if we had a pandemic or similar emergency, we need to move really quickly. And so this executive order laid out the framework for how Guarda and the defense would work with private sector to set up a public-private partnership in the event of a pandemic. And this ended up, you know, like I said, great timing because this was the framework that was used for Operation Warp Speed. And Operation Warp Speed, while we didn't nationalize Pfizer,
01:07:15
Speaker
We didn't nationalize Moderna. We did have a former board member of Moderna running open Operation Warfare. And we did have enormous advanced market commitments that pulled forward the vaccine in months. And so my anticipation is that we're, you know, this isn't the 1950s, it's not the 1940s anymore. We don't, we very rarely do we nationalize companies outright. Instead, we we have quasi nationalization for national champions via these public private partnerships, via contracts,
01:07:44
Speaker
and via the revolving door of who sits atop the governance of these these organizations. so there's the There's the question of how likely it is that we you get like pseudo-nationalization going on, but there's also how risky it is. this If the US government were to nationalize and perhaps ah combine companies in front of AI development at the frontier of AI development,
01:08:06
Speaker
That would result in an enormous concentration of power. I do worry about this, and I do worry about how how such an entity would act in relation to other countries, in relation to, say, other private companies. Are there any lessons from history there? I'm thinking specifically of, ah say, the end of the Second World War, or the collapse of the Soviet Union, where we have the US at the kind of height of its relative power.
01:08:32
Speaker
what what What can we learn about the US? How the US would act if it had enormous, enormous power? I mean, the US has had enormous power, right? We've been effectively the global hegemon for and instead the end of the Second World War. so And you know we've made mistakes. I think even critics of and US s imperialism and so on would say that relatively speaking, we've been benevolent. We've had foreign excursions that were probably heuristic and we flew too close to the sun, but we haven't been out there trying to dominate other countries.
01:09:08
Speaker
And to the extent that we have you know interfered with other countries, it's pursuant towards them liberalizing or democratizing. And so I don't think that basic cultural foundation of US power will change. So the the question then becomes, and what are our options? is Are our options to have this enormous power being concentrated within private hands or to have some public and democratic involvement? And I think the latter to me is potentially the better scenario part because you Silicon Valley in particular is founded by a bunch of ah congregationalists who you you know didn't like hierarchy. and
01:09:46
Speaker
you know you know it's it's There's a lot of sovereign citizens in Silicon Valley, let's put it that way. you know People who think the law doesn't apply to them. And you know some of these people may be leading a few important companies. right And if they get delusions of grandeur or a messiah complex, you know they may say, well, you know take this AI model from my cold dead hands and abscond on a private jet to El Salvador. You can't can't necessarily rule that out, let's say that. And so you know having, especially early on, um some basic involvement from from national national security apparatus, I think it's actually about, I think there there are people who worry that this will bias
01:10:28
Speaker
the development of AI towards militarization? That would be my main worry that that is so a sort of pseudo-nationalization would lead to AI being developed mostly for military use and AI being ah used to to compete with China for for kind of global dominance in a way that would be perhaps very risky but for the for the world. And I can totally see that. I would ah just ask, what are the counterfactuals? Is this priced in? Is that going to happen anyway?
01:10:57
Speaker
you know The Department of Defense has a billion-dollar budget for directed energy weapons annually. I haven't seen these weapons. I don't know if we're using them. you know It's a lot of money to be spending on microwaves. Directed entit energy weapons, is that is that a smart way of saying lasers or or am I misunderstanding something? More like a very focused to microwave beam or a neutrino beam. There's different versions of it. Then this sparks conspiracy theories about Havana syndrome and maybe we're like shooting neutrinos to the to the Earth so we can give people a headache on the other side of the planet. and I don't think we're doing any of that stuff, but the point point being, it's a very speculative technology that you can make the case is ah is an example of pentacon waste, um and yet we're we're willing to expend a billion dollars a year, this sort of vaporware. And so I would imagine we're also willing, the US military is willing to spend a billion dollars a year on something that actually is useful. And you know the hundred billion dollar training run will probably
01:11:52
Speaker
first be done by a government or with government involvement or advanced market commitments of some form, you know, different public private partnerships. And so I tend to think that some kind of joint venture, some kind of close arm-flank collaboration between the private sector and government is inevitable. It's already happening piecemeal. OpenAI has a partnership with Los Alamos, right? The same Los Alamos that we know from history.
01:12:20
Speaker
to red-team their models for, I think, biology biological and radiological risks. Anthropic is deepening its partnerships with the national security community for preparedness and defensive technology and cyberpreference in particular.
01:12:34
Speaker
And I think it's actually part of their corporate duty right and and and civic duty, really, to share their insights with ah with the US government. And you know the US government cannot simply have a secret AGI project that no one notices, in part because we would see the Gigafactory with satellite.
01:12:53
Speaker
satellite imagery. And also because all the talent is in these labs. So I'm not worried about like there being a crowding out where these private companies get co-opted and all their energy is going towards building the AI Stuxnet that is going to be used to
Dual-use Technologies and International Collaboration
01:13:09
Speaker
counter China. Rather, I think they're going to continue building what they're building.
01:13:12
Speaker
And as those capabilities cross dual-use kinds of capabilities and thresholds, the U.S. government will learn that what's possible, what's nearly possible, probably inevitably apply those techniques to the and NSA and other other agencies for cyber defenses and cyber offenses. And whether that's good or bad, we can't really stop it.
01:13:31
Speaker
Do you think AI involvement, say US government involvement, specifically in and anthropic and open AI and deep-minded, does that accelerate AI progress or decelerate? it Or perhaps tell me about how that's, as we discussed, a false dichotomy, but yeah how does it affect the progress of ah specifically and of frontier AI?
01:13:52
Speaker
It may slow down on the margin. right So I think that the more important implication is it it enables the kind of learning by doing within government. So you know we now have the AI Safety Institute at NIST, and there's a similar organization in the UK. And these are government bodies or quasi government bodies that are for the first time having to you know develop evaluations for models, safety evaluations, safety benchmarks. And they've never done it before. And there's really no one in the world who's done it before.
01:14:20
Speaker
no we we've We've done it maybe in the last few years, there's people who have developed this expertise. and And so they need to have you some training wheels. They need to have some practice to develop ah the muscle and organizational capital to do this when it matters. And so it's important to start doing it now. And that will necessarily involve aspects of the security state because if we want to model, if we want to red team a model for knowledge of how to build a cyber weapon, know the the secrets of how to build a cyber weapon are secret. So we need people who know those secrets to test the model. but and And this is an important precursor because if we do need to intervene, we need that muscle already in place.
01:14:59
Speaker
And we need some like more objective criteria of what dual use even means. I mean, that will only come through the these kind of partnerships. And so you know building these triggers. And the other the other thing I think about is dual use has a very particular meaning. and In the US context, we're under Export Administration Regulations and ITAR, which governs international you know weapons mass destruction. and and other dual use technologies. If OpenAI or Enthropic or any of these labs were ever declared dual use, they would have to lay off half of their employee base. Because and under US law, companies can only build dual use technologies with US persons. This is why SpaceX can't hire non-Americans. And this could be a major inhibitor to ever making the determination that a technology is dual use.
01:15:46
Speaker
right Because the moment you do that, the whole not only does the whole sector you know hit a wall because all their Polish engineers have to go back to Poland, but also it's counterproductive because suddenly all those internal secrets are being cast to the wind and all the people go back home and share the secrets with their citizens. And so we need to be thinking ahead to not only how to define the triggers for dual use, but by thinking about what is the right framework for for what dual use implies in the in context of AI.
01:16:16
Speaker
you know there could There could be techniques that are being learned at Suno, the music company that make ah make their models much more sample efficient. And those techniques could be equally applicable to a biological platform. And and so there's not really like you know there's there's ah there's a huge substitutability between these different AI platforms in in the terms of the technical knowledge that they're acquiring. And so we we shouldn't you know restrict advanced AI development to US persons.
01:16:44
Speaker
We need some other framework for that. And having that framework in place, and yeah the planning for that needs to happen now. And because it's going to be multilateral. There'll have to be a major diplomatic effort from the State Department to define you know maybe some like AI talent network for allied countries and democracies.
01:17:00
Speaker
get to work on advanced AI systems, even if they're dual use in nature. Let's talk about the economics of AI. You have a post where you push back a little bit on kind of the notion of an intelligence explosion, but you also note that in that post that you are, perhaps relative to the, say, average economist, you are you're more convinced that that that something akin to and to an intelligence explosion is possible.
01:17:28
Speaker
But maybe just to sketch out how how would you model such an explosion? What would it mean for the economy to move from 3% growth rates to 30% growth rates, for example? Yeah, so people mean different things by intelligence explosion. I think the standard meaning people use is you cross some threshold of like the AI researcher that then can build a successor at infinite of you have like this bootstrapping towards super intelligence. I'm a bit skeptical of that.
01:17:55
Speaker
and in part because I think one of the big lessons from deep learning is that there is a no these systems are are powerful insofar as they are very highlyment have a lot of dimensions, right? There's thousands of different dimensions. and And there's limits to how you compress those dimensions into something smaller.
01:18:13
Speaker
you know, a kind of irreducibility to the the vector space, right? and And so, you know, going from merely human level to fastest superhuman, isn't just a matter of, you know, figuring out better algorithms, right? Or, or somehow, you know, introspecting on your, your model weights and finding a better configuration of weights. Chances are those model weights are probably in some kind of local optima to begin with. And so s scale will still be a factor. You can't, and so even if there is sort of in the bounds of compute,
01:18:42
Speaker
a process where we cross some threshold and then the models get much, much smarter within those bounds, they will still be bounded, right? And we will still be rate limited in the intelligence explosion by the availability of compute and other polynomics like energy and power. So that's point number one. but When it comes to the economic implications, so You know, there are worlds where, so, you know, one of the ways where AI could drive, you know, it's dependence growth and in the near term is is through services, right? And US economy is about 70% services. Most modern economies are service dominated. The services are labor intensive. They are often non-tradable, right? And non-tradable goods don't have a law of one price, right? We, because you need to be in person to experience them. And so, you know, one of the reasons we've had relative stagnation for the last state
01:19:30
Speaker
30, 40 years is because the economy of services is dominated and services are resistant to productivity improvements. But because of 70% of the economy, you know if you get major productivity accelerations in services, that could lead to a much higher rate of growth, precisely because it's it's been the stagnant part of the economy for so long. That makes sense to me.
01:19:50
Speaker
Maybe maybe a for for the non-economists listening, talk about the the difficulty just of maintaining 3% growth rates and what it would mean to to enter a regime of 30% growth rates. How insane that would be? yeah Merely maintaining the 1-2% growth that weve we've had as a frontier economy for yeah to the end of the the century implies you know enormous <unk> you know changes, transformational change to how the economy is structured. I think if you just extrapolate say one or 2% growth out to the end of the century, then US per capita GDP is in the six figures. right And so you know thinking about the median person having you know a Bentley or something that. And the resource utilization implied by that is also it's also
Limits to Economic Growth and Consumption
01:20:35
Speaker
huge. And so so you know something's got to give. And if there is a major sort step function increasing in growth on top of that, you know I don't think that means we have 30% growth indefinitely.
01:20:48
Speaker
it merely means that we and have a sort of sigmoidal spurt of growth that lets us catch up to the new ah steady state and and ah in a and a rapid period of time. And that could be you know very very dramatic. you you You'd almost rather smooth a 30% growth phase out over 10% growth over three years rather than 30% growth over one year and because of the scale of disruption. But I don't think we just maintain 30% indefinitely and The models that suggest that are all models where humans exit the picture. right they're They're models where AI engineers and researchers are just generating inputs that go into better AI engineers and researchers. and At the end of the day, like the economy is for people, and we're consumers. and What gets built in the economy is downstream of consumer demand, and you can't just have more production without more consumption.
01:21:39
Speaker
and so growth, like really, really explosive growth is inherently bottlenecked by our ability to consume. This doesn't enter the equation of some of these models where where basically labor share of income goes to zero. And yes, you can have hyperbolic growth, but then but then we're just, it's ah it's an economy of AIs building things for AIs.
01:21:57
Speaker
yeah But are there limits to how much we can consume? You could imagine under, say we have an aligned superintelligence that just presents all of us with a the lifestyle of a present-day billionaire. hey you but I think people would still be interested, like perhaps today's billionaires are, in increasing their living standards, obtaining something more, something different.
01:22:19
Speaker
You have a ah post again that I've read where you discuss the the pounders boundedness of the subjectivity of value. What are those bounds? What are the limits to how much we can consume? i mean there there There are intrinsic limits because there's a certain bit rate that passes through our eyes, ears, and mouth, and nerves. right and you know so In a literal sense, there we can only consume so much media,
01:22:46
Speaker
we can only eat Michelin star restaurants you know every every day for eternity. And so at some at some level, there are inherent limits to growth just in terms of human utility. right human yeah there's There's a kind of saying in economics that wants are infinite. that's only That's only kind of true. right They are potentially infinitely you know you And the, insofar as we care about relative status that can, maybe we can enter into an infinite the loop where we're always just trying to one up each other. But especially when you get into worlds where we could simulate or emulate a mind in a virtual environment, then, you know, GDD becomes basically irrelevant, right? Because I could, you know, virtualize, you know, simulate my 72 virgins and a Lamborghini and, you know, but but whatever people want, right? they could You could, you live in, it you live in,
01:23:38
Speaker
heaven at that point with you know for the cost of six kilowatt hours, yeah whatever the electricity rate is. right The subject of subjectivity of value ends up breaking down in more sci-fi AI scenarios because we're so used to being within this regime where GDP growth means incremental and productivity improvements where we have a better mousetrap.
AI's Impact on Jobs and Future Roles
01:23:59
Speaker
Great. so So we talked about the consumption side of the economy, but there's also the production side. And there you've you've you've written something that's that's somewhat skeptical about humans kind of exiting the economy, no one having jobs. You write technological employment is only possible if capitalism collapses. and Maybe you could explain that. Yeah, this goes back to what I was saying about the consumption constraints on production. So a stylized fact of advanced economies is is that over time, the labor intensive part of the economy becomes a bigger and bigger share of the economy. This is this known as Beaumel's Cost Disease. ah You have rapid productivity growth in make fat manufacturing, but you still need teachers, you still need nurses, you still need childcare workers. And so their wages have to rise in proportion with other sectors of the economy to retain that workforce. And over time, the most productive
01:24:52
Speaker
ah Parts of the economy are super deflationary, price of things goes zero, and everyone is a nurse or a teacher or or the things that remain resistant to automation. Now, that doesn't change in an AI post-AGI world, in part because we're still the ultimate consumers. Everything that's being produced is being consumed and is being produced for humans. And humans need some income to be able to afford that production.
01:25:17
Speaker
and And that couldn't just be some form of universal basic income or another form of transfer. It could be, but I don't think you you might you might need that to, you know, provide some kind of floor or basic standard of living, but the economy itself, the circular flow of demand within the economy will but drive in general equilibrium incomes to rise, right? And maybe that means we're all like, you know, pet therapists or something like that.
01:25:45
Speaker
whatever the new service sector jobs in the future will be, but but we're still going to be earning income and consuming the fruits of AI labor. When I interview economists, this is often a point they make that new jobs will arise to to replace the jobs that are now being done by AI, like we saw in the transition from say agriculture to to factories and then to offices.
01:26:10
Speaker
I often wonder whether there is there could be a ah failure of imagination on both sides here. There could be a failure to imagine these weird jobs that might exist in the future, like pet therapists, but there could also be a failure to imagine that things might be different because AI is different from other technological transitions.
01:26:29
Speaker
I, it's which jobs could be available. I mean, pet therapist is kind of semi-serious, I guess, but, but we can't all be pet therapists, right? Which jobs do you imagine being available when we have say, say very advanced AI? Yeah, just I just saw on TikTok ah yesterday, a professional back scratcher. What woman has long, long acrylic nails who will gently caress your back ah for probably, you know, a hundred dollars an hour or something.
01:26:59
Speaker
And then they also get TikToks of like the $1 head massage you can get in India where the guy just slaps your head a bunch. It's hard to predict what these jobs will be. And we talked about this a little bit in the past about the kinds of sources of intrinsic scarcity. And those include positional goods. They also include rents, rent seeking. I work in the nonprofit sector. And I think my job, I have a modicum greater job security than my ah colleagues in the private sector, because if their job can be automated, it will. Whereas my job depends on the good graces of large foundations and philanthropists. And what, and what they're paying for is not necessarily my written output, because maybe I could get Fachibati to write my white papers for me. What they're paying for is in part, the combination of my ideas with my personal relationships. Right. The fact that, you know, in my case, I work with members of Congress and, and staff on the on Capitol Hill.
01:27:55
Speaker
And there's a legal structure that will require that we still have members of Congress making important policy decisions. And whether you have access to those members of Congress does not necessarily depend on you writing the most persuasive essay, but whether you have a trusted relationship with them. And maybe that trust will become more important and in the future because of ah you know the growing eligibility of society and not knowing what to believe.
01:28:16
Speaker
this is This is extrapolating into perhaps the shorter median term. I'm i'm interested in in kind of the end state scenario where you have AI capable of doing, say interacting with the legal system or the government or perhaps AI is the government ah in the limit what jobs could humans do. So when I look around the world, what countries or or regimes most closely approximate a post scarcity society today?
01:28:46
Speaker
I think of the Gulf States. right and these are you know Saudi Arabia has a giant spigot that can turn on and off, and oil comes out, and then they can make up projects in the desert. right But as a consequence of that, you know this this is you know used to be known as Dutch disease, the classic resource curse, is that countries that have phenomenal resource wealth, like Norway or or Saudi Arabia,
01:29:11
Speaker
often have very dysfunctional politics because the economy does not is not built on the basis of human capital or productivity growth or labor productivity. It's based on who do you know? Are you a member of the royal family? So on down the line. And so there ends up being battles over rents and that leads to very corrupt qualities. And the countries we look and we look out at, the countries that exist today that are relatively stable like Saudi Arabia is because they've overcome rent-seeking problem through the consolidation of power in a you know a royal family in Altarqui. Norway is a bit different. you know Norway has you know a sovereign wealth fund that is managed by professional fund managers, and everyone lives a very high standard of living. And even their even their prisons are nice. right And so so future a future world where we do have explosive growth and there is a kind of post-scarcity economy is one where the remaining you know where where everything kind of becomes a rent. And
01:30:07
Speaker
you know the worlds where we do have technological unemployment are, are like I said, worlds where basic market capitalism kind of breaks down. And we move to, you know, there's different versions of this. You can imagine moving back to a more forager style lifestyle, where, you know, sort of high tech, a hundred gatherer societies where and everything can be produced locally because of localization and the fact that we've got factories in our backyard and fusion hugesion powering it and so on. But, you know, no one needs to really work, but for you know making each other nice necklaces and the dream catchers that we ah but we barter over at a gift economy. and And then there will be other qualities that go the more Gulf state route where they restore the you know the counter of the forces of rent seeking and the corruption that brings through true power consolidation and the imposition of a strict moral code that yeah regulates among other things, you know whether you can have your
01:31:05
Speaker
jailbroken phone in public and stuff like that. Those are all plausible to me. I think the reason economists have a trouble intuiting these scenarios is because they're imagining that market capitalism and continues on in more or less and normal form. And so the worlds where technological employment is realistic are are also worlds where where capitalism kind of fails and is replaced by something different. And it's not free market capitalism plus a UBI. it is some new form of economic social organization. Let's talk about differences between AI cognition and human cognition. that This is perhaps relevant to the to the question of whether AIs can replace humans fully or automate everything that humans do.
AI vs Human Cognition
01:31:52
Speaker
so Just in a general sense, how how similar is AI cognition to human cognition? so I think it's similar in kind, even if it's different in specifics. I think ah
01:32:04
Speaker
useful example of this comes from a project at New York university where a team of researchers were wondering, you know, what what would happen if we just attached a GoPro to a baby's head and recorded everything it saw as it crawled around. And this was, this was partly to explore, you know, what is the sort of sample efficiency of these models for learning language, right? Cause one of the critiques of large language models is they're trained on trillions of tokens. You know, they've read every book ever produced um and and they're just now sort of at maybe a graduate level of of reading comprehension.
01:32:33
Speaker
Whereas humans, you know we go yes we we we take years to get to that level, not three months on a supercomputer, but we we see many fewer examples and we're able to learn language. And so this team at New York University, they they put this GoPro on the kid's head and discovered that using a very simple language architecture, that from just the the vocalizations of the parents in the room and everything the baby was hearing around it,
01:33:00
Speaker
They were able to begin piecing together basic syntax and semantics ah within this AI model. Now, you know, the baby also has multiple advantages. We are innately wired to acquire language, right? We probably have a number of inbuilt priors that make it, make it so we can leap forward in the gradient descent and get to language fluency faster than purely brute force mechanism. But those inductive priors are cons a priori or evolutionary a posteriori, right? did the All the a priori knowledge we have encoded in our brain were post hoc knowledge acquired by evolution through natural selection. And most of that is not hard coded in our DNA, right? Our DNA, human DNA maybe has 20,000 base pairs of code for proteins. ah There are a number of other parts of the DNA that code for epigenetic functions and protein ah functions, but then the rest is kind of junk. And so, you know, you can't really encode a blueprint for a human body within 20,000 genes.
01:33:56
Speaker
But what instead biology seems to do is harness gradient descent and the fact that balls like to roll down hills to set up ah and the hyper parameters and the reward signals that then guide human development from embryo on up.
01:34:10
Speaker
through processes that closely resemble gradient descent with the right hyperparameters. And you also see this in the evolutionary literature where you know in in many ways a cow and a giraffe are the same animal, just the giraffe has a longer neck. right And so um mammalian DNA encodes for a basic body map and then has and attributes that can be molded. And this actually reduces that this, in a sense, smooths this more this morphology, consistency of morphology, smooths the lost like landscape for or evolution and to operate over. Because if natural selection were just purely blind, you know we would never have evolved because the lost landscape is so so jagged. And so you know we we sort of evolved the ability to evolve, right? We have Darwinian mechanisms that then select for or sort of
01:34:59
Speaker
You know, kind of meta-learning within within but in a genome. But I think efforts to guide machine learning in this way or build in some hard-coded kind of heuristics, those have have often been outcompeted by simply making the model bigger or training on more compute. ah This is ah you know Richard Sodden's bitter lesson. So I think maybe the interesting question here is, what can we learn from how humans learn? And can we use that to improve AI?
01:35:29
Speaker
can we make AIs more sample efficient so that they could learn from, say, what's available to ah to to a baby instead of thousands of years of examples? I would just to dispute the premise a little bit. So, you know, there is there is a version of the bitter lesson that says, you know, scale is all you need and you shouldn't hard code in any priors. And and that that that is true for, you know, certain kinds of priors. Like if you hard code in like you know the the way a spider has an innate knowledge of how to make a web. you know there's there's very There's actually very few examples and in evolution where you have that degree of hard coding. What you instead have are architectural features that imply a certain prior. right So equivariance or permutation invariance, in the case of transformers. Transformer architectures have a permutation invariance where that that builds in a kind of prior into how it processes tokens.
01:36:26
Speaker
image models that build in an equine variance so so that an image that is flipped or just transposed to the left or the right doesn't get treated as a totally different image but as the same image. Those kind of priors, if you want to use that that word, are poor to sample efficiency. It's why these models can learn quicker if they if they can and If an image model can say that and this this is the same face, just moved over a few pixels, it doesn't have to learn the face a second time. And you know my intuition is that those are the kinds of priors that matter most through sample efficiency. And they're they're not mutually exclusive with believing the bitter lesson. like The bitter lesson just says with those priors in place and then scale and search is the the thing that matters.
01:37:10
Speaker
Another difference between AIs and humans is that humans are just, for now, much better in in highly varied physical domains. Humans still outperform robots. Do you expect that to to continue?
Advancements in Robotics
01:37:22
Speaker
Do you think that we can we can learn how to do robotics? Is robotics mostly a software or a hardware problem? yeah When we last spoke, ah which I think was in the fall of last year, I said that I expected to see rapid progress in general purpose robotic foundation models.
01:37:40
Speaker
And I think we have seen, right. We've seen companies like, like figure one or Tesla's humanoid robot make really rapid progress to the point where, where I can remember a year or two ago talking to a machine learning researcher, science policy guy who was like, well, you know, I'll believe it when AI can fold laundry. Cause there's been competitions going back decades now where engineers submit, you know, RL trained systems that can barely fold a T-shirt. And new now we have.
01:38:09
Speaker
robots within within the span of a year that can you not only fold a t-shirt, but can fry an egg and flex and fold your all your laundry, ah go park a car potentially. right um And now they have like touch sensors. right So the Tesla humanoid robot can actually ah adjust the pressure of sensitivity of its hand when it's picking up an egg versus picking up a more solid object. And you know there has been some neuroscience research suggesting that if transformers are are similar to anything that goes on in the brain, they're most similar to motor cortex and the cerebellum, where there is evidence that cerebellum has a residual stream and it's sort of sequence by sequence. And so it may it may turn out to be the case that transformer architectures are actually better suited to motor control than even language. And we it certainly see early signs of that where also these general purpose foundation models are able to learn via demonstrations.
01:39:01
Speaker
You may have seen the the demos where the you know the robot watches somebody playing drums and then it goes and picks up the sticks and plays the exact same drum line. so I think progress in in robotics is going to be but parallel progress and in other areas if if it's slower because we're using mechanical actuators and we don't have the you know half a half a billion years of evolution that optimize for you know very metabolically efficient biological organisms that that require very little energy that you know repair themselves if they get cut or injured. right the This is a level of competence that is hard to imbue into things made of metal and wires and transistors.
01:39:44
Speaker
You think robots will be limited like self-driving cars by their mishandling of edge cases, so by their lack of reliability, lack of very strong reliability. I think it would be difficult to convince people to let, say, a robot fold their laundry in their house if that robot was not extremely reliable, if it's walking around among people. Yet, do you foresee that as a problem for for widespread robotics? Not necessarily. It depends on the use case, right? So maybe you need a high level of reliability if it's it's in a nursing home or a hospital ward, but we already have Roombas, right? So I provided the the amount of damage that can do you buy and by you know getting the edge case wrong is is limited. I think the adoption will be quite quick. In the case of a Roomba,
01:40:37
Speaker
if it tries to vacuum up some dog poo, and it could ruin your carpet, right? and And so there are edge cases even there, but still people adopt it. And, you know, these system-use e-mode robots are going to have hard in-built, you know, grid limits where, you know, they're they're stuck at walking three miles an hour. They're never going to become like iRobot and start chasing you down the hall. But, you know, maybe they could if there was like software vulnerability and someone didn't order the air exploit, right? And so that becomes an issue.
01:41:07
Speaker
um But this is why ah hardware security becomes that much more important.
AI's Knowledge and Processing Speed
01:41:11
Speaker
Another difference between AIs and humans is AIs are much more knowledgeable than humans in a certain sense. They are knowledgeable across a very wide range of domains. What should we make of AI nailing a bunch of, say, high school tests and a college tests and perhaps the LSAT and so on, tests that require knowledge to to be retrieved and presented and in certain ways.
01:41:40
Speaker
and So there's one view that that argues that the of large language models on say the bar test or LSAT or whatever is a bit of a ah illusion of intelligence because rather than learning general principles, these models are developing kind of complex inductive biases that let them copy and paste from patterns that they've seen before. And there's a lot of truth to that. But it's also the case that if you ask a cognitive psychologist that human reason work similarly. we we are We don't sit down like we're a Russian Russell or some logical positivist and think in terms of the logical atomic structure of thought. We apply various reasoning heuristics that copy and paste from similar structures that we've seen before and bill build up from that and then formalize it
01:42:31
Speaker
with the help of language because the language lets us externalize our thought into the environment. And so you've seen this already where like there's people who, you know, complain that if you ask a large language model to count the number of ah words that are in the sentence or or count the number of letters that are in this prompt, that it will get it wrong. But if you give it a chain of thought example of how to do that, where you you first take the the sentence, break it down and and enumerate each letter that the model will just do it, right? He'll do it, do it right away.
01:43:01
Speaker
And that's an example of applying a kind of pattern and reapplying it. But it also shows that these models, if we think are limited in a certain way, actually aren't if we give them a little bit of a nudge in that direction. So, you know, well, I do take the critique from folks like the ah the creators of the arc context where, you know, LLMs struggle to solve basic sort of spatial reasoning problems that they haven't seen before because it's not in their training set. Totally, totally true.
01:43:31
Speaker
But I think the the way to think about this is you know if if the tokens that the AI is seeing are like its input it's sensory input, you know the data that we receive as humans is highly multimodal. right Every photon that hits our eye is like a token. These these models, you know they maybe have 50,000 tokens, so they're like, yes of course they can't interpret a visual image where they where they've been really trained on subparts of words.
01:44:01
Speaker
anyway they they They are, they're you know from that perspective, like Helen Keller, you know they they are they are blind to those kind of century inputs and maybe better tokenization would help that, but also multi-modality and transfer learning through through a large number of modalities could end up leading towards better performance on on and such tasks. I think it remains to be seen, but I wouldn't count out that these existing approaches could eventually you know pass the art challenge with flying colors and also be able to like dynamically adapt every edge case and in a self-driving car or things of that nature. There could still be new tipping points that we've yet to cross. The part because, you know, when you look at like Epoch AI's projections of compute scaling, you know, we're, 2030 is like the the crossover for like ah a brain sized model. So the models we're using today are still very, very small in a certain sense.
01:44:52
Speaker
which is which is a weird fact given the... I mean, when when you look at what current models can do, they seem to be approaching human level in in in some domain, at least, writing a high school essay, for example. But yeah, they are when when i'm when I'm interacting with models, i'm I'm constantly kind of disappointed that they can't you know push for something truly creative or create something kind of at at the at the edge of of human abilities. And of course they can't, right? ah Because we're we're not there yet. But that that That's just been my my experience. I think one question is, you know what what should we make of the fact that AIs work incredibly quickly? They can output something very fast. They can output like a pretty good high school essay very quickly and a pretty bad scientific paper just as quickly. In which domains does it matter that you're that you're very fast and that you that you output very quickly? I think it matters really in almost every domain. right Because there is this
01:45:51
Speaker
trade-off that's been discovered where multi-agent architectures that are able to, you know if with very fast inference, ah generate you know thousands of outputs and then ah have some selection process to pick the best outputs and then iterate on that can act can actually ah compensate for scale. but it It can trade off at scale and you can get you can approximate what it would be like to have a 10X bigger model with a smaller model, but with many agents operating in some kind of team. And the applicability of that will pay dividends to anywhere where AI is applicable.
01:46:21
Speaker
Beyond that, you know I think there's a sense in which you know we're we're we're kind of biased by you know the temporal and spatial scale that our mind works at. Yoshua Bach has speculated that maybe plant systems have some low-level agency that we just can't really detect because they move at such different time scales. But if you run a time lapse of a plant, you can see the plant sort of reaching towards the sun and and so on and so forth. and And you know because they don't have a central nervous system, they communicate sort cell to cell, and that's that moves at the speed of sound or the speed of light. And so you know maybe there is some kind of information highway, some kind of forest internet that the trees are communicating where the water's where where the water's coming from and so forth. And you know maybe yeah so this is another dimension of of AI risk, where merely having systems that are human level,
01:47:16
Speaker
but could operate at blazing fast speeds. you know From the perspective of those AIs, if those AIs have some kind of inner experience, they will be looking out at the world and seeing humans as if they are slow moving trees and plants.
01:47:27
Speaker
Yeah, which would be an enormous advantage if if there's a say a competition for for power in the world. Just speed alone. You could imagine me having, ah say, 10 years to form my next sentence as opposed to having to do it on the fly when talking to you. And, you know, speed becomes its own capability if you can react quickly enough.
01:47:51
Speaker
Yeah, absolutely. This is why yeah there have been proposals. Yeah, I think Yann Talon has suggested there's maybe max teamwork as well to ah have some kind of speed limit in these systems. And probably not a bad idea, but also hard to imagine how that gets enforced.
AI and Spiritual Beliefs
01:48:04
Speaker
Let's talk about AI and religion. how do you How do you think AI, specifically quite advanced AI, could affect religion?
01:48:12
Speaker
there will be different reactions depending on sort of where you exist within sort of the the esoteric versus exoteric span of knowledge. right You've got to explain that one, I think. so i mean like What is the person on street grants versus be the rationalist sitting in their cave who is who has thought about all these problems and can sort see through the miasma of deep fakes and so on? you know I think back to a journal entry Kierkegaard wrote when the Mormons visited Denmark.
01:48:40
Speaker
in the 1850s. And Kierkegaard noted that he thought that their theology was, I think he described it as a retrograde step in theology, from the spiritual to the concrete. Because historically, the trend in theology has been for for God or concept of God to become more and more abstract, right? more The world becomes more disenchanted and God becomes like this abstract being. Whereas the Mormons told Kierkegaard that God wasn't everywhere, that he wasn't omnipresent, but rather moved great rapidity from star to star. And Kierkegaard surmised that this was probably due to the influence of the telegraph and early real network. but and And so there's a sense in which Mormon theology, because it was emerged during the Industrial Revolution, had a much more materialistic conception of God that saw God as not separate from the world, but bound by the same laws of special relativity and so on that everyone is. and And so I think AI
01:49:33
Speaker
also has the potential to influence theological conceptions of what AI is. At first order, this goes to the title of the Ray Kurzweil book, The Age of Spiritual Machines. We have a sense in which what a spirit is is somewhat demystified. A spirit is an operating system. A spirit is software. A spirit is a law. Software is a law like property where the same software or the same logic gates work here and they as they would ah on in and Andromeda, right, regardless of the substrate. So Yoshua Bak has talked about this as leading towards a kind of cyber animism that we begin seeing spirit as something that is actually normal to talk about, you know, at our scale. Spirits exist, right? And, you know, going back to the discussion about forests and plants, whether they have some kind of agency. Yeah, they used to believe, and and the Shinto's in Japan still believe that, you know,
01:50:28
Speaker
forest or fill this with berries and have a kind of ah agency of their own. maybe Maybe something like that was actually kind of true. Maybe we went too far in the post Protestant reformation of sort of disenchanting the world and pushing spirit into something super abstract. And maybe this is also why we have you know the hard problem of consciousness and challenges conceptualizing what it means to have spirit and matter coexist.
01:50:53
Speaker
Do you think do you think that we could really be drawn to to these spirits, say, if we understand that the spirits are actually, say, just models running on our phones, or models running in ah in in trees, or or we've embedded models everywhere in the world? but it Does it seem magical if it's fully understood, I think it is is the main question? ah There's a difference between intellectual understanding and and ah visceral brokking.
01:51:22
Speaker
that I think, you know, but what separates magic from not magic, right? If a magician reveals how he does his trick, and actually it's like the super simple thing, but you know, just some misdirection pool to you, it's like very disappointing, right? It's disillusioning. But you can study the foundations of deep learning and theoretical deep learning all day long and still feel kind of spark or some kind of magic when you interact with advanced AI systems.
01:51:49
Speaker
because it's a very high dimensional thing that is beyond our pastures to grasp and is inherently that as such, right? Because it is a, we are computationally bounded. And so anything that exceeds our computational bounds is strictly uninterpretable to us, not cognizable to us. And the way these models get integrated into the world is going to be, yeah, there'll both be the very big models that you go,
01:52:14
Speaker
talk to like oracles, but then it'll also fuse society, right? We, we already have algorithms that, you know, if you ever scroll on TikTok or Instagram and suddenly you're being shown things that were on your mind, but you never articulated. It's like, how did it know that? Well, there's probably all kinds of ambient cues are being put off and the machine learning algorithm running the recommendation feed ah has picked up on those cues. They're not made explicit, but the world becomes much more agentic.
01:52:40
Speaker
there's like There's a common claim that people believe that their phones are listening to them. And I don't think phones are actually listening and recording conversations and using that to serve of ads. It's just that these recommendation systems are so good that it seems like phones are listening. Yeah, and you're and you're mostly blind to all the signals you put off. This is, you know, like, why cold reading works, right? ah Mentalists can somehow know that your grandmother had buried a locket in the backyard and say, how did you know that? Well, you probably gave a lot of instill you didn't even realize. The corollary of this is that we're all becoming a bit more schizophrenic. For the most part, there are no conspiracies in the world. no like There's no lizard people sitting somewhere in in the world economic forum that are like coordinating the world. But there will be as if agencies, meta agents, and there already are. Corporations, ah cultures, nation states are already meta agents. There will be more of those meta agents.
01:53:37
Speaker
And we will move into a world that is much more, in a sense, polytheistic. Joe Henrich's work on the rise of what he calls big gods sort of illustrates his point. So prior to yeah the the start of history, right, with early civilizations, Sumeria and so on, religion was mostly animistic, right? A hunter-gatherer society sort of saw this sort of unity between the spiritual and the concrete, symbolic and the natural. You know you could do a dance to bring on the rain,
01:54:07
Speaker
and so on. And it's only with ah the the earliest city-states and the nation-states where gods take on the identity of a people. And at first, there are multiple of these gods. You can hold pantheon where there's the god of Sumer and the god of the Assyrians and the Canaanites and the Judean god. And over time, as nation-states consolidate, we get the first monotheistic religious. And these are the big gods.
01:54:35
Speaker
And you can sort of think of these gods as representing the meta-agent of that nation, right the cybernetic system that gives rise to an agency that's more than the sum of its parts. And that's an incredibly useful social technology right because it can help a align a people to you know collective action. And you know over time, as governments have gotten stronger and we've had you know the scientific enlightenment, we sort of pushed our notion of God even further into the periphery where you know maybe God is not just like some um abstract concept of humanity or something like that. But I think this could end up running in reverse, where the fracturing of society that could occur under an intelligent explosion, where existing nation-states start to fragment and yeah know more localized forms of production become possible. Once again, yeah you think about the you know the town, local schoolhouse you know that gets displaced by the
01:55:31
Speaker
large consolidated school districts because of labor costs. and you know Now we have AI tutors, maybe we could re-consolidate and go back to more communitarian modes of living. um But by the same token, there will be a proliferation of these meta-agents and thus a proliferation of things that we could speak of as demigods. This is a much more polytheistic equilibrium than than our current one and goes to, I think, a through line through of what am I thinking on this, ah that you know inspired by Robin Hanson, that there's there's this kind of these kind of two modes of morality, the kind of forager morality in the agricultural society morality. And we are agricultural society peoples. like we are All our myths go back to the Garden of Eden, which is really an early agricultural society myth. But the forager style of thinking is actually much more natural to us, which is you know also the reason why Freud wrote, civilization is discontent, because civilization is a very unnatural thing.
01:56:24
Speaker
and we long to be part of more communal social organizations. That could end up being a path of pathway these resistance, therefore, because this is the ah way we we we prefer to live. It just hasn't been any it hasn't been possible in modern society because of the economies of scale that are offered by large ancient states and cultures. Do you think AIs will become conscious? Do you think current AIs are already conscious?
AI Consciousness and Moral Alignment
01:56:48
Speaker
I think the question is is harder than a lot of people give you credit.
01:56:51
Speaker
right We don't have a final theory of consciousness to begin with, so it's hard to say that a ah large transformer model is incapable of consciousness. I don't know how anyone has the grounds to say that. That said, i don't I don't view consciousness as some kind of like primitive, right there' some kind of panpsychist belief that you know everything is a little bit cautious. Nora's consciousness is just just ah a product of like a lot of information processing. This integrated information theory would have it. Consciousness seems very specific and very functional.
01:57:21
Speaker
but we We are mostly unconscious. and We are 99% philosophical zombies ah for for almost everything that we do. And the conscious experience we do have is very limited in some ways. you know Our brain tricks us into thinking we're we're more conscious than we really are. But then you introspect and think about you know what is what are we conscious of? And we tend to be conscious of things that surprise us, new things enter the scene when we're in the process of learning a new skill, we're hyper-conscious, practicing something that we were already adept at, like driving a car, we can go on autopilot and sort of steer and drive and get to our destination with barely paying attention. And so consciousness in terms of the sensational experience and inner theater that we sense that we have seems deeply related to learning.
01:58:08
Speaker
and and our ability to form coherence models of the world. I guess the question there is often for for any function you can name or any evolutionary pressure that could have resulted in consciousness, you could always imagine some agent that can do that thing without being conscious. But this goes to the the point about sort of us being mostly unconscious, right? We are mostly When we're driving a car, if we're an experienced driver, we are driving unconsciously. and And so consciousness only enters the equation if we need to learn new things. So consciousness seems very deeply interconnected with both online learning, ah our ability to learn in real time, and how we allocate attention, as well as coherence. You cannot be conscious of something incoherent. Our mind strives to make the things that we perceive in some ways coherent.
01:58:58
Speaker
If if it it fails, then you just see noise, right? And I'm really just recapitulating Yoshua Boxview, because I think he's done the best thinking on this, where he talks about consciousness as a kind of coherence operator. And you know he kind of flips the script. He says, well, actually, you maybe consciousness is the first thing we develop as babies, right? Because we are cautious from our earliest moments, at least from memory. and only and But that consciousness seems integral to our ability to then acquire skills that then build upon those skills and get us to a high level of agency. So consciousness may come first, not last. And and if that's the case, then it may actually be much simpler than people imagine. It may be a product of self-organizing forms of of intelligence. We don't come equipped with a trillion parameters, then with you know initialized weights, and then know and our our brain is a self-organizing system.
01:59:53
Speaker
that builds upon itself from the inside out. And Cauchy's, from that perspective, may be the most natural kind of learning algorithm that is acquired by such self-organizing systems. Do you think it'll be replicated by AI then? Do you think we will learn to create artificial consciousness for the purpose of learning more efficiently perhaps? Yeah, and this this is also Yosh's conjecture is that one of the reasons why current generation models struggle on and ah with with both coherence and agency is deeply related to their lack of consciousness. And you know it may be that future systems need to have some kind of proto consciousness to be as sample efficient as we want them to be, to have the coherence over time that we want them to have as agents.
02:00:39
Speaker
Interesting. I guess on on the flip side, you could also worry that consciousness is computationally costly in some way. And so any system that's conscious would be outcompeted by a system with the same function, but without consciousness. but with that system So if consciousness is integral to online learning, would that system be frozen in amber?
02:00:59
Speaker
I think you can get to basically any level of intelligence without being conscious. And at least I think i think it's not necessary to reach a certain level of of intelligence. I can imagine a superintelligence without any level of consciousness at all. like I can as well, but I think it would, if you want a system that can and learn from experience and learn from new sensory data and new inputs,
02:01:27
Speaker
cautious Consciousness may be the way you do that. And because we are embodied and embedded in our own inner theater, it's hard for us to step outside and imagine something that's not. But like I said, there are costs to consciousness. People who are very productive at work, we call that getting into the flow state. And in some ways, being in a flow state is being less conscious. I sometimes can only get into the flow state late at night where parts of my brain are starting to power down and I can sort of just lose myself in my work. And i'm you know I'm slightly conscious, but the fact that I'm so productive is actually due to the fact that i my inner monologue and my self-consciousness is dialed down. But on the other hand, if you are a sleepwalker, you have all the faculties, know your brain is still there, but you've lost the coherence operator that is letting you act coherently in the world.
02:02:23
Speaker
and so there there may be a little trade-off here. And if Yosha is right about this, about consciousness being first rather than last, then it probably isn't that computationally intensive. It's probably just what it is what it feels like to be a unified system that is striving for coherence with a degree of recurrence and you know the kind of strange loop going on and in how it allocates attention.
02:02:48
Speaker
What does this mean for AI safety? I guess one worry is that the future will be dominated by just very smart systems that are unconscious and therefore some some form of moral value will be lost. But perhaps if if if this theory that you're laying out here is is right and and consciousness is deeply involved in learning. Perhaps there are some future AI systems with consciousness will will learn ah something about morality, ah something about what it what we mean when we say pleasure and pain, good and bad. Do you think do you think it it would be good for for AI safety if if if consciousness is is somehow involved in in the and the process of becoming more intelligent?
02:03:33
Speaker
i think I think so. you know Again, I'm um um really echoing Yoshua on this where there's ah there's a lot of danger in building kind of unconscious golems that that can do everything we can do but have no inner light. In part because one of the other forms, you know a lot of human learning is done through social learning. Arguably most of our learning is done through social learning, through mimicry, imitation, adopting the norms and wars of of the our peer group. and That capacity to learn is also related to consciousness.
02:04:03
Speaker
and this is why we we say when you're, you know, you so you feel self-conscious when you're you know in an environment that you feel awkward in because you're sort of attending to your own ah behaviors and mannerisms to make sure that they match the the convention. What that means in practice is that like consciousness is integral to our ability to form resonance with other agents, right? and when When you're in a deep conversation with somebody, you can feel a kind of the boundary between your consciousnesses start to dissolve somewhat. And this is, I think, integral to empathy and why empathy is also related to learning. And if we want advanced AI systems to have sort of be in our image right and so and to see themselves and ask them vice versa, ah then they need some capacity to to resonate with us on a conscious level. That would be my intuition. And then and i also I also just think that like
02:05:00
Speaker
if things do go haywire and Rogay eye takes over and the humans are dead end in the evolutionary tree, I'd much rather that that new species have consciousness. But this is this is also, I admit, a chauvinistic point of view.
02:05:14
Speaker
Yeah, I guess all of this is is extremely speculative, but it's still important. I mean, so someone has to think about these topics. yeah You write somewhere that if you imagine a future and it doesn't seem like sci-fi, you're on the wrong track or something like that. And and this is perhaps but what we're doing here. All right. Do you want to talk about AI and children? Sure.
Raising Children in an AI-driven World
02:05:35
Speaker
Yeah. ah how How should we think of having kids if we are on the verge of of very advanced AI?
02:05:41
Speaker
ah You should still have them. you know There's a major dearth of kids. Again, I i am showing my cards. I am a species. I am a human chauvinist.
02:05:53
Speaker
you know there's a question you know With these questions about whether we should pause or delay AI or make sure that we're civilizations capable for superintelligence, there's always the question of, okay, let's say we pause for 50 years. What are we hoping to gain? What is the world that we want on the other end?
02:06:10
Speaker
yeah know what what but future do we want to build? And who are we building it for, but for another generation? Yeah, so we want we want to align advanced AI with the interests of humanity, I think, if if we choose to pause. And that's the project we are undertaking during the pause. Right. And the future of humanity will be our progeny. Right. And maybe Richard Sutton wants them to be our our AI mindchildren.
02:06:36
Speaker
But I think there's lots of ways that that kind of post human transition could go terribly wrong. Yeah. One of the most obvious ways is just leads to a kind of splintering. We share a lot of cognitive architecture. We share a lot of software and hardware as members of the same species. And this is what enables language games and culture and everything great. Without that common software layer, you know, we'd be but suffer from, you know, Wittgenstein's private language argument. We wouldn't be able to talk to each other. And there's worlds where, you know,
02:07:04
Speaker
ah posthuman, the the the opening ah of the mere technological possibility of sort of altering our genetic code and you know adding new cognitive prostheses and having new sensory inputs and so on could lead to totally different kinds of humans that don't share any of that common experience. And that would be a loss. I think it would be a loss precisely because even if you had something that was individually superior, it wouldn't have a shared community.
02:07:31
Speaker
yeah I wonder and wanted to what extent we should protect our children from living in a world that's it's too far removed from the world we evolved in, where there's too much of this kind of mismatch between our evolutionary psychology and the technology we're encountering. I guess there's TikTok, and you could you could imagine advanced versions of kind of AI-enhanced TikTok,
02:07:59
Speaker
Is that something you want to hand to your to your child? I don't think so. But on the other hand, you can't hide your your your child from the world forever, unless you want to go the army shroud. So how do we how do we integrate technology into our lives in a wise way? Yeah, there's a sense in which you could, you know, we already have a rogue AI and his name is Mr. Beast. You know, he is, is monumentically optimizing for the YouTube algorithm.
02:08:28
Speaker
and using our cognitive ticks to addict children to ah clicking subscribe because they might win a Tesla, you know, and they never do. But my hope, and this applies to kids and adults as well, you know, an ideal world is one where AI helps us level up in our rationality rather than to be used to exploit the foibles and and the gaps in our rationality, right? And I tend to think,
02:08:58
Speaker
that rationality is not a property of an individual or even a reasoning engine. It's it's a social achievement. We are rational because we have other people.
02:09:09
Speaker
right because We have the ability to check each other's biases, to check each other's work, to externalize our cognition into into the built environment. And that enables us to be more rational. you know Someone who is has a gambling addiction is is less rational than somebody who lays their clothes out, running clothes before bed, so that they can be prepared to go running in the morning. right and and all that yeah But that self-control and that rationality that people achieve is is mostly not from their ability to think harder.
02:09:38
Speaker
or because of innate factors. There are innate factors that correlate with self-control and so forth, but what it really is is they have better social scaffolding, right? It's easier to diet if you don't have cereal in your cupboard, right? But there's a person intervening if that person notices you spending too much money on gambling sites. Yeah, Oprah Winfrey can afford to have a person whose only job is to slap the donut of her hand, right? And that enables her to be in some level more rational.
02:10:07
Speaker
but have more agency. right most at The most competent CEOs have very competent executive assistants and all kinds of social staff living around them that makes them more agentic and more in control of their higher order preferences. And in some level, rationality is not just, can you execute a chain of logical reasoning, but can you align your practical reason, your behavior with your higher order preference?
02:10:34
Speaker
so I have a piece on implications of AI for ADHD, for instance, where you know I would love to have, as someone of ADHD, I would love to have super intelligent executive assistant that not only was making sure my bills were paid, but was sort of my my tandem partner in work to keep me on track um and to you know give me the nudges and a very sophisticated yeah many hidden layers deep way to to make me a better version of myself. You're imagining the the kind of AI life coach, assistant, a psychologist all packaged into one model, perhaps. I would i would like such a thing too. and i And I would like it to be aligned to my interests, not the interests of cooperation. And I and
02:11:19
Speaker
but but also your specific higher order interests, right? Because everyone has the their aspirational self and their and their actual self, right? This was going back to Aristotle talked about the weakness of will, right? And humans, it'll be very easy for AI to pass us and in terms of agency because humans are very weak agency. It's very hard. you know we go to We pay money to go to spin classes, right just so because it's easier to bike if everyone else is around us is biking. And so that collective social achievement of rationality is something that AI could worsen or could undermine or enhance. It could exploit our or my myopia, or it could build a kind of codex around us to warn us when our myopia is being exploited by someone else.
02:12:05
Speaker
Yeah, it's it's a difference between laying in bed, isolated, so scrolling TikTok and getting a notification from your AI assistant on your phone telling you, yeah here's why you should go to bed early and lay out your running clothes and your vegetable smoothie for but tomorrow morning. Or ideally the robot does that for you.
02:12:25
Speaker
Or the robot does that for you. Yeah, that's true. That's true. and so So I guess your general point is that we should implement technology in ways that that make us stronger. Yeah, and it and it just in precisely analogous way that we use a variety of scaffolding to make AIs better at reason.
02:12:43
Speaker
And to be even more agentic, we should use AIs to be scaffolding to make us better at reasoning and in agency. Does it mean that we should avoid simply outsourcing tasks ah to the AI and to the AIs? And and is that possible where when when AIs are increasingly better and than we are in in many domains? How do we avoid saying,
02:13:03
Speaker
Say your child comes to you and say, why do I have to learn calculus? This AI is is much better and will always be much better than I am at it. ah Why do I need to learn something? is it is it is it Does education and self-improvement become more of something you do for its entertainment value? Yeah, what's the argument there? It's a good question. I mean, we already have surveys where kids would rather be YouTubers than astronauts, right? so and some way you want to achieve. Is there really a lot of value in and learning calculus? I think that there's value in understanding yeah the first fundamental theorem of calculus and understanding some of the more conceptual parts of math, but you don't necessarily need to know how to grind out partial differential equations.
02:13:51
Speaker
that That's true, but I guess the the generalized form of of of the argument i'm I'm trying to make is just how what is what is what what is your answer when your when your kid asks you why why he should stay kind of informed and active in the world and trying to to hold on and understand what's going on, if it's so much easier to simply outsource tasks and goals to AI systems?
02:14:16
Speaker
maybe they shouldn't, right? I think we have a kind of conception of the liberal arts and humanities and what it means to be an informed citizen. I think this is part of the aspirations of our current psychological regime. No, 500 years ago, what mattered was, did you honor your mother and father? Did you work hard? And did you follow the right liturgy or something? And insofar as if AI does lead to a kind of organization and re-localization and kind of more communitarian world, then I think there won't be just one sort of right way to interact with the world. I think we'll have more deep diversity, right? We'll have different conceptions of the good. And ideally this is not just that you have a high score on the clash of plans, but that you are being fully human, right? That you are touching grass, that you're
02:15:13
Speaker
forming relationships and living the, you know, the Shakespearean tragedy of of life rather than, rather than just having everything drone delivered to you. Okay, last question. Should we accept relationships, friendships or romantic relationships between AIs and humans? ah We should tolerate them. yeah she's worth We should accept them.
02:15:38
Speaker
Again, this goes to whether we use AI to enhance our rationality or to exploit the weaknesses in our rationality. And the long-term answer to this won't be simply passing a loss and you can't have an AI girlfriend. It will be building building better tools that ah make you a better person so you can go get a real girlfriend. And so yeah this is, again, one area that's sort of differential. you know Maybe the D and&D act should be dating life.
02:16:05
Speaker
That's good. Okay, Sam, thanks for talking with me. It's been a real pleasure. As always, thanks, guys.