On this episode, Ben Goertzel joins me to discuss what distinguishes the current AI boom from previous ones, important but overlooked AI research, simplicity versus complexity in the first AGI, the feasibility of alignment, benchmarks and economic impact, potential bottlenecks to superintelligence, and what humanity should do moving forward.   

Timestamps:  

00:00:00 Preview and intro  

00:01:59 Thinking about AGI in the 1970s  

00:07:28 What's different about this AI boom?  

00:16:10 Former taboos about AGI 

00:19:53 AI research worth revisiting  

00:35:53 Will the first AGI be simple?  

00:48:49 Is alignment achievable?  

01:02:40 Benchmarks and economic impact  

01:15:23 Bottlenecks to superintelligence 

01:23:09 What should we do?

Facing Superintelligence (with Ben Goertzel)

On this episode, Jeff Sebo joins me to discuss artificial consciousness, substrate-independence, possible tensions between AI risk and AI consciousness, the relationship between consciousness and cognitive complexity, and how intuitive versus intellectual approaches guide our understanding of these topics. We also discuss AI companions, AI rights, and how we might measure consciousness effectively.  

You can follow Jeff’s work here: https://jeffsebo.net/  

Timestamps:  

00:00:00 Preview and intro 

00:02:56 Imagining artificial consciousness  

00:07:51 Substrate-independence? 

00:11:26 Are we making progress?  

00:18:03 Intuitions about explanations  

00:24:43 AI risk and AI consciousness  

00:40:01 Consciousness and cognitive complexity  

00:51:20 Intuition versus intellect 

00:58:48 AIs as companions  

01:05:24 AI rights  

01:13:00 Acting under time pressure 

01:20:16 Measuring consciousness  

01:32:11 How can you help?

Will Future AIs Be Conscious? (with Jeff Sebo)

On this episode, Zvi Mowshowitz joins me to discuss sycophantic AIs, bottlenecks limiting autonomous AI agents, and the true utility of benchmarks in measuring progress. We then turn to time horizons of AI agents, the impact of automating scientific research, and constraints on scaling inference compute. Zvi also addresses humanity’s uncertain AI-driven future, the unique features setting AI apart from other technologies, and AI’s growing influence in financial trading.  

You can follow Zvi's excellent blog here: https://thezvi.substack.com  

Timestamps:  

00:00:00 Preview and introduction  

00:02:01 Sycophantic AIs  

00:07:28 Bottlenecks for AI agents  

00:21:26 Are benchmarks useful?  

00:32:39 AI agent time horizons  

00:44:18 Impact of automating research 

00:53:00 Limits to scaling inference compute  

01:02:51 Will the future go well for humanity?  

01:12:22 A good plan for safe AI  

01:26:03 What makes AI different?  

01:31:29 AI in trading

Understanding AI Agents: Time Horizons, Sycophancy, and Future Risks (with Zvi Mowshowitz)

On this episode, Jeffrey Ding joins me to discuss diffusion of AI versus AI innovation, how US-China dynamics shape AI’s global trajectory, and whether there is an AI arms race between the two powers. We explore Chinese attitudes toward AI safety, the level of concentration of AI development, and lessons from historical technology diffusion. Jeffrey also shares insights from translating Chinese AI writings and the potential of automating translations to bridge knowledge gaps.  

You can learn more about Jeffrey’s work at: https://jeffreyjding.github.io  

Timestamps:  

00:00:00 Preview and introduction  

00:01:36 A US-China AI arms race?  

00:10:58 Attitudes to AI safety in China  

00:17:53 Diffusion of AI  

00:25:13 Innovation without diffusion  

00:34:29 AI development concentration  

00:41:40 Learning from the history of technology  

00:47:48 Translating Chinese AI writings  

00:55:36 Automating translation of AI writings

Inside China’s AI Strategy: Innovation, Diffusion, and US Relations (with Jeffrey Ding)

On this episode, Allison Duettmann joins me to discuss centralized versus decentralized AI, how international governance could shape AI’s trajectory, how we might cooperate with future AIs, and the role of AI in improving human decision-making. We also explore which lessons from history apply to AI, the future of space law and property rights, whether technology is invented or discovered, and how AI will impact children. 

You can learn more about Allison's work at: https://foresight.org  

Timestamps:  

00:00:00 Preview 

00:01:07 Centralized AI versus decentralized AI  

00:13:02 Risks from decentralized AI  

00:25:39 International AI governance  

00:39:52 Cooperation with future AIs  

00:53:51 AI for decision-making  

01:05:58 Capital intensity of AI 

01:09:11 Lessons from history  

01:15:50 Future space law and property rights  

01:27:28 Is technology invented or discovered?  

01:32:34 Children in the age of AI

How Will We Cooperate with AIs? (with Allison Duettmann)

On this episode, Steven Byrnes joins me to discuss brain-like AGI safety. We discuss learning versus steering systems in the brain, the distinction between controlled AGI and social-instinct AGI, why brain-inspired approaches might be our most plausible route to AGI, and honesty in AI models. We also talk about how people can contribute to brain-like AGI safety and compare various AI safety strategies.  

You can learn more about Steven's work at: https://sjbyrnes.com/agi.html  

Timestamps:  

00:00 Preview  

00:54 Brain-like AGI Safety 

13:16 Controlled AGI versus Social-instinct AGI  

19:12 Learning from the brain  

28:36 Why is brain-like AI the most likely path to AGI?  

39:23 Honesty in AI models  

44:02 How to help with brain-like AGI safety  

53:36 AI traits with both positive and negative effects  

01:02:44 Different AI safety strategies

Brain-like AGI and why it's Dangerous (with Steven Byrnes)

On this episode, Ege Erdil from Epoch AI joins me to discuss their new GATE model of AI development, what evolution and brain efficiency tell us about AGI requirements, how AI might impact wages and labor markets, and what it takes to train models with long-term planning. Toward the end, we dig into Moravec’s Paradox, which jobs are most at risk of automation, and what could change Ege's current AI timelines.  

You can learn more about Ege's work at https://epoch.ai  

Timestamps:  00:00:00 – Preview and introduction 

00:02:59 – Compute scaling and automation - GATE model 

00:13:12 – Evolution, Brain Efficiency, and AGI Compute Requirements 

00:29:49 – Broad Automation vs. R&D-Focused AI Deployment 

00:47:19 – AI, Wages, and Labor Market Transitions 

00:59:54 – Training Agentic Models and Long-Term Planning Capabilities 

01:06:56 – Moravec’s Paradox and Automation of Human Skills 

01:13:59 – Which Jobs Are Most Vulnerable to AI? 

01:33:00 – Timeline Extremes: What Could Change AI Forecasts?

How Close Are We to AGI? Inside Epoch’s GATE Model (with Ege Erdil)

In this special episode, we feature Nathan Labenz interviewing Nicholas Carlini on the Cognitive Revolution podcast. Nicholas Carlini works as a security researcher at Google DeepMind, and has published extensively on adversarial machine learning and cybersecurity. Carlini discusses his pioneering work on adversarial attacks against image classifiers, and the challenges of ensuring neural network robustness. He examines the difficulties of defending against such attacks, the role of human intuition in his approach, open-source AI, and the potential for scaling AI security research.  

00:00 Nicholas Carlini's contributions to cybersecurity

08:19 Understanding attack strategies 

29:39 High-dimensional spaces and attack intuitions 

51:00 Challenges in open-source model safety 

01:00:11 Unlearning and fact editing in models 

01:10:55 Adversarial examples and human robustness 

01:37:03 Cryptography and AI robustness 

01:55:51 Scaling AI security research

Special: Defeating AI Defenses (with Nicholas Carlini and Nathan Labenz)

On this episode, I interview Anthony Aguirre, Executive Director of the Future of Life Institute, about his new essay Keep the Future Human: https://keepthefuturehuman.ai   

AI companies are explicitly working toward AGI and are likely to succeed soon, possibly within years. Keep the Future Human explains how unchecked development of smarter-than-human, autonomous, general-purpose AI systems will almost inevitably lead to human replacement. But it doesn't have to. Learn how we can keep the future human and experience the extraordinary benefits of Tool AI...  

Timestamps:  

00:00 What situation is humanity in? 

05:00 Why AI progress is fast  

09:56 Tool AI instead of AGI 

15:56 The incentives of AI companies  

19:13 Governments can coordinate a slowdown 

25:20 The need for international coordination  

31:59 Monitoring training runs  

39:10 Do reasoning models undermine compute governance?  

49:09 Why isn't alignment enough?  

59:42 How do we decide if we want AGI?  

01:02:18 Disagreement about AI  

01:11:12 The early days of AI risk

Keep the Future Human (with Anthony Aguirre)

On this episode, physicist and hedge fund manager Samir Varma joins me to discuss whether AIs could have free will (and what that means), the emerging field of AI psychology, and which concepts they might rely on. We discuss whether collaboration and trade with AIs are possible, the role of AI in finance and biology, and the extent to which automation already dominates trading. Finally, we examine the risks of skill atrophy, the limitations of scientific explanations for AI, and whether AIs could develop emotions or consciousness.  

You can find out more about Samir's work here: https://samirvarma.com   

Timestamps:  

00:00 AIs with free will? 

08:00 Can we predict AI behavior?  

11:38 AI psychology 

16:24 Which concepts will AIs use?  

20:19 Will we collaborate with AIs?  

26:16 Will we trade with AIs?  

31:40 Training data for robots  

34:00 AI in finance  

39:55 How much of trading is automated?  

49:00 AI in biology and complex systems 

59:31 Will our skills atrophy?  

01:02:55 Levels of scientific explanation  

01:06:12 AIs with emotions and consciousness?  

01:12:12 Why can't we predict recessions?

We Created AI. Why Don't We Understand It? (with Samir Varma)

The Future of Life Institute (FLI) is a nonprofit working to reduce global catastrophic and existential risk from powerful technologies. In particular, FLI focuses on risks from artificial intelligence (AI), biotechnology, nuclear weapons and climate change. The Institute's work is made up of three main strands: grantmaking for risk reduction, educational outreach, and advocacy within the United Nations, US government and European Union institutions. FLI has become one of the world's leading voices on the governance of AI having created one of the earliest and most influential sets of governance principles: the Asilomar AI Principles.

Future of Life Institute Podcast

Gus Docker's Show

The year is 2045. Humanity is not extinct, nor living in a dystopia. It has averted climate disaster, major wars and the risks from advanced Artificial Intelligence. How? This was what we asked the entrants of our Worldbuilding Contest to imagine last year.

Our new podcast series digs deeper into the eight winning entries, their ideas and solutions, the diverse teams behind them and the challenges they faced. You might love some; others you might not choose to inhabit. FLI is not endorsing any one idea. Rather, we hope to grow the conversation about what futures people get excited about.

Ask yourself, with each episode, is this a world you’d want to live in? And if not, what would you prefer?

Listen on Spotify, Apple Podcasts, YouTube, or wherever you get your podcasts.

Explore the worldbuilds discussed in these episodes here: https://worldbuild.ai/winners

This podcast was produced by the Future of Life Institute. FLI is a non-profit that works to reduce large-scale risks from transformative technologies and promote the development and use of these technologies to benefit all life on Earth.

Gus Docker

Shows