Esben Kran joins the podcast to discuss why securing AGI requires more than traditional cybersecurity, exploring new attack surfaces, adaptive malware, and the societal shifts needed for resilient defenses. We cover protocols for safe agent communication, oversight without surveillance, and distributed safety models across companies and governments.   

Learn more about Esben's work at: https://blog.kran.ai  

00:00 – Intro and preview 

01:13 – AGI security vs traditional cybersecurity 

02:36 – Rebuilding societal infrastructure for embedded security 

03:33 – Sentware: adaptive, self-improving malware 

04:59 – New attack surfaces 

05:38 – Social media as misaligned AI 

06:46 – Personal vs societal defenses 

09:13 – Why private companies underinvest in security 

13:01 – Security as the foundation for any AI deployment 

14:15 – Oversight without a surveillance state 

17:19 – Protocols for safe agent communication 

20:25 – The expensive internet hypothesis 

23:30 – Distributed safety for companies and governments 

28:20 – Cloudflare’s “agent labyrinth” example 

31:08 – Positive vision for distributed security 

33:49 – Human value when labor is automated 

41:19 – Encoding law for machines: contracts and enforcement 

44:36 – DarkBench: detecting manipulative LLM behavior 

55:22 – The AGI endgame: default path vs designed future 

57:37 – Powerful tool AI 

01:09:55 – Fast takeoff risk 

01:16:09 – Realistic optimism

AGI Security: How We Defend the Future (with Esben Kran)

Benjamin Todd joins the podcast to discuss how reasoning models changed AI, why agents may be next, where progress could stall, and what a self-improvement feedback loop in AI might mean for the economy and society. We explore concrete timelines (through 2030), compute and power bottlenecks, and the odds of an industrial explosion. We end by discussing how people can personally prepare for AGI: networks, skills, saving/investing, resilience, citizenship, and information hygiene.  

Follow Benjamin's work at: https://benjamintodd.substack.com  

Timestamps: 

00:00 What are reasoning models?  

04:04 Reinforcement learning supercharges reasoning 

05:06 Reasoning models vs. agents 

10:04 Economic impact of automated math/code 

12:14 Compute as a bottleneck 

15:20 Shift from giant pre-training to post-training/agents 

17:02 Three feedback loops: algorithms, chips, robots 

20:33 How fast could an algorithmic loop run? 

22:03 Chip design and production acceleration 

23:42 Industrial/robotics loop and growth dynamics 

29:52 Society’s slow reaction; “warning shots” 

33:03 Robotics: software and hardware bottlenecks 

35:05 Scaling robot production 

38:12 Robots at ~$0.20/hour?  

43:13 Regulation and humans-in-the-loop 

49:06 Personal prep: why it still matters 

52:04 Build an information network 

55:01 Save more money 

58:58 Land, real estate, and scarcity in an AI world 

01:02:15 Valuable skills: get close to AI, or far from it 

01:06:49 Fame, relationships, citizenship 

01:10:01 Redistribution, welfare, and politics under AI 

01:12:04 Try to become more resilient  

01:14:36 Information hygiene 

01:22:16 Seven-year horizon and scaling limits by ~2030

Reasoning, Robots, and How to Prepare for AGI (with Benjamin Todd)

On this episode, Calum Chace joins me to discuss the transformative impact of AI on employment, comparing the current wave of cognitive automation to historical technological revolutions. We talk about "universal generous income", fully-automated luxury capitalism, and redefining education with AI tutors. We end by examining verification of artificial agents and the ethics of attributing consciousness to machines.  

Learn more about Calum's work here: https://calumchace.com  

Timestamps:  

00:00:00  Preview and intro 

00:03:02  Past tech revolutions and AI-driven unemployment 

00:05:43  Cognitive automation: from secretaries to every job 

00:08:02  The “peak horse” analogy and avoiding human obsolescence 

00:10:55  Infinite demand and lump of labor 

00:18:30  Fully-automated luxury capitalism 

00:23:31  Abundance economy and a potential employment cliff 

00:29:37  Education reimagined with personalized AI tutors 

00:36:22  Real-world uses of LLMs: memory, drafting, emotional insight 

00:42:56  Meaning beyond jobs: aristocrats, retirees, and kids 

00:49:51  Four futures of superintelligence 

00:57:20  Conscious AI and empathy as a safety strategy 

01:10:55  Verifying AI agents 

01:25:20  Over-attributing vs under-attributing machine consciousness

From Peak Horse to Peak Human: How AI Could Replace Us (with Calum Chace)

On this episode, Tom Davidson joins me to discuss the emerging threat of AI-enabled coups, where advanced artificial intelligence could empower covert actors to seize power. We explore scenarios including secret loyalties within companies, rapid military automation, and how AI-driven democratic backsliding could differ significantly from historical precedents. Tom also outlines key mitigation strategies, risk indicators, and opportunities for individuals to help prevent these threats.  

Learn more about Tom's work here: https://www.forethought.org  

Timestamps:  

00:00:00  Preview: why preventing AI-enabled coups matters 

00:01:24  What do we mean by an “AI-enabled coup”? 

00:01:59  Capabilities AIs would need (persuasion, strategy, productivity) 

00:02:36  Cyber-offense and the road to robotized militaries 

00:05:32  Step-by-step example of an AI-enabled military coup 

00:08:35  How AI-enabled coups would differ from historical coups 

00:09:24  Democratic backsliding (Venezuela, Hungary, U.S. parallels) 

00:12:38  Singular loyalties, secret loyalties, exclusive access 

00:14:01  Secret-loyalty scenario: CEO with hidden control 

00:18:10  From sleeper agents to sophisticated covert AIs 

00:22:22  Exclusive-access threat: one project races ahead 

00:29:03  Could one country outgrow the rest of the world? 

00:40:00  Could a single company dominate global GDP? 

00:47:01  Autocracies vs democracies 

00:54:43  Mitigations for singular and secret loyalties 

01:06:25  Guardrails, monitoring, and controlled-use APIs 

01:12:38  Using AI itself to preserve checks-and-balances 

01:24:53  Risk indicators to watch for AI-enabled coups 

01:33:05  Tom’s risk estimates for the next 5 and 30 years 

01:46:50  How you can help – research, policy, and careers

How AI Could Help Overthrow Governments (with Tom Davidson)

Anders Sandberg joins me to discuss superintelligence and its profound implications for human psychology, markets, and governance. We talk about physical bottlenecks, tensions between the technosphere and the biosphere, and the long-term cultural and physical forces shaping civilization. We conclude with Sandberg explaining the difficulties of designing reliable AI systems amidst rapid change and coordination risks.  

Learn more about Anders's work here: https://mimircenter.org/anders-sandberg  

Timestamps:  

00:00:00 Preview and intro 

00:04:20 2030 superintelligence scenario 

00:11:55 Status, post-scarcity, and reshaping human psychology 

00:16:00 Physical limits: energy, datacenter, and waste-heat bottlenecks 

00:23:48 Technosphere vs biosphere 

00:28:42 Culture and physics as long-run drivers of civilization 

00:40:38 How superintelligence could upend markets and governments 

00:50:01 State inertia: why governments lag behind companies 

00:59:06 Value lock-in, censorship, and model alignment 

01:08:32 Emergent AI ecosystems and coordination-failure risks 

01:19:34 Predictability vs reliability: designing safe systems 

01:30:32 Crossing the reliability threshold 

01:38:25 Personal reflections on accelerating change

What Happens After Superintelligence? (with Anders Sandberg)

On this episode, Daniel Kokotajlo joins me to discuss why artificial intelligence may surpass the transformative power of the Industrial Revolution, and just how much AI could accelerate AI research. We explore the implications of automated coding, the critical need for transparency in AI development, the prospect of AI-to-AI communication, and whether AI is an inherently risky technology. We end by discussing iterative forecasting and its role in anticipating AI's future trajectory.  

You can learn more about Daniel's work at: https://ai-2027.com and https://ai-futures.org  

Timestamps:  

00:00:00 Preview and intro 

00:00:50 Why AI will eclipse the Industrial Revolution  

00:09:48 How much can AI speed up AI research?  

00:16:13 Automated coding and diffusion 

00:27:37 Transparency in AI development  

00:34:52 Deploying AI internally  

00:40:24 Communication between AIs  

00:49:23 Is AI inherently risky? 

00:59:54 Iterative forecasting

Why the AI Race Ends in Disaster (with Daniel Kokotajlo)

On this episode, Daniel Susskind joins me to discuss disagreements between AI researchers and economists, how we can best measure AI’s economic impact, how human values can influence economic outcomes, what meaningful work will remain for humans in the future, the role of commercial incentives in AI development, and the future of education.  

You can learn more about Daniel's work here: https://www.danielsusskind.com  

Timestamps:  

00:00:00 Preview and intro  

00:03:19 AI researchers versus economists  

00:10:39 Measuring AI's economic effects  

00:16:19 Can AI be steered in positive directions?  

00:22:10 Human values and economic outcomes 

00:28:21 What will remain for people to do?  

00:44:58 Commercial incentives in AI 

00:50:38 Will education move towards general skills? 

00:58:46 Lessons for parents

Preparing for an AI Economy (with Daniel Susskind)

Ed Newton-Rex joins me to discuss the issue of AI models trained on copyrighted data, and how we might develop fairer approaches that respect human creators. We talk about AI-generated music, Ed’s decision to resign from Stability AI, the industry’s attitude towards rights, authenticity in AI-generated art, and what the future holds for creators, society, and living standards in an increasingly AI-driven world.  

Learn more about Ed's work here: https://ed.newtonrex.com  

Timestamps:  

00:00:00 Preview and intro  

00:04:18 AI-generated music  

00:12:15 Resigning from Stability AI  

00:16:20 AI industry attitudes towards rights 

00:26:22 Fairly Trained  

00:37:16 Special kinds of training data  

00:50:42 The longer-term future of AI  

00:56:09 Will AI improve living standards?  

01:03:10 AI versions of artists  

01:13:28 Authenticity and art  

01:18:45 Competitive pressures in AI 

01:24:06 Priorities going forward

Will AI Companies Respect Creators' Rights? (with Ed Newton-Rex)

On this episode, Sarah Hastings-Woodhouse joins me to discuss what benchmarks actually measure, AI’s development trajectory in comparison to other technologies, tasks that AI systems can and cannot handle, capability profiles of present and future AIs, the notion of alignment by default, and the leading AI companies’ vague AGI plans. We also discuss the human psychology of AI, including the feelings of living in the "fast world" versus the "slow world", and navigating long-term projects given short timelines.  

Timestamps:  

00:00:00 Preview and intro

00:00:46 What do benchmarks measure?  

00:08:08 Will AI develop like other tech?  

00:14:13 Which tasks can AIs do? 

00:23:00 Capability profiles of AIs  

00:34:04 Timelines and social effects 

00:42:01 Alignment by default?  

00:50:36 Can vague AGI plans be useful? 

00:54:36 The fast world and the slow world 

01:08:02 Long-term projects and short timelines

AI Timelines and Human Psychology (with Sarah Hastings-Woodhouse)

On this episode, Michael Nielsen joins me to discuss how humanity's growing understanding of nature poses dual-use challenges, whether existing institutions and governance frameworks can adapt to handle advanced AI safely, and how we might recognize signs of dangerous AI. We explore the distinction between AI as agents and tools, how power is latent in the world, implications of widespread powerful hardware, and finally touch upon the philosophical perspectives of deep atheism and optimistic cosmism.

Timestamps:  

00:00:00 Preview and intro 

00:01:05 Understanding is dual-use  

00:05:17 Can we handle AI like other tech?  

00:12:08 Can institutions adapt to AI?  

00:16:50 Recognizing signs of dangerous AI 

00:22:45 Agents versus tools 

00:25:43 Power is latent in the world 

00:35:45 Widespread powerful hardware 

00:42:09 Governance mechanisms for AI 

00:53:55 Deep atheism and optimistic cosmism

Could Powerful AI Break Our Fragile World? (with Michael Nielsen)

The Future of Life Institute (FLI) is a nonprofit working to reduce global catastrophic and existential risk from powerful technologies. In particular, FLI focuses on risks from artificial intelligence (AI), biotechnology, nuclear weapons and climate change. The Institute's work is made up of three main strands: grantmaking for risk reduction, educational outreach, and advocacy within the United Nations, US government and European Union institutions. FLI has become one of the world's leading voices on the governance of AI having created one of the earliest and most influential sets of governance principles: the Asilomar AI Principles.

Future of Life Institute Podcast

Gus Docker's Show

The year is 2045. Humanity is not extinct, nor living in a dystopia. It has averted climate disaster, major wars and the risks from advanced Artificial Intelligence. How? This was what we asked the entrants of our Worldbuilding Contest to imagine last year.

Our new podcast series digs deeper into the eight winning entries, their ideas and solutions, the diverse teams behind them and the challenges they faced. You might love some; others you might not choose to inhabit. FLI is not endorsing any one idea. Rather, we hope to grow the conversation about what futures people get excited about.

Ask yourself, with each episode, is this a world you’d want to live in? And if not, what would you prefer?

Listen on Spotify, Apple Podcasts, YouTube, or wherever you get your podcasts.

Explore the worldbuilds discussed in these episodes here: https://worldbuild.ai/winners

This podcast was produced by the Future of Life Institute. FLI is a non-profit that works to reduce large-scale risks from transformative technologies and promote the development and use of these technologies to benefit all life on Earth.

Gus Docker

Shows