AI Computing Hardware - Past, Present, and Future

A special one-off episode with a deep dive into the past, present, and future of how computer hardware makes AI possible.

Join our brand new Discord here! https://discord.gg/nTyezGSKwP

Hosted by Andrey Kurenkov and guest-hosted by the folks from Latent Space

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.

Sponsors:

• The Generator - An interdisciplinary AI lab empowering innovators from all fields to bring visionary ideas to life by harnessing the capabilities of artificial intelligence.

In this episode:

 - Google and Mistral sign deals with AP and AFP, respectively, to deliver up-to-date news through their AI platforms.
 - ChatGPT introduces a tasks feature for reminders and to-dos, positioning itself more as a personal assistant.
 - Synthesia raises $180 million to enhance its AI video platform for generating videos of human avatars.
 - New U.S. guidelines restrict exporting AI chips to various countries, impacting Nvidia and other tech firms.

If you would like to become a sponsor for the newsletter, podcast, or both, please fill out this form.

Timestamps:

• 00:00:00 Introduction
• 00:03:08 Historical Recap: Early AI and Hardware
• 00:11:51 The Rise of GPUs and Deep Learning
• 00:15:39 Scaling Laws and the Evolution of AI Models
• 00:24:05 The Bitter Lesson and the Future of AI Compute
• 00:25:58 Moore's Law and Huang's Law
• 00:30:12 Memory and Logic in AI Hardware
• 00:34:53 Challenges in AI Hardware: The Memory Wall
• 00:37:08 The Role of GPUs in Modern AI
• 00:42:27 Fitting Neural Nets in GPUs
• 00:48:04 Batch Sizes and GPU Utilization
• 00:52:47 Parallelism in AI Models
• 00:55:53 Matrix Multiplications and GPUs
• 00:59:57 Understanding B200 and GB200
• 01:05:41 Data Center Hierarchy
• 01:13:42 High Bandwidth Memory (HBM)
• 01:16:45 Fabrication and Packaging
• 01:20:17 The Complexity of Semiconductor Fabrication
• 01:24:34 Understanding Process Nodes
• 01:28:26 The Art of Fabrication
• 01:33:17 The Role of Yield in Fabrication
• 01:35:47 The Photolithography Process
• 01:40:38 Deep Ultraviolet Lithography (DUV)
• 01:43:58 Extreme Ultraviolet Lithography (EUV)
• 01:51:46 Export Controls and Their Impact
• 01:54:22 The Rise of Custom AI Hardware
• 02:00:10 The Future of AI and Hardware