Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
84. (Part 2) Intelligence in Nature v. Machine Learning - an Interview with Brit Cruise
2.2k Plays1 year ago
Help Support The Podcast by clicking on the links below:
- Start YOUR podcast on ZenCastr!
Use my special link ZenCastr Discount to save 30% off your first month of any Zencastr paid plan - Visit our Patreon
Part 2/2 of the interview with Brit Cruise, creator of the YouTube channel "Art of the Problem," about interesting mathematics,, electrical and computer engineering problems.
In Part 1, we explored what 'intelligence' may be defined as by looking for examples of brains and proto-brains found in nature (including mold, bacteria, fungus, insects, fish, reptiles, and mammals).
In Part 2, we discuss aritifical neural nets and how they are both similar different from human brains, as well as the ever decreasing gap between the two.
Brit's YoutTube Channel can be found here: Art of the Problem - Brit Cruise
Transcript will be made available soon! Stay tuned. You may receive a transcript by emailing us at breakingmathpodcast@gmail.com.
Become a supporter of this podcast: https://www.spreaker.com/podcast/breaking-math-podcast--5545277/support.
Recommended
Transcript
Introduction with Britt Cruz
00:00:06
Speaker
Welcome to part two of a two-part interview featuring Britt Cruz of the YouTube channel Art of the Problem.
Recap of Intelligence Across Species
00:00:12
Speaker
In part one, we explored examples of intelligence in nature, including how brains work in fish, mammals, reptiles, birds, insects, and humans.
AI and Machine Learning Overview
00:00:21
Speaker
And in part two, we'll be exploring how artificial intelligence works with machine learning and neural networks and how various architectures in machine learning have allowed machines to think a little bit more like humans.
00:00:35
Speaker
The transcript for this entire episode is available to anyone who's interested by emailing us at breakingmathpodcast at gmail.com and will be posted shortly to our website at breakingmath.io. And for listeners interested in a commercial free experience, all episodes are posted to Patreon for our $3 monthly tier.
Early Neural Network Models
00:00:59
Speaker
talk a little bit about the early research into machine learning and some of the clumsy early models of a neural network, ones where you had to manually change each dial. I was hoping that you could give us just a quick little preview of early research and the clumsy early models.
00:01:20
Speaker
Sure. Yeah, I'm glad you brought up a dial. So the original dream was let's make a mesh of wires with just neurons and wires connected. And a neuron is like, in electrical terms, it's a transistor. If it gets enough energy, it turns on.
00:01:37
Speaker
So you need something to be your neuron, and that thing basically you can use a transistor.
Learning Mechanisms in Neural Networks
00:01:44
Speaker
And the only other thing you need is connections. But we need to be able to change connections. And so this is why I use, and I made this up. I don't think they didn't use this, but I use the dimmer switch to hint at this idea of a dimmer switch as a variable resistor, which allows you to change the strength of that connection electrically.
00:02:07
Speaker
That's all you need. Then you have to give it, again, machine learning, all of learning, input, output, how to act. And so the first experiment was so great. Rosenblatz was just like he used like 50 neurons all connected together, thousands of wires, but he would either draw on like a light, bright style screen, very low resolution, a circle or a square, and then have the machine learn circle versus square.
00:02:34
Speaker
And how do you learn? Well, you've got to give it some experience. So he draw a circle, put it on an initially random mesh of connections. And that I need to amplify super clear because there's not much to machine learning if you got the core right. Random mesh of wires. Well, it doesn't work at all at first. What's not work me? Well, the output is forced to be one of two things.
00:02:55
Speaker
And we can call it circular square. And in this case, that's what he wanted to do. But it does nothing at first. So you put a circle in this machine, and what happens with the two light bulbs at the end? The machine doesn't know anything. They're both kind of lit up just kind of randomly.
00:03:11
Speaker
And so it has to learn. What's learning? Well, in this case, the human kind of cheats a bit where we show it a square. It does some random thing that doesn't work. Then one by one, we go through and wiggle each dimmer switch. And sometimes going one way will help, meaning, oh, I'm an eye human. I'm going to call, let's call the top light bulb a square and the bottom light bulb a circle.
00:03:36
Speaker
Any time I put a square in and I do a wiggle, when the right light bulb gets brighter, I'm going to keep that wiggle. If it doesn't help, I'm going to go the other way. And you literally just do that through all the neurons.
Generalization in AI Systems
00:03:49
Speaker
Keep doing that. And you hit a point where you put circles and squares in this network, and it doesn't need you to do any more wiggling. That's the time that it has generalized, which means it'll work on what it was trained on.
00:04:02
Speaker
But most importantly, it will work on new circles and squares you draw, different people with different handwriting, different pixels basically. And that is like what we're still doing today. Nothing really has changed.
00:04:17
Speaker
I was watching your videos is so theoretically, you could build a neural network out of like anything, you know, a bunch of breadboard pieces, you could build manual switches and like have it out on a table. And yes, it would be very clumsy to go there and manually change everything. And that's one of the reasons why machine learning didn't really take off until the hardware caught up. And I think the gosh, 2009, when graphics cards became available, I think that was a major turning point in artificial neural networks. Is that right?
Advancements in AI Processing
00:04:45
Speaker
That's right, because it can multiply a big chunk of numbers instead of one at a time. It wasn't anything special beyond a happy accident. Very cool. Very cool.
00:04:57
Speaker
There's two videos that we're going to watch from your series here that'll be up here in a second. One of them is on deep learning. It involves the paper folding analogy. And the other one is from the exact same video. Shortly after, it's on probing neural network layers, where if you've got a neural net that is trained on recognizing images, what happens when you peel back every single layer and you
00:05:18
Speaker
look at what's built up throughout. So I think without further ado, let's go ahead and do it. Allegra, if you could play the first video, I think it's called Paper Folding.
Deep Learning and Neural Network Layers
00:05:29
Speaker
Organic brains use layers of neuron activations to process their inputs. The importance of depth, or many layers, is the least understood aspect of neural networks. So let's pause and consider a simple analogy to understand why multilayered networks are better at partitioning the perception space than a single-layer network. Imagine this is our perception space and we have two kinds of input data types. Each neuron we add in the first layer acts like a fold in this space.
00:05:51
Speaker
With two neurons, we can make two folds, like this. And we could keep going, folding and unfolding the paper to carve out regions to separate these points. This will take six separate folds. This allows us to then group regions containing the same type of points using a final neuron, which activates if any of those regions are active. But now consider what happens if we layer our folds. That is, we don't unfold after each fold. So let's do the first fold again, then the second, then the third fold across that layer like this.
00:06:15
Speaker
That ends up carving the space in the exact same way, using three folds instead of six. And if we were to continue this process with a fourth fold, it results in 16 regions. Practically, this means that neurons deep in a network are not simple linear partitions, but are instead activated by a complex pattern of linear partitions.
00:06:37
Speaker
Awesome. Awesome. And you may have noticed there in order to fit all these videos in this awesome interview, I sped them up to 1.5, except the very last section that went back to its normal speed. But still, I think that was a fabulous video. Can you tell us a little bit about that video and just put it into more layman's terms, even though it's already in layman's terms? Yeah, you can't get, um, so when we're talking about neurons, they just have inputs and an output.
00:07:03
Speaker
One way when you want to think about it mathematically, and let's just use the example of a temperature where you're trying to make a neural network that can just, let's go to a thermostat, that you just set what you want your temperature, your dividing line. A neuron really in math world where you're just plotting temperature on a line, like 60 degrees versus 62,
00:07:24
Speaker
What is the neuron? It's actually just a divider. Think of it as a point on that line. Or in 2D space, if you have a couple variables, like a pressure sensor also attached, it's a line. It's a partition. It's just dividing sets of points, which are senses. A sense is just, in a mathematical term, a point on a line or a point on a plane into two categories. And that's how it knows one thing or is another or versus another, that essential difference that we build everything on.
00:07:55
Speaker
So that's all a neuron is. And when you put lots of neurons together, you get lots of partitions. And what do you do with lots of partitions? Well, you can define complicated shapes. And that allows you to capture more complex concepts. And so this is why there's a result in AI where you only need one layer, really, to do anything. A neural network.
00:08:20
Speaker
with an infinitely wide layer of neurons and all the connections going in and out. In theory, it can do everything. But in practice, it doesn't work because you can't make something big enough in practice to do problems. You need too many partitions, too many neurons. And the trick they found was if you layer neurons, so instead of trying to do 100 million neuron wide network, you just do 10.
00:08:46
Speaker
and then another layer connected to 10, and another layer connected to 10. Those actually multiply together, leading to the same amount of partitions with very few neurons. And so this sounds so complicated. So if you're confused, it makes sense. So I needed a physical analogy. And my physical analogy is
00:09:08
Speaker
you fold a piece of paper, that fold is a neuron. And so if you try to fold and unfold the paper to do all your folds, you have to do one fold per fold. But if you fold and then fold again and do that classic thing, or if you fold a piece of paper like 12 times, it goes to the moon. When you unfold your paper, every fold you end up with is like a neuron. So with just a few folds, you can get many folds.
00:09:37
Speaker
Wow. I think that concept right there is key to understanding some of the alchemy that machine learning is. You know what I mean? It's just how our conclusions arrived at through the combination of neurons. So that's a very cool one. And then we've got this next video. And then coming up, we're going to talk about a whole lot of examples of AI.
AI in Real-World Applications
00:09:53
Speaker
This is one of my favorite parts of the conversation. Then we're going to talk about AI and business as well. Just to give a quick heads up of what's left on this episode.
00:10:00
Speaker
So again, this next episode, we're going to see, I don't know what AI this was, but it's something that's trained on images. And we're going to see what happens at multiple layers. So let's see here. Allegra, if you could, can you play the third video? I sped this one up too, so it would go a little faster.
00:10:23
Speaker
Okay, I think they're gonna fight it. We probe neurons in the first layer of the network, we find they are detecting these patterns, which are looking for edges and points. Then they move to the next layer deeper into the network and probe those neurons to see what activates them. What they found was the next layers are activated by different kinds of textures. And deeper into the network, these textures get more specific. And as you move deeper into the network, the textures get more complex.
00:10:54
Speaker
And the deepest layers contain individual neurons that are activated by entire objects, such as dogs, wheels, houses, or trees. These complex activation patterns are possible due to the layered structure of the network. And so if we cut open a neural network, we'll find that deep layers contain representations of a perception based on what level of different things or patterns they contain, which is defined by how active those specific neurons are. Very cool. Yeah, I think that was a fabulous video. Where did you find that example, if I may ask?
00:11:22
Speaker
I believe that visualization is probably from the work of Chris Olah. They've done and distill have done a lot of great work visualizing the innards of neural networks. Nice, okay. I may ask you for that link. I'll include that in this video as well. If anyone wants to see some more visualizations of neural networks, I found some amazing ones myself. I'm a visual guy,
AI in Art and Language Generation
00:11:46
Speaker
very, very, very helpful.
00:11:47
Speaker
Next part of this podcast, I have a section where I want to talk about some of our favorite examples of AI that we've used and why, as well as some of the examples that are brought up in your videos that we haven't discussed yet. Do you have any favorite examples of AI in any use in industry or entertainment or elsewhere?
00:12:11
Speaker
What I find most impressive is not any specific use, but the pattern of what's happening. And I just put it simply. It can handle such messy output, input, unstructured, messy, noisy input, that all we need to do now to interface with computers is utterances.
00:12:34
Speaker
and gestures or one or the other. Just with the most simple utterances or simple hand waving, the machine has everything it needs to do what it needs to do. And so whether that means there's new graphic design software where I just kind of go make an ocean. And even when I say it that the sound's all fuzzy and maybe I pronounce it wrong and I'm like, and make the wave look like this. And I just whoosh my arm around. It's going to output a beautiful painting.
00:13:02
Speaker
And I remember as soon as this was possible, I made some for one year for holiday gifts. And so you just see that again and again and again, the simple utterances, look at chat GPT, but also just simple gestures is all you need to do the interface with the machine. And I think that's just, we're still getting our heads around that.
00:13:21
Speaker
That's amazing. I kind of think of like when a husband and a wife know each other after many, many years and you know, just from their facial expressions, how they feel about a certain situation, you know? So it's a, you know, I'm so well, you know, not even that, you know them just from the types of the types of silence, just like the way people breathe. It's incredible how our brains can absorb just what it needs to know.
00:13:41
Speaker
Oh yeah, yeah, absolutely. Now, there's a part at the very, very end of our outline here where we're talking about your talk at Google, and you had mentioned some really impressive AIs. I think one of them was, was it a student who you saw build a social media platform just with AI in a very, very short time? Can you talk about that a little bit?
00:14:02
Speaker
Yeah, so this is an example of a woman who kind of had no coding experience. And this is a you'll see this pattern everywhere now. That's why this year is exciting. And that said, oh, I want to kind of make a doing go meets Netflix for learning languages.
00:14:21
Speaker
And how the app works isn't so much important. What's important is she was able to prototype it in three days and be walking around on the street handing her app to somebody. And anyone who's tried to build anything knows going from idea to someone holding it in their phone testing it sometimes never happens or takes years or takes months. It doesn't take three days with some simple utterances.
00:14:45
Speaker
And so she was able to do that, and that's where we live now. So everything's being kind of restructured around that truth. That is amazing. Okay, so she said, Netflix meets Duolingo. So something, I mean, do you have more details? I'm curious about the app now, actually.
00:15:03
Speaker
I think it was. I know, right? So traditionally, you just get subtitles. This game, I'll just put it, gamify the subtitles. So imagine all, no one's even thought of that, which is, I don't have the app, but just imagine some words are different colors or maybe questions pop up on top of it. I don't know, but it was this idea of gamifying subtitles, like how brilliantly do lingo gamifies things. Yeah. Yeah.
00:15:28
Speaker
Oh, fascinating. Fascinating. Yeah, actually. And so how would you build that? You'd need multiple engineers and you'd need super, like you need a couple of engineers or one brilliant one and a lot of time, but she didn't. Okay. That is amazing. That's amazing. At work, we talk about AI all the time. And one thing we've thought about is a coworker of mine really loves the idea of AI avatars and he's trying to think of some use cases for them. We thought it would be amazing if you had some kind of an entertainment, like a
00:15:55
Speaker
murder mystery and you set out your audience as the investigators and they can interview AI witnesses or AI ghosts of characters that have passed on and get some information and obviously as an AI
00:16:12
Speaker
You could have all this information in it, but have some hidden knowledge where if you ask the right question with the right reference, it'll give you some key information. Then you'd see how successful distributed problem-solving really, really is, because we've been terrified at how
00:16:29
Speaker
how intelligent fans are at figuring out what movie writers and screenplay writers are doing in mystery. So that's one application that we haven't done yet, but I'd love to see it.
AI in Medical and Diverse Fields
00:16:43
Speaker
That would be really, really cool.
00:16:44
Speaker
I wanted to bring up this section about AIs specifically because it's the, I don't want to say strange, it's the unique ones that really show the capability of AI that I find most interesting. The example I previously mentioned is the AI called BakeryScan that was built for a Japanese bakery that is now used or a model of it is now used to identify
00:17:08
Speaker
pre-cancerous screens or tumors in an MRI screen. What fascinated me about this AI is in this Japanese bakery, there were two AIs that were created.
00:17:25
Speaker
one of them was done through reinforcement learning over a period of five years by a team of programmers that really understood the bakery and another one was done entirely unsupervised through deep learning, only machine learning. For the original purpose, which was identifying
00:17:42
Speaker
Baked goods in the bakery they were about on par they were about equal but then there's a critical task where one of them outperformed the other the critical task is learning new baked goods and you know i get to introduce a new shape or a new recipe you got a new item.
00:18:01
Speaker
how soon until your tool, your machine has learned it. And what they found, and I've got my citations here, what they found is that the supervised learning model learned it multiple orders of magnitude faster than the deep learning model. So that to me was fascinating. I didn't see that coming. I literally thought the deep learning model would learn it much faster, but that's not the case. So in that use case itself, I'm dying to know
00:18:28
Speaker
Why is it that the model trained by a human who knew the bakery inside and out and he knew what to look for, why is it able to learn new things faster than the deep learning model? I don't have an answer for that, but that's the kind of questions that are relevant right now.
00:18:47
Speaker
There's one potential confusion there, which is both are deep learning, meaning they're using layers of neurons, but they're using two different approaches, which is supervised and unsupervised. That's the key difference.
00:19:04
Speaker
What's neat about supervised, what it really means is the humans providing the reward every time. You give it bread and say what it is. Give it bread, say what it is. Do counter examples on both sides versus unsupervised where you're letting it learn that itself. In many cases and still to this day,
00:19:25
Speaker
I call this narrow siloed AI. When you use deep, again, it is deep learning, but it's humans providing the labels and the examples over and over. You get very high performing systems, but they're not generalizable, so they can't do other things. And that's why there's a cost benefit there when you go to more general systems that may fail narrow.
00:19:48
Speaker
Yeah, I hope that was helpful, but it definitely was. Yes, you're right. Thank you for your correction. I always want to be corrected. That's how I do things best. They are both deep learning. One of them is supervised, one of them is not. I also thought that in terms of acquiring knowledge for a specific task, in this case, the task is a human-centric task. Humans are the ones that make the baked goods. Humans are the ones that eat them and that evaluate what's good or what's not good for human purposes. Well, frankly speaking, we have insider knowledge on what makes something good for us.
00:20:15
Speaker
So that also is very helpful for something that learns new tasks. And that is, yeah, so I think that that's optimistic that humans will be in the loop for a lot of tasks in the future. We won't be obsolete quite so soon. At least that's my assumption. I don't know if you'd argue on that one.
00:20:32
Speaker
I don't want to argue. I want to throw in one fun example I didn't tell you about. Again, is this a failure mode or a feature? So on a different cancer detection test, they had an AI looking at cells and it could tell whether or not it was cancer. They gave it two examples, two batches of data, had it learned.
00:20:49
Speaker
And it did it better than a human. And then so, hey, you could stop there and say, wow, it's brilliant. They looked into why. And again, it learned how to act, act means say cancer or no cancer with what we thought was a hack, which is it noticed that in all the cancer images in the data set, there was a ruler in the image. And in all the non cancer, there wasn't a ruler in the image.
00:21:12
Speaker
And so what did it learn? It just learned immediately. You see ruler, it's cancer. And so that's a fun example of like, it'll learn what it needs to learn to do it. So if you take that ruler out, well, guess what? It's going to know there are other subtle differences and it will work. But I love that ruler example because is that a bug or a feature? I would say it's the feature.
00:21:31
Speaker
But as humans have to be careful with our data sets. That's the key. The data set can really, when we try to hand engineer it, I'm also very skeptical of that. That's the humans who want to get on the controls again. And I'm usually like, I don't know if that is the way forward.
00:21:45
Speaker
So obviously the lesson here is be very, very aware of your data set. And also, even if you're doing the deep learning unsupervised model, just be very aware of what you're training it on and what you're not training it on, what's missing. And for our purposes, that requires a lot of awareness there. Oh, and I keep being told that we are down to under 10 minutes.
00:22:12
Speaker
I want to ask my producers real quick if we can extend it a little bit, if that's all right. Mark, is it okay if we extend a little bit past our 10-minute deadline? Thank you, sir. I do appreciate that. Awesome. Awesome. Because this conversation is just so fun. So one other thing I'll talk about is any examples here. There's countless examples we could talk about. Even the history of chess and AI, we
00:22:36
Speaker
that early models were programmed with what we thought of as the best tricks from humans into AI. Later models said no, let the AI figure out how to play chess, and it was extremely successful. From the last I read, from 2020, so this is almost four years out of date now,
00:22:55
Speaker
is that in some cases, the best chess players were combinations of humans and the AI for certain tasks. I don't remember the parameters. I only saw a bunch of releases and mainstream media. But I don't know if you heard about humans and AI working together at certain tasks and how successful they are.
00:23:16
Speaker
Yeah, and broadly speaking, it never hurts. But again, it's tempting to be like, oh, yeah, that means human was why. And I'm like, no, it's actually just that the state of the art in the AI plus human doesn't get worse. And that's not so amazing to me, but it is the case. Yeah, for sure.
00:23:41
Speaker
Very good. And I guess this next section, and given our time limit, I'm trying to prioritize what we talk about in the rest of the conversation because this conversation is fascinating to no end. So I may jump around a little bit here. Are there any interesting complete failures of AI that you're interested in talking about?
AI and Physical Experience Limitations
00:24:00
Speaker
Maybe just that when I mentioned earlier, it's
00:24:07
Speaker
I want to say it's lack of awareness of itself, but I don't know how strong I feel about it right now. The current kind of boundary of what it can't seem to do is, on the one hand, people say it doesn't have a world model. It doesn't really understand the physical world because it didn't grow up crawling around and getting dirty.
00:24:29
Speaker
But word, so that's the boundary, like it doesn't really, with language, it can sound intelligent. But people's, their argument still, this is not my argument, is when a, when chat community says water, it hasn't felt water, fine, but that is,
00:24:46
Speaker
Currently the edge of research right now and it's amazing. I sent you one video of just teaching a simple robot to walk in the grass. We went from taking impossible to 20 weeks to two hours and this thing's babbling around learning to walk.
00:25:04
Speaker
So that video wasn't cool. The failure. Yeah. Right. Yeah. Yeah. It was absolutely incredible. And then actually, when you talk about how quickly the show about learn how to walk, the reason why we brought this up is I tried to argue on the side of humans. And I said, wait a minute. AI isn't as good as humans at everything. And I showed them some videos about
00:25:21
Speaker
the awesome robots from Boston Dynamics. These robots do amazing gymnastics and cartwheels and all these scary things when you look at them and you're like, oh no, it's like the Terminator. These things are better than us. But those specific examples, those robots have a very hard time with a task like sitting in different chairs that are different sizes or walking on a surface that's not flat. They're clumsy and they fall down if they don't perfectly understand their environment. So I thought,
00:25:49
Speaker
naively, that that was an example of where AIs have a long way to go. And I sent that to you, Britt, and you just came back with a video and you're like, ah, ah, ah, not so fast. So as you said earlier, in a period of two hours, a robot was able to learn.
00:26:07
Speaker
how to walk on all kinds of different services just because it was it was trained with modern machine learning methods on that task so that that was incredible. One of my questions real quick is an artificial brain.
Energy Efficiency in AI vs. Biology
00:26:20
Speaker
Yeah one of the questions that I've been emailing you a lot about lately is I'm trying to think about
00:26:25
Speaker
digital learning through a neural network versus a biological brain. Because we can learn to play chess. I imagine there's some rewiring. I think there's rewiring. I'm not sure how much actual physical rewiring is happening in our brain. But I think about power consumption of our brain versus a machine and what are the limitations and what
00:26:52
Speaker
A machine can learn in two hours, but really we're fighting against the fact that our rewiring is not just a digital thing. It's actually a physical thing, so it's more like a muscle. This is where my neuroscience lacks. I don't know if you know a lot about the actual human process of learning and pruning of neurons. How is that different than machines?
00:27:15
Speaker
I'll go with two examples on this power consumption thing, which is, of course, evolution has found a million awesome hacks. So one is that neural networks in the brain don't actually are on or off all the time. They're sending spikes at different rates. So fast is on and slower is off, and they can be continuous. And so on the one hand, that's smart just because it's only sending bits of energy once in a while for an on, not staying on. Duh.
00:27:44
Speaker
But also it gives it an ability to send continuous signals, which can process much more information. So it's always, can you do more with less? And another one, what we call in machine learning is pruning, as we know with the human brain, you could go in there and often just snip random neurons, and nothing's going to happen.
00:28:03
Speaker
It's only when you damage a big chunk, like billions of neurons, even then sometimes nothing happens. It's incredible. You have to like torch your brain and pull chunks out for it to fail unless you happen to hit this perfect neuron that's critical. So pruning, eh.
00:28:19
Speaker
In AI, what we do in machine learning is you just randomly, instead of having a fully connected network where everything's connected to everything, you just randomly start deleting stuff until it stops working. This is called a sparse network. You can get by with a very sparse network. Nature is doing those things and many more things we don't understand, but it's just doing more with less power. It's really cool.
00:28:46
Speaker
Yeah, that's incredible. Wow. There's a whole bunch of topics in this area. One thing I also thought about is we exist for some, on average, I don't know what the average lifespan of a human is, we obviously shed our mortal
Meta-Learning and Self-Improvement in AI
00:29:01
Speaker
coil. We die at some point, but we also have kids. We can propagate our DNA. And through mutations, we can get new hardware. That got me thinking about new hardware for a neural network.
00:29:15
Speaker
Obviously, a neural network is fixed on a server right now. There's no talk about a neural network itself producing new hardware. Now, I don't know how far along we are. That's where it gets scary. A neural network could be aware of its own hardware and design its own next iteration. So maybe we're not that far off. I don't know. Have you ever thought about a neural net designing its own next iteration?
00:29:42
Speaker
Oh, the Avengers. What was the Avengers movie with Age of Ultron where he kept designing his next iteration? Yes. And so this is called meta-learning and the results are amazing. And so of course it helps and it works. And so there's a lot of work that's been done on the, you're kind of going up to the meta level where the neural networks
00:30:05
Speaker
changing its own structure and to be more efficient. Okay. And it's cool. Check it out. Uh, metal learning, metal learning. Let me add that to this outline. Cause I need to add that, um, uh, yeah, metal learning. Okay. Uh, new hardware. We're going to add that to our outline and we're going to do a lot more on this topic. By the way, I want to do a quick little break here.
00:30:26
Speaker
Obviously my passion right now is machine learning and I'd love to go to be involved in machine learning research. We have a community that I've just started. There's a Discord server on the Breaking Math podcast and there's also a, oh gosh, what's the app that businesses use to collaborate?
AI Community and Future Prospects
00:30:45
Speaker
My wife just told me to download it and to use one for this podcast. Oh, Slack. Thank you. Sorry. That was kind of funny. Totally forgot the name. I just created a Slack and it's either Breaking Math Pod or it's Breaking Math Podcast.
00:31:01
Speaker
We're going to put interesting papers here on both Slack and Discord and we're going to have interesting concepts and explore these further and talk about what will be on future episodes. We're also going to provide the papers that we're talking about and the resources so you as a listener can get smart
00:31:19
Speaker
get smarter than we are right now by reading along. I'll also put Brit's channel as well. And I found a professor that teaches data science and machine learning who just put out a textbook. His Twitter handle or his X handle is EigenSteve.
00:31:36
Speaker
And he has an entire YouTube series with hundreds of videos where he goes through his textbook on machine learning. And it's phenomenal. I know that at his Twitter or X handle, he's got his entire textbook for free on PDF. He also has all of his Python code, as well as his Matlab code for almost all of the examples in his textbook.
00:31:58
Speaker
I only want to mention this because for our listeners and our viewers who really want to dive in deeper and be part of the conversation and see what's on the cutting edge of this, I want to make those resources available for you. It'd be wonderful to publish something. I'm not opposed to collaborating on things or just sharing these ideas. When we first began this interview, Brett, I wanted to say I was very encouraged because my background is
00:32:23
Speaker
more creative. I come from a storyteller background and I'm okay at engineering, but I found people who with traditional engineering tasks are just amazing and I'm just not that way. I'm much more creative, at least historically. But I find with designing machine learning architecture, this is a chance for creative to really shine. Can you talk about that real quick, just in the context of what you said to me before the interview started?
00:32:51
Speaker
Yeah, well like in any field that can stagnate and whatever, regardless of what you're talking about, you need what we think of as fresh ideas. Fresh or novel ideas are just ideas that aren't common.
00:33:03
Speaker
And so what does that mean? Often experts know so much about a certain area that it can blind them. It doesn't mean they're not creative, because you can have experts who are both. But what it also means is non-experts are almost as important as experts, because it's looking at something new and being like, why isn't it this way? Why don't you turn it around? Generative networks are running a neural network backwards.
00:33:30
Speaker
Actually, when you go through and look at the recent discoveries from Ian Goodfellow and onwards, it's like he said, he was out with his friends, they're having some beer, and someone's like, what if we had two networks compete with each other? And it led to all this explosion in image generation. So being creative, I used to think of as just either embracing or being good at unknowns or being around unknowns. So it's nice that you're new to a field
00:33:56
Speaker
and you're excited about the unknowns. If you're excited about the unknowns, that can be very useful. Nice. Very good. Now, one thing I also want to mention is you had mentioned before our episode today is that you have some future videos that you're working on with IEEE and just in general. Is there anything at this stage that you're able to tell us about? Or if not that, then other ideas that are in the future?
00:34:23
Speaker
Yeah so I make videos sometimes with the IEEE Information Theory Society and so we should be working on two this year and I think we're going to do one on quantum communication and the other one on I keep bugging them like information theory has to say something about modern day neural network kind of the explosion in applications and
00:34:49
Speaker
And because at the bottom of an information theory is kind of core, even more core than all the research. And there is information theory research happening, but want to pull out what we call the information bottleneck problem. You could think of it as given a neural network.
00:35:05
Speaker
how much can you squeeze it before it stops functioning? And this gets your question of optimization and how nature would do it. So information bottleneck and quantum, that's the two videos I wanna do with them. And the video I also just thinking about in my own time is the followup to my last video on chat GPT, which I think is the end of the AI series, and it will be on reinforcement learning.
00:35:28
Speaker
Awesome, awesome. And actually you sent me an email asking me about some resources I think I gave you two. The absolute best quantum mechanics book I've ever read and I've wanted to get the host, the author on this show actually, Jim Baggett. His book is phenomenal. Now that's the history of quantum mechanics going back from Max Planck inspired by Boltzmann and it goes all the way to
00:35:53
Speaker
Oh, about 2008. I believe it doesn't get far past that, but I've never ever read something that is as well spoken as that. The other channel I'd love to give a shout out to is the, um, through the Looking Glass universe. The name is changed. I used to be really good. Yes. She's awesome.
00:36:10
Speaker
Oh, yeah. And she's she talks about the nuance of things like there's no action, like, you know, the collapse of the wave function is a very misleading term because there's no actual collapse, you know, and she's very much a many worlds proponent, whereas somebody likes to be in Hosenfelder.
00:36:26
Speaker
who's also very, very good, just doesn't specialize as much in this area specifically. She offers some very, very good critiques of the many worlds problem. Yeah, there's some great resources. I hope those are at your disposal. I would love to work with IEEE. If they ever need a deep dive dialogue, oh man, if you know somebody, please send them my way. I haven't established a relationship with them.
00:36:50
Speaker
But that would be my dream. I'd love to work with IEEE. For our audience who doesn't know, IEEE, I sometimes say they're like the Vatican of electrical engineering. Essentially, it's the Institute for Electrical and Electronics Engineering. If you pick up
00:37:07
Speaker
That's right if basically if you pick up any just reach the nearest quarter electronic thing you have and if you look close enough with the numbers in there you're gonna find something i triple e. Was part of creating yes yeah i should believe is the standards the standards i'm underlying everything.
00:37:26
Speaker
They're incredible. There's so many things on this outline that sadly we didn't get to that I know we are running out here on time. I want to do one more video. There's one video that talks about art and language and it involves the field of linguistics, which is a human activity. I'm hoping to play this video real quick. Allegra, can you play the TikTok from the creator Etymology Nerd? There's actually a few of them, but this creator who made a video on art and language and I want us to watch it thinking about chat GPT and
00:38:00
Speaker
I don't think people realize that there is no difference between art and language. If you start with the earliest K paintings, they do the same thing words do. They serve as symbols representing a concept. Just like we can use the word ox to talk about the concept of an ox, this picture also communicates the idea of an ox. And it's precisely because of the blurred boundary between art and language that writing systems developed in the first place.
00:38:19
Speaker
Eventually, people realized that an ox could represent more than just an ox, so they start using stylized ox heads to represent sounds, and then that evolved into the letter A. And the same thing happened when the Impressionists and the Post-Impressionists realized they didn't have to paint reality as it is, but that they could abstract the representations of it. Then slowly but surely, painters like Picasso realized they could push their representations further and further from reality, which is why I hate it when people look at a painting like Elena Koning's abstraction of a standing bull and say, oh, my child could draw that.
00:38:42
Speaker
This is actually an anarchic critique of the relationship between symbols and meaning, and all poetry does the same thing. This description of a pair of oxen doesn't describe the oxen as they exist in reality, but it serves as a representation that each of us will visualize slightly differently. At its core, art and language do the same thing. There are inextricable, desperate attempts to capture pieces of our conscious experience, which is so beautiful.
00:39:01
Speaker
I don't think okay. Awesome. Thank you. Watching that video and watching all of your videos, I had this whole idea about how, uh, an image generator or something like, you know, mid journey or Dolly or any of those things that they have, you know, they'd been trained on these images and their knowledge of any concept, like a cat or a dog exists distributively in their head. But just like humans at some point have the ability or the desire or the need to draw a minimalist depiction,
00:39:27
Speaker
of these images in their head and they were recognized, there's this key data that's captured in the drawing of a cat that multiple people can recognize. So I'm curious how soon it is until Mid-Journey or Dali is able to create minimalist pictures and then further abstract them into their own alphabet. I'd love to write a paper on an artificial alphabet created entirely by an image
00:39:56
Speaker
generation network. What do you think about that idea?
00:40:00
Speaker
Yeah, it's a fun area to think about. And I love information theory because communication at its core is how one mind influences another. And how efficiently can you do that? And so of course, our language and art are the same. But if I wanted to tell you something, and I'm going to draw a whole picture of it, that's going to take so much time. And so the more compressed you can get that representation, the better. And yeah, I've seen a bit of research on neural networks talking to each other in their own code or their own language.
00:40:28
Speaker
super fruitful area to think about, especially when we want to download our current mental states and recall them later. Yeah, I am. Now I want to recognize from our lovely producers that we are on borrowed time right now, so I'm not going to go too much longer real quick, but the areas that we haven't gone into yet, we didn't talk, for example, I'll just give a little preview here. We didn't talk at all about the innovation that was called attention or transform networks that was introduced or it was it,
00:40:55
Speaker
introduced in your last video on chat GPT-4. So for our viewers, I recommend you go to Britt's channel and watch his last video on the history of chat GPT. When I first saw attention networks and I saw a layer within layers where it links every neuron to itself,
00:41:15
Speaker
I thought, whoa, that network layer is self-aware. What I mean by that is it considers the information in each individual neuron and how it relates to other parts. And to me, I don't know, that seemed like an emergent self-awareness. So then I thought, well, could you have more attention networks at higher levels and you have a higher level of self-awareness? And I'm intentionally using that term.
00:41:42
Speaker
At least that was the significance that I brought with it. Is that roughly what you would say or am I talking woo-woo science at this point?
00:41:52
Speaker
No, you're on the right path. And just going back to our three layers of learning, that mysterious third one where you're learning by closing your eyes and imagining things. Yeah, that's not growing new connections and attention is a way to create a kind of synthetic connection, a short term one as needed. And that's what attention allows you to do. It's really neat. And so just a lot to do there and thinking about where it goes.
00:42:19
Speaker
Yeah, it's that technical term, attention, which was the term adopted as it's used in machine learning. I encourage your audience to watch Brit's video and ask yourself if that could be thought of as emergent self-awareness at an individual layer, and if you took that same concept at higher and higher and more complex layers.
00:42:40
Speaker
what that might mean. Oh, that sends shivers down my spot. All right, before we wrap it up, is there anything else that you would like to talk about? Anything you'd like to plug?
00:42:54
Speaker
Sure, this was a lot of fun. Outside of YouTube, the other thing I do is I have a company that created a product called Story Experiential that's experiential with an X, with my two partners, Tony and Elise. And it's a kind of online peer-to-peer learning model I'm really excited about, thinking about
00:43:15
Speaker
What won't AI do is facilitate human to human creation. It won't get rid of the humans. And so I'm really interested in this peer to peer learning model online. So that's, I wanna build other experientials and other domains as well. So check out Story Experiential.
00:43:34
Speaker
Awesome. Awesome. Thank you so much. Oh, yeah. That was our last topic. We didn't even touch on businesses that have gone under or been severely challenged by chat GPT. So that's very interesting. In short, I think our audience should
00:43:50
Speaker
You know, those interested should really familiarize yourselves with AI and think about how you can build your businesses. And I'm totally stealing, not stealing, but riffing on your idea here. How can your business be community centric so that your business model stays afloat, even if AI automates certain things? So, all right. Well, thank you so much for coming on the show. I've had a blast. This has been a journey, both with preparing,
00:44:16
Speaker
this outline, but also just preparing the body of knowledge that is artificial intelligence and where we're at. One of the biggest challenges was to do this in a way that breaks it down for an audience that may not be familiar with it. I tried very hard to do that, but again, we will continue this conversation if you're on Slack. I'll put all the links in the YouTube video as well as in the podcast link. You can continue this conversation
00:44:39
Speaker
through email or through Twitter or X or through Slack or Discord. All that information will be there. Also everything we mentioned from the textbook by Professor Steve or who goes by Eigen Steve on Twitter and everything else. So I think that's it. Thanks again. Britt, it's been a pleasure. I hope to continue our conversations as time goes on. Thank you, Gabe. It was a lot of fun. Yes, sir. Sure was. All right. Take care. Till next time.
00:45:10
Speaker
Ah, oh man.