Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
49: Thinking Machines II (Techniques in Artificial Intelligence)
475 Plays4 years ago
Machines have been used to simplify labor since time immemorial, and simplify thought in the last few hundred years. We are at a point now where we have the electronic computer to aid us in our endeavor, which allows us to build hypothetical thinking machines by simply writing their blueprints — namely, the code that represents their function — in a general way that can be easily reproduced by others. This has given rise to an astonishing array of techniques used to process data, and in recent years, much focus has been given to methods that are used to answer questions where the question or answer is not always black and white. So what is machine learning? What problems can it be used to solve? And what strategies are used in developing novel approaches to machine learning problems? This episode is distributed under a CC BY-SA 4.0 license. For more information, visit CreativeCommons.org. For more Breaking Math info, visit BreakingMathPodcast.app [Featuring: Sofía Baca, Gabriel Hesch] References: https://spectrum.ieee.org/tag/history+of+natural+language+processing
Ways to support the show:
-Visit our Sponsors: theGreatCoursesPlus.com/breakingmath Get a free month of the Great Courses Plus while supporting this show by clicking the link and signing up! brilliant.org/breakingmath Sign up at brilliant.org, where breaking math listeners get a 20% off of a year's subscription of Brilliant Premium!
Patreon Become a monthly supporter at patreon.com/breakingmath
Merchandise Purchase a Math Poster on Tensor Calculus at our facebook store at facebook.com/breakingmathpodcast
---
This episode is sponsored by
· Anchor: The easiest way to make a podcast. https://anchor.fm/app
Support this podcast: https://anchor.fm/breakingmathpodcast/support
Ways to support the show:
-Visit our Sponsors: theGreatCoursesPlus.com/breakingmath Get a free month of the Great Courses Plus while supporting this show by clicking the link and signing up! brilliant.org/breakingmath Sign up at brilliant.org, where breaking math listeners get a 20% off of a year's subscription of Brilliant Premium!
Patreon Become a monthly supporter at patreon.com/breakingmath
Merchandise Purchase a Math Poster on Tensor Calculus at our facebook store at facebook.com/breakingmathpodcast
---
This episode is sponsored by
· Anchor: The easiest way to make a podcast. https://anchor.fm/app
Support this podcast: https://anchor.fm/breakingmathpodcast/support
Recommended
Transcript
Introduction and Humorous Game Show
00:00:00
Speaker
Okay, round two. Name something that's not boring. A laundry? Ooh, a book club. Computer solitaire, huh? Oh, sorry. We were looking for Chumba Casino. That's right. Chumbacassino.com has over 100 casino-style games. Join today and play for free for your chance to redeem some serious prizes. Chumbacassino.com.
00:00:30
Speaker
Hey guys, it is Ryan. I'm not sure if you know this about me, but I'm a bit of a fun fanatic when I can. I like to work, but I like fun too. It's a thing. And now the truth is out there. I can tell you about my favorite place to have fun. Chumba Casino. They have hundreds of social casino style games to choose from with new games released each week. You can play for free anytime, anywhere, and each day brings a new chance to collect daily bonuses. So join me in the fun. Sign up now at ChumbaCasino.com.
New Science Show and Licensing Humor
00:01:00
Speaker
This is Gabrielle Hesh from the Breaking Math podcast. I am thrilled to announce that we have teamed up with a particle physicist and a science fiction author named Dr. Alex Alaniz to create a brand new science show on YouTube. Our new show is called the Touring Rabbit Holes podcast. It's named after the mathematician and father of modern computer science, Alan Turing. You can find us at youtube.com slash touring rabbit holes podcast.
00:01:25
Speaker
and that's spelled T-U-R-I-N-G rabbit holes podcast. Be sure to like and subscribe for great visual storytelling about the history of mathematics, physics, and computer science and the impact that they have had on society. Again, that's youtube.com slash touring rabbit holes podcast. And now back to the breaking math podcast.
00:01:47
Speaker
This episode is distributed under a Creative Commons attribution share like 4.0 international license. For more information, visit creativecommons.org. Somebody stole our website. Oh no, whatever shall we do? I mean, I guess you could go to the new website, http://breakingmathpodcast.app with no www for all you old timers.
00:02:15
Speaker
So breaking me up podcast dot app. I mean, if you're into that sort of thing.
Podcast Creation and Machine Learning History
00:02:21
Speaker
Hey, Breaking Math fans. First, I want to thank you for listening. I have an important message for everyone. You can start your own podcast right now with Anchor. Anchor lets you create and distribute your own podcast. Just get an idea, record, and upload. It's just that easy. Anyone can do it. I'm on my way to accomplishing my dream, and you can too. Just get on your device's app store and download Anchor. It contains everything you need to make a podcast. With Anchor, you can put your podcast on all the big platforms.
00:02:50
Speaker
Apple Podcast, Spotify, Google Podcast, Amazon, and more. Reach the whole world with Anchor. Best of all, Anchor is free. You have nothing to lose with a free platform. Download the Anchor app or go to anchor.fm to get started.
00:03:10
Speaker
Machines have been used to simplify labor since time immemorial and simplify thought in the last few hundred years. We're at a point now where we have the electronic computer to aid us in our endeavor because it allows us to build hypothetical thinking machines by simply writing their blueprints, namely the code that represents their function, in a general way that can be easily reproduced by others.
00:03:32
Speaker
This has given rise to an astonishing array of techniques used to process data, and in recent years, much focus has been given to methods that are used to answer questions, but where the question or answer is not always black and white. So what is machine learning? What problems can it be used to solve? And what strategies are used in developing novel approaches to machine learning problems?
COVID-19 Humor and Creative Sketch Talk
00:03:51
Speaker
All of this and more on this episode of Breaking Math, Episode 49, Thinking Machines 2.
00:04:02
Speaker
I'm Sophia. And I'm Gabriel. And you're listening to Breaking Math. Now we have a few plugs. Oh, before the plugs just say about COVID-19 is always stay safe, wear masks and social distance. That's right. We're wearing a mask right now as we're doing this podcast. That way we won't, you know, pass our germs on to you or something.
00:04:26
Speaker
Yeah. Um, germs can be spread through the radio, but I'm not just kidding, but like, we're going to have somebody wrote us for that. Don't worry. We're just kidding. Right. Yeah. Just kidding to the episode. Yeah. Yeah. Okay. They're going to write an angry Facebook post. We are wearing masks on this podcast in solidarity with you. How, how's that?
00:04:48
Speaker
But yeah, like if you, I mean these rules might seem silly to some people because some people have said, how does the virus know the difference between a grocery store and a restaurant? And the answer is it doesn't. But the less you go out, the less the virus has a chance to spread. So that's why it's limited to certain things. I mean, they had to choose certain things, you know, and
00:05:08
Speaker
restaurants are high risk because you're in a enclosed space for like an hour with a bunch of people who are chewing and wiping their spit and yeah, it's a high of germ environment. Yeah, that's true. In fact, the grocery stores that I've seen have been doing a fantastic job of limiting the amount of people who go in at any given time and they also have somebody constantly wiping off the carts in between each use. That is outstanding. That is a great way of clearing off any germs.
00:05:34
Speaker
not to mention everybody has their masks on. And in a restaurant, you wouldn't always have your mask on. There's just more probability. So nonetheless. And not to be a Debbie doubter, but I just want to remind everybody listening that this is only wave one of the pandemic. There's never been a recorded pandemic in history, not the 1919 flu, not the plagues.
00:05:57
Speaker
not bouts of typhus and cholera. They always have a second wave that is more deadly than the first. So I just thought I'd bring that up because it's a harsh reality. Absolutely. Better safe than sorry.
00:06:11
Speaker
But yeah, so because of COVID, episodes one through 10 and the Black Hole series, episodes, it's the abyss into the abyss and gaze into the abyss. And the comedy that we made after that have no ads during this time.
00:06:27
Speaker
Yay, no ads, because we all know how much we love ads. That's our gift to you. No ads at all on those episodes. Those were a lot of fun. Black holes have been probably my single biggest fascination for the last 15 years. So that was a lot of fun. And we wrote a sketch. I should say, Sofia wrote the sketch, and I may or may not have inspired it in some parts. I don't know. But it's almost a sci-fi, Monty Python-esque sketch.
00:06:56
Speaker
And we have it on our podcast. It is called Black Hole Heist. It's just pure silliness. After the deep dive research into black holes, we felt like our brain needed a break. So we decided to write a funny sketch for your enjoyment. Yeah, I mean, I'll play a few seconds of it now.
00:07:20
Speaker
The entrances and exits into and out of Tokyo have all been devoured, and all that remains is a mode of lava. Time to kick some butt.
Neural Networks and AI Capabilities
00:07:27
Speaker
How are you going to attack Nadia? Rocket launcher. I swear you were just so forgetful. We talked about this like five minutes ago. That won't work. She just absorbs it and grows stronger.
00:07:39
Speaker
I'm general specific. And I'll say how we attack this thing. Soldiers, fire the big guns! Sir, please, you're just making her angry. I'll tell you something, kid. I've been a general for 40 odd years, and in that time I can proudly say that I have never negotiated with space-time anomalies. She's not an anomaly, she's the black hole found throughout the universe. The sound changed? Was that sound supposed to be the home of the black hole or the containment field? I'm confused.
00:08:09
Speaker
And we also have a Patreon poster tier. I mean, we have Patreon. You can donate $1 and get, what do you call it? Mention on the show if you want. Nobody's taking us up with an offer. For $5, we put up the shows without ads and the, what do you call it? Outlines that we used for the episodes on Patreon. And we're pretty good about that lately.
00:08:35
Speaker
And we also have a poster tier, $22.46 I believe, where we will send you one of our posters if you don't want to pay monthly because you just want a poster. We have our Facebook poster store at facebook.com slash breaking math podcast and just click on store. It is e to the e dollars or $15.15 for a tester poster plus $4.50 shipping and handling, which we've been able to get down significantly on both of those things.
00:09:00
Speaker
meaning that it's only $19.65 for a poster. It's very big. It's like 36 inches by 24 inches. It has a great deal of explanation of tensors, which are imperative to explaining Einstein's theory of gender relativity, because otherwise it would be something like 80 or 160. I can't remember differential equations.
00:09:22
Speaker
Yeah, that poster is amazing. So that, that's available on the website, on the Facebook marketplace, as well as on Patreon. So real quick, just a note, I think you'll notice that there are some blips on my sound wave because my son has been outside banging on the door. It's not super loud.
00:09:39
Speaker
Oh, yeah. And honestly, like, I don't think that it'll matter too much. Like, I mean, we're doing the pandemic, whatever, you know, like, and he didn't sound like he was like dying or anything. He was like, hey, I'm a baby. I need attention. I need I need something, you know. Yeah. We were on the Twitter and Facebook part where we have all of our updates, you know, including at Breaking Math Pod for the Twitter handle. And then our Facebook page is just Facebook.com slash Breaking Math Podcast.
00:10:05
Speaker
Oh yeah. And we have the website at breakingmathpodcast.app. A pirate stole our other one. We're very sad about it, but we've moved on to greener and happier pastures. So it's breakingmathpodcast.app. Just don't put www. Yeah. That's yes. Yeah. No, www. HTTP colon slash slash breakingmathpodcast.app.
00:10:28
Speaker
or just break your math podcast at APP with all the HTTP, no WW. I'm old school. Alrighty, so this episode will be an interesting one. This is a follow up to our last episode. In fact, this is more about what we can do with machines, what neat things we can do with machine
00:10:47
Speaker
learning and thinking and communicating. Because although there have not been any attempts, you know, since the time of Leibniz to seriously make a universal alphabet, in fact, we still hold the position that such a thing is not possible, we certainly have gotten machines to get better at learning and teaching themselves things. And that's what this episode is all about.
00:11:12
Speaker
categorization, also extraction of features from data that sounds surgical and trained. Yeah. I mean, it is kind of surgical cause like, you know, these are the kind of problems like, you know, like, you know, you have a picture of like a bunch of, uh, like, you know, dogs in a meadow and it's like outline all the dogs, you know? Oh yeah. And that's something that can be done by neural networks now.
00:11:35
Speaker
Yeah, yeah, it's crazy. And I have not read, oh, I shouldn't say this. I have not read the latest, but then why am I podcasting about it? I mean, you read it like a bunch of this stuff on the yeah, we had breaking math, maintain the position that without perfect knowledge of a subject, you shouldn't ever even be talking about it. Keep your head down in the ashes and you should drag yourself in the streets by your knees.
00:12:01
Speaker
Yes, yes. Don't even open your mouth. That's why we know everything about every topic we've mentioned, right? No, I'm just kidding. Oh yeah, we've never made a mistake with a show.
00:12:10
Speaker
Of course not, of course not. So yeah, so in this episode, we'll be talking a lot about Markov chains, but what they are, where they came from, what they're currently used for. Also something called LSTM. How would you describe LSTM? They're a type of neural network neuron that can be used for things like sequences.
00:12:32
Speaker
So for example, if you're trying to get a neural network to compress HTTP, you can use an LSTM. Or if you want to get a neural network to try to generate music based on some samples of music, an LSTM might be the way to go. And as we'll see, it's used because it has the ability to forget information in specific ways.
00:12:54
Speaker
Interesting yeah, intentionally forgetting hmm. What a what a concept. Yeah, and there's many many Variants on that in fact that that section alone on our outline. I think is about three pages here Is that what I'm seeing that it's like two pages to pay. Okay. Well, okay. I sometimes stretch the truth
00:13:10
Speaker
Yep. And then we also are going to talk about properties and probabilities of random variables and Bayes theorem and Bayesian networks and how probabilities are used in a machine learning. So quite a hefty episode today. Oh yeah. And we'll talk a little about the future of this type of research. Yeah. And yeah. I'd love to hear from some folks out there who are working in neural networks to hear some of the cool things that they've been up to.
00:13:41
Speaker
So as we dive into the more nitty-gritty aspects of this episode, we are going to start with something called LSTM, which I did not know before doing the research for this episode. Any guesses on what LSTM stands for? No spoilers. LSTM stands for Long Short-Term Memory. I'll say that again. Long Short-Term Memory. Sophie, I care to elaborate on what Long Short-Term Memory is.
00:14:07
Speaker
Yeah, basically what LSTMs are cool for is with a normal neural network, if you want to connect the neural network to itself, so that it's in a loop.
00:14:22
Speaker
Using normal back propagation, which is used to train neural networks, it's kind of like you're thinking about the algorithm that is used for learning. You get these like divided by zero and like weird answers with that because basically the error as you go in a circle keeps propagating so much that goes into infinity.
00:14:39
Speaker
So a long short term network is a modification of this type of neuron that where each neuron has what's called a forget vector that tells it what parts of what information to forget or to not like factor into the output. So basically it can have a short term memory because like you know the forget vector is strong or long term. I believe that's why it's called long short term memory. I haven't looked that up actually very hard but that's what's cool about it is that it can be used for recurrent neural networks.
00:15:08
Speaker
Interesting. Now, real quick, do a few things in that description, and I'd like to real quickly, for somebody who is not familiar with how neural networks work, how would you explain the bare bones basics of how a neural network itself works? You had said some terminologies here. I think we've said in a previous episode called artificial thought, but just for the sake of review, how would you describe, oh gosh, I'm not backpedaling, but back propagation.
00:15:32
Speaker
Well basically the way that a neural network works is that you have all these inputs and as they go forward they're added together like the ones that are connected together are added together and they're added together and put through a function that's called a sigmoid function usually that goes from negative one to one as X goes from negative infinity to infinity so basically like negative a thousand would be like negative point nine nine nine or something zero would be zero and a thousand would be like point nine nine nine and like
00:15:58
Speaker
Something like that. What's cool about that though is that for some reason that's not totally well understood. It's getting better understood and there's some papers on computational geometry that deal with this and also just neural networks.
00:16:13
Speaker
Like, you're able to approximate functions using these. And the problem with backpropagation, what that is, is that you see how much error there is in your approximation by judging it based on some criteria. And there could be any criteria, it could even be human criteria.
00:16:33
Speaker
You go back in time basically and say, okay, so based on like this and it's based in calculus because these neurons have very simple functions. You go back in time and say, okay, this neuron is responsible for this much error. So that means we have to change it this much. Like, you know, it's like a perturbation.
00:16:50
Speaker
Interesting. Yeah. Wow. That's quite a bit there. That's quite a mouthful. So it sounds like it not being completely understood. It's a little bit of a black box, sort of a, the analogy I use is in terms of understanding it, like tangled Christmas lights. Like it works, you plug it in, it lights up, but you can't really quite see where everything is going and how exactly it works. But we're again, as you said, we're getting better at that. Yeah. And yeah, that's a type of research that, you know, is ongoing is how to make sense of neural network solutions, you know,
00:17:20
Speaker
Yes. What's funny is on my end, I know that there is a sound of a baby crying and baby knocking. And on your end, there's a cat meowing. It'd be funny if at this entire episode we have like random sound effects that are in the background, like we even know. Oh yeah. Let me let that cat out real quick one time. Okay.
00:17:36
Speaker
Yeah, in this episode it'd be funny if you had like, you know, ice cream truck sounds, and then like explosions, machine guns, and like other random backgrounds. I was like, what? Sorry guys, working from home. Yep, working from home. You know, banging on pots and pans. Yeah.
00:17:53
Speaker
Anyway, so that is the single thing I heard from your description of neural networks more than anything else was the cat meow. No, I'm just kidding. I'm just kidding. So yeah, they are quite interesting. And then when you get to LSTM, as we said earlier, the thing is you have the ability to forget and how that's utilized in machine learning.
00:18:14
Speaker
Now, I think there's some equations that we have in this episode that we're going to talk about. We're not going to bore you to tears with them, but we have some important variables to talk about. What are the variables? Yeah, we're not going to read the equations out. We're going to talk about what the equations do, but we will talk about the variables that are involved.
00:18:29
Speaker
Yeah, probably worth it to just go, yeah, just to discuss, as you said earlier, just to discuss the variables, even if we're not gonna, you know, hit you over the head with the equation itself. Because we know that that's what you all came here for, right? You know, just to read you equations. Okay, so one of the most popular variables in all physics and math is the letter C. This equation is brought to you by the letter C. You want to explain what C stands for with LSTM?
00:18:53
Speaker
C is just an activation vector, which is a fancy name for, hey, it's a little piece of information that helps it do its job. And it's also the output. It's not the output, but it might be the output. It's used heavily in the output. C is the activation vector. When you say that, it just sounds high tech. I just imagine these machine sound effects like...
00:19:17
Speaker
C is the vector for the cell state. The reason why it's a vector is because these are solutions that have a lot of information on them. For example, generating text or something like that.
00:19:32
Speaker
And so you have an input and output vector as well, and you have a forget vector. And what's cool about all these vectors is that they all have weights on them that tell them how to connect to one another. And they all at the base are connected to these vectors x and h, which are the hidden vector and just this other vector that is just used in calculation. And what it turns out to be is something that can forget because the
00:20:02
Speaker
one of the gates that is using output, the forget vector is, the higher the forget vector is, the less that this neuron cares about its previous state. So like I remember these neurons are used for generating sequences, so that's why they care so much about their previous state sometimes, but other times when it doesn't matter and it would cause a huge back propagation error, it turns out the forget gate allows that to nip it in the bud.
00:20:31
Speaker
Wow, you know what else I'd like to do real quick? As we're getting more into the technical details of these different neural networks, I want to take a moment and explain what kind of like why neural networks, what kind of problems we're trying to solve with neural networks.
AI Sentience and Cultural Biases
00:20:44
Speaker
I know we've gotten into the how, but again, just to provide a little more context here, why.
00:20:50
Speaker
So with LSTM, but as well as neural networks in general, can we, I realized this, this may be appropriate earlier in the episode, but we'll just do it now. Just because what are some problems that neural networks are being used to solve?
00:21:05
Speaker
Well, I mean, what they could be with an early project that a lot of people learn when they're doing a neural network is they develop a neural network that could identify written digits, which surprisingly only takes like 20 minutes to write using Python. If you know what you're doing, like if you know what you really don't, you're doing only takes like two or three minutes.
00:21:22
Speaker
You could, there's also a website, thispersondoesnotexist.com. Gabriel, have you got to that? No, I've never gone to this person does not exist. Go there real quick. Okay. This person does not exist. Oh, we're in for some excitement here. Oh my gosh. Okay. Dude, he's smiling. Yeah, he doesn't exist. It's generated by neural network.
00:21:48
Speaker
Wow, that is insane. Yeah, keep refreshing it. Is it possible to have competing neural networks where you have one of them outputting these pictures of people who don't exist and someone else trying to identify real from fake? I only bring this up. It's funny that you're bringing it up because that's literally how that works. It's called a generative adversarial network.
00:22:10
Speaker
And you have one neural network that is trying to fool the other one, and the other one is trying to outthink the other one. And together, they start generating data based on sometimes very small sample sizes. Naughty. This is insane. I mean, now I'm thinking of like, okay, fine. How many neural networks could you have in on this? Is it like a huge gang fight? You know what I mean? And would it be possible to
00:22:35
Speaker
I mean, is this just something where it just gets closer and closer to approximating perfection to where it's not, you just can't tell? Or once you figure out a way to tell, then pretty soon the way this production works, that issue that you've discovered will be taken care of. You know what I mean?
00:22:53
Speaker
Yeah. And also like anybody can play around with deep fake now, like a deep fake. You can make videos of, I mean, just take videos of world leaders and you can put your face on them. If you just like take a video of you for a few minutes talking, like, uh, it's, it's, it's really cool. I mean, there's also deep voice, which is the same thing with voice. Um, like it's really cool. The kind of stuff that we're doing with throw networks now, like the potential is barely scratched the surface. Fascinating.
00:23:21
Speaker
Okay, so again, yeah, just going to thispersondoesnotexist.com, it's these pictures of these people. It looks like real people. This is just wild. Throughout the production of the Breaking Mouth podcast, there have been several opportune times that I've sought after to look for times to bring up the show Rick and Morty. I know it's a comedy show. A lot of folks who I bring this up, a fair amount of folks see it as rather juvenile.
00:23:45
Speaker
Nonetheless, Rick and Morty have some very, very deep scientific and philosophical ideas, and one of them relates to this website I'm looking at right now, thispersondoesnotexist.com, as well as the power of neural networks in problem solving and even in creation. And it's funny, I just mentioned creation. This is similar to
00:24:04
Speaker
In our previous episode, Godfrey Leibniz imagined an alphabet, a perfect alphabet that through basic combinatorics or complex combinatorics could come up with any knowledge or any creation. Well, that can't be done, but we do here have a process rather than an alphabet per se, a process that it is not perfect.
00:24:28
Speaker
perfect, but it is astonishing nonetheless, a process of competing neural networks that is able to produce things that have never existed before. So tell me, is what I'm looking at right in front of me, is this digitized? Is this automated creativity?
00:24:48
Speaker
Isn't it? I would say that, uh, and I'm sure it's automated creativity or if it's automated expression of the creator's creativity, like, like the question is like, when does a program get, would get like complex enough that you could call it its own creativity? And I would say that that point, uh, that point happens when the machine can defend its own creation.
00:25:14
Speaker
Fair enough. And we wouldn't maybe maybe the machine would consider something art and we would not consider it art. I'm still going to come around to the Rick and Morty piece here. I just need to get all of these things out because, you know, I'm thinking of them as we go. You had mentioned that that this this website, which is working based on two competing or two or more competing neural neural networks. I believe it's two in this case. I'm sorry. Oh, only two. It's two in this case.
00:25:44
Speaker
where it creates these faces that don't actually exist. So this might be an expression of the original creator's creativity, kind of like how, you know, what an addle-addle is, where somebody would use an addle-addle to increase their throwing, where the human being is still doing the throwing, but the addle-addle adds a whole lot more power to the throw.
00:26:07
Speaker
If you've ever seen Hialy, it's this long stick with a hook at the end that you train to put a spear in or bounce it against and throw it extremely quickly.
00:26:24
Speaker
So, these neural networks, although they themselves are not automating creativity per se, they are certainly contributing to the original author's creativity, perhaps, like an Attle-Attle would increase a human's
00:26:39
Speaker
throw power. Is that a fair analogy? Yeah. Yeah. And I'd also say is like our whole definition of creativity is based in a world that did not allow, that did not account for this kind of thing. So maybe we're going to have to come up with new words and stuff to describe human artistic experience.
00:26:57
Speaker
You're exactly right. In one of the issues with recognizing artificial intelligence, or one of the issues with recognizing other intelligence at all, whether it's from octopi or other things, is that our recognition of intelligence is what we consider intelligent. That is to say, it's culture and context dependent. So much of it is based on our own
00:27:20
Speaker
our own culture and our own context, not all the time, but so much of it is. The example I have here is in a recent article this last year, it was shown that fish, I don't remember which fish, I'm gonna have to find this article, but there are some fish that were thought to be emotionless, that were discovered to have complex emotions that we just don't recognize because of facial expressions. Their faces are pretty- Yeah, I saw a freaky video of an eel being pet,
00:27:47
Speaker
by a scuba diver and it was cuddling like it was like a like a dog. Yeah I saw that. It was and not only that there's an experiment done recently with octopus octopuses that um where they gave them uh MDMA um methylene deoxy methamphetamine which is also known uh as uh ecstasy um in its pill pressed form or molly and its uh street uh crystallized form. Can you imagine?
00:28:14
Speaker
Wow Imagine this is a pathogen in humans, but have you heard about this experiment? Yes, I have and I haven't read more about it and I'm imagining yeah Wow, basically what's weird about it is that octopus don't even have a pleasure-reward pathway Their brain is structured fundamentally different than the way that human brains are but for some reason they had the exact same qualitative experiences that humans do on MDMA like a tactile sensitivity and
00:28:42
Speaker
increased social ability, playfulness, things like that. Wow, that's crazy. I was going to say, could you imagine being on the committee that evaluated this scientific test? Whoa, these guys want to what? They want to give octopuses drugs?
00:29:01
Speaker
Oh, yeah. I mean, I will advocate right now, though, for MDMA. It was used in the 1980s by a lot of psychotherapists for a while in an increasing fashion until it was scheduled by the government because of its perceived association with subcultures. And basically people were partying with it and people considered it dangerous. But honestly, like it's a miracle drug for PTSD when he was in a clinical setting.
00:29:30
Speaker
I could also advocate for ketamine and psilocybin, not to mention cannabis. There's a lot of substances that have been, unfortunately, because they have a strong effect and because they're a kind of new science, we have done a very human thing and we have become scared of them and we've locked them away in a cabinet.
00:29:52
Speaker
Yes. Yeah. So I've heard. I've heard. Wow. Fascinating. Now remind me of the acronym that you used for the drugs that were given to the octopi. MDMA. MDMA. What is that commonly known as something else? Ecstasy. Ecstasy. Okay. Okay. Got it. Got it. Wow. That's fascinating. And also I had no idea that their pleasure pathways are different than ours. That's
00:30:12
Speaker
Yeah, they literally don't have the same plate because they have a donut shaped brain. So the entire way that their brain works from the ground up as different. So how these molecules created the same response is that was they didn't expect anything like that. Like it's, it's one, it's, it's actually one of the most mysterious experiments that I've read about in my lifetime.
00:30:34
Speaker
As I was saying earlier, this is all going to come around full circle. And I had mentioned Rick and Morty for a reason. The first thing I wanted to say is that recognizing intelligence might be a difficult thing to do. Now you can do it when you have problem solving abilities, but expressions of culture, we might, you know, they might be so alien to us that what is a, you know, a vibrant culture and a real culture we think is just noise or barbarism. You know what I mean? Exactly. I've always thought of that as a fundamental flaw with the traditional Turing experiment.
00:31:04
Speaker
For those unfamiliar, the Turing test is you get a computer and you connect it to a human in a way that the human can't tell if it's a computer or a human being. And then you have them talk to it for a while and guess whether or not it's a human being. And if it can fool most human beings, then it has passed the Turing test and is considered by this test conscious.
00:31:24
Speaker
And while it has a lot of benefits, I can see this test being used against artificial intelligence beings after that because, I mean, in the United States, there's so many different cultures. There's a culture in the Northeast. I mean, there's even ethnic subcultures. There's black people. There's Irish people, Hispanic people. There's some people in Minnesota with a very strong
00:31:50
Speaker
German culture and all these people will have little cultural tells like there's I don't think it's very Likely that a artificial intelligence has wouldn't develop their own probably bizarre culture So within probably a couple years of these artificial intelligences being able to interact with one another People would be able to tell via cultural signals that they were not a human and they might use that as saying like oh see then They're not conscious because I could tell they're not a human and I can see that you being used by artificial intelligence bigots
00:32:19
Speaker
So we need a better, yeah. That's scary. Yeah. And again, I don't, so we've talked a very, very common thing on this theme, on this podcast. And by the way, those who are waiting for my Rick and Morty thing, it's coming, it's coming. Trust me, it all relates to this. But a common thing that we talk about is Godel's incompleteness theorem.
00:32:38
Speaker
And you know that that talks about how mathematics is always incomplete and you will always have axioms that you can't prove are true Theorem say using the axioms yes, and what I'm talking about with not being able to tell if a culture is again we
00:32:56
Speaker
we're quick to identify a culture that is not our own as barbarism when it might not actually be. It might be a valid culture that we just don't recognize. You know what I mean? Like again, there's just there's nuance and there are things that you just can't know.
00:33:11
Speaker
at first glance until you have much more information and even then you'll ever know you'll never know for sure. In fact Orson Scott Card writes a lot of those themes into his science fiction where you have inevitable miscommunications that's especially in the sequels to Ender's Game including Ender, not Ender's Shadow, more so Speaker for the Dead, Xenocide and Children of the Mind. I just I love those books and a huge theme in that book is
00:33:35
Speaker
different cultures where you simply don't know something and you think that people are, that a different culture are savage killers when they're in fact something else. So all this to say this relates to, you know, the dawn of artificial intelligence where even if it existed, if it existed right now, like right now, we would just see it as noise, as just like bits and bits and bytes and you know, just noise. Like if Google were actually self-aware, we wouldn't know at this stage.
00:34:03
Speaker
So that's kind of a scary thought, but that's at least my thought here.
00:34:07
Speaker
Yeah, and then yeah, and there's even others if you get into like for example Like if humans are seen as neurons and the way that they talk to each other considered neural connections We could consider the fact that no one has ever figured out a way to unlock a way to talk to the human consciousness as a whole I mean to be cool if we had made some kind of weird machine that used like principle principles of Economics and propaganda and all these things to literally talk to the human subconscious, but that's getting into weird sci-fi. I
00:34:35
Speaker
Yes. In the series of articles that led up to part one of this episode, it was the IEEE Spectrum articles that I cited. One of the later articles was a review of the chatbot created by Microsoft that turned out to be extremely racist.
00:34:53
Speaker
the chatbot learned from people who interacted with it and some people were rather unsavory and wrote very awful things and the chatbot adapted that as part of its habits so it is possible to have a wild you know a chatbot that is self-learning turn into what we consider awful and very very racist so
00:35:11
Speaker
Yeah, and it's like there's also a Facebook algorithm because of the way it was trained improperly because of a Unrecognized at first cultural biases. They were identifying a black people on Facebook as gorillas even though it was an AI Yes, and it's like and and then they they fixed it and now it doesn't do that anymore but for a while he was doing that and like, you know, it's it's it's
00:35:33
Speaker
It reminds me of a, there's a computer science cone. A cone is originally is this Buddhist kind of story that makes no sense. That's supposed to lead you to enlightenment and computer scientists made their own. But there's this cone that says that a computer science guru came to see a student and the student was programming a neural network. The guru asks, what are you doing? The student said, I'm making a neural network play tic-tac-toe. The guru said, how are you wiring it?
00:36:02
Speaker
the student said randomly the guru said how come the student said so it'll have no preconception of how to play the guru closed his eyes the student said why are you closing your eyes the guru said so the room will not exist interesting and just meditate on that and it'll make sense after a while okay yeah let's just play some music while you meditate
00:36:25
Speaker
Full disclosure, I said, what's up? The hint is that there's that basically, even when you try to be as impartial as possible, you are still somebody who's making something based. So there's always going to be a subjective component. Like even the objective measures that you take are subjectively chosen.
00:36:43
Speaker
Yeah, interesting. Now, real quick, before I lose my train of thought here, the Rick and Morty episode that I'm talking about is called Close Encounters of the Rick Kind. It is from season two. And in that episode, there are these mental parasites that create false memories in your mind. And the false memories are of somebody that never actually existed, but suddenly these artificial
00:37:04
Speaker
memories appear. And it seems as though they did. And it relies heavily on deeply sentimental memories of these individuals. So we would say that if those virus, if those parasites existed in real life, it'd probably be involved with not only producing false imagery and other memories, but producing a heavy hits of oxytocin. So there's a lot of sentiment behind it.
00:37:25
Speaker
In this episode, we later find out, I may have a little spoiler here, but there's a way that you find out that these parasites that produce these memories, there's a way to tell what memories are real and which are false. I'm not going to spoil that part for you because that's a very good part of the episode.
00:37:42
Speaker
But this is all part of this. I am completely fooled by the people that don't exist on the website. This person does not exist. And this is just from a visual inspection. That makes me think about what other ways do I recognize dealing with a human and how could I be fooled that they exist, similar to the Rick and Morty mental parasite example,
00:38:07
Speaker
whether it's oxytocin being pumped, like let's just say either an artificial intelligence or some alien life figured out how to do that, how to pump oxytocin and made us doped up on feel good feelings and think that we're in a safe place and we could be taken advantage of then. It's the same kind of fear, like this is real. It's possible for us to be fooled and that's a very, very scary thing. And then it just gets me thinking of other ways that humans can be fooled. We are not so perfect after all. It's terrifying.
00:38:37
Speaker
Oh, yeah. I mean, there's a whole field of fooling human senses that sometimes is poorly paid attention to. But I mean, look at illusion and magic.
00:38:46
Speaker
Yes, yes, absolutely. That's to fool human and it's really simple. Wow. This was a wonderful, long, long, long tangent that I thought was great. That was fantastic. I know that we were in the middle of LSTM. Is there more that we need to say about LSTM specifically and what it does that other traditional neural networks do not do?
00:39:08
Speaker
Well, it's really good at producing sequences. That's the best thing about LSTMs.
Bayesian Networks and Probability Applications
00:39:14
Speaker
It's really good at producing sequences. It's also cool because it uses a rudimentary form of memory, which is its forget vector, because it knows what's important and what's not.
00:39:25
Speaker
But it's also been used to great effect with data compression. Basically, you train a model using the data that you want to compress. I can't remember exactly how it works, but not only data compression, but other stuff I've seen done with LSTMs.
00:39:43
Speaker
is imitation of cursive handwriting. I've seen generating not only generating half handwriting of one type and half of another because you can add these networks together sometimes. It's really cool what you can do with these. Wow, that's fascinating.
00:40:08
Speaker
All right, so now we're going to talk a little bit about Bayes theorem and just Bayesian probability and stuff. So yes, as Sophia was saying, the next section is on Bayesian networks, which obviously involves Bayes theorem. So naturally we'll be talking about Thomas Bayes who originated the theorem and how it is now used and how it is different than LSTM networks. There will be a test, pay attention.
00:40:33
Speaker
Yeah, and Tony's bass was kind of a contemporary of people like, you know, like, um, what's his name, Ben Franklin and stuff, a little older than Ben Franklin. He was 1701 to 1761. But, uh, basically his whole thing was that, um, was that instead of probability being counted as something, uh, based on a frequency of it happening, it should be based on like this weird kind of like internal likelihood of it happening.
00:40:57
Speaker
which seems like a small difference but basically it asks sometimes the question like what is the probability of the null hypothesis compared to the alternative versus what is the total probability of the null hypothesis which can be more useful sometimes because sometimes even though like the null hypothesis might be a lot more likely than nothing it might not be much more likely than the alternative.
00:41:19
Speaker
Yeah, so subtle subtle but useful differences now the original base theorem I think it was it was used to try to prove God or God's existence, I believe
00:41:30
Speaker
Was that Bayes theorem or was that Pascal's? I think it was also Bayes. I'll prove it real quick. Let me do insert typing sound. Check it out, yo. So from thank you to qz.com, the most important formula in data science was first used to prove the existence of God. That is a June 30th article by Dan Kopp. How do you say his name? Correct. That's right. That's where it came from. Yep.
00:41:57
Speaker
Interesting. Yeah, it's used now in inferring probabilities because, I mean, we'll go through some identities. Basically Bayes' theorem, sometimes it relies on the notation, the probability of A given B. So what does that mean?
00:42:15
Speaker
So let's say in our culture, for example, people who identify as men don't tend to wear skirts, people who identify as women wear skirts some of the time. So let's say one out of five women wear a skirt, but only one out of a hundred men wear a skirt. That means that the probability of someone wearing a skirt given the fact that they're a woman is one out of five, but the probability of someone wearing a skirt given that they're a man is one out of a hundred or whatever it might be.
00:42:39
Speaker
And so the first formulas we want to do is the probability of A given B is the probability of A and B happening, both of them happening, divided by the probability of just B happening alone. Interesting.
00:42:55
Speaker
Yeah, and then I can see this is where you can do all kinds of, you know, uh, uh, combinations. I don't say combinations, but from that you can have a whole set of, um, what's the term I'm looking for? Not theorems. Formulae? Formulae, yes. Formulae all derived from that by itself. So basically, if B is assumed to already have happened, then it sort of amplifies the likelihood of the probability of A given B relative to the probability of A and B both happening. For example,
00:43:22
Speaker
If a third of all days are rainy, half of all those days are parade days, and all rainy days are on parade days, then that means that two out of every six days are rainy, and three of six days are parade days. Since all rainy days are on parade days, that means that we have two rainy days for each three parade days, if you guys are following along on that.
00:43:46
Speaker
That means that given the assumption that it was a parade day, the chance that it will be rainy is not one-third, but two-thirds, which is one-third divided by one-half. And that section is not totally necessary to understanding Bayes' Theorem, like, necessarily, but if you're interested in it, I would suggest listening to that a couple of times, because, yeah, especially the way that Gabriel read it was very clear.
00:44:12
Speaker
Yeah, yeah. I mean, again, it's just probabilities given other events. And that's, again, a nuanced thing here, but obviously a very, very relevant thing to talk about. Yeah, probability is all about assumptions of what you know, what you don't know. Because if you think about it, probability is all about what you don't know. Because if you did know it, there wouldn't be a probability associated with it.
00:44:35
Speaker
Yes, and understanding the relationships between related events. For instance, you know, if I have peanut butter on my sandwich, what's the likelihood that I have jelly as well? Or rather, you could say, you know, you may want to revisit the likelihood that I have jelly given that I have peanut butter on my sandwich as opposed to anything else, you know.
00:44:54
Speaker
Yeah, like if you put a piece of bread on the counter, maybe there's a 50% chance that the next thing you put on is turkey. Maybe 50% that is jelly. Yes. But once you put on jelly, there's always 100% chance that it's going to be peanut butter, maybe a 5% chance that it's going to be bananas. Correct. But if you but like there's almost no chance there's going to be cheese.
00:45:14
Speaker
Yes. And by the way, I thank our non-American listeners for putting up with that because Gabriel, did you know that outside of the United States, people don't eat peanut butter and jelly sandwiches? Y'all are missing out. You guys got to try it. It is delicious. I know. Like don't think of peanut butter as being a savory food. Just put some jelly on it and taste the magic. It's American. We are experimental, you know, but we figure out some cool stuff and is, oh, we've got a little visitor here. He's in my room.
00:45:44
Speaker
Hey, baby boy, what's up? Augie, do you like peanut butter? Do you like peanut butter, Augie? He'll be joining us here for a little bit for this last part of the episode. Yeah, it's my my my little guy. For those of you who are listening elsewhere, it's very good. Just regular bread. OK, well, what's considered regular elsewhere? You know what I mean? Like, I would say just like a slice of wheat bread, the kind of bread that is long and square.
00:46:10
Speaker
Yes, for morning toast. Just Google, go to whatever internet search tool you like and type peanut butter and jelly and try it. So the thing about it is it's filling, it's very, very filling and it's sweet. Yeah, it's sweet and it's delicious but not too sweet.
00:46:30
Speaker
And what's nice about it too is that, oh, and also just, I'm sure British listeners are fuming at the ears right now because jelly over there means jello over here. But yeah, peanut butter and jam, okay? We appreciate all our British listeners. But a Bayes theorem is cool because it relates the probability of A given B to the probability of B given A using only a probability of B and probability of A.
00:46:55
Speaker
Wow. And what it is is the probability of A given B times the probability of B is equal to the probability of B given A given the probability of A. Whoo! That one's fun. And yeah, it could be derived from, I mean, the probability of AB is the probability of AB. You multiply that by the probability of A divided by the probability of A. And then you use the identity that the probability of AB divided by the probability of A is the probability of B given A.
00:47:24
Speaker
But the probability of AB divided by the probability of B is the probability of A given B. But that's not that important. But basically, what this allows us to do is, I mean, for example, answer questions like, how effective is a test for a virus? If it has a certain positive rate and a certain negative rate, how often does it actually detect the virus correctly? And sometimes the number is a lot lower than what we think it should be because of the way that the math works out.
00:47:51
Speaker
because humans are bad at interpreting probability. And that's why I want humans to learn probability. I want it to be taught in preschools using games. We teach so many things using games like cooperation and things like that. Why not probability? Hey, do you remember, what do you call those 17th and 18th century things where it was an automaton that was supposedly right?
00:48:18
Speaker
Those are just called automatons, but yeah, they totally did write. There's some restored ones now. You can see YouTube videos of them. So some of them can write, and I'm sure you're aware of the automaton fraud as well, where some of them... Oh, the Turkish chest plane robot. So basically, there is this thing called the mechanical Turk, and it was this stereotypical Turkish guy. He was early 1800s or whatever.
00:48:44
Speaker
And it was some robot that was said to be able to play chess with any human. And it was extremely good at chess. It would defeat masters and stuff like that. And it turned out after a very long time that they had just gotten a chess grandmaster or master or whatever to agree to help them with this. But it was somebody in the machine itself manipulating all kinds of knobs. So it was half robot, half machine.
00:49:10
Speaker
It was like, and it would play chess based on all these weird signals. And he was a big hoax, but I would recommend reading about it. Mechanical Turk. So in this day and age of deep fakes and neural networks, I'm sure that there are neural networks that also have human beings behind the curtain as well, making it seem just a little more real. You know what I mean? Like, I'm sure that's a thing. Or rather, that's at least part of it.
00:49:38
Speaker
Well, I mean, it's interesting that because what you've gotten into is the difference between a train and an untrained network. And let me expand on what I mean by that. Basically, you train a network when you have a human telling it like, yes, he did good. Yes, he did bad neural network. But it's untrained or oh, no, no, not trained. Sorry. Supervised or unsupervised. But it's unsupervised when you use some mathematical criteria, some function to do that work for you. And there's hybrids too and stuff like that. For example, like the
00:50:08
Speaker
the generative adversarial networks that we talked about earlier, where they fight each other. Wow. Wow. Interesting. Okay. Yeah. So that's also a thing too. We also have a lot of human involvement as well. Interesting. Wow. But yeah. So if you put a bunch of Bayesian variables together, if you think about P of B given A as being like an arrow from A to B,
00:50:29
Speaker
You can think of all these different associations that things have with each other, like what's related to what? And Judea Pearl, who was born in 1936, and he's still alive, he's an Israeli mathematician and computer scientist, and he popularized the idea of Bayesian networks. And let me give you a basic description. Let's suppose we can win the lottery, right? And that's a completely random event, right? It's not really based on anything, it's based on a draw.
00:50:56
Speaker
And let's say on the day that we can win the lottery, it might rain and the ground might get wet for the rain versus just evaporating immediately like some rain does. So let's call these variables L for lottery, R for rain, and W for getting wet for the rain.
00:51:12
Speaker
W obviously depends on R, right? Because it can't get wet for the rain unless it's rainy to begin with. So those variables must, you know, relate and say epistemologically relate. Yes. So the joint probability of R and W that is rain is going to get wet for the rain is the probability of rain times the probability of it getting wet given that it's raining.
00:51:35
Speaker
So basically you combine variables like that if they're related. And since L is independent, we just multiply it by that. So the probability of L, R, and W all together is the probability of R times the probability of L times the probability of W given R. Okay, I think that makes a lot of sense.
00:51:51
Speaker
So let's say that we want to figure out the probability. Let's say we have all these different variables, right? And we know how they relate to one another. But we want to know the probability of, for example, let's say we have this gigantic network and it's raining. So basically, you combine variables like that if they're related. And since L is independent, we just multiply it by that. So the probability of L, R, and W altogether is the probability of R times the probability of L times the probability of W given R.
00:52:20
Speaker
Okay, I think that makes a lot of sense. So what we let's say that we want to figure out like, you know, the probability, let's say we have all these different variables, right? And we know how they relate to one another. But we want to know the probability of like, you know, for example, like, let's say we have this gigantic network, the probability that we're going to buy red shoes, given the fact that we are just visited by a politician.
00:52:43
Speaker
or something weird like that. You can figure that out by summing up all the probabilities of the variables that aren't RRS together. But the thing is, it takes a very long time. So let's say we have like 10 different variables, right? Yeah. 2 to the 10th is 1,024. So let's say we had 20 different variables. 2 to the 20th is like a million. So the more variables you add, it gets very, very complex, very quick.
00:53:07
Speaker
So we use the technique of variable elimination, which helps a lot. Um, and you could see this on Wikipedia, or if you're a Patreon member of the $5 class or more, you could see our outline on this. But basically we have to use estimation techniques in all practicality, because, uh, except for small problems, the problem is NP complete, meaning that it uses approximately exponential time, non-polynomial time.
00:53:33
Speaker
And polynomial time is like, you know, two to the n. I mean, not two to the n, like n to the x. So like, let's say the problem is like three times as big. So it takes like three cube times as long or three to the 20th time as long. Those are all polynomial. But as long, but as soon as it gets to like two to the 20th or two to the 30th or something like that, it's called NP complete because it is no longer polynomial. I see.
00:53:57
Speaker
And it's part of a huge problem called P equals NP. But we've talked about that a little bit in past episodes, but we'll talk about that again in the future. OK. Yeah. So I know that with respect to applications, we talked a little bit about medical applications. I think that understanding probabilities in medicine, Bayesian statistics are huge because they really help. Oh, yeah. Automated. Oh, yeah. And automated. What do you call it? Diagnostics.
00:54:24
Speaker
For example, given all these probabilities, like this person is presenting with this symptoms, or this ethnicity, they weigh this much, they have this level of potassium in their blood. What is the probability that they have this disease? And how much can we reduce the uncertainty given these series of tests?
00:54:45
Speaker
So it really helps with, you know, things like that. I mean, even to help with things like paleontology, you know, like where to go look for things. And it's just different from neural networks because Bayesian networks are used when we specifically have causal or probabilistic relationships.
00:55:02
Speaker
So in machine learning, obviously, it's a field that is experiencing a renaissance. And we could really call it a renaissance because there's this thing in the 80s called the AI winter, where our basically computers weren't at the degree that they needed to be to do continue AI research.
Future of AI and Machine Learning
00:55:19
Speaker
So people just got really depressed and stopped researching AI until like the mid 90s, really. And it didn't really get big until like the 2010s or so, or like, that's when it really started taking off again.
00:55:32
Speaker
So we have a lot of new paradigms that will exist. Like LSTM models only date back to, I think, 2011 or 2012. So, and there's new models popping up all the time. So it is a fertile field. Like if you have a good idea for a type of neuron, implement it, write a paper about it, put it on archive.org. That is ARXIV.org. And you might get an academic career out of it. That'd be really cool. Yeah. It's a great time to be doing that.
00:56:00
Speaker
Yeah, and I mean Alex, and we'll interview about this soon, possibly. I'm not going to promise anything, but we're thinking about it. Basically, Alex, who's been on episode 41, reality is more than complex, about group theory and how it relates to physics, has made a chatbot based on some gleamings that he's taken from computational geometry.
00:56:21
Speaker
So he's created a paradigm all of his own and it just goes to show how fertile this field is because I mean Alex is a brilliant person but he's only one person of a million brilliant people just like him and I'm sure several listen to this show. So if you have the gumption go do it. Yeah absolutely.
00:56:39
Speaker
Because this is the future. This is what is going to create the future of organizing, the future of data collection, the future of, I mean, honestly, you name it. This is the ground floor. That's a good way of putting it. That's an exciting time.
00:56:58
Speaker
It's an exciting time to be a person studying thought because we seem to be getting closer and closer to a point where we understand not only what thought is, but how to manipulate it and design algorithms that potentially make it more efficient and useful. It is also an exciting time because there are many challenges to be solved. In the 21st century, we're going to face more problems due to the complexity of climate change, politics, environmental changes due to pollution and climate change,
00:57:22
Speaker
and even just organizing society in a way that benefits the most people while starting to be human in that most quintessential human way, learning everything possible about the world around oneself. I'm Sophia. And I'm Gabriel.
00:57:35
Speaker
And this has been breaking math. Cool. So yeah, visit us on facebook.com slash breaking math podcast, or Twitter breaking math pod, our website breaking up podcast.app. And yeah, anything else that you think to blog or email breaking math podcast at gmail.com.