Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 50: Neural Interfaces
85 Plays1 year ago
Joe and Rolf discuss new work in neural interfaces that is helping paralyzed individuals communicate.
Based on the recent Nature article:
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., ... & Chang, E. F. (2023). A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 1-10.
Recommended
Transcript
Introduction to Neuroprosthesis and Speech Decoding
00:00:09
Speaker
Hello, and welcome to Cognation. I'm your host, Rolf Nelson. And I'm Joe Hardy. And on this episode, we're going to talk about recent work in neuroprosthesis for decoding information from paralyzed individuals so that they can communicate using neural network.
High-Performance Neuroprosthesis for Speech Decoding
00:00:31
Speaker
The article that we're going to be talking about is called a high performance neuroprosthesis for speech decoding and avatar control. And this just came out in Nature on August 23rd. Authors are Sean Metzger, Kayla Littlejohn, a whole bunch of other authors, and then the final author is Edward Chang.
00:00:55
Speaker
Yeah, I feel bad for sometimes for people who are the fourth author because I guess apparently several of these people contributed equally according to this. And so, but forever it will always be Metzger et al. There you go. Life isn't always fair.
00:01:17
Speaker
So okay, so this is a cool study and the basics. So the basics of this are that they recorded, they used brain recording and
00:01:32
Speaker
allowed a person who has been paralyzed in I believe in locked-in syndrome for the past 18 years or so to be able to communicate and use an avatar to to Speak words and generate facial expressions Yeah, I mean but was this person Like locked in would you describe it as locked in? I don't know that that's that's really
00:02:01
Speaker
what we're dealing with here.
Case Study: Patient with Brainstem Injury
00:02:06
Speaker
She's paralyzed, she's quadriplegic, and she's lost the ability to speak or articulate speech in a way that was comprehensible.
00:02:15
Speaker
But she was still able to communicate. She was using something like one of those head based tools. I think it was, I think it was, this was a Stephen Hawking level communication thing. Yeah. That she could, and from what I heard is that the device that she had been previously using gave an English accent. So for her, for most of the time that she was paralyzed, people thought she had a British accent, although that was incorrect.
00:02:43
Speaker
Yeah, interesting. Yeah, so yeah, so she's paralyzed. She's not able to speak. She had I think it was a brain stem injury. So like a stroke that did some damage to the brain stem, which which was pretty pervasive. Life threatening, obviously something that could could normally be quite deadly. But she was mostly paralyzed.
00:03:10
Speaker
Right, exactly. And so, you know, she had this array of electrodes implanted into her brain, partially with the goal, at least of, you know, seeing if they could help her communicate more effectively. Was there another reason to have that implanted? Or is that the only reason? Do you know?
00:03:29
Speaker
I know in some patients, they'll implant electrodes because they might be looking to find the locus of a seizure activity or something like that. Sometimes there'll be deep brain stimulation as well. I think this is purely just for recording for communication for this kind of study.
00:03:49
Speaker
Yeah. So, but it was a pretty, you know, it's like a sort of a flat, uh, sheet of electrodes that were late. They were essentially they laid it over part of her cortex, which this part of the cortex was the, um, uh, and included the motor cortex. So the part of the brain that is associated with moving different parts of your body.
Decoding Speech Intentions with AI
00:04:12
Speaker
So I think 253 channels, that's how much it can resolve. And yeah, like you said, over the motor cortex and there's somewhat posterior to that and also
00:04:29
Speaker
temporal as well. Okay. And they're specifically trying to get at the regions that would activate if she were speaking facial muscles and things like that. Yeah, exactly. So they took an approach where the idea of... So the goal is to help her communicate more effectively because she has this device that allows her to communicate, but it's slow. And so it's not very high resolution.
00:04:57
Speaker
Um, and in order to help her communicate more effectively, they wanted to see if they could decode her intention to speak. So the idea is that they're the brain activity in the motor cortex.
00:05:09
Speaker
would be the same as if she had the ability to move those muscles to create articulatory speech, but she's not able to because there's a disruption in those neural networks. But she can still have the thought of moving. She can try to move her, not supposed to speak. Exactly. And so they took the approach in this particular paper, and there's many different ways to approach this, but this, you know,
00:05:35
Speaker
In this particular study, they took the approach of trying to utilize that articulation intention and decode the neural activity associated with the attempt to speak and then correlate that with different things that she was trying to say. And then using artificial intelligence, create a model that then anytime she tries to say something different, it can identify what she's trying to say.
00:06:04
Speaker
Right. So they're simultaneously getting articulatory positions or where the where the parts of the mouth and tongue are moving as well as phonemes for meaning. Right. Exactly. So they're they're trying to decode into there's different ways to approach this. Like, right. Like one level of abstraction would be
00:06:33
Speaker
just try to look at the entirety of the brain and see like what if you could decode what happens when someone tries to say a word or a sentence and that is not effective currently in the current level of technology we're not able to do that like that's not possible consistently but what they were what they found was that actually if you start to try to pick out the specific
00:06:57
Speaker
fine movements associated with producing a certain phoneme, like a sound, like a particular kind of sound, you can actually essentially recreate those phonemes and predict words based on what the prediction of the phonemes is. They were also able to do it through a lower level, kind of more fine grained approach to actually trying to more closely deconstruct how the actual mouth would be moving.
00:07:26
Speaker
and what sounds that would create, which was slightly less effective. Yeah. So it's interesting. So presumably what she is experiencing when she's doing this is, I mean, you probably have to think very clearly what it is you're trying to say instead of, so it's less of sort of a vague thought that may be passing through her head as something you're really trying to articulate and trying to actually say.
00:07:53
Speaker
Right. And so she's actually trying to move her mouth in that way.
00:07:59
Speaker
it relies on the fact that she had previously had the ability to speak. It's interesting, too, because her stroke was something like 18 years before they put this implant in. So that means she'd been unable to move those muscles for 18 years, but they're still mapped onto something in the motor cortex that can be decoded. So I think that's pretty fascinating. Yeah. Now, that is a really interesting aspect to that.
00:08:29
Speaker
But yeah, exactly. It depends on the fact that she already had that ability in the past and it was preserved through the stroke and through the passage of time.
00:08:37
Speaker
But yeah, I guess the, the, the advance here, oh, this, this kind of thing has been done in the past, both with, you know, electrodes that would be placed, you know, as I said, on, you know, EG, like on the outside of the brain, much less effective. Yeah. And then also less resolution, right? Much less resolution. And then also in other patients who have had electrodes implanted in their brains for different reasons.
00:09:01
Speaker
As I mentioned, this deep brain stimulation is one, you know, other aspects, you know, trying to find the parts of the brain that are involved in epilepsy, for example. But yeah, so this has been done before.
00:09:16
Speaker
to help people recover some abilities in terms of communication. But the idea of the advance here is that it's more higher vocabulary and more accurate than previous and faster. Sort of like, you know, in terms of real time, this is like relatively accurate and relatively large vocabulary compared to what other folks have seen in the past.
Enhancing Neuroprosthesis with Neural Networks
00:09:38
Speaker
So this is putting out, I think, what is about about 80 words a minute.
00:09:43
Speaker
as opposed to earlier models that were a lot slower and really sort of plodding through something. Yeah, exactly. And they know how accurate it is. This is this question of like, how do you know how accurate it is? Because you don't know what that person was trying to say. But experimentally, the way they get around that is they basically give her sentences that she's
00:10:04
Speaker
then tries to speak. And that's, they train the model on those sentences. And then the model tries to predict which sentence of those that she's trying to, or word that she's trying to read or say. And so that's basically how they know what she's trying to say. And that's what, what's the accuracy here?
00:10:23
Speaker
It was kind of in this range of like 75 to 80%, depending on the metrics, a whole range of different conditions that they tested that range from like, you know, fairly low accuracy of 50% up to like closer to 90, depending on the conditions. But, you know, the average seemed to me like kind of in that 75% range where, you know, you produce ultimately some statements, you know, in sort of, um,
00:10:50
Speaker
in the wild, if you will, that might be difficult to predict, might be difficult to interpret. But you also do do a decent amount of communication with that as well. It seems like she was able to actually communicate quite a bit. Well, certainly more than nothing, right? I mean, it's amazing. I mean, because she's not producing any other sort of communication besides, you know, what's, what's coming out of that motor cortex.
00:11:15
Speaker
Right. Well, she had that thing where she could use her head at some point to select different letters, but that's very, very slow.
00:11:26
Speaker
So I guess one of the questions here is, is this 75%? Is that pretty good and honing in on 90% or 100% where this stuff is going to be great? Or is that where the last little bit of this is going to be really difficult and sort of intractable, where most of this is getting that last little bit of it?
00:11:52
Speaker
Yeah, I mean, it's a good question. I guess the first question is like, how usable is it at 75%? And partly that's going to be a bit of a user interface question. Like, is this rig something that you can use on a daily basis? I don't know.
00:12:09
Speaker
They gave some illustrative text decoding examples from one of the sets. So the targets, like for example, in one case, the target sentence that she was trying to read was, you should have let me do the talking. And the decoded sentence, like what the computer outputted was, you should have let
Ethical and Practical Issues of Brain Implants
00:12:27
Speaker
me do the talking. Perfectly exactly accurate. And then in the middle, something like, why would they come to me?
00:12:37
Speaker
Why would, and it was decoded as why would they have to be, it doesn't actually, you're not getting the meaning of the sentence out of that. Right. Right. Yeah. The error rate is only 33%. It's 67% accurate, two thirds accurate, but it's totally not what it's not comprehensible. Yeah. When you get down to like error rates of 75%, you know, how is your cold is translated as you're old.
00:13:05
Speaker
which is not anything. It's 0% of the information right there. Yeah, exactly. Exactly. So yeah, so I guess it's, it's cool. And it's very cool. And so certainly in advance, but
00:13:20
Speaker
One of the things that was interesting was they were able to differentiate for the electrodes where the information was coming from in terms of the information that was being used by the algorithms to decode the speech. And they found that it was actually those areas of the motor cortex that would be associated with speech.
00:13:43
Speaker
They were able to differentiate that between like, for example, motor movements, attempted motor movements of the hand, for example, which were they were able to to actually detect from some of their electrodes, but they were able to find that actually it was the parts of the brain that you would expect to be involved in speech that were being decoded that led to the performance.
00:14:07
Speaker
So that's a good, this is a good opportunity to talk about the homunculus, which I think we always should talk about the homunculus whenever there's an opportunity to talk about the homunculus. So the motor cortex homunculus. Yeah. Or the sensory homunculus. I think this is, we're talking here, I think about the motor, motor homunculus. Right. Yeah. So do you want, do you want to break down the homunculus?
00:14:34
Speaker
So the homunculus is just this representation on the motor cortex of different places in our body. And it's represented with different sizes for different regions. So it's a topographic mapping.
00:14:50
Speaker
It looks like a little guy, the shape of where the different parts of the body are represented. Contiguous parts of the body are represented by contiguous parts of the motor cortex. But it creates a weird shape, and it's not like a- Well, for the sensory homunculus, I'm thinking he has giant hands because he got lots of sensors on your hands, and then a very small back because there's not really any sensors on the back. Big lips. Yeah. In the motor cortex, you get a lot of
00:15:19
Speaker
representation for the parts of the mouth and jaw that are responsible for speech. Yeah. So that's, that's kind of part of this is like, speech is like highly over represented in the motor cortex. It's partly how this, why this is effective, I think.
00:15:39
Speaker
So what do we attribute some of the success to here? Because some of this is just improved neural network modeling and language models, right?
00:15:52
Speaker
Absolutely. Yeah. I mean, so they're, they're taking advantage of deep learning models. They're taking advantage of new language models. There's also the speech, speech, articulation models that have been banned. So those are, those are already kind of pretty well developed. There's a made less drastic gains recently, but, uh, certainly the neural network models, the deep learning models have improved and the speeds of the, the, the, you can do the training sets on, you know, we're talking about it's a matter of weeks now that they're able to train these.
00:16:21
Speaker
data sets up, which is just a vast improvement. So it's coming into just practically useful. You know, it's something that you could actually take advantage of with one individual. I mean, that's kind of one thing about this is that it's really particular to this individual. Right. That's right. If you took the same algorithm out and put it in another person, it wouldn't work.
00:16:46
Speaker
probably would give you 0% accuracy, you would need to retrain the model, but the procedure couldn't could work for another person. Well, and actually, it could even do better in another person to after training, because presumably, in most, most other people, you're not going to have this long term paralysis.
Potential and Limitations of Thought Reading
00:17:07
Speaker
Right. So there's a question of like, how, how effective could it be in someone who still had their intact speech? Right. But then you might not probably get the opportunity to implant a big, uh, electrode array. That is true. Motor cortex up there. Otherwise it's okay.
00:17:23
Speaker
Yeah. So the things that are limiting this from perfect performance are some of the neural network models, but then the resolution of the signal that's coming in too. So getting the signal from the right place, hitting that motor cortex, and getting good resolution there, and just having enough resolution in general, I think, is always going to be something that you want. Because if you've got
00:17:47
Speaker
86 billion neurons firing in your brain and you have this low resolution impression of just a really small percentage of those. It's going to be hard to get. It's going to be hard to reconstruct what it is your brain was trying to do. The higher resolution we get there, the better. Absolutely. No, for sure. I think this is where we talked about
00:18:14
Speaker
EEG, electroencephalography, where you have electrode arrays on the outside of the head, on the actual scalp to the extent possible. Those are going to have much less fine grain resolution because not only are they capturing a huge proportion of the brain in terms of just the area that each electrode is capturing from, but also the strength of the signal is really weak because it has to pass through this electrical signal
00:18:44
Speaker
has to pass through the skull, which is thick and doesn't transmit electricity very well. So that's where this electrocorticography approach, which is like laying the electrode array on top of the cortex underneath the skull itself. So you actually excise a piece of the skull, place the electrode array, then sew the skull back up.
00:19:10
Speaker
That's a lot more effective because you can, first of all, you can get much, you know, the strength of the signal is stronger because you just don't have to pass through the skull. It's also much closer. You can get a finer grain detail, but it's still, yeah, it's still each represent, each electrode is picking up signal from millions of neurons. Right. Right. Ultimately.
00:19:32
Speaker
And certainly hundreds of thousands strongly represented in there for each electrode. So it's still quite limited in terms of its resolution.
00:19:45
Speaker
But what's an interesting thought experiment is for each of these types of devices, depending on where the signal is coming from, what is the sort of logical, what is the mathematical limit of how much you can do with them? Right. How much can you extract out of it?
00:20:06
Speaker
And we did have one of our earlier episodes was with Adrian Nester on EEG and extracting information like this too. So how much can you extract out of pure EEG off the surface of the skull? And you know, how much can you extract from, you know, single cell recording, if you can get that?
00:20:27
Speaker
Yes, exactly. Yeah. So it's going to be very different. And so, I mean, this is like, you know, the Elon Musk stuff where he's the neurolink stuff. Yeah. He's really interested in, you know, implanting electrolyte arrays into people's skulls.
00:20:45
Speaker
High resolution, super high resolution, but yet you still don't want to destroy the brain that you're recording from too. So yeah, there's a lot of tricky, there's a lot of tricky things involved. Yeah. It's a high risk business in terms of getting your skull operated on, uh, your brain operated on. Um, and, and right now the value for the average consumer is probably not there. Yeah, that's probably right.
00:21:12
Speaker
But I mean, it's an interesting thing. I mean, like, would it be cool to, like, you know, control all of your electronics using just your thoughts? Would it be cool to, like, be able to, like, you know? Well, it sounds less cool, I have to say, if you have to articulate your thoughts into words, right? Because then you might as well just say it, right? And then it's just like you're talking to Alexa.
00:21:32
Speaker
Exactly, exactly. Exactly. At that point, it's like, if I still have to move my mouth, what a drag. It's exhausting. It is exhausting. It's exhausting. So yeah, the value is substantially less. Yeah, so I think it kind of speaks to also just the thing with
00:21:55
Speaker
electrical recordings from the brain broadly, which is to make sense out of them, you need a lot of data about the correlation between a particular, whether it be thought, movement, et cetera, and the electrical signals themselves. So it all comes back to psychology because you have to, the ground truth is always the person's intention. So you need to be able to collect that person's intention. So I think the way that a lot of these devices
00:22:24
Speaker
are represented in the media in terms of like the potential, but even like the reality of the existing, you know, they have these EEG devices that people use for like meditation and stuff like that nowadays. How well those are actually correlated with what, you know, they are said to be representing, I think is, you know, is a topic of, you know, that's just, they're not very good right now, I think is this kind of the,
00:22:51
Speaker
Because of some of these fundamental limitations, it's not that there's not enough compute out there to make them good. It's that the strength of the signal, the fidelity of the signal, the fine-grained nature of the signal is such that
00:23:07
Speaker
It's, it's difficult to gather that information from electrode on the surface of the skull, especially, especially if you're moving around in the actual world, you're not in an isolated chamber, you know, immobile with, you know, 256, um, electrodes on your skull. Yeah. Yeah. Yeah. So this, yeah. So right now the best brain computer interfaces are still, you know, talking, typing,
00:23:36
Speaker
Well, that's interesting. Okay, so would you want Okay, so I mean, thinking about this idea between reading your thoughts versus reading your sort of articulation from your what your mouth is doing. There is something strange about reading someone's thoughts when they haven't organized it into sort of a stream or a sentence or something that's kind of, you know,
00:24:03
Speaker
Yeah, some sort of linguistic form to it anyway. And that's what, so this paralyzed omen has, I mean, she's putting all of her, could be somewhat stray thoughts into a more organized linguistic form. I think it'd be weird to have a thought reader that sort of read my thoughts where I didn't, I wasn't able to sort of, you know, articulate them even just to myself in a way, so they'd be a lot more random.
Public Perception vs. Reality of Brain Interfaces
00:24:30
Speaker
Yeah, and I think it's it's also it's an interesting thought experiment from the perspective of what would the nature of that representation be like what what would how would you represent someone's thoughts.
00:24:44
Speaker
when it was not explicitly them trying to speak. It's when it's just a lot of things sort of floating through their head. It may not represent their- It may not really come in. It may not be represented easily in words or sentences, but then, you know, it's interesting because I, like we've talked about this before, but like I personally have like a very strong like verbal inner monologue in the sense of like, and I talk to myself in my head all the time. Yeah. And I thought about that. I'm not, I can't tell sometimes whether I'm doing that, but
00:25:11
Speaker
I don't do it all the time but certainly there's times when I do do that and that would make perfect sense to represent that as words and like I could imagine someday somehow running an experiment where you recorded from someone's brain and then they were able to then reproduce I mean I think you would have to ask them later what they were thinking because if you're asking them what they were thinking now they'd be talking and
00:25:35
Speaker
or typing. It would break the kind of, right? Right, right. So like, how do you encode it in the first place? How do you create the tags? You'd probably have to think something and then stop. What were you thinking about? And then you'd have to match it up. Then you'd have to match it up. And then I guess one of the things that was really hard in this paper was getting the timing right.
00:26:03
Speaker
So just figuring out when she was meaning to be talking. Because otherwise, it's a substantially bigger problem because these neurons are not just sitting there quiet and then when someone starts to talk. It's all continuous. It's a continuous flow of information.
00:26:24
Speaker
And so knowing when someone was trying to talk is a huge benefit for the model to reduce the amount of information that needs to be processed and encoded. Yeah, so I mean, I guess the other thing is if you're representing, because I think in the movies, the movie representation of this is the thought reader
00:26:49
Speaker
Oh, you're in my head. You can read what I'm thinking, you know, version. It would be. Yeah, it would be interesting to think about
00:27:01
Speaker
what that would be like, what the representation of that would be like, you know, with a thought reader, like, sure, sometimes you get a sentence, but maybe what you'd want is just like someone's general mood. Are they, you know, is there like activation level highs or activation level low, you know, some other sense of feeling or tone, but something that might be different than what their inner monologue is.
00:27:23
Speaker
That's right. It might not map exactly to the words. And I think that's what people imagine that these devices can do already to a certain extent, which is, I think, a question. I think we could explore. I guess, to some extent, it's possible to do some of that, some of the time. But largely at this point, under limited laboratory conditions.
00:27:45
Speaker
I wonder if some of this can improve over time with a single individual too. The more she uses this, the more she might identify with the thoughts that come out of it too. Right, exactly. You could imagine a coupling where she figures out the way to
00:28:04
Speaker
the way to attempt to articulate a certain sound that the model can then represent. Yeah, it's as though the errors are just sort of mismatches right now. Yeah, exactly. Yeah, so that'll be interesting to see if they follow up with
00:28:18
Speaker
Well, this is cool stuff. I mean, they keep coming out with new models of this every couple of years and they are advancing quickly just like, you know, all of the advances in neural networks in general, large language models.
Future Developments and Applications
00:28:32
Speaker
So it's exciting to see this stuff. And, you know, this is an application where they're actually doing some good and there's some real potential for positive change here. Absolutely. And you could totally imagine, you know, eventually a model that
00:28:44
Speaker
you know, uh, allowed, say for example, like a parallel, someone who's paralyzed and all four limbs, you know, a mech, like, you know, that they were able to move and walk with just like, as though they were naturally walking by the intention to walk. Like Ironman, like an Ironman suit sort of man, mech kind of thing. Exactly. What about driving? Driving might be dangerous.
00:29:08
Speaker
Yeah, I think, well, I think, yeah, the thing with driving is it's, we're already getting like humans are, don't, are, you don't need to drive because they just have the, just have the computer drive. It's actually just better. Why have a signal from a person in the street? Exactly. Yeah. At that point, the personal person's signal is not, we're not going to really even develop that technology because it won't be necessary or useful. Cool. I think that's probably a decent place to wrap it up. Um, you know, quick little check-in on this technology, something we've been tracking.
00:29:37
Speaker
But in terms of getting in touch with us, cognationpodcastatgmail.com. I'm at JL Hardy PhD on Twitter. We're at NationCog on Twitter. We also just started a YouTube channel. We've got one video with actual visuals. We're talking about probably
00:30:00
Speaker
doing just a logo and then the audio behind it, because it's just too much work to make the actual videos. We'll see what feedback looks like. We'll see if anybody looks at that. But that's also Cognition Podcast. That's the YouTube channel at Cognition Podcast on YouTube. And you can find us on all the podcast services. Rolf, do you want to give your Twitter handle as well?
00:30:25
Speaker
I would love to. I have forgotten it since I never use it. I can't do that. It's probably. Yeah. If you say something to me about, uh, about cognition, I'll make sure we're all fears about it. Awesome. All right. Thanks for listening, everyone. Thanks.