Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 16: Dopamine with Michael Frank
21 Plays5 years ago
Dr. Michael Frank of Brown University talks to us about dopamine -- how it works in the brain, what his research has done to elucidate the function of dopamine circuits, and some of the genetics behind it. A really fascinating dive into a great topic!
Papers:
Dopamine and free will:
Dopamine and learning:
Recommended
Transcript
Introduction and Acknowledgements
00:00:15
Speaker
Before we start the episode, we wanted to give a big shout out to Will Mason, who has created the music for the podcast. So you notice some new music at the beginning. That is Will Mason. Thanks, Will. It's awesome to have some fresh beats.
What is Dopamine?
00:00:28
Speaker
Our guest today is Michael Frank, who's a researcher at the Cognitive Linguistic and Psychological Sciences at Brown University. And he is here today to talk to us about dopamine. So dopamine, what is it?
00:00:43
Speaker
What's it doing in the brain? What is our popular conception of how dopamine works? What has recent research told us about how it works? Welcome to the show, Michael, and thanks for coming on. Thanks for having me. Should be fun. Thanks, Michael. It's always great to have someone from my alma mater on the show. I appreciate your coming on. Let's just start out with a basic question. What is dopamine?
00:01:11
Speaker
So dopamine is a brain chemical. It's a neurotransmitter that's made by neurons that are mostly in the midbrain. So very deep in the brain and shared amongst evolutionarily quite old. So it's shared amongst many other species and involved in a host of processes that we'll try to unpack a little bit about today.
00:01:36
Speaker
Good, and your approach to understanding how the dopamine system works in the brain is computational, correct? Yeah, so I mean, I tend to think that in general for trying to understand the links between the many levels of analysis from brains, molecules, genes, all the way up to complex behavior, computation is one way of trying to formalize theories that are testable.
Dopamine's Role in Reward Systems
00:02:00
Speaker
And so they both allow you to quantify how animals and humans
00:02:05
Speaker
behave and why they behave in certain ways or what problems they're trying to solve and make you sort of think about that from an evolutionary perspective and also from a normative perspective, like what's the correct way of solving different problems, but also just for quantifying a theory and forcing you to have some unified approach to your theory that prevents you from sort of waving your hands and saying, oh, my theory could account for all the data out there.
00:02:33
Speaker
when often verbal theories couldn't if you actually sort of implement them in mathematics. So from the perspective of the dopamine system, there's a lot of popular discussion of dopamine as a neurotransmitter and what it's doing in the brain. A lot of the discussion in the popular media comes down to the idea that dopamine is responsible for essentially like reward
00:03:02
Speaker
When you do something that's pleasurable, you get a shot of dopamine. And we know that this is a little bit not precise and perhaps not even quite correct. Do you want to talk a little bit about just at a high level, what your understanding of the role of dopamine is and particularly in the systems that you're looking at? Sure. I mean, I can say more. I mean, I think you're right that that's the popular press notion and it's not unfounded based on
00:03:31
Speaker
They saw nothing. Originally, a long time ago, it wasn't known that dopamine was a neurotransmitter or a chemical at all. But it was shown that it was important for movement disorders like Parkinson's disease. So it was one of the main successes, I'd say, of
00:03:50
Speaker
applying basic neuroscience research to help clinical conditions is Parkinson's disease because it was discovered that the neurons that make dopamine die in Parkinson's disease and that leads to all sorts of motor problems that maybe we'll talk a little bit more about later.
00:04:08
Speaker
And so there was two different competing sort of historically views on what dopamine might do. And one of them was that it's simply a motor neurotransmitter, that it promotes motor movement. But then there was a lot of research by, for example, Roy Wise and Olds and Milner have a classic paper showing that if you take a rat and you stimulate the brain systems that
00:04:37
Speaker
give rise to increases in dopamine electrically. And you can make it so that the rat will press a lever and it's essentially stimulating its own dopamine system. That the rat will continue to do that essentially forever and at the expense of real natural rewards so it won't eat and so forth. And it actually will kill itself because all it's doing is pressing the lever in order to obtain the dopamine. And so that and many other studies along those lines
00:05:08
Speaker
sort of supported that idea that dopamine might be involved in pure pleasure or reward. But then in the 90s or the late 80s and then 90s, there was a lot of studies that became extremely influential afterwards and influenced me as well quite a bit, starting off with Wolfram Schultz, who's a primate neurophysiologist who recorded from dopamine neurons in monkeys.
Research and Experiments on Dopamine
00:05:35
Speaker
And he showed that
00:05:37
Speaker
If you pair a, let's say just a light or a tone in a few hundred milliseconds or a couple of seconds before a reward, an unpredicted reward for a monkey might be some apple juice or something that when the reward is unpredicted, the animal doesn't know to expect that it's going to happen. You see that the dopamine neurons kind of start spiking like crazy. And that looks like so far that it's consistent with the idea that dopamine is involved in reward.
00:06:05
Speaker
But the important finding was that after a lot of Pavlovian conditioning was applied, so the monkey really expected the reward, you know, a couple seconds after the stimulus, the lighter the tone. What you see then is that when the animal gets the reward, he seems just as happy to lick it and still thirsty or enjoys the apple juice, but the dopamine neurons don't do anything to that reward. So it doesn't seem like they're actually conveying the reward anymore.
00:06:35
Speaker
And moreover, they do start getting very happy in response to the light or the tone that predicts the reward. So it's as if the light and the tone has taken on the reward properties, but now the reward itself is not evoking any dopamine release. And then what's further pretty conspicuous in the pattern of results is that if you then sort of play a trick on the monkey and you withhold the reward, so you show the light or the tone and you wait a couple of seconds and then you don't give it anything,
00:07:04
Speaker
then the monkey should be disappointed because it should expect that it's going to get some reward. And there you see that the dopamine neurons actually stop firing for a few hundred milliseconds and they sort of decrease below their baseline activity. So that pattern of results, which then was confirmed in a whole host of other experiments, was interpreted by actually computational neuroscientists that
00:07:28
Speaker
The pattern that you see in the dopamine neurons looks a lot like the pattern that you see in reinforcement learning algorithms that were developed by computer scientists and actually initially motivated by psychologists, where what you really need in order to learn something is to learn from the difference between what you expect and what you get. And the pattern of dopamine neuron activity looks like that reward prediction error. That is, when the monkey gets a reward that's not predicted, okay, that's a prediction error.
00:07:57
Speaker
But then if you learn to expect a reward, now the key thing in these reinforcement learning algorithms is that they spend across time. So it's not really just about reward per se, but it's about the amount of predicted future reward. And so if you're in a situation, you don't know that there's going to be any reward available. And suddenly some stimulus comes and tells you, hey, now you're in a context where maybe things are going to be better.
00:08:23
Speaker
That thing itself is an error in predicted reward because you're suddenly in a context that's better than you predicted before. And so that evokes a reward prediction error. And then when the reward does actually occur two seconds later, you shouldn't really be surprised by that because you've already expected it. And that's what you see is that you don't see any of these prediction error signals. But if the reward does not occur, then you should be surprised and negatively surprised.
Dopamine in Disease: Parkinson's and More
00:08:46
Speaker
So it's worse than expected. And that's why you see this decrease. OK, so let me try and piece this together
00:08:52
Speaker
or just kind of summarize what you're saying so far, which is the first ideas about dopamine were that it was something to do with either motor initiation, in other words, getting movement started. And we had some info from that from people with Parkinson's disease that dopamine levels seem low and there's some difficulty initiating motor movements in Parkinson's. There was the Oliver Sacks
00:09:22
Speaker
There's an Oliver Sacks movie with Robin... Awakenings, right? Awakenings, right, yeah. So that was in the 60s, I think, right? So it was when Oliver Sacks was a younger researcher and he was on some early trials to test out whether the drug L-Dopa, which
00:09:44
Speaker
which promotes dopamine production in the brain could help out people with Parkinson's disease. And since then, I mean, this is a whole nother topic. We could talk about the way in which dopamine works in Parkinson's and the ways in which you can administer or increase amounts of dopamine. And then the other idea about dopamine is that if you have too much dopamine in your brain, you can produce schizophrenia or schizophrenia seems to be a result of too much dopamine. So at the lower end, Parkinson's disease where you can't move.
00:10:15
Speaker
or have trouble with motor movements. And at the higher end, you have disordered thought and sort of like your mind is racing too much. And those seem like odd things to reconcile that it's the same sort of thing. Okay, so then in the 90s, we have research that suggests that dopamine is not just about reward, it's about the prediction of reward.
00:10:40
Speaker
specifically about the error in prediction of reward. So it's a signal of either a good surprise, so a good surprise would be an error in your favor, like something that you didn't expect, but something that's very positive, and would promote the release of dopamine, and then something that is a bad surprise, something negative, would decrease the available dopamine, is that correct? That's correct, yep. Okay.
00:11:11
Speaker
And that's about where we, and then we're, and then we're tying this into the idea of reinforcement learning. So in computational terms, the way reinforcement learning is used from the, especially the computer science and artificial intelligence community is not the simple notion that you want to reinforce stimulus response, associative links. That is, it's not just that when something good happens, you do it again without thinking about it. Although there are algorithms that allow that to happen.
00:11:40
Speaker
But it's broader than that. It's more like anything in which an agent, which could be a computer or an animal or a human, is trying to maximize the sum of its future rewards, where rewards are defined however you want them to be. But everybody has their own value function or things that they care about. And you're attempting to maximize that.
00:12:05
Speaker
The way in which you try to maximize that has to involve often some learning because you don't know which actions or states of the world are associated with the best outcomes. But the way that you learn could be through what's often referred to in computer science as model free where you don't represent
00:12:22
Speaker
anything about how the world works. You just learn incrementally through lots of experience that some states happen to produce reward prediction errors. And when that happens, you assign those states with high value and those or the same thing to actions. So, you know, you just out of habit, you learn that when you tend to move in a particular way, for example, if you're playing a sport, maybe you learn after practicing 10,000 hours that, you know, certain
00:12:51
Speaker
find motor movements are actually useful to you and you don't know exactly why you're not predicting every time you move in a particular way exactly what the next state of the world is and you're not thinking about the impact of that next state on your future choices, you're just doing it. And so it's important to have some algorithms that allow you to just do it, but it's equally important to allow yourself to sort of plan in a more flexible way when you go to a novel context and
Theories and Models of Dopamine
00:13:15
Speaker
You have some questions about how the world works. You might want to use other cognitive strategies and hypothesis testing, whatever, to try to maximize the chances that you're going to get reward. And as long as you're learning from that, that's still reinforcement learning. Based on what you were saying earlier about the role of dopamine in this, it sounds like dopamine is kind of involved in updating your expectations for reward. Is that, am I following that correctly?
00:13:44
Speaker
Uh, that's right. Yeah. So it, the change in the spike rate of the dopamine neurons seems to reflect something like a reward prediction error. So it goes up when it's better than expected and goes down when it's worse than expected. And it just doesn't change if you get what you expected. And that signal is used to learn about earlier states and actions that produced those prediction errors. So in the case of the monkey, if you'd saw a light and later there was a reward,
00:14:13
Speaker
and it didn't fully predict the reward at that time, then you would get an increase in dopamine. That's a prediction error. And the impact of that prediction error is to assign it the states that preceded the reward higher value so that they take on sort of reward predictive value themselves, such that the next time you see that light, now you're going to have a little bit increase in your prediction that you're going to get reward. And then you'll actually have a smaller
00:14:40
Speaker
uh, reward prediction error in response to the reward itself because you were a little bit better at predicting it. And so when you iterate many times that converges, meaning it'll fully be predicted if the reward occurs reliably at the same time. And so the algorithm will produce a value that is consistent with the true expected value of the state that the agent is in. If that makes sense. No, it does. Yeah.
00:15:07
Speaker
I'm wondering if there's something that we could think about in the real world where these kinds of prediction errors would come into play that we could paint a scenario where that's not just lights in an experimental box. Well, so one of them might be in a casino. So if you go to the gambling and there's different slots machines or there are different games that you could play like Blackjack and craps and so forth,
00:15:37
Speaker
Uh, and let's say you don't know the odds of how well you could do given the optimal strategy. Like in blackjack, we know what the optimal strategy is, right? You can write it down what you should do for each hand. Uh, and if you were to do that in play, according to the ideal strategy, I think the odds are something like 48% that you're going to win. You still lose, but more slowly over time. Yeah. So that's why the casino makes lots of money, but people.
00:16:05
Speaker
feel like, you know, maybe they're going to get lucky or maybe they have control over, you know, they don't necessarily act according to 48%. But if you think about it from a slot machine perspective, let's say there are different slots that have different mean payoff rates. And sometimes a casino tries to lure you in to think that one of them is just nearly missing or is going to actually is going to pay off for a bit. And then on average, it'll have a lower expected value.
00:16:35
Speaker
you still might want to have a system that would allow you to learn which are the best slots to play, right? So if it was true that you could actually win on average in some of the slots, then you would want to have a system that would allow you to choose the ones that would give you the best payoff. And of course, in the real world, outside of the casino, there actually are situations where it makes sense to do the best thing possible, like
00:17:03
Speaker
learning skills, like there are certain actions that on average, if you're learning to ski, it's one of my favorite examples about motor skill learning is some actions, you'll have fun, but you might look a little bit silly. It might not be that efficient. And so you'll get some prediction errors. Hey, maybe if I try this, things go a little bit better than expected. I got down the hill. Um, but if you try something that is a little bit more aligned with what maybe an instructor would say, and eventually you ingrain it in, that will have a higher probability of a payoff. You'll feel better. You'll go faster and so forth.
00:17:32
Speaker
Yeah, that's super interesting how that all kind of ties together there. I think that's a pretty good high level overview. I wonder if it makes sense to dive into some of your specific work in modeling and maybe get into some of those experiments that you've done as well.
Dopamine's Role in Motivation
00:17:50
Speaker
Yeah, sure. I mean, one way maybe we could start is to go back to what Ralph mentioned before in terms of the connection to Parkinson's disease and
00:17:58
Speaker
Motor stuff because, you know, we started off by saying there might be a motor theory of dopamine and then there's a reward theory and now reinforcement learning. How could those actually relate to each other? Maybe it's clear enough already how reinforcement learning relates to reward, because it's just sort of a modification of it. It's not really about reward. And so I can give you a little bit more context of that before talking about my specific research. So there's Kent Berridge in
00:18:28
Speaker
Michigan has made a lot of progress on understanding the role of dopamine, not really from the reinforcement learning perspective, but from what he calls incentive salience or incentive motivation. In his view, dopamine is involved in motivation in a way that causes an animal or human to want something. And that wanting is very different from liking. So this is where it differs from the popular press, where
00:18:57
Speaker
Uh, we know that a dopamine depleted animal, uh, if you give it reward, if you just squirt some juice into, uh, its mouth and let's say it, you know, it would have liked juice. Uh, one of the things that Kent has done is sort of quantify in terms of like the emotional, uh, ex facial expressions and other things. It looks like the animal is enjoying that food just as much as an animal who doesn't have dopamine depletion. So by depletion, I mean levels of dopamine are low in the brain.
00:19:28
Speaker
But what's different is that the animal who has low levels of dopamine will not work for that same reward, so they will not press a lever many times to obtain it. If you think about it in terms of humans, it would be like having a volition that if you got some rewards, you tasted some food that was really tasty, maybe you would say, hey, that's really tasty, but you still might not get up out of bed in the morning to go and
00:19:54
Speaker
go for a run to make yourself feel better. Even though you might know that after doing exercise, for example, maybe it's rewarding or maybe your food will taste better or whatever, if you're not representing the potential value of those actions when you're selecting them, you're not going to actually select those actions. So it's really about assigning motivational value to action selection rather than, you know, experiencing the hedonic pleasure per se. So that's the sort of dissociation between liking and wanting.
00:20:23
Speaker
or dopamine does not seem necessary for liking. There are other things like opioids that are involved in liking, but it does seem to be involved in wanting and motivation. Yeah, and this is a really interesting part. There's a book out just last year called The Molecule of More, which is about the idea that dopamine is essentially about, as you say, about wanting and about motivation. I wonder how much you feel that
00:20:53
Speaker
This aspect of dopamine explains motivation in human beings. In other words, I couldn't help when reading the book, I guess, thinking that there were some overgeneralizations that are made, that dopamine is responsible for the striving of humanity and why we basically do just about anything. I wonder how you think about it in terms of what role dopamine plays in motivation. Is it a fairly local role?
00:21:22
Speaker
I mean, is it everything that we do that we have some sort of motivation for? Is that dopamine in action? Yeah, I mean, so I think there does tend to be a tendency for people who study dopamine to think that dopamine is the answer to a lot of things. And clearly, the brain is more complicated than one neurotransmitter. And in fact, when you try to even ask what that one neurotransmitter does,
00:21:48
Speaker
In some sense, in my view, it doesn't really make sense to even ask that question, because it's like asking, what does glutamate do or what does GABA do? These are the most common neurotransmitters that are present in pretty much every brain area. And they communicate information. They're involved in exciting neurons or inhibiting neurons. But you can't say that they do kind of one thing, like motivation or reward or memory. They do lots of things. And so the impact of anything, any chemical
00:22:17
Speaker
function will depend on the effect of that chemical in that particular brain area and the architecture of that brain area and how it's embedded in a larger system and so forth. And so obviously things are more complicated than that. But also, if we talk about motivation in humans, there's all sorts of neural circuits that are involved. Dopamine interacts with a lot of them. And I think one of the reasons why we tend to think dopamine does everything is that it does have quite profound effects on a lot of behavior when it's manipulated.
00:22:44
Speaker
And it's one of the cases in which I'd say, theoretically and experimentally, we have some handle on how it does modify motivation. Even if it's acting, its effects would not occur by themselves. It's really only dopamine in the presence of other neurochemicals and neural networks that makes it have an effect on motivation. But it's also certainly true that other chemicals like serotonin and
00:23:11
Speaker
norepinephrine also have effects on sort of distinct aspects of motivation. But I do think that in the real world, so maybe just also to continue on the line of thought on the incentive, like sort of wanting versus liking and how that relates to motor function.
Decision-Making and Dopamine
00:23:27
Speaker
I think that a nice connection to that is, is in fact, what came out of that Oliver Sacks movie and book Awakenings.
00:23:35
Speaker
where the example there was if you have very severe Parkinson's patients that, it was a particular kind of Parkinson's, but for the sake of this discussion, just assume that it's severe Parkinson's of patients couldn't move and they were sitting in wheelchairs. The canonical example there was that if you throw a baseball at them, they can actually suddenly raise their hand and catch it. Even though when they're sitting in their chair on their own, they're actually not
00:24:05
Speaker
moving very much and they're having a lot of trouble with movement. A similar example is if you have a Parkinson's patient that has a lot of trouble walking and they're in a room and they have trouble walking, but then you give them a room where the floor has black and white checkers and they see those black and white checkers, that gives them these contextual cues about where might be the better place to put their feet and now they're actually a lot better.
00:24:33
Speaker
Those kinds of findings and lots of others from neuroimaging and lesions and so forth have led to the conclusion that this system that's impacted by dopamine and the underlying structures that dopamine innervate strongly, like the basal ganglia, are involved not really in the execution of motor actions per se,
00:24:54
Speaker
but in the assignment of value to those motor actions that allow you to sort of volitionally select what actions to take when it's sort of ambiguous what you should do. But if it's clear what you should do, like in the black and white checkered floor, or if a ball is coming to your face and there's not really that sort of like the one salient action, then you don't need that system to sort of boost that action selection process. It just sort of happens from the constraints that are provided by the environment in the rest of the brain, in the cortex and so forth.
00:25:25
Speaker
So that's interesting. So the word you used was volitionally select. When you have to volitionally select something or if it's ambiguous what to choose, then dopamine is important for allowing you to do that, but it's not needed in order to actually execute the motor action. Another connection to that is like related to this wanting liking dissociation. You just go back to the rodent literature.
00:25:48
Speaker
So one of the classical examples is if you give a teammate, so you just an animal's in a maze and it can go either left or right. If it goes left, it gets two pellets of food. And if it goes right, it has to climb a barrier of like, say a foot high. So it's a rat. And if it climbs that barrier, it'll get four pellets of food. So the healthy rat will climb that barrier to get four foot pellets of food instead of go for the easier two pellets of food.
00:26:14
Speaker
If the animal has depletion of dopamine, so there's not a lot of dopamine in its basal ganglia, then it will not climb the barrier. It'll go for the easy two. And that experiment by itself is not clear whether it's just that you need dopamine in order to execute the motor behavior. So you would think, well, okay, if dopamine is involved in motor performance, like one thought in Parkinson's disease, then it makes sense. You just simply can't do it. It's hard to climb the barrier. But critically, if you do the same experiment where the
00:26:43
Speaker
the animal gets, let's say, four pellets of food for going over the barrier and zero for going to the left, then it will sort of happily climb the barrier, even if it has dopamine depletion, which suggests that it's not really about the ability to execute the motor action, but it's about the sort of cost-benefit trade-off associated with the reward that you could get and the costs of the actions needed to reach that reward. And the costs essentially loom larger when you have low levels of dopamine.
00:27:14
Speaker
And that so that would be sort of the neural correlates of laziness that you just have low dopamine and you're going for the easy two pellets rather than expending effort on the climbing the wall. And there is a lot of research on that, that, you know, that physical effort tradeoff does depend on dopamine across species. But it's not only about laziness because it's also risk taking, for example, right. And in a gambling situation, they say you can have one dollar for sure or a 50 percent chance of two dollars.
00:27:43
Speaker
The expected value on average is the same, but some people prefer taking the risk and some people don't. We know again from across multiple species that if you elevate dopamine levels in the basal ganglia that people or animals will be more likely to choose the risky choice and if you lower it, they'll be more likely to choose the safe choice. I remember the thing that I was going to ask about which is,
00:28:10
Speaker
Okay, so the brain is super complicated and all neurotransmitters have multiple functions and it's difficult to parse out individual effects of any one. I think the other relevant piece of information here is the idea that it's not just washing neurotransmitter over the whole brain that there are particular circuits that are involved in dopamine and its activation of these circuits that is the promotion of the effects of this neurotransmitter and maybe some
Dual Role of Dopamine
00:28:40
Speaker
neurotransmitters are more widely available throughout the brain and dopamine has a couple more specific pathways. I wonder if you could, without going into too much detail about, I guess, particular brain area, say anything about dopamine pathways and what this can help tell us. So, you know, what dopamine is actually doing in facilitating a particular pathway? Yeah. So most of what I've been talking about is in terms of the motivational value, you know,
00:29:08
Speaker
of actions and reinforcement learning and motor movements all pertains to the effects of dopamine in the basal ganglia which are themselves a complex notoriously convoluted circuit because it's a bunch of different areas that connect to each other with many indirect routes like one area projects to another that projects to another with negative connections or like inhibitory synapses and trying to work out how that whole circuit works is something that people have been
00:29:36
Speaker
working on for a long time. And that's also where these computer models help to sort of establish theories that are hard to sort of work out in your head. But that system is the one that sort of the basal ganglia is thought classically to be a motor structure. And the presence of dopamine is most by far strongest, or there's more dopamine in the basal ganglia, especially in the striatum, which is the main sort of largest input area of the basal ganglia, in which there are these different
00:30:05
Speaker
populations of neurons that seem to represent things like the costs and the benefits of alternative actions in different populations of neurons that have different receptors for dopamine. And trying to unpack how all that works in a form like this would be hard, but we could just think about it canonically that you have some neurons that represent the good possibilities of what happens if you select this action and some that represent the possibilities of
00:30:31
Speaker
something bad happening and you have that for all the possible actions that you're considering in any one moment. Dopamine will act like a knob on tuning how much you emphasize or amplify the benefits by activating the neurons that are encoding the benefits and at the same time suppressing the neurons that encode the costs. Whereas when dopamine levels go down, it has the opposite effect and amplifies the costs of the actions.
00:30:59
Speaker
And that happens at both the time, according to some of our models and research, but also a lot of data that sort of tries to unify these sort of reinforcement learning from the action selection versions of dopamine is that essentially there's both effects and they happen for a common reason. That dopamine changes the excitability of the neurons representing costs and benefits, both while you're selecting actions and in response to the unpredicted rewards themselves.
Dopamine and Disorders
00:31:26
Speaker
so that you could have a way of modifying this sort of cost benefit decision making and modifying how much you learn when things do turn out better or worse than expected. But then there's also effects of dopamine in other brain areas like the prefrontal cortex, which I know you've talked about in some of your other podcasts, involved in executive function and working memory and more cognitive aspects. And I think the role of dopamine in that circuit is modeled mostly by other people, but some of our models as well.
00:31:56
Speaker
It's not about this cost benefit decision making. It's about completely different horizon of time. It's not like you're suddenly increasing dopamine and decreasing dopamine about reward prediction errors, but more sort of how robustly do you hold some thought in memory and use it to plan some course of action over the course of, let's say, many seconds. And there is an interesting connection between the role of dopamine in those sort of more cognitive circuits and the role of dopamine in the basal ganglia
00:32:27
Speaker
in a way that I think sort of interesting that if you modify dopamine levels sort of systemically, so if you give someone a pill that changes their dopamine levels, you see effects both on the sort of motivational and reinforcement learning aspects of behavior, but also in higher level cognitive functions, working memory and so forth. And it's not obvious a priori why there should be a relationship between the effects of dopamine on one and the other. But if you think about it,
00:32:56
Speaker
as if what's happening in the midbrain, the dopamine neurons themselves that projects both to the basal ganglia and to the frontal cortex, generally those neurons do convey information about motivationally significant
00:33:08
Speaker
outcomes in the world, so things that are better or worse than expected, or whether they would be better if you took some course of action. Dopamine neurons seem to reflect that. And it might make some sense for a system that was sort of evolved to work in the motor domain to then extend that out to more cognitive actions. Like if I hold something in memory, maybe that's going to be useful for the task that I'm trying to solve. And if I hold something in memory that's distracting,
00:33:33
Speaker
That's going to prevent me from performing the task. And so I'm going to actually get less reward in some sense, right? If you're trying to solve a math problem, it doesn't depend on it as being a primary reward. You get some reward for just intrinsically if you solve the math problem, right? Or maybe if you're in school, you get, you know, a better grade or something. Essentially, we think that the same, roughly the same rules apply in the cognitive domain as they do in the motor domain that you've
00:33:59
Speaker
learn which things to hold in mind, which operations to do and which ones not to do because it will prevent you from performing the action well. And so there's sort of that interesting connection between motor and cognitive behaviors, which I think also connects a little bit to what you highlighted before about high levels of dopamine associated with psychosis and schizophrenia, which does, schizophrenia is also very complicated and there's a lot of things going on besides dopamine, but it does hold up that
00:34:28
Speaker
prior to somebody actually being diagnosed with schizophrenia and having a full psychotic break, it has been observed that those people who are at risk for developing schizophrenia do have higher levels of dopamine in the basal ganglia. So it's exactly the inverse of Parkinson's disease, and especially those that go on to then have a psychotic break later on. When you go back and look at their dopamine levels earlier on, they had higher levels than the ones that didn't.
00:34:58
Speaker
The connection I'm trying to make here is that if your threshold for what you attend to in your mind and your memory is too low. Then something that a healthy person would consider, but then sort of ignore because it's distracting and not useful.
00:35:14
Speaker
somebody who has too high levels of dopamine will actually treat it as if that's a good thought, hold it in mind and make some connections between two things that shouldn't really be connected together. And that could lead to something like delusions. And in the case of the visual system or the auditory system, it could lead to hallucinations. That's a really good explanation. And this may be one example where
00:35:39
Speaker
learning something about the brain really tells us something specific about a relation between two different kinds of disorders. So we wouldn't necessarily have expected Parkinson's and schizophrenia to be related in this kind of way or existing along a continuum, but neuroscience does some good. Not that I have anything against neuroscience, but sometimes it seems hard to
00:36:04
Speaker
understand the relation between some of these brain processes and then the cognition that it relates to.
Genetic Variations and Dopamine
00:36:11
Speaker
Yeah, I mean, I think another area that, Michael, your work specifically sort of points to some really interesting connections between these systems and behavior relates to some of these pathways that you were discussing. Especially when reading some of your work, I was very interested and somewhat confused, actually, about the difference between the positive and negative aspects of the reward prediction errors.
00:36:40
Speaker
So it seems like there's two distinct elements where one is in regards to the positive reward prediction errors, and then the other one regards the negative prediction errors. And this might actually be related both to free choice and also to genetics. I thought that was an interesting set of connections. And I was a little bit, honestly, a little bit confused and following the storyline on what you were saying there about the difference between the positive and negative aspects.
00:37:09
Speaker
Yeah, so I mean, I think maybe it'll help to follow along the same line of thought with Parkinson's disease because, you know, my original foray into this field was in building neural network models as a sort of computational models of the brain areas and how dopamine might be involved in reinforcement learning. And then I realized, this was when I was a graduate student, that the model, while I was trying to account for other people's data that showed that
00:37:35
Speaker
For example, Parkinson's patients benefit from their medications that elevate dopamine levels on some cognitive tasks, but it actually hurts their performance on other cognitive tasks. And so I was trying to develop a model that could explain both ways in which dopamine could help and ways in which it could hurt. And I realized in building the model that it made a simpler prediction that hadn't been tested yet, which was simply that, you know, if you think that positive reward prediction errors are conveyed by increases in dopamine,
00:38:04
Speaker
then downstream in the basal ganglia there has to be a mechanism that allows the neurons to learn from that increase to repeat the actions that are most likely to lead to the highest amount of reward prediction errors. And there are mechanisms in the basal ganglia in neurons that have receptors for dopamine called D1 receptors that do exactly that. They listen to large increases in dopamine and then the synapses from
00:38:31
Speaker
the cortex, which represents the sensory state and the action and some internal state, for example, those all project to those D1 neurons and those synapses get strengthened. So the connections between the states of the world and the possible action that you just selected that produced the reward, those actually get stronger. But when dopamine levels go down, you have a negative prediction error. The effects downstream in that same system is not to activate
00:38:59
Speaker
those neurons that have D1 receptors, but actually to activate another set of neurons that have a different receptor for dopamine that are called D2 receptors. It makes it easy because D1 and D2. Those neurons then actually are normally suppressed by dopamine. Dopamine inhibits them. But when dopamine goes down, that means that suppression goes away, so they get excited, and then they learn at that same time because neurons that tend to transiently get excited,
00:39:28
Speaker
tend to learn about the connections. And so that allows the system to learn to not select actions that lead to negative prediction errors. Because I guess what I didn't say is that those D2 neurons are involved in a circuit that sort of suppresses the selection of those particular actions. And so that sort of convoluted idea is essentially simply just that when dopamine levels go down, there's a population of neurons that learns that action was a bad thing to do. And so in Parkinson's disease,
00:39:58
Speaker
If you have low levels of dopamine, it's harder to have those big increases to the positive prediction errors, but it's easier for the, the dopamine levels only have to go down a little bit in order for those D2
Dopamine's Influence on AI and Learning
00:40:11
Speaker
neurons to represent the costs because dopamine levels are already low and it doesn't take much to lower them further so that those D2 neurons get excited.
00:40:20
Speaker
The prediction that the model made there was that Parkinson's patients should be able to learn in reinforcement learning scenarios where they're having to select arbitrary actions and learn which ones of them lead to the highest probability of something good happening or the highest probability of something bad happening. That if you have low levels of dopamine, like Parkinson's patients, you should be able to learn from the negative and not from the positive. Then when that same patient is medicated,
00:40:48
Speaker
with L-DOPA that Ralph mentioned before, it sort of increases the release of dopamine from their existing dopamine neurons that have not degenerated yet, then that actually should improve their ability to learn from positive prediction errors because when those dopamine neurons fire in response to the good events, they will release more dopamine and now the patient should be able to learn from that. But the concomitant effect is that because they're taking a medication
00:41:18
Speaker
that's raising their dopamine levels. It's no longer perfectly under the control of the brain's natural regulation of dopamine. So you're essentially stimulating some dopamine all the time with the medication. And so when you normally would be disappointed and get a negative prediction error, that is not really sensed by the downstream D2 neurons anymore because they're continually being stimulated by this medication. And so the patients will not learn from negative prediction errors.
00:41:45
Speaker
And so that was a finding that was part of my PhD is that Parkinson's patients learn more from negative than positive prediction errors, but when they're medicated, it reverses. And the thing that's sort of interesting about that from the real world is Parkinson's disease, as we've already talked about, can be thought of as a disorder that's associated with assigning low motivational value to action. So it makes it hard to select actions. And that's part of the motor.
00:42:13
Speaker
Part of Parkinson's disease, although there's, there's other things like tremor that also arises from this circuit, but not as straightforward to explain exactly that way. But then Parkinson's patients that are given their medication, a subset of them, they get better in their motor symptoms for a while, but a subset of them shows impulsive behaviors like pathological gambling.
00:42:33
Speaker
compulsive shopping, hyper sexuality. And we've sort of proposed that this very basic mechanism of an asymmetry in learning from positive and negative prediction errors could explain some of that, not just the learning part, but also the sort of choice part, having a different weight, cost and benefits.
00:42:52
Speaker
So just going back to the blackjack example, if there's a 48% chance of winning blackjack, but your brain has to interpret that as 60% because of an imbalance in their dopamine system, that's going to make you repeat those actions. I don't know if I lost my train of thought from what your original.
00:43:09
Speaker
No, that's exactly along the lines of what I was interested in knowing more about, especially the distinction between the D1 and D2 neurons there. Yeah, super interesting stuff. And again, interesting how you can
00:43:25
Speaker
make a prediction based on a model of how the system is working and then actually relate that to someone who has a disease and then has a disease and then is either medicated or not medicated. It's pretty powerful stuff. Maybe we should take a short little break here.
00:43:55
Speaker
Okay, we're back. So now we are going to get into some more speculative stuff. And let's see, maybe I will ask you, Michael, what do you see as the end result of some of this research on dopamine? And I guess, I mean, one thing you could ponder is to what extent building computational models or building artificial intelligence that takes advantage of some of these kinds of
00:44:25
Speaker
ways of thinking about a motivational system. Is that something that you think it's an integral part of sophisticated enough artificial intelligence that it might almost be necessary to build in a motivational system like like the dopamine pathways that we have? Do you think that's or is it an optional thing? Yeah, that's interesting. So I mean, I mentioned that the first interpretation of the dopamine patterns of firing in terms of
00:44:54
Speaker
Reinforcement learning essentially is taking a computational theory, a theory that comes from, even though some of it was motivated by psychology in the first place, it was then took on its own life in artificial intelligence to use those kind of algorithms to train computers to do stuff like learn how to play games and learn how to control vehicles and all sorts of things. And so then interpreting dopamine in terms of those signals was very useful for advancing
00:45:25
Speaker
theory in neuroscience. And I think having theory is really important for making some headway because otherwise you're just collecting huge amounts of data and just writing them down and not sort of organizing it in any way. But it's also important to, you know, even though the engineers and computer scientists have come up with smart ways of making computers smarter, they also clearly have lots of limitations. You know, they're much smarter than us at some things.
00:45:52
Speaker
like the amount of processing capabilities they can do in per unit time, for example. But they're way, way, way far away from us in terms of being able to do reasoning and actually complex thought in lots of different ways. And so one of the things people are trying to figure out in artificial intelligence is how to make those machines more flexible and what would be needed to make them smarter and have a
00:46:19
Speaker
capability for language and have capability to optimize performance across diverse different environments and to transfer knowledge from one situation to another. And for that, it seems like we should be motivated to understand how humans and how biological agents have been able to solve some problems over the course of evolution. So what are the the cognitive level strategies? What are the computational level strategies and what are the neuroscience implementations of those things?
00:46:47
Speaker
Maybe we can look at these reinforcement learning algorithms and say, oh, dopamine kind of looks like that. But we shouldn't stop there and say, this is what dopamine does. Instead, we should say, well, how does dopamine work in that circuit? What kind of processes is involved in? What does the circuit look like? And when you think about what that circuit is doing in the brain after looking at a lot of experiments, does it really perform like these reinforcement learning algorithms say, or does it perform in somewhat different ways? And some of the research
00:47:13
Speaker
that we and others have done suggests that actually it performs, you can summarize the overall performance as a reinforcement learning algorithm, but it's not necessarily exactly the way that computer scientists dreamed it up. Of course, there's still a lot of debates on exactly what the right way of doing that is, but just to keep on the theme of what we mentioned before, even the basic idea of this opponentcy where you have like these D1 and D2 neurons and they're representing costs and benefits separately,
00:47:41
Speaker
That's quite different from what's usually done in most of the artificial intelligence computer science community where there's many, many different algorithms for doing reinforcement learning, but they don't tend to separate things out in the way that it works in the biology.
00:47:55
Speaker
And so you could try to say, well, okay, here's the theory of the human brain and other animals brains that overlap in some ways that the dopamine system is quite similar across those species.
Practical Applications of Dopamine Research
00:48:06
Speaker
And you could say, well, how does that theory allowed us to explain a bunch of observations like, you know, dopamine manipulations change, risk taking and effort seeking in Parkinson's disease and schizophrenia. But it's another thing to say, why was it built that way in the first place? How did evolution get to that particular solution and why is there
00:48:24
Speaker
Is that something that you think seems like a reasonable goal? If you're building an artificial intelligence and somebody brings to your attention the idea that while you need some sort of motivational system that goes along with what you've got, you could add something to a pre-existing system that just adds some kind of motivation.
00:48:46
Speaker
but the human brain likely didn't evolve that way. It likely evolved as a sort of big complex mess where you couldn't really describe it as motivation throughout the whole process, but it ends up functioning like a motivational system, but it's deeply embedded in all of the rest of the cognitive systems that we've got. How do you just throw motivation on top of what you already have? Or do you have to really understand the brain in a much, much more
00:49:15
Speaker
detailed sense before you can build some kind of artificial intelligence that takes advantage of the way that, you know, of the kinds of things that dopamine may do for us. Well, I think that's a great question, and a deep one, and really trying to understand the evolution of the brain across is really difficult. Right. It's a jumbled mess in a way. But I mean, you could start with a less lofty goal of understanding how it all develops in the first place.
00:49:44
Speaker
to say, well, it's already the case that in artificial intelligence algorithms, a lot of them do have reinforcement learning mechanisms that are trying to, because a lot of the situations in the world, well, maybe to step back for a minute, the alternative to reinforcement learning in AI is, the main one is supervised learning, where you just train, you take these agents and you just tell them this was the correct thing to think about. So vision algorithms tend to work that way, where you, if you,
00:50:13
Speaker
have an artificial agent that looks through a camera at pixels and you want it to be able to say, is that a house? Is that a car? Is that a license plate? What are the letters in the license plate? And you want it to just automatically classify those things. That's, you know, artificial intelligence has made a huge amount of headway on that computer vision problem, which is actually really hard, even though it might not sound that way. But the way in which it usually works there is you just give it access to huge amounts of data, really large,
00:50:41
Speaker
neural networks, artificial neural networks. But they're always trained where you tell the network, in this image, there was a house and that's what your output should have been. And if you guessed that it was a cat, no, it was not a cat, it was a house. And then on the next trial, it sees something else and it makes another guess that's wrong. And if you do that over millions of training samples, eventually it learns
00:51:05
Speaker
something that allows it to sort of generalize to like a new house it's never seen before, it will still be able to say that's a house and it does really well at that. But the real evolving brain in animals and humans doesn't have those supervised signals all the time in terms of labels. There must be ways in which- In other words, you don't have the ground truth of this is a house accessible to you. Yeah, especially if you don't have language. And I think we don't really doubt that other animals can
00:51:34
Speaker
can see things and categorize them and know what they are. But you do have access to something like, I predict that if I move a little bit, the house that I'm looking at now is going to look a little bit different because I'm looking at a different angle. And when you're moving your head around and moving around, you do get to make those predictions. And then your predictions turn out to be wrong because you didn't know everything about the physics of the world. And so you can learn through those kind of signals.
00:52:04
Speaker
When it comes to things like, how should you act in the world? What are the right strategies? When is it appropriate to greet somebody? And when is it not appropriate to greet somebody? Or how should I play the game of chess? Or how should I learn to ski? What words should I use when I'm talking? What's the syntax? All those different things. For the most part, you don't have supervised labels that say, this is what you should have done. This was the correct strategy. But really, you just do things. You might be able to copy other people.
00:52:33
Speaker
But you get to find out the outcomes of whether things turned out well or not. And so you need to have a sophisticated way of using that feedback that doesn't really tell you the optimal way that you could have done something, but use it to improve what you're going to do in the future. And AI algorithms currently do use some kind of reinforcement learning for those kinds of things. And so all I was trying to say was maybe we can learn something about the way the biology solves reinforcement learning to ask
00:53:01
Speaker
whether if we take into account the algorithmic description of the biology, so the way in which we think by looking at the circuit, it's doing something, and maybe we can make the AI algorithms smarter in some ways. And we have some initial ideas on that, but maybe I'll let you guys follow up with whether that's in line with what you're thinking. Yeah, no, absolutely. I mean, I know that, for example, the Sirius Cybernetics Corporation has thought about this a little bit. You guys familiar with the Sirius Cybernetics Corporation?
00:53:31
Speaker
I am not. You guys read the Hitchhiker's Guide to the Galaxy? Yes. This is the company that makes the robots in the Hitchhiker's Guide to the Galaxy and of course the elevators. But basically what they figured out is the robot needs two emotions, basically. Happiness and boredom.
00:53:52
Speaker
And then they work out the rest of themselves. That's basically the takeaway. And it kind of reminds me of the idea of having the positive and negative prediction error systems is being somewhat separable. You've got one system that's basically driving towards happiness and one system that's sort of driving away from being bored. And then you kind of let the system just run off on its own. Yeah, that's interesting.
00:54:13
Speaker
Well, okay. So this has nothing to do with what you're researching, but if you were building AI systems like this, in which, in what way would they go horribly wrong? Well, I think it all gets back to values, right? I mean, again, it's all, you know, this is the whole thing about what is like a positive reward and what is a negative reward.
00:54:35
Speaker
And I think that all comes down to what is the system valuing? I mean, for humans, right, we're valuing appetitive things, things that keep us alive or propagate our genes into the future generations. But with robots, of course, you have to assign to them what will be rewarding to them, right? And I think that's where things start to go wrong. Yeah, that's exactly my intuition as well. I remember a talk that I saw from a roboticist
00:55:05
Speaker
She was trying to figure out if you're trying to train a robot to learn on its own how to behave in the world, then of course you have to give it some objective function. You have to give it some goals that would count as rewards. But often your intuitions for how to assign those, even if you think you know what you want it to do, how to assign those values to the robot so that it actually does what you want to do and not something that would sort of
00:55:32
Speaker
trick you into getting the most amount of like points according to your reward system. So an example is if you do this instead of for a robot for an artificial learning algorithm that's playing a video game, there's an example of like, if it's just trying to get points in the game, but you want to train it so that it's going to learn to play the game in a way that's sort of like how a human would play it. And hopefully it would be able to transfer some of that
00:55:56
Speaker
knowledge to other games and so forth. There's an example of where if you use the just the points and you just allow it to do reinforcement learning, it can find these little like quirks in games that allow it that it ends up going like I think instead of going from the beginning to the end of a level and to the next level and so forth. There is like some quirk in a game where it will find that if just does little circles around some little
00:56:20
Speaker
barrier between two things, it tends to get some bonus points and it can sort of optimize that and just like cycle around that and do nothing else. And so yeah, we'll get more points than not doing that or, but it might also get stuck in a local minimum. If you, if you think the goal is to actually solve the whole game and get to the very end and kill the dragon and win so far.
00:56:40
Speaker
It'll never do that, but that's because you haven't really given it the right reward function so that it does what you really want it to do. Instead, you just said, try to win as much points as possible. And so that's a problem. So by simplifying it to the level at which we can instantiate it in a system, it might be too difficult to control in more complex situations, I suppose. And this is, I mean, this seems like, have you thought about,
00:57:09
Speaker
writing some sci-fi books about this because there's probably some rich territory here. I remember thinking of probably in the 90s or 2000s thinking about what it would be like if you had more control over, say, your dopamine system so that you could motivate yourself a little bit more when you knew that you wanted to be motivated, you know, that when you just don't want to get out of bed. If you could
00:57:34
Speaker
optimize that as much as you wanted. You could conceivably reward yourself or motivate yourself to do all the things that you feel like you don't want to, like eating your broccoli,
00:57:48
Speaker
or going to do exercise or going to do exercise. All those things that we're not, we're not exactly tuned for that are, you know, our taste system is built to get as much sweet food and salt as we can. And that doesn't necessarily translate into a healthy, it doesn't translate to a healthy lifestyle in the 21st century. But if you could tinker around a little bit with neurotransmitters that are modulating your motivation,
00:58:15
Speaker
Yeah, that may be, this can go horribly and right too, obviously, but yeah. I mean, that was what I was thinking about when we were talking earlier is like, I wonder how plastic is the dopamine. Are these dopamine systems themselves? In other words, like to Ralph's point, like how trainable is this motivation system is the valid. Yeah. I mean, there's different parts of reinforcement learning and some of them are like, what would be like the inputs to the dopamine system or what counts as a reward and how do you predict and.
00:58:44
Speaker
What counts as a predictor of future reward versus the outputs, which is, you know, once you've decided that something was better or worse than expected, how should you change your behavior to allow you to be more likely to achieve that again in the future or to generalize to other situations? And those are, uh, and I think what you're talking about now is the, the input side is like, how do you change the value function itself? Right.
00:59:06
Speaker
And yeah, the brain processes associated with that are even more complicated than the ones at the output side. So I'm not going to talk about that. Um, but, and also I think that concept of like trying to sort of trick your motivational system or hijack it to do what you want is, is an interesting one because, I mean, it's also kind of, I thought about this myself sometimes that it's strange that you can sometimes have this phenomenon that you want to want something that you don't want it. So you want to decide that maybe you should.
00:59:33
Speaker
go do exercise, but you don't really feel like doing it. Yeah. If you had a button that you could push that would stimulate basal ganglia and provide you with dopamine when you, I don't know how to describe it because you want it. Yeah. So you could do that both at the learning sense. So like after you've done something like a broccoli or done a hard exercise or whatever,
00:59:56
Speaker
You could try to increase dopamine further so that you are ingraining that and learning more about it. So that you become addicted to the things that you want to be addicted to, not addicted, but reinforced for anyway. Yeah, so that could help, but I think at least now we're into introspection. But on my own, I like running and doing exercise and so forth.
01:00:18
Speaker
after I do it, I almost always feel good. And it's not that I feel like I would need an extra boost of good feelings at that time or, and probably I'm getting dopamine as well, that even though it's not involved in the liking part, it is going up for other reasons that is providing the reinforcement. But it's when you're in a situation when you wake up in the morning and it's maybe you're in, it's a fall day and I cognitively know that even though it's not like that bright, sunny summer or spring day where it would be fun to run,
01:00:48
Speaker
that I still would feel good if I went running and I won't be cold anymore once it's done and everything will be great, you just really don't feel like doing it. And so what you really want to do is energize yourself in that moment, not at the end. Yeah, that's right. Yeah. Yeah. Yeah. And I think, I don't know, there are probably some tricks you could take there. I don't know, I guess you could do it pharmacologically or you could, I think, you know, there's also some psychology here where people who have bad habits, I think there's this
01:01:17
Speaker
absurdly simple rules that some people have talked about in terms of solving those things. Like you just tell yourself if this particular situation happens, then this is what I'm going to do. So like if somebody offers me a cigarette, I'm going to check my watch and go away for a second and come back. I don't know if that's the exact example, but just giving yourself some concrete rule about how you should act in a situation that would normally require a lot of resisting of temptations. Um, and I wonder whether you could use something like that to
01:01:46
Speaker
sort of trick yourself that when you're in a situation where it's time to decide whether you should go for a run or sleep in more or do something else that you could give yourself a little bit of a carrot for for increasing your motivation to do that, even, you know, just through psychological tricks.
Balancing Dopamine's Effects
01:02:02
Speaker
But, you know, just knowing how the dopamine system works might help for that. I'm not sure.
01:02:07
Speaker
But it would be much easier if you had a button that you could push. Well, yeah, everybody wants a quick fix, right? There's no feedback. So there's a woman named Allison Adcock at Duke and others that are looking at how if you put someone in a brain scanner and you measure activity from the dopamine system from the midbrain, and you give them feedback in a real time about what that's doing, and then you tell them
01:02:31
Speaker
You know, move this little ball so that that thing goes up. They can figure out how to do that to sort of reinforce themselves. And it's possible that if you train again, it's possible to do that, but I guess it's you. I would wonder whether
01:02:45
Speaker
If you figure out, okay, what are the things I need to do to do that? Could you then just use that in your regular life without any imaging or thoughts about dopamine? Just like get yourself in that state and use that to energize yourself. So you just have to think the right thought that that gives you that dopamine production. Yeah, possibly. Or this is the tricks that I play myself. So I sometimes like to do pushups. And if you're really tired and you don't feel like doing it, then you just might not do it. But if you say, okay, I'm just going to do
01:03:15
Speaker
So let's say you're used to doing more than 10, but you're like, you know, whatever, I'll just do 10. That seems like a small cost. So you just do it. But once you're getting going, that you know that once you get going, you actually are able to continue to persist afterwards. And it's really the initial decision to enter into that high effortful situation that is costly. And so you can trick yourself by saying, oh, I'll just give myself a little bar. And it's fine if you just meet that little bar. But a lot of times you won't, and you'll actually continue.
01:03:44
Speaker
Yeah, this is some of these techniques are related to things that people are using in like cognitive behavioral therapy. So for example, you know, for treating addictions, you know, setting up situations where you can identify triggers for, you know, different behaviors and then having
01:04:01
Speaker
plans for action and also token systems where you can give yourself rewards for so many days of sobriety, for example. I think there's some very direct applications to that type of thinking to treatments for different types of cognitive and behavioral disorders. That sounds right. The other thing I was thinking when you were talking about that, the exercise part was,
01:04:26
Speaker
There is a big thing that happens in, uh, specifically in running and aerobic exercise in general. I don't know if you guys have experienced this before, but I've definitely noticed it myself and I've seen it in a lot of other distance athletes as well, which is when you're out of shape and you haven't been training. It's super hard to get motivated on a day to day basis to go out. And it's like every single time you go to exercise, it's difficult. But once you've actually gotten.
01:04:50
Speaker
in shape and you've actually gotten those boosts that you get that those feelings of positive reinforcement that you get after exercise and that's you've been doing that for like, say weeks or months, it's suddenly much, much, much easier to go out and do it again. You know, each day, it's easier and easier and easier. And you actually really want not just want to want it, but you actually do want to do it. And when you miss a day, you actually feel like it's aversive. Absolutely.
01:05:18
Speaker
Yeah, I mean, there's probably a lot of things going on there. You get the habit of doing it. So there might be like a day where you don't quite feel like it, but because you've been doing it by habit, it's easier to get over that hurdle. And then there's the reinforcement that you get from having done it. And then also, if you're out of shape, you're probably experiencing a lot of negative prediction errors initially because it's harder, but you're also not going.
01:05:42
Speaker
as fast as you could have before or you couldn't, you can't run as long as you used to be able to. So you just feel better. So in this case, negative prediction errors refers to the pain that you're experiencing while you try to run when you're in bad shape. Yeah. Pain or just like if you care about how far you went or how fast you went, you're always measuring yourself relative to your expectations and your expectations tend to be what you were like before.
01:06:05
Speaker
Or maybe your friend, if you're running with a friend and you're not going as fast as they are or as long as they are. So there's the case where lowering the bar, if you can lower the bar for yourself a little bit, like you say, might be a good strategy. Yeah, I think it is a problem in that.
01:06:21
Speaker
The reinforcement learning system is designed to continually adapt its expectations so that you get prediction errors that are useful to improve, to continually improve in the world. But one of the flip side problems with that is that it's hard to ever, people tend to have these set points of happiness or satisfaction that might be partly related to that, right? Because if you're always establishing a baseline for your expectations of how well you're doing at life and whatever you're doing,
01:06:51
Speaker
then you're never going to be satisfied unless you're at least meeting those expectations, if not exceeding them. And if you're getting better, then those expectations get higher and that's what allows you to get yet better, but it makes it hard to really be satisfied. And if you do get worse, like you get out of shape, then it's going to take a while for those expectations to go down and then take a while for you to get better again, for you to start meeting those expectations and beating them. And so it all sort of interacts.
01:07:19
Speaker
Well, one of the and one of the things that I think this may relate to is your discussion of how different systems are in charge of strict rewards. So opiates or serotonin or other systems that may take
Conclusion and Future Implications
01:07:35
Speaker
charge after you've actually performed the act to sort of reward it. So whether they're selling beyond the dopamine system that we can think about in in motivating behavior and well, that
01:07:48
Speaker
So you don't want to live a life where you're always disappointed because you can't quite, you know, you're, you're motivated to do something, but when you actually get it, it's not as satisfying as you thought it would be. That sounds like a recipe for a terrible life. Yeah. So at some point you want to be satisfied with things the way that they are. And having an overachieving motivational system is only going to lead to depression about the way that things always turn out. But on the other hand, like you say,
01:08:18
Speaker
If you're a perfectly satisfied person and you have no aspiration, you have no motivation to sort of improve yourself, then do you need a balance between the two? I suppose. So do you think about this as dopamine providing some motivation? And then when you get the thing that you were motivated for, you switch over to another.
01:08:38
Speaker
you know, serotonin, serotonin, serotonin or opioid system that then it's sort of being, having the striving part, but also having the being satisfied part that you can switch over to, you know, is it, is it a strategy that you can use to switch over from dopaminergic to other systems that you can be more satisfied? In other words, you can, you can strive enough, but also be satisfied with what you get.
01:09:06
Speaker
Yeah, I mean, I think like you could still have the hedonic feeling of pleasure or other kinds of feelings of satisfaction without dopamine. I think that that's supported by the literature when dopamine doesn't change, when rewards are expected, you still have that. It's just that, you know, according to the reinforcement learning part of it, you wouldn't then further learn to continue to do that behavior. Or I guess if you produced something
01:09:34
Speaker
If you did produce some behavior that led to hedonic pleasure, but it was not as much as you expected from your prior experiences, you would still experience that pleasure in that moment. But according to what the dopamine system would be doing, it would be like telling you that whatever you did that led to that was actually a bad thing. The trick is to try to figure out.
01:09:57
Speaker
how to allow your dopamine system to be happy with the behaviors that produce that. So like, for example, if you're getting older, you're probably not, and you're talking about sports or something, you're probably not going to reach your peak performance that you had when you were in your twenties or thirties or whatever. But that doesn't mean that you should unlearn all of your skills, even though they're produced, they're, you're not meeting your expectations. You kind of have to change your expectations, but that's maybe okay because you can initially, while you might feel disappointed,
01:10:24
Speaker
while you're feeling the high of having run or whatever it is, that disappointment doesn't only make it so that you should not run anymore. It actually changes your expectations themselves. So your expectation can evolve. The idea is if you're getting better at something, your expectations go up and up and up so that you can get better and better and better. But if you're getting worse, they should go down. And there's that trick where if you've been exercising a lot and then you're just out of shape and you're not
01:10:54
Speaker
necessarily older, but you're just out of shape, then you might be disappointed the first few times you go because you're not meeting, you don't think that you're as out of shape as you were, but you really are feeling it now. But if you keep going, then your expectations will start to come down and you don't feel as disappointed by them. And then they should start going up again because you're actually getting better again. So maybe there's something to what you're saying also with while you're feeling the hedonic pleasure, even though you're not meeting your expectations, you can still use that
01:11:22
Speaker
in other ways besides dopamine to decide to do that again the next day, like cognitive strategy. I know that if I exercise more, I'm gonna get better, even though my dopamine system doesn't quite know that right now and in the way it's experiencing the outcome. It's a bit of the one-to-one. Yeah. Right. Yeah. The metacognition
01:11:43
Speaker
just knowing a little bit about how these processes work and being able to take advantage of it in a way that's beneficial. Yeah. I mean, I'm not sure you really need to know about how the brain works to do that, but you need to know that, yeah, I tend to feel disappointed when I don't meet my expectations, but I know that if I just work harder that I'm going to meet my expectations again. It might be as simple as that, but that's underlying all that is this circuitry that
01:12:11
Speaker
is computing those expectations, learning from them, and also experiencing the pleasure and having cognitive strategies of what you need to do in order to get better and so forth. Yeah. So I was wondering when you were talking about some of these topics with more or less motivated, more or less responding to rewards, this gene that you mentioned in your paper, the reinforcement learning mechanism responsible for valuation of free choice, the NeuroReport one from 2014?
01:12:40
Speaker
Yeah. The DAR-PP32. Yeah. Does that have functional consequences for people who had different polymorphisms that we can see out of the laboratory? Or is that just something that's super subtle that you only see in the laboratory? Yeah. Well, actually, if I could just switch to a different gene for a second. Sure, yeah. Absolutely. Which we did talk about a little bit in that paper, but in earlier papers as well. So there are genes that relate to how much the dopamine system impacts
01:13:10
Speaker
activity and learning in the basal ganglia and those D1 and D2 neurons that I was talking about before. And the simplest one to describe is the D2 gene. So it's a gene that controls essentially how many D2 receptors you have. So if you have a polymorphism, which is essentially a genetic mutation or variant,
01:13:29
Speaker
then you may have more or less D2 receptors than other people. And that doesn't mean it's good or bad because it could be consequences that are just different. There are different personality types, different degrees to which
01:13:44
Speaker
You know, if you take it literally, it's involved in like how much you're going to be a risk taker or not, or how much you're going to sense being conservative and not take actions that might have some costs and so forth. And you can imagine a story in which it makes sense to have people of all types in the world in general. It's not something that's going to kill you unless you're super risk taking and so forth. So anyway, we see this gene that if you look like you have a lot of D2 receptors according to your gene,
01:14:13
Speaker
because we know a little bit about how that variant controls the D2 receptors, then you learn more from negative outcomes in the same way that a Parkinson's patient who doesn't have medication learns more from negative outcomes. And if you have the other variant, then you learn less from negative outcomes. And we also see that in not just as a function of genes, but if you image, you take a brain scan of somebody, you can measure the amount of D2 receptors in there
01:14:41
Speaker
basal ganglia, and you also see an association there. So there's individual differences in how much people learn. But still, all of what I'm describing now is in these sort of contrived lab settings where we create these experiments to allow us to precisely measure how much people learn about different probabilities of good and bad outcomes, which you need to do to make it sensitive enough to detect in the lab the effect of these single genes. And
01:15:05
Speaker
For good reason, a lot of people are somewhat skeptical of these single gene effects on behavior because there's sort of a long history of psychiatric genetics where people thought that we could look at single genes and link them to things like schizophrenia and autism and so forth. And it turns out that, of course, things are more complicated than just a single gene causing those kind of complex disorders. And even though there were some reports on some genes being predictive of different kinds of illnesses or IQ and so forth, those tended to not be replicated. And when you had to have very large samples to show
01:15:35
Speaker
effects, and usually they're not single gene effects. But nevertheless, we noticed that in these paradigms, when it comes to the dopamine genes, a lot of the effects that we have observed are replicated in several samples of modest size, like 80 to 100 to 150 subjects. And the same effects show up across multiple experiments, not every single time, but they've never shown up in the opposite direction.
01:16:00
Speaker
So enough for me to be convinced that there are genetic predictors of behavior in these tests, even though I don't try to make the claim that they extend to full on disorders or personality types, because after all, there's just one gene in one brain system and they have to interact with lots of other things. And so maybe you need these lab tests to identify them.
01:16:20
Speaker
So very interestingly, somebody, a colleague at Harvard, Randy Buckner just emailed me last week about this new database in Oxford called the Oxford Biobank. And they have a website, so they've collected data from hundreds of thousands of people. And they have their whole genome, so all the genes, and the information about what diagnosis they had and when they had medical visits. So they didn't do any of these lab tasks.
01:16:47
Speaker
but they might have a diagnosis of cancer or of liver problem or of eyesight problems or Alzheimer's or Parkinson's, whatever it is. He looked at a paper that we had, not the one that you just mentioned, but a different one on three different D2 genes, three different variants of the D2 gene, I should say, that we links to learning from negative outcomes in these lab tasks. It turns out that those variants were strongly associated with personality traits of
01:17:15
Speaker
miserableness, feeling guilty after doing something bad and forgot. But there was a couple of other ones that were pretty clearly related to negative feelings associated with not performing well enough. Wow. Actually, that's wild. Yeah. All right. Because so is that is that something you can like get from your like 23 and me kit or is it? Yeah. Yeah. Sure. I have that one. I'm 100 percent sure.
01:17:44
Speaker
That's wild. Are those people more risk averse as well? Would that be a prediction or no? It would be a prediction from our models and some of the other data, but I didn't see that in. They don't have risk averse as like a problem. The problem with that biobank is it's really rich data, but it's just, you know, diagnoses. So somebody's not going to get that. They might have a diagnosis of like over too much risk taking or mania. And that gene was not associated with that, but yeah.
01:18:15
Speaker
Well, this seems closely related as just to tie this back into future hellscapes, mostly related to the Gattaca scenario where people might eliminate or people might, you know, if you have a choice between being miserable and being relatively happy, people might choose the happy version. But, you know, of course, that can have some unintended consequences, too. So is there any advantage to, you know, if you have a, if you want
01:18:44
Speaker
a number of different versions in the population, is there an advantage to having people who are anxious and never happy? Maybe I would encourage you to talk to one of my other colleagues on that Amitai Shenha who studies anxiety but also anterior cingulate cortex and things about cognitive effort and
01:19:08
Speaker
cognitive control and maybe the utility of, I don't know, I think it's fascinating to think, why are so many people who are smart? Why do they worry so much? How can you be smart and still plan without having that negative affective component of worrying? And I think it's not just about the D2 receptor and the striatum and the other parts of the cortex and other brain systems that are related to anxiety and worrying, but I guess
01:19:32
Speaker
Maybe to put it back to like, if you have this mechanism, let's say it is as simple as genes that bias the asymmetry and how much you represent positive versus negative outcomes. Is that a good or bad thing? Well, irrespective of whether it's good or bad to have distributions across the population. If you think about for any one person, it's easy to spin a story in which the same thing would be either good or bad, right? So a gambler, if they have that asymmetry might become a pathological gambler and continued and
01:20:02
Speaker
lose all their money and lose their friends and so forth. But a scientist who has that same asymmetry, we get rejected with our papers and our grants a lot. The ones that are successful have motivation to continue to persist in the face of adversity. Persistence and having some anxiety about performances. I mean, you can see how it could be a valuable thing for higher achievement. Yeah. So tenacity, but it could be the very same genetic trait, but against the background of other ones and other
01:20:32
Speaker
cultural backgrounds and other genes and other brain systems will produce an adaptive thing in one case and a maladaptive thing in another case.
01:20:41
Speaker
Well, maybe we should stop here. This has been a great discussion and hopefully we can have you back again sometime and discuss all the things we didn't really get to. We didn't even get to the idea of volition or choice and what that has to do with dopamine. Yeah. So maybe another time. Michael, thanks so much for being on the show. Thanks a lot. It was fun.
01:21:20
Speaker
The sounds as they appear to you are not only different from those that are really pleasant,