Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Balazs Szigeti: Unblinding Effects in Psychedelic Research
0 Plays2 seconds ago
For the third time, Joe and Rolf talk to Dr. Balazs Szigeti about psychedelics for the treatment of depression. This time, Balazs discusses his recent paper that shows an equivalence between psychedelics and traditional medications in effectiveness for treating depression. This is surprising, given the size of effects that have been demonstrated in some early psychedelic trials, and can be accounted for, in part, by patient expectations.
Williams ZJ, Barnett H, Szigeti B. Psychedelic Therapy vs Antidepressants for the Treatment of Depression Under Equal Unblinding Conditions: A Systematic Review and Meta-Analysis. JAMA Psychiatry. Published online March 18, 2026. doi:10.1001/jamapsychiatry.2025.4809
Recommended
Transcript
Introduction & Guest Welcome
00:00:08
Speaker
Welcome to Cognation. I'm your host, Joe Hardy. And I'm Rolf Nelson. And today we have friend of the pod and returning champion, Balazh Seghetti.
Role of a Clinical Data Scientist
00:00:19
Speaker
Balazh has been on the show a couple of times already, including some of the topics that we're going to be discussing today. We covered in the last episode that he was on last summer.
00:00:30
Speaker
Balazh is clinical data scientist at UCSF's Translational Psychedelic Research Program.
Blinding Problem in Trials
00:00:36
Speaker
He has served as lead scientist for ah lead data scientist for two of the top labs in psychedelic research, Imperial College London and UCSF, putting him at the center of some of the biggest psychedelic trials in recent years, including the landmark psilocybin versus eschatolopram study and the recent psilocybin for Parkinson's disease trial.
00:00:59
Speaker
The main the thread across his work is the so-called blinding problem, the awkward fact that in psychedelic trials, participants can usually tell whether they got the real thing or a placebo, which throws a wrench into the gold standard of double-blind placebo-controlled trial design in psychedelic research specifically.
Comparing Psychedelic Therapy and Antidepressants
00:01:19
Speaker
And today we're going to be talking about a paper that he wrote with his colleagues, Zachary Williams and Hannah Barrett, called Psychedelic Therapy versus Antidepressants for Treatment of Depression Under Equal Unblinding Conditions, a Systematic Review and Meta-Analysis.
00:01:37
Speaker
And this paper was ah published in the Journal of American Medical Society ah Medical Association. Is that right? i think it's Association. Yeah, Journal of American Medical Association Psychiatry.
00:01:52
Speaker
And so, yeah, Bosch, welcome to the show. Thank you for having me. Yeah, great. So why don't we just dive right into it. ah Tell us about ah the the paper. What made you want to pull this ah review together and you know what was sort of the background there?
Response to Blinding Critique
00:02:13
Speaker
So actually, there's a funny background story here because this paper started out as a really short piece that was meant to be a response to the letter. that was written by a researcher at the University of Toronto. His name is Matt Burke. He's a friend and a colleague.
00:02:28
Speaker
And in, I think it was 2022 or 2023, when the um tip psilocybin versus acetylopram study was published that you also mentioned in the intro.
00:02:39
Speaker
And when that paper was published, there was one of the letter to the editors penned by ah Matt. which brought up this issue is that no blinding data was collected. And I was working at the Imperial College at the time. And, you know, that being my area, I was like, you know, I was asked that like, hey, like, you know, how you how would you respond to that? And, you know, initially I was like trying to make some cookie cutter argument about like, you know, why the blinding should not matter in psychedelic studies. But I was thinking about it. I realized that, okay, like, you know, the issue is, is that psychedelics are practically always open label.
00:03:13
Speaker
then you could make a fair comparison with traditional antidepressants if they are open label, which is the technical term for the open label is the technical term for my patients know what they are getting.
00:03:25
Speaker
So, you know, i was formulating this like short response, like I think it was like 300 words, but then I realized that actually this is a really good idea and I'm not going to fit it into 300 words. So let's spin it out to its ah own paper.
00:03:37
Speaker
And, you know, there was a lot of working between that moment and this paper being published. But really, it started out just as ah ah a response to this letter. And, mean, you know, some credit to Matt as well that, like, you know, yeah, in a sense, this letter inspired this work.
00:03:54
Speaker
basic idea here and the big, you know, sort of driving concept here is that it's very hard or too impossible to run a true double blinded control study of psychedelics.
00:04:10
Speaker
Correct. And this is true in a variety of different areas. This is true in some behavioral research as well, but it's particularly true in psychedelics research because You can give a person a placebo ah versus ah psychedelics.
00:04:27
Speaker
but they're gonna know within a half hour to an hour, you know fors like for psilocybin, ah whether or not they've received the treatment drug or the kind a placebo. Because if you've received the treatment drug, it's quite a strong effect and with strong subjective experiences that that are you know quite well known and and well established and pretty much always happen. So if you've received the placebo, you realize,
00:04:57
Speaker
within a relatively short amount of time that you've received the placebo and vice versa. And so this basically, you know, do you want to talk a little bit about why that is a problem for for so-called blinded trials or the concept of a blinded trial?
FDA Focus and Broader Treatment Parameters
00:05:14
Speaker
Yeah, sure. So like, you know, I'm in a little bit of an awkward position here because I am you know, my research and my my career, you know, has been known as like this this this blinding issue, as you have mentioned at the top of the show, but The longer I spend on this topic, the less and less I personally care about But the FDA does care about it a lot. So like you know I think you know the reason why we care about it, and here by we, I mean you know the broader research community, is partially driven by that what the FDA is approving drugs is based on something that is called the between-arm difference. or sometimes it is also called the treatment versus control difference. So how much better your drug did relative to the placebo or whatever control condition you have in your trial.
00:06:02
Speaker
And it's a little bit more complicated than just subtracting two numbers because you need to use the pool standard deviation, yada, yada, yada. But for the purposes of this podcast, we can't just like think about it as the difference between what was the patient improvement in the treatment arm minus the patient improvement in the control arm.
00:06:20
Speaker
So let's say the patient improvement in the treatment term was eight points on whatever scale and five points in the placebo arm, then this between arm difference is three points. When the FDA is approving drugs, they are looking at this difference. And this is really the only metric that they are going to look at.
00:06:37
Speaker
So that is why this is like such a such of a big importance to researchers. but That being said, like you know there is something to be said that I think you know this is a quantity that captures something important. like you know I'm not dismissing you know the importance of the between-and-difference, but I also do feel that on the other hand, because this is what the regulators are pushing, we are forgetting that there are a lot of other different parameters of a treatment that we are missing. For example, what is the actual patient improvement And this is something that hit me a few years ago. that If you think about it, isn't it really, really, really, really weird that when we are approving medications, we are not considering the magnitude of the patient improvement. The FDA does not look at it.
00:07:21
Speaker
Now, call me crazy, but if you are developing medicine and the purpose is to help patients, then I think patient improvement is something that we should definitely consider. The issue is, just to give you a numeric example, is that let's say this between arm difference is three points. And again, like it can be on whatever scale.
00:07:41
Speaker
If the placebo arm did 12 points, then 12 plus three, the then the patient improvement is 15 points. Let's say the placebo arm, the improvement there was only five points, then five plus three, that's eight points.
00:07:55
Speaker
There's a big difference between those two scenarios in terms of the patient improvement of the treatment, but because the regulators are only looking narrowly at the between arm difference, they're completely missing that, like, you know, that v between arm difference is measured relative to what baseline.
00:08:10
Speaker
Effectively speaking, your control condition is your baseline and you're measuring the effect on top of that. But this baseline is shifting from study to study. And I think this is like one of the unsaid assumptions of placebo control studies that it's a relatively ah stable baseline.
00:08:26
Speaker
But it turns out that it is actually not. In particular, in the case of psychedelic studies, it seems to be much lower than with other kind of depression
Placebo Effect in Psychedelic vs. SSRI Trials
00:08:36
Speaker
interventions. And it leads to this illusion of a large efficacy.
00:08:40
Speaker
But a lot of the between-arm difference is not explained by patients doing better in the treatment arm, but it's rather explained by patients doing much worse in the control arms compared to other kind of treatments.
00:08:52
Speaker
That's your sort of missing placebo effect in psychedelics or... Yeah, the technically the correct term would be the placebo response. Placebo effect is a component of the placebo response, and placebo response is but the total effect what your patients are experiencing in the um placebo arm. But to give the listeners just some ah hard numbers, so if you're looking at something called the Hamilton depression scale, which is the most common way of measuring depression,
00:09:21
Speaker
then in your SSRR trials, the average placebo response is about eight points. In psychedelic studies, it is about four points. So it's basically we are missing half of the placebo response. And like that is that's quite a meaningful effect, like you know that four points, and bear mind all of the numbers that I'm going to give you are going to be average effects.
00:09:42
Speaker
But at four points on average, that's pretty substantial. The difference between placebos and traditional antidepressants is two points. And there is a lot of discussion in the literature whether the two points is meaningful or like you know does it have like any practical clinical relevance, yada, yada, yada.
00:10:00
Speaker
But here, what is missing is is twice as much as what we have, um ah you know the difference between placebos and traditional antidepressants. like I just want to highlight that this is not like you know some sort of like a small effect that like you know you really have to look for it to find it in these large sample of studies. No, this is a big chunk of what's going on.
00:10:20
Speaker
The reason why people wanna look at placebo, yeah so take the treatment effect or the treatment response, ah you know compare it to the placebo response is that the theory is that if you take ah the difference between those two, you subtract out the placebo response from the treatment response, then the resultant difference is the so-called true effect of the drug. That's sort of the traditional approach and the way that people have thought about this.
00:10:50
Speaker
Yes, that's the the specific effect of the treatment. like you know There are non-specific factors of a treatment, like you know doctors paying attention to you, being in a clinical study, so on and so on.
00:11:00
Speaker
And then what is on top of the placebo response? That's the specific treatment. So that, yeah, so the theory there being that, you know if you don't see a difference, then obviously you're not, there's no specific response to the drug. so there's no point in giving the drug because you could get it from just putting someone in a so scenario that mimics the other components of the study that don't involve the drug. So you could get away without, you know you don't need to use the drug. What's been found in the field is that This difference, the specific drug effect difference as calculated by this treatment response minus placebo response is considerably larger in psychedelics trials than it is in so-called traditional antidepressant.
Meta-Analysis Findings on Treatment Efficacy
00:11:45
Speaker
drug trials like SSRI or SNRIs. People so you know have been touting this, that that psychedelics are more effective for treating depression ah than are traditional antidepressants. This has been the kind of the claim in the field. and particularly looking at a scale like the Hamilton Depression Rating Scale. So do you want to I think that's a good lead-in to just talking about what you found in this particular analysis and and you know how that may read against what's been claimed in the literature so far.
00:12:19
Speaker
So one thing you know that is different from our study is that, again, because we looked at open-label traditional antidepressants, in a lot of those studies, there is no comparison arc. So 99% of meta-analysis papers that you can find out there, they are only going to compare this between-arm effect because that's what people care about.
00:12:39
Speaker
But here, because with traditional antidepressants, we looked at studies without a controlled condition, we needed to look at the patient improvement. The technical term is the within-arm effect. So we looked at the patient improvement of opal-labeled traditional antidepressants, and we looked at the patient improvement in a psychedelic studies.
00:12:57
Speaker
And in both cases, we find that on average, the the patient improvement is about somewhere between 11 and 12 points on the Hamilton depression scale, the aforementioned Hamilton depression scale.
00:13:11
Speaker
So that was a big, like, you know initially when we set up the study, like to be fair, like, you know, i wanted to show the opposite. Joe knows my history that I tend to, I'm Pretty bad scientists because every every hypothesis I have turns out to be incorrect. It was the same with microdosing that I wanted to show the opposite that my other big paper ended up showing.
00:13:32
Speaker
I wanted to show that even if you consider the subliminal effects of psychedelics, the effects are so much so big that even if you look at open-label antidepressants, psychedelics are still better.
00:13:45
Speaker
But what we have found is not quite the opposite because this is important. We did not find that psychedelics would be worse. We just find them that they are not better than traditional antidepressants. And there is a big difference there.
00:13:56
Speaker
Not being worse than the current best available treatment, that's not a dead blow of psychedelics. It just means that like you know maybe the miracle cure narrative has been a little overblown with that.
00:14:07
Speaker
um So that's the core finding. And this is like this was the the first hypothesis that we have. And this was a pre-registered meta-analysis. So like you know these hypotheses were registered and you can find them online before we started collecting the data.
00:14:23
Speaker
And then there were two minor hypotheses that we have also tested, and they are about the effect of blinding. And the first one that we did is that we compared, again, the within-arm effects, so the patient improvement, in open label psychedelic studies, there are such studies, and and the traditionally blinded psychedelic studies. So where you go through this blinding procedure, we know it's probably not working, but like you know there is still like a blinding procedure. and There, the our hypothesis was, is that there's not going to be a difference between the two simply because there is going to be, ah because the the studies are effectively always open label. And indeed, that is what we have found that in terms of patient improvement, blinded and um open label psychedelic trials yield the same patient improvement.
00:15:11
Speaker
And the third hypothesis that we have is basically the same question, but with traditional antidepressants. is there a difference in the patient improvement with blinded SSRIs versus open label SSRIs? And there we did find a difference, although it was a relatively minor, I would say it was surprisingly small. It was about one and a half point on the aforementioned Hamilton scale.
00:15:31
Speaker
I think you know if you are a kind of like a clinical data scientist geek like myself, like that's that's that's also something interesting that we make such a big deal out of blinding, but the effect of blinding is actually fairly minor, at least in the case of SSRIs.
00:15:47
Speaker
So I wonder if you can speculate at all about reasons for this difference. And I'm thinking, OK, so um clearly if i'm in if I'm in one of the two conditions, if I don't get a psychedelic, it's pretty obvious that I didn't get it, right? And I think you report something like 90% to 95% of people know which condition they're in.
00:16:13
Speaker
Whereas if I'm getting an SSRI, it's about a flip of the coin, and I really can't tell just from sort of subjective experience which condition I'm in. um So do you see this as being um minimal expectation effects once participants know that they're not in the psychedelic group, and also increased expectation effects for those who are in the psychedelic group? And I'm wondering if there might be a correlation between the strength of the experience that they have and the strength of what they believe um it's helping them with or doing to them.
00:16:53
Speaker
So just to go back a little bit to the SSRI, so it's not a flip of the coin at what rate they can correctly identify that. Oh, okay. It's a little bit better. It's about 59%. Okay. That's the largest meta-endritic on this paper. so The technical term is functional unblinding when patients realize what treatment when they are getting. And there is some functional unblinding with SSRIs as well, but the magnitude of the problem is much smaller compared to psychedelics. As you said, there is like 90 to 95% of patients can figure it out.
00:17:24
Speaker
So the issue is the issue exists also with SSRIs, but you know it's and um the magnitude is much smaller.
Impact of Expectations and Nocebo Hypothesis
00:17:33
Speaker
Now, when it comes to like you know these like expectation effects, and I'm going to ah you know go a little bit ahead of the data, but I would say that right now there is emerging evidence that, and you know for something that's very counterintuitive, that maybe these expectation effects matter a little in the treatment terms, that they do matter a lot in the control arms of these studies that we are running. And the best example of that yeah is um ketamine trials.
00:18:01
Speaker
ah Now, with ketamine, the same issue of functional and blinding exists. It's also a strong if psychoactive substance where it's easy to identify whether you received ketamine or placebo.
00:18:13
Speaker
But we have been researching ketamine for much longer, and it turns out that there is a drug called midalazam. it's ah Basically, it's ah another benzo. And if you are using that as an active placebo, then those studies are not functionally involved.
00:18:29
Speaker
So what's cool of with ketamine is that you have examples of studies that use an inactive placebo where there is functional blinding, similar to psychedelic studies. But if you use this drug called midalazone, that's going to be not perfectly blinded, but you're going to get a much better quality of blinding.
00:18:47
Speaker
So that's something that presents some sort of experience to the person, but different than actual ketamine. Correct. so like you know But it's close enough that patients cannot no longer reliably identify whether they have received midalazone, the active placebo, or the ketamine.
00:19:05
Speaker
So it definitely solves this functional unblinding issue. And what's interesting there is that if you're looking at the patient improvement between these two studies, like the patient improvement in the ketamine arm is is the same if you're looking at the unblinded studies or the blinded studies. And here by hereby the unblinded studies, I mean with the versus inactive placebo, the blinded studies are versus this rock called mid-alazone. There is no difference in how much patients are improving ketamine.
00:19:35
Speaker
Traditionally, you would think that in the unblinded studies, the patient improvement after ketamine is going to be much higher because you know that you got into the good drug group, like you know you're doing ketamine, it's all cool, yada, yada, yada. But it turns out that's not the case. The patient improvement is the same.
00:19:52
Speaker
However, if you're looking at the placebo-controlled condition based ones, then there is a massive difference. On the MADRA scale, which is another depression measure, if you are looking at the unblinded studies, so versus an inactive placebo, then the improvement in the placebo arms is merely one point on this MADRA scale, it is which is basically just like noise. There's basically no improvement.
00:20:15
Speaker
But if you're looking at the blinded studies versus middleism, then in the middleism arm, the control conditions improve about seven points on the MADRA scale. So it seems to be that like, you know, the presence or the lack of blinding does not make a difference to the active treatment, but they do make a lot of difference in the control condition.
00:20:37
Speaker
And again, to go back to like what we were talking about before, regulators are looking at the difference between two. So like, you know what does that mean is that if you're looking at unblinded studies, there's going to be a lower placebo response. So it means that this between-up difference is going to be measured against a much lower floor.
00:20:54
Speaker
This work that I told you about ah with ketamine, it was done by Samuel Dielkenson, who is at Yale. It's one of those papers, and this is, I think, you know the the highest compliment that one scientist can give to the other. I wish I would have done that paper because it is so cool. But anyway, I just wanted to highlight that that's not my work, but it was done by a group at Yale.
00:21:18
Speaker
So for that one in the in the placebo control condition there that you described where with this drug that is like ketamine but but different, do they think that there is that it has like real antidepressive ah effects or is the is the hypothesis that it's it's really just a placebo? It it does not have, middle-azom is not known to have an antidepressant effect. Of course it would have, then it would be a bad control condition. Like typically, you know, for the control condition you want, like, you know, no specific effect to the medical condition that you are improving.
00:21:55
Speaker
I am not, like, I have not looked into that literature, like, you know, in terms of like, has it been tested, like middle-azom has antidepressant effects or not, but the people who have, like, it's not that there's just like one middle-azom controlled ketamine studies. There are several of these.
00:22:10
Speaker
And I am sure that they have done that background work. I personally have not, but it is ah kind of like one of the standard control conditions in of Just to dig in a little bit more to this idea of, or this much smaller placebo effect in the psychedelics trials.
00:22:24
Speaker
So what do you think is going on there psychologically? Like why is there this this suppression of the placebo effect? so it's not just that they're not getting a really enhanced placebo, but there's very small to like negligible placebo effects in these psychedelics trials.
00:22:39
Speaker
I'm going to answer this question with ah a little story. So it happened last fall. I was watching NFL. Joe knows that I'm a big football junkie. And I asked one of my daughters to get me a beer from the fridge. And she did.
00:22:53
Speaker
But she accidentally brought me non-alcoholic beer. I like non-alcoholic beer. That's why I have it in my fridge. But at that instance, I wanted to drink something with alcohol. So like, you know, quickly opened it, like, you know, my eyes were on the game. And when I took a sip or two, that's when I realized that, hey, this is actually non-alcoholic.
00:23:12
Speaker
And like, you know, I had like this feeling of a little bit being upset. And again, it's not like I i do like non-alcoholic beer, but at that instance, my expectation was to have an alcoholic beer.
00:23:24
Speaker
I was disappointed, you know, that I did not get what I was expecting. And I got something, you know, which is regarded, you know, as a, kind like a lower ranking, non-alcoholic beer compared to alcoholic.
00:23:36
Speaker
And I think something similar is going on with these trials, that everybody who is signing up for one of these trials is signing up, hoping to be in the treatment. arm And this is not specific to psychedelics. Like in all medical trials, everybody wants to be in the active group, right? Nobody is signing up, hoping to be in the placebo condition.
00:23:55
Speaker
So I think you know when you will realize that you're in the active arm, it's business as usual. You're getting what you were expecting, like you know like why what is the fuss about? you know But on the other hand, if you don't get what you're expecting, that I think induces this disappointment effect. And actually, when we talk to some of our patients at the studies that we're running at UCSF, day they they often report these feelings. I'm not therapist, but when I talk to our study therapist, they're often saying that Like, you know, when a patient had a very low intensity session, they expressed that, oh you know, some some level of frustration, some level of disappointment, like, you know negative feelings.
00:24:33
Speaker
You know, again, signing up for these studies, it's it's a lot of hassle. You have to complete a lot of questionnaires. It's a lot of time. You have to go into the medical center X times over, you know, a certain period. It takes effort to participate.
00:24:46
Speaker
And then, like, you know, all you get in return was a placebo. I mean, I know I would be disappointed, of course, like, you know, that's just like human nature. um i't So this is what we are calling the nocebo hypothesis. So nocebo is kind of like the evil twin of the placebo effect. It's the negative effects of what happens in the placebo control conditions. And it's kind of have been playfully rebranded as nocebo as like, you know, that you were getting the you are in the bad drug group.
00:25:14
Speaker
And there is no hard data on
Efficacy and Side Effects of SSRIs vs. Psychedelics
00:25:16
Speaker
it. But I would say that, you know, in the field, there is this strong kind of like narrative convergence, that is probably what is happening in these trials. And in particular, keep in mind that in all of the psychedelic studies, there's a strong therapy component. So there's barely any placebo response, although you know they are getting a a therapy, which of course like you know is going to help the patients, even if they are not getting the drugs. So we have a minimal placebo response despite the strong therapy.
00:25:44
Speaker
Like for example, in the SSRI, in blinded SSRI trials, there is no therapy there. Yet the placebo response is basically twice as large as in the psychedelics studies is where patients do get therapy. What do you what does that where does this leave you with regards to your your feelings about how effective SSRIs are and how effective psychedelics are as treatments?
00:26:09
Speaker
So I would say, and I'm going to answer this question in a little roundabout way, but let's start that For every treatment, there seems to be efficacies falling as you go through the different phases of drug development. Drugs always like you know look really cool and promising in phase one.
00:26:27
Speaker
Phase two, studies are a bit more rigorous. There is more emphasis on efficacy. There is a drop. And in phase three, the falling just keeps continuing. This is very general. This is true for every drug that you are developing. And I think you know part of what we are seeing right now is the same with psychedelics, is that that you know if you are if you're looking, you know i think it's figure one in the paper where we are like listing the effect of their psychedelic studies, the largest effects are the ones that are the earliest.
00:26:54
Speaker
And then in the phase two and the phase three studies, the effects are smaller and smaller. And again, that's a general phenomena. However, I do feel that the magnitude of that fall is much larger compared with SSRIs. I don't have like hard data on this. Like I haven't looked at it quantitatively, just sort of like, you know, i mean, I spend a lot of my time, like, you know, looking at various trial results and that's, that's, um you know, just ah a strong hunch that I have, but I would not claim that to be proven.
00:27:21
Speaker
And in terms of like you know how I feel about it, like I would say that I was like i was definitely like negatively surprised by these results. Again, when I set up the study, like you know I wanted to show the opposite result that we are having. And it was you know it was a little bit of a reality check, it felt like. you know That like, hey, like can actually open-able SSRIs, psychedelics are not better or ever.
00:27:46
Speaker
But two things to keep in mind. One of them is that, of course, these are on average effects. So like you know that does not mean that like you know patients they are going to be patients who are going to benefit more from one or the other treatment option.
00:27:58
Speaker
The other thing to keep in mind is that there are many other differences between these treatments. like you know Here we are just looking at you know these core depression measures, and what they are measuring is technically what's called symptom reduction. All of these measures are basically a checklist of symptoms that you are rating according to their frequency or how disturbing they are to your life.
00:28:19
Speaker
But there are other kind of like out outcome there are other parameters of a treatment that matter. So for example, psychedelics, like you know you do the psychedelic once or twice. With SSRIs, that's a continuous treatment. That is a huge difference.
00:28:34
Speaker
Also, it seems to be that the side effect profile of psychedelics are more favorable compared to SSRIs. Although, and here like a little like side note is that SSRIs, we have given them to hundreds and thousands tens of patients in a clinical trial context and in a primary care context, millions of people have taken With psychedelics, both of those numbers are much, much, much, much smaller. So like, you know, this this this could be shifting. Like there's more and more people are trying psychedelic treatment. There could be like, you know, some other side effects that are emerging. So for example, you know, the sexual dysfunction that was not really, um
00:29:09
Speaker
noticed in the early SSRI studies. But hey, if you give them to millions of people, you know new things are going to emerge. And especially there's a big difference in a clinical trial context versus those like primary care.
00:29:21
Speaker
So for example, with SSRIs, most of the trials are eight to 12 weeks long, but everybody has probably many friends, unfortunately, who have been taking SSRIs for years. With psychedelics, we are testing them as like you know patients taking them once or twice in a relatively short period of time.
00:29:38
Speaker
But I am 100% sure that there are going to be people who are abusing it and who are going to do psychedelics, I don't know, on a biweekly basis or something like that. There are going to be probably some new side effects that are emerging and some negative outcomes as well.
00:29:52
Speaker
So I would say that like you know they we still need to you know, stuff like, get more, know, we have so much data with SSRI, there's going to be nothing new emerging there. But with psychedelics, we have just an order of magnitude less data.
00:30:05
Speaker
And therefore just like, you know, one or two can like highlight that, that, you know, like new things could emerge with psychedelics as well. But overall, I would say that I was Joe knows that I'm psychedelic advocate for sure. And you know I still, by and large, believe in the potential of psychedelic medicine.
00:30:24
Speaker
But I felt disappointed when I saw that these are going to be the results. I don't think this is a kill blow by any means to psychedelic, but it's a little bit of like, maybe it is a
00:30:37
Speaker
like you know this ah Maybe just like evidence that like some of those early claims have been, you know, the demure-cure narrative I think has definitely been overblown. I wonder if, um and this again is just some speculation, I wonder if part of what might be driving this might be some self-selection in the participants who are willing to be in a psychedelic trial. That by now, since SSRIs are so common, it's not ah much of a hurdle to volunteer for an SSRI trial.
00:31:07
Speaker
But maybe the type of person who volunteers for a psychedelics trial already has some positive thoughts about it or some high expectations about it. And I wonder how that might be playing into this, maybe some reduction in effects over time.
00:31:23
Speaker
That's 100% a part of it. Or like, you know, it's like not 100% in the sense that I have hard data to prove it, but that is a very, very, very reasonable assumption. And not just me, but I would say 99% of the field is operating that that is part of what is going on.
00:31:39
Speaker
So for example, the um about a month ago, the top line results for the two phase three studies of COMPASS came out, which is a psilocybin for treatment-resistant depression.
00:31:51
Speaker
And the patient improvement in both of those studies are much, much lower than in the previous studies. And those are large studies, like involving, I think, three and 500 patients that do trials. So like know these are like big studies. and part of it is why the efficacy is lower is certainly that like you know kind of like this like early self-selecting, enthusiastic, sorry for the lack of a better word, but like know did the hippies, they are just like, i don't know if hippies anymore in these trials.
00:32:19
Speaker
Yeah, we had Dr. Levine on from Compass Pathways a few episodes ago, and he discussed the the results of that trial. The headline there was that the results were disappointing, while staistic statistically significant, to disappointingly small effect sizes.
00:32:35
Speaker
Yeah, so like, you know, what Compass emphasized is that i think in one of the studies, I think it was the COM005, there was a 3.6 point difference against placebo. But the placebo response was merely 1.2 points. So 1.2 plus 3.6, that's about five points total patient improvement. That's that's really not much.
00:32:55
Speaker
you know The 3.6, that is kind of like similar to what you see in SSRIs, although slightly larger than what you see with SSRIs. But the baseline against with which you are measuring it against, it's so depressed that it's just a between-up difference like that does not...
00:33:10
Speaker
tell you the the whole story. And you know to go back to our meta-analytic study, which is like you know about all of these studies, like know basically, here is the math of like know why we got the results that we got.
00:33:23
Speaker
As I said, on the Hamilton depression scale, the patient improvement in SSRI is about 11 to 12 points. And I also said that the between-arm effect is two to three points. And now all of the numbers I'm going to give you are on the Hamilton depression scale.
00:33:37
Speaker
And the placebo response is about eight points. So like, you know, eight plus two to three points, you land, you know, somewhere land 11 points. With psychedelics, the bit between our effects is six to seven points, but the placebo response is four points. again, four plus seven, again, you are landing around 11 points.
00:33:56
Speaker
And the reason why, or like, you know, a big part of the reason why psychedelics have, um, you know generated so much interest because everybody was just looking at that between arm difference.
00:34:07
Speaker
Hey, with SSRIs, the between arm difference is three points. With psychedelics, it's seven points. Patients are improving so much better. Look at this. This is a very promising new treatment.
00:34:18
Speaker
But that's false. like The between arm difference is much larger, but the reason why the between arm difference is larger is not because patients are doing better in the treatment arm, because patients are doing much less well in the control arms of those A lot of the sort of like this kind of the between-arm difference of psychedelic studies is not explained by patient improvement in the treatment arm. It's ah it's explained by the lack of improvement in the control.
00:34:44
Speaker
I just mentioned a point about the COMPASS trial because this came up in our conversation with with
Community Response and Study Controversy
00:34:51
Speaker
Dr. Levine from COMPASS Pathways. They didn't really do therapy in that trial.
00:34:57
Speaker
And that was part of their sort of their theory of ah of action there was to try to make it as much ah as ah like a drug trial as possible. So it's really not a ah a therapy arm a therapy study, which is important because you know in the context of what you were discussing previously, Most of these trials do involve, it's like so-called psychedelic assisted therapy, where the therapy is sort of a centerpiece of the treatment and the drug is interacts with that with that therapy.
00:35:28
Speaker
And in the COMPASS pathways, they really tried to make it as much as possible, not a therapy trial, but really a drug trial. So that's just just important context. Yes, that's correct. And this probably also ah yeah partially explains why the effects were small.
00:35:43
Speaker
Right, exactly, exactly. and And it could also, you know, both on the placebo and on the on the drug side. Correct. would would We would hypothesize that that might reduce the size of the effect. there's actually like even a more extreme example of this in the GH001 trial, which is a formulation of 5-MeODMT.
00:36:02
Speaker
And they recently published a placebo-controlled trial in GEMF Psychiatry, in the same journal where our paper was published. and their patients in the placebo arm actually got worse, which is something that like you never, ever see. And they also operated in this sort of like minimal therapy framework. But I looked at the largest meta-analysis of SSR trials, and there, I did not find a single example out of 304 studies where patients in the placebo arm would get worse. But in psychedelic studies, it does happen.
00:36:37
Speaker
that's Yeah, that's dramatic. That's a dramatic difference. Yeah. So I'm curious to get your your thoughts on, there's been some controversy, to say the least, I would say, with regards to your study and the impact of of these results you know within the psychedelic community.
00:36:54
Speaker
And I would say there you know the um responses have been all across the map from, I think you're just like, I think you're wrong, Baj, like this is, your your methodology is flawed in some way, to actually this is, you know, this is true and it's it's an important eye-opener to like, actually this shows that psychedelics just don't work or aren at least not like shouldn't be approved. So like I'd just be curious to hear your response and thoughts on on sort of, well, first of all, just being part of a news cycle on this. And also like what your you know what your responses are to some of these both criticisms, but also you know the positive responses as well.
00:37:39
Speaker
Yeah, sure. So i think like you know this is just that if you publish anything high impact and you know something like, how to put it, like i don't I don't consider this study to be controversial.
00:37:50
Speaker
But I would, like, it definitely stirred up the pot. And, like, you know, even the criticism is not like, you know, that we did, like, you know, something like technically wrong or anything about, like, you know, that.
00:38:01
Speaker
But I do think that, and this is very familiar with me with respect to my microdosing studies. I think you know a lot of what happened both with the microdosing study and you know like a super brief recap, like you know in 2021, we published the largest placebo-controlled psychedelic microdosing, and we said that it's it's not better than placebo.
00:38:21
Speaker
And then a lot of the community turned against us and because they interpret interpreted it as negative. But not being better than placebo, that just means that you are not better than an already a pretty large effect, the placebo effect. So like you know I did not view the results of the study as being negative.
00:38:37
Speaker
And I think you know something very similar happens here is that we are saying that the patient improvement after psychedelics is not better than after open-label antidepressants. To me, that doesn't sound like you know like something like a deadblower, like, oh my God, psychedelics don't have any effect or something like that.
00:38:54
Speaker
But I think a lot of the psychedelic advocates are kind of over-interpreting that, or maybe not over-interpretation, but they are perceiving it as you know something that's kind of like negative. And That's just the way the world works today is that like you know it's ah it's a 4,000 verse long article like you know full of like technical terms. There's a lot of nuance in the paper.
00:39:17
Speaker
But like you know most people are just going to focus on that like you know one or two sentences at the end, you know which is about, like hey, we did not find a difference. I think a lot of the nuance gets lost there. and you know I expected this sort of reaction you know in the um from this study. I knew that like a lot of people with psychedelic period are not going to like it. And a lot of psychiatrists who feel much more negatively about psychedelics are going to celebrate it. it's It's a weird situation for me because I want to be a cool psychedelic researcher who shows that like all of these concerns are not valid. But again, I just keep getting the the wrong result. and
00:39:54
Speaker
kind of like instead of like hanging out with the cool psychedelic bros, I hang out with like the old school psychiatrics who are suspicious about psychedelics. But hey, it's is the nature of research. ah Integrity is a bitch.
00:40:07
Speaker
Well, yeah, I can confirm ah for those who are listening that Balas is not anti psychedelics. Thank you. As much as the ah the results of your studies, um you know, but that's actually to me, yeah, it does is is an indicator of your integrity because like As you said, these are results that were counter to your original hypotheses and what you're not what expecting to find. And so, you know, kudos to you for for publishing those and and sticking to to the results and what you you know, what the numbers told you.
00:40:38
Speaker
Yeah. For the folks who are advocating for psychedelics, one of the things that they would point out is that like, all right, you're comparing apples to oranges, you know, where in the one hand, you've got a set of studies that are controlled trials. So those are the psychedelics assisted therapy trials that do have a control group. And then you're comparing it to an open label, traditional anti like antidepressant study that that is does not have a control group. They're just different setups.
00:41:07
Speaker
Is it fair to compare the effect sizes across those two different setups? What is your response to that? I think, you know, it is it is still better than to compare a controlled SSRI study versus a controlled psychedelics. Like, it's not perfect. Like, you know, I'll i'll acknowledge that.
00:41:23
Speaker
But given like you know that we think like you know about the importance potential of blinding and yada, yada, yada, I would say that this is still better than compare against the controlled SSRI studies where blinding, by and large, works versus the controlled psychedelic studies where blinding does not work at all.
00:41:40
Speaker
So like you know I do hear this criticism, and like I'm not against it, but like you know this is definitely, how to put it, like um I would say it is less biased than the alternative comparisons against with the controlled studies than the controlled studies. You know, of course, this is not equivalent to like a an RCT.
00:41:59
Speaker
That being said, like, you know, in the acetylopram versus psychedelic RCT, which is really the only head-to-head comparison of these two ah drugs, you know as you know, Joe, then the the primary outcome psychedelics did not beat acetylopram.
00:42:13
Speaker
On some of the secondary outcomes, of psychedelics ah or psilocybin in that case was better than acetylocrine. But I just want to highlight that our results are kind like, it's not like that they're out of line, like, you know, with this head-to-head comparison trial. Our results are perfectly compatible with with that one head-to-head study that is in the literature.
00:42:32
Speaker
I think that also highlights both points, which is that on this sort of headline self-rating scale, there are no differences in the overall effect size, but there are different effect profiles, both to your point about side effects, but also actual positive effects as well.
Functional Improvements and Data Challenges
00:42:50
Speaker
um You know, there are different effects that SSRIs and psychedelics have. So depending yeah on, yeah it's not it's not like there's, it's it's not a univariate assessment. Correct, yes. And there you know there's this point that we also raise the paper is that beyond the symptom reduction against with this checklist questioners measure, there's also something called like you know kind of functional improvements. Like, you know hey, how is your life? like you know it like You may have the same amount of symptoms, but like you know you're just like having a better quality of life.
00:43:23
Speaker
And I initially really wanted to include that kind of measures in this analysis, but unfortunately those are just like rarely measured. So like, you know, that's, it's kind of like, um you know, just the the data is just not there, unfortunately. So like, you know, we, we, I stand by the results that like know, it comes like symptom reductions, like, know, these are the results, but it's not, that does not mean, you know, that it's, they're kind of like,
00:43:47
Speaker
equal in every respect of it. and we And we made sure to highlight that in the paper. But just back to the point of the apples and oranges comparison question, the obvious comparison would be to compare open-label psychedelics to open-label traditional antidepressants. Why was that not the approach?
00:44:04
Speaker
Oh, that was done. Like that's one of the, it's in the supplementary materials that just kind of like one of the robustness analysis. And the is there, you can find the numbers. Qualitatively is the same as with the big analysis. Again, blinding doesn't really make a difference for psychedelics.
00:44:18
Speaker
So you're just kind of like looking at the subsection of the psychedelic studies, but we've we've done that and the results are the same. Okay. So and the reason why you didn't lead with that was just because it's like less data or why would why not lead with that?
Solutions for Placebo Issues in Trials
00:44:33
Speaker
Because the whole point is is that psychedelics are practically always open-label. So I don't see a difference between a blinded and a traditionally blinded psychedelic study. And and we confirmed that when we compared the reading arm efficacy of blinded versus, I mean, formally blinded versus open-labeled psychedelics.
00:44:51
Speaker
Yeah, I mean, all right, now I'm just gonna dig in a little bit on this Bosch, because have a couple little knits to pick with the paper, and this is we're gonna get into that territory. I hope you're okay with that. Go it. Oh, yeah, absolutely.
00:45:05
Speaker
Great. Roast me. that No, it's not a roast. It's not a roast, but it is well I would say they're knits. So with regards to this one, you got into the blinded versus open-label psychedelic-assisted therapy, and you compare them.
00:45:20
Speaker
when you did the so-called frequentist analysis, you saw no difference. And that that's really not, there's not and even a trend. Like, you know, the p-value is like 0.93. So it's like very really no difference between um the effect in the treatment arm of a blinded psychedelic assisted therapy study and an open labeled psychedelic assisted therapy however when you did when you um compared it using the bayesian approach and now this is where you're gonna have to help me out because i don't really know enough about the analysis that you did there to know why those things are different but in the bayesian approach you saw a a beta of 2.14 which is
00:46:03
Speaker
essentially saying that, you know, in the HAMD units, this Hamilton depression rating scale units, equivalent to a little bit more than two points difference. And you you concluded that these results provide evidence for hypothesis three, which is that there is a difference between open label and and controlled trials for psychedelics. So I guess why lead with the frequentist result and not the Bayesian result? And then why are they different?
00:46:32
Speaker
So the something there's something called the random effect term in these models. And with the Bayesian models, you could just back more into the random effects because they are more flexible.
00:46:42
Speaker
ah So that's why there is some difference between the two. The random effects setup is different between the frequentist and the Bayesian. And to be honest, it's just... The reason why we have Bose analysis in the paper is just because and i'm I'm trying to get into like more Bayesian analysis. And it was just almost kind of like an exercise for myself that like, you know, like, hey, and I want to do Bose. And then just like, you know, once I had the numbers for Bose kind analysis, why not disclose like, you know, Bose in the paper? To be honest, that's that's that's really it. It's a little self-serving because it was a kind of like a practice run for myself.
00:47:16
Speaker
um We kind of like led with the frequentist in most of the discussion because much of the literature is based on frequentist statistics. that's That's really it. like you know It's not like you know some sort of deep reasoning. It's just more familiar to most of to more of the readers.
00:47:30
Speaker
Yeah. yeah yeah i but it was just i thought It was interesting because you know with the traditional antidepressants, you saw the opposite pattern where The Bayesian approach so showed no difference, but the frequentist approach showed a statistically significant difference. So I would say that like the Bayesian also showed a difference there, but like with the frequentist, and this is why they are popular is because there is like this, like in terms of the p-value, there is like this like- A cutoff. Traditional and arbitrary cutoff for when you have an effect, and when don't. With the Bayesian approach, there is no equivalent to that.
00:48:02
Speaker
So like, you know that's kind of like just makes it easier to frame the paper a little bit. So that's- that's um Also, one of the reasons why I think like you know most people like frequent is because it gives you like, it's arbitrary, but it gives you a yes, no kind of like answer. With Bayesian, it's always a bit more like shades of gray.
00:48:20
Speaker
Yeah, understood. All right. Well, that that that makes sense. um Yeah, okay, that that all makes sense to me. um I guess ah going forward, what do you think, what are the, like, so we've got a problem now, like, you know, it's, you don't have ah a really great way to do a placebo control trial in psychedelics. ah what' Like, what's the solution? Like, how do we make a decision about, um you know, whether these drugs are effective?
00:48:55
Speaker
So I would say like know right now, you know, there is kind of like a new generation of psychedelic trials are coming up that are trying different tricks, are basically a combination of various active placebos and something that's called incomplete disclosure, which means that, you know, when patients sign up for these studies, like we don't tell them exactly what are the possible options that they are going to get and Like, you know, different people, different trialists are thinking of different approaches. And I think, you know, this is kind of like we have used active placebos in psychedelic studies, but to be honest, pretty stupid ones like niacin, which is going to give you little buzz, but like nobody is going to compare, ah confuse that with psilocybin.
00:49:40
Speaker
But like, you know, what happens if you give them a salveum or a really high dose of THC or something like that, or or maybe a combination of both, like ah they can give you an intense experience Like maybe you know, maybe we can blind these studies. Like i I don't think that the field has completely given up on the idea of blinding psychedelic studies. Like so far we have really not succeeded, but to be honest, we have only tried the most obvious kind of like active control conditions at this point. So that's just like a little disclaimer that, you know, I would not say that the field has given up on blinding the psychedelic studies.
Future Research Directions
00:50:19
Speaker
but In terms of like you know the regulations and the approval and so on and so on, I think you know the it's an interesting situation because you know psychiatry as a field has struggled for a long time to distinguish itself from psychology. And you know kind of like you're in the field, like you know psychology is like this look down, woo-woo version of psychiatry, which is like proper science.
00:50:44
Speaker
and The difference between the two, and of course, I'm simplifying, is that in psychiatry, you run placebo-controlled trials. In psychology, you are just talking. The point is is that psychiatry, kind of like the placebo-controlled trials, that's kind of like its image of why is it not woo-woo, because we are running objective research. We are running placebo-controlled studies.
00:51:07
Speaker
But And, you know, as I said, like with SSRIs, there's also functional unblinding, but the entire field, including scientists and regulators, they just put their head into the sand and kind of forgot about this issue because nobody knew how to deal with it.
00:51:22
Speaker
Now with psychedelics, like, you know, the issue is so big that that strategy no longer works. But then there is an element of, like, mea culpa here. Like, you know, did we do these trials wrong? Do we need to go back and now demand like actually blinding SSR trials?
00:51:35
Speaker
like It's kind of like this like weird position because if you change the policy, then you are admitting that what you did previously is wrong. So i kind of feel like you know what is going to happen is that regulators are going to ask for some sort of like active comparator groups and then like can you just basically say that, hey, these trials are doing the best they can and pretend that they are placebo.
00:52:00
Speaker
But what I don't see as a future when the regulators are saying is that, hey, actually what we did that research in the past 50 years is all wrong because probably there's functional binding in all of these studies. Let's get all of the drugs off the street and like let's like you know play the whole reapproval game all over again. But the regulators and the field need to do that without losing face.
00:52:23
Speaker
So i think that's kind of like the tricky for the field to navigate is that, oh yeah, like know there's functional binding everywhere. But in like, you know, with psychedelics, we are demanding that.
00:52:33
Speaker
Anyway, like, you know i think there's some tension here that I hope you can feel that like, you know, how we are navigating it. But I think, you know, what's going to be the compromise is that we're going to ask some sort of like active competitor, do the best that you can. And then we are pretending that it's functional and blinded and then just approve as usual.
00:52:51
Speaker
By usual, I mean like using the usual criteria.
00:52:55
Speaker
Well, it does it does seem like there is you know a different standard in some ways being applied with psychedelics over time and how they've been treated by the psychiatric community.
00:53:08
Speaker
I would say yes and no. like you know At face value, it's true. But again, functional and blinding, it happens with other drugs as well. But the magnitude of the problem is less.
00:53:18
Speaker
So that's why I think I do agree with you that like kind of like ah it's kind of like a shifting goalpost in a sense that like now, you know, we' psychedched this is an issue. But the problem is really kind of like the magnitude is is much larger in psychedelics.
00:53:32
Speaker
Yeah. Makes sense. All right. Well, last question that I would like to ask, what are you excited about in terms of research going forward? What's coming up next for you? Whether it's your own work or something else in the field, are really excited about seeing coming up?
00:53:49
Speaker
So I really want to go after this, like does blinding matter in the treatment arms and does it matter in the control arms? And we have started a large-scale investigation of that. So I gave you the the ketamine study. And again, there's some kind of like is this placebo suppression in the psychedelic studies and you know, we saw that the blinding makes a little difference in SSRIs, but you know, there's a kind of like bits and pieces here and there, but I want to make it more ah systematic.
00:54:17
Speaker
And like, kind like that's that's what definitely gets me exciting, you know, as regards to my own work. And like, you if you like step, you know, make make one step backwards,
00:54:28
Speaker
and like look at the field, then what I'm excited about mostly is that just like looking at the real world data that's coming in from Oregon and from Colorado and from other places, like, you know, at the end of the day, there is a very large difference in the clinical trial context versus real world. Like, you know, I'd say that the example with SSRIs, like how they have been approved, it's very different how they are being used.
00:54:50
Speaker
And I think, you know, something similar is going to also play out with psychedelics and especially like with psychedelics because of the therapy and because of its context dependency, like, you know, maybe the the gap between real world and clinical research is going to be even larger. so I think it's going to be like really, really exciting to see, you know, real world data coming in and see how that relates to what we have been cooking up in the lab.
00:55:15
Speaker
Great. Well, Bosh, thank you so much for being on the show again. Always great to talk to you and yeah really excited to see what comes