Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Large Language Model Chatbot Simulation for Suicide Risk Assessment Training in Postgraduate Psychology
50 Plays3 months ago
The study being discussed sits within the Psychology Mechanisms and Innovations program at the Healthy Brain and Mind Research Centre - ACU, and was led by Dr Elizabeth Pizarro-Campagna in collaboration with Dr Simon D’Alfonso (UniMelb), Prof John Gleeson (ACU), Dr Thomas McGuckian (ACU), Hannah Giddy (ACU Psych Hon student), and Cassandra Mount (MPsych Clin student).
Apologies , Kerry had some audio issues on this episode.
Recommended
Transcript
Intro
Introduction and Topic Overview
00:00:41
Jane Frost
Hi everyone, name is Jane.
00:00:44
Jenny
Hi, I'm Jenny.
00:00:46
Jane Frost
And today we're going to be talking with Liz Pizarro-Campagna and Simon D'Alfonso. And the topic of today a research
Meet the Guests: Liz and Simon
00:01:15
Jane Frost
Would you like to tell us a little bit about yourself? I'm going to start with Liz, is that okay?
00:01:18
Jenny
Thank you.
00:01:21
Liz Pizarro
Oh, hi Jan, thanks for having us. So yeah, I'm Liz and I am an academic at Australian Catholic University based in Melbourne, i'm in Fitzroy.
00:01:33
Liz Pizarro
And I teach postgraduate psychology students here. I coordinate one of the postgrad psych programs, the Master of Professional Psychology.
00:01:44
Liz Pizarro
And as part of my broader academic role, I also do
00:01:46
Jenny
okay
00:01:48
Liz Pizarro
to research and some of the research that I've been doing lately has been collaborating in collaboration with Simon and we've been looking at tools, simulation tools, particularly the use of AI in training psychologists.
00:02:07
Jane Frost
Great. Thank you, Simon.
00:02:08
Simon
And I'm Simon. I'm a senior lecturer in digital health from the School of Computing and Information Systems at the University of Melbourne. So I'm a computing guy and my research explores the application of AI to mental health and psychology more generally.
00:02:25
Jane Frost
Great. Thank you. So I'm fascinated to talk to you. But before we go any further, want to welcome our co host Kerry, who has just joined. Joined as hi, Kerry.
00:02:35
Kerry Anne ReidSearl
Hi, Jane, and
AI Chatbots in Psychology Training
00:02:36
Kerry Anne ReidSearl
welcome to Women and Liz as well.
00:02:36
Jenny
and Tara, we are
00:02:40
Jane Frost
OK, so my first question is what prompted you to explore AI chatbots as a simulation modality for suicide risk assessment?
00:02:51
Liz Pizarro
Well, in training of postgraduate psychology students, one of the limitations that we have is that we don't really have very many opportunities for practice of the skills that students are learning.
00:03:11
Liz Pizarro
Main reason for that is because it's really expensive to pay actors to play the role of clients. essentially we rely
00:03:22
Liz Pizarro
on role plays with their peers, which, you know, as educators, we know that it's, you know, peers don't really simulate clients very well.
00:03:24
Jane Frost
Thank you.
00:03:36
Liz Pizarro
they They giggle, they don't stay in role. So it's not very true to life, genuine, realistic practice for students.
00:03:45
Jenny
Yeah.
00:03:49
Liz Pizarro
So I guess we've been trying to think about watch what can we do to
00:04:19
Liz Pizarro
and I guess with advancements in technology, we have some new tools available, to us, and LLM chat pots is, is one of those potential, you know, tools that has a lot of potential, for being, affordable and scalable,
00:04:22
Jenny
and continue to work with humans, and of course, the world's been really doing to And we are going to talk about what we're doing to us because we're going to do this. And we're going to talk about what we're doing to do.
00:04:34
Jenny
And we're going to talk about what we're doing to do. And we're going to talk about what we're doing
Advancements in AI Training Tools
00:04:41
Jenny
to do. And we're going to talk about what we're doing to
00:04:41
Liz Pizarro
realistic, which is, I guess, really important or more realistic than, than role playing with your peers.
00:04:49
Jane Frost
Do you mind if I just go back a step and say, so we're talking about AI chatbots. What do you mean when we're talking about an AI chatbot? For our listeners, what are we talking about here?
00:04:57
Jenny
So I'm just going to ask you a question.
00:05:02
Liz Pizarro
Thank you.
00:05:03
Jenny
I'm just going to ask you a question.
00:05:03
Simon
So a chatbot, we've heard of ChatGPT these days, or I think most people have.
00:05:06
Jenny
I'm just going to ask you question. I'm just going to ask you a question.
00:05:10
Simon
was around 22, 2023, I think that but they couldn't large language models started to become more prominent via systems such as OpenAI's ChatGPT and Google's Gemini platforms.
00:05:11
Jenny
I'm just going to you a question. just going to ask you a question. I'm just going to ask you a question. I'm just going to ask you a question. I'm just going to ask a question.
00:05:25
Simon
And these frameworks and these systems make it easier these days to just craft your own prompt, send that prompt to the system and tell it to act in a certain way and have a conversational engagement.
00:05:42
Simon
So even 10 years ago, there was considerably more work involved in trying to get one of these chatbots off the ground to create your own chatbot.
00:05:52
Simon
It's become much easier over the last few years with these large language model systems that are readily available.
00:05:58
Jenny
Thank you.
00:05:59
Simon
So if you've used ChatGPT, that's what a chatbot is essentially.
00:06:04
Jane Frost
Okay, I'm going to follow that up with that. So is this a chatbot? So it's purely text? Or are they verbally responding?
00:06:12
Simon
Good question. So we use...
Client 101: A New Training Platform
00:06:15
Simon
these LLMs for our own platform, which we call Client 101. And Client 101 is a web platform that houses various chatbots that simulate mental health clients.
00:06:26
Simon
The first modality is textual interaction. We are currently adding voice interaction to that. And perhaps a future possibility might be some visual mode or an avatar.
00:06:37
Jenny
Thank you.
00:06:40
Jane Frost
Okay, great. Thank you. So did you want to tell us a bit about your research?
00:06:47
Liz Pizarro
Oh, so I guess to extend on that sort of broader challenge of being trying to think of of ways to provide better scaffold in more realistic simulation, practice for post-grad psychology students.
00:07:02
Liz Pizarro
One of the areas that, is really important and pressing and in need of further practice is, where there's risk to clients.
00:07:07
Jenny
Thank you.
00:07:13
Liz Pizarro
So, looking at using chatbots to allow students to practice suicide risk assessment skills we felt was really important and the literature shows that graduates and early career psychologists don't feel competent, confident in undertaking suicide risk assessments and in fact often will avoid it
00:07:36
Jenny
Yes.
00:07:41
Liz Pizarro
which obviously puts clients at risk.
00:07:59
Liz Pizarro
would simulate a client presenting with depressive symptoms, but particularly who would present with a a medium suicide risk that would allow the students to practice a suicide risk assessment.
00:08:15
Liz Pizarro
So that's kind of the beginning of the project is the idea of developing the chatbot for students to practice suicide risk assessment.
00:08:25
Jenny
All right.
00:08:27
Liz Pizarro
So from that, suppose we ran a pilot last year. And we were wanting to ask, I guess, two main questions that we explored quantitatively and also qualitatively.
Pilot Study and Results
00:08:43
Liz Pizarro
And this was collaboration. There was contribution from a number of people, including two of my students. So I had an honours student and also Hannah Giddy was my honours student and I had Cassandra Amount who was a master's student who was completing a clinical master's degree.
00:09:01
Liz Pizarro
And so they looked at qualitative and quantitative aspects of the research question, which was, is it useful?
00:09:07
Jane Frost
Great.
00:09:11
Liz Pizarro
Do students, do post-prite psychology students find it useful, beneficial to practice suicide risk assessments with a chat bot? And does it improve their self-efficacy in undertaking suicide risk assessments? So that's that was our research question.
00:09:37
Jane Frost
And what did you find?
00:09:39
Liz Pizarro
What did we find? So we found that, yes, they did really enjoy interacting with the chatbot and practicing suicidal risk assessment skills.
00:09:52
Liz Pizarro
They found that it, we found that it improved their self-efficacy. So we did, we got them to practice four times.
00:10:04
Liz Pizarro
So it was very structured and time limited studies. So we invited them to participate. immediately after they had completed training in suicide risk assessment.
00:10:16
Liz Pizarro
So they completed a unit in the postgraduate program that teaches them how to do suicide risk assessment. And then following that, they were invited to participate in the chatbot.
00:10:29
Liz Pizarro
So we wanted to make sure that they'd had the training. This wasn't a training tool, but rather a practice tool. And so we... invited them to practice four times over a period of four weeks, so once a week over four weeks and this was scheduled and it was done online remotely.
00:10:51
Liz Pizarro
They didn't have to come in so it was very practical and accessible for them and we measured So they completed very brief online questionnaires at baseline, so before any practice and then after each of the four practices as well.
00:11:11
Liz Pizarro
And what we measured on each occasion was their self-efficacy. So was very brief self-efficacy and risk assessment tool. And at the very end, we also asked them some questions about acceptability.
00:11:26
Liz Pizarro
So how useful, beneficial and acceptable they found the tool. And yeah, we found that they enjoyed interacting with it.
00:11:37
Liz Pizarro
They found it beneficial, they found it useful and it was quite consistent across the quantitative and qualitative findings. We did find that practice, the number of practices mattered. So the benefit was in the quantitative data, we saw benefit at week three and week four compared to the baseline level of self-efficacy.
00:12:08
Liz Pizarro
So the first practice, we didn't find a significant difference. So it took a few practices, which I guess is consistent with learning theory that says that you know practice
Limitations and Future Improvements
00:12:16
Liz Pizarro
is really important for increasing skill and self-efficacy.
00:12:22
Liz Pizarro
um'm We did find though that there were some limitations to that. So you know it wasn't perfect and the students did. So some of the in the qualitative data, some of what came up in that was really interesting and helpful in terms of you know future development of these types of tools that
00:12:41
Jenny
yes
00:12:43
Liz Pizarro
one of the challenges was obviously that, what we used was only text-based and so the students weren't able to see a client. And when you, you know, working with clients, you really rely on those, nonverbal cues.
00:12:59
Liz Pizarro
You rely on, body language, you rely on, uh, the tone, of the voice and other, Such cues and so obviously you don't have that when you're interacting with text So that was a limitation and the other one was repetition so Because we it was a pilot we only had one client we had Kyle So there was between session repetition because they practiced on each occasion with Kyle so obviously practicing with some clients repetitious but I guess
00:13:14
Jenny
and the other one. What's your reputation to be able to create? That was a good idea. Yeah. And I was going say, well, reputation when you come to the education community, and you can see the benefit to the education community.
00:13:27
Jenny
And I'm going to say, it's going to be a good reputation for the community. And I'm going to say, well, it's going to be good reputation for the community. I'm going to say, well, it's going to be reputation for the community.
00:13:36
Liz Pizarro
more interesting was that within session repetition where sometimes the student would try to ask a question in a different way to try and undertake a more thorough assessment, but the chatbot would respond with almost an identical response, which is one of the limitations and Simon could probably talk a little bit more to that and what work we're doing and thinking of.
00:13:42
Jenny
to the next few years to the next few years.
00:14:04
Simon
Sure. So client evolution is an active research and development problem, trying to get the chatbots to organically or inorganically develop, especially if a student will be interacting with the same chatbot on more than one occasion.
00:14:24
Simon
In this case, it was this caveat that we actually created four instances of Kyle, Kyle 1, 2, 3, and 4, and we deliberately made each instance a fresh slate when the students started with that instance for the week.
00:14:34
Jenny
Great. So what's the light?
00:14:37
Simon
So there was bound to be repetition there just because of the structure of the study.
00:14:38
Jenny
So what's the light? So what's the light?
00:14:42
Simon
But at any rate, yeah, this idea of trying to have a chatbot somewhat evolve as a student comes back to it the second week, the third week.
00:14:44
Jenny
Just the light? So what's the light? So what's the light? So here is the light? So what's the light?
00:14:50
Simon
So there is some type of something developing, I suppose.
00:14:52
Jenny
So what's the light? So what's the light?
00:14:54
Simon
And we have some basic basic mechanisms at play.
00:14:55
Jenny
So what's the light? So what's the light?
00:14:57
Simon
We now have a memory system.
00:14:57
Jenny
So what's the light? So what's the light?
00:14:59
Simon
functionality whereby say a summary of previous sessions will be appended to the current prompt that we send to the large language model.
00:15:00
Jenny
So what's the light? So what's the light?
00:15:08
Jenny
This is a very interesting question.
00:15:09
Simon
And you will just have an instruction like in formulating your, you know, next response, you can consider all these bits of previous information that have occurred in the sessions.
00:15:21
Simon
And then you could even do possible things like, you know, if there's something out in the real world that's happened, you can drag that bit of news into the context and tell the chatbot about that. So like, just pretend like that the chatbot is this entity in the real world that is learning some new things throughout the week prior to having the next session with the with the therapist.
00:15:42
Jane Frost
Can you talk me through a session, as in they've got four sessions, so the student would come in and use or be allowed to use the chat bot, just the words and have a conversation?
00:15:47
Jenny
OK. OK.
00:15:58
Jane Frost
Is that what you're saying?
00:16:01
Liz Pizarro
The students had access to the chat bot for the four week period and they could access it at any time, but to prevent that we created a schedule.
00:16:09
Jane Frost
Yep.
00:16:16
Liz Pizarro
So the students who were working on the project collaborated with the participant to create a schedule of practice so that each participant was practicing only once per week over that four week period.
00:16:32
Liz Pizarro
And when we look at the data, we can confirm that that is what happened, that participants were practicing once a week over the four week period.
00:16:43
Simon
Although
00:16:43
Jane Frost
So they just had one conversation with the chatbot, is that what they're doing?
00:16:49
Simon
one per week over four weeks, though some students like sometimes doubled up.
00:16:51
Jane Frost
Yeah. Yeah, OK.
00:16:55
Simon
like They were only supposed to interact with the first instance of COAL once in the first week, but they might have done it twice when they weren't supposed to, but that's okay.
00:17:02
Jane Frost
OK.
00:17:03
Liz Pizarro
Yeah.
00:17:04
Jane Frost
Yeah.
00:17:05
Liz Pizarro
Yeah. Hmm.
00:17:07
Jenny
I've got a question please, Jane. I'm really curious.
00:17:14
Jenny
Suicide risk assessment, what safety did you put in around, what were your safety considerations in building these scenarios and how did you manage student stress?
00:17:31
Liz Pizarro
If I could flip that question around actually, because one of the things that we're trying to do here is in some ways expose the students to the stress that they will encounter when they're undertaking suicide risk assessment.
00:17:44
Liz Pizarro
So in training psychologists, one of, you know, we're preparing them to work with clients and people in distress, people who are unpredictable.
00:17:46
Jenny
Thank you.
00:17:56
Liz Pizarro
there's, they don't come with a script and you never know what a client is going share with and one of those things that clients share with you is suicidal risk.
00:18:08
Liz Pizarro
and they, you know, don't necessarily come out with it immediately. It's something that you have to kind of be attuned to and know what you're listening for, looking out for, knowing how to ask the questions in in a way, in a sensitive way, but that is also,
00:18:24
Liz Pizarro
able to elicit that information from a client so that you can support them so guess we're not trying to protect the students from distress we are wanting them to experience as realistic as we can what it is to come in contact with a distressed client who might be presenting with suicidal risk and And, you know, I guess if we're sort of thinking of guardrails, because this was a pilot, we involved, we invited students who had already received suicide risk assessment training. And these are provisionally registered psychologists in their fifth year of training.
00:19:08
Liz Pizarro
So that's really the main guardrail, that they've got a fair bit of training under their belt already. They're provisionally registered psychologists and and had had suicide risk assessment training, but the actual practice with the chatbot itself, we weren't trying to protect them from, from distress, I guess.
00:19:28
Jenny
Thank
Supporting Students and Educators
00:19:30
Jenny
you.
00:19:30
Simon
Although you did provide provide some reminders and links to mental health service resources if they just wanted to debrief afterwards, or were concerned.
00:19:34
Liz Pizarro
Yes.
00:19:38
Liz Pizarro
Yes. Yeah, we did. So we provided the Lifeline number and also within the university we offer free, there's free counselling that is separate from psych training discipline and we, know, reminded students that they have access to that.
00:19:58
Jenny
Lovely, thank you.
00:20:02
Jane Frost
just going ask Kerry if you've got any questions.
00:20:03
Kerry Anne ReidSearl
Yeah, look, sorry, there's a bit of echo here, but I'm just wondering if you had a takeaway for educators curiously experimenting with AI, what would you
00:20:18
Liz Pizarro
I didn't quite catch that, sorry.
00:20:22
Jenny
I think what Kerry was asking Liz was if you had a takeaway message for educators.
00:20:22
Jane Frost
So I think
00:20:28
Jenny
Is that it Kerry? Yeah. What would be your takeaway message for educators in working with chat bots and AI?
00:20:38
Simon
So I think the first important point to make now is that the possibilities and opportunities are ripe now.
00:20:39
Liz Pizarro
Oh, yeah, Simon's done heaps, he'll work here.
00:20:45
Simon
The entry barriers are pretty low to just trying to find you do a Google search if you're interested in this and you will probably find some option to just set up something relatively easy and just start experimenting. You can craft your own prompts that can influence what you want to generate and just see how you go.
00:21:04
Jenny
So do I have to have any knowledge, Sam? I've got no experience.
00:21:08
Simon
Yeah, so to have a really, really basic system, you don't need to have any knowledge. You can try out your own, go into ChatGPT and prompt it to behave in a certain way.
00:21:18
Simon
So write a prompt, go, you are this type of client. These are some of your characteristics. Now respond to me as i as if I am a therapist and we are having a conversation and it would go back and forth as a starting point.
00:21:26
Jenny
Thank you.
00:21:31
Simon
But then to use these technologies and develop your own platform, obviously you need to have some web development experience and know how to call what are called application programming interfaces and all this stuff.
00:21:42
Simon
But there individuals like myself who know how to do the computing things and you know we're open to collaborations and such and there are others like me out there probably so just find some computing people I suppose who'd be interested in working together if you're say from the mental health disciplines or the allied health services and try it out.
00:21:53
Jenny
so
00:22:04
Liz Pizarro
I think if I could add to that a little bit, I wouldn't have been able to develop this on my own. working with Simon, really helped guide how you develop a prompt for a client.
00:22:21
Jenny
So
00:22:22
Liz Pizarro
trialed it, obviously we tested it out before, getting the student, you know, before running the pilot, we tested it with, with staff, you know, we played around with it, we challenged it, we did all sorts of things to to make sure that it was acting like a client.
00:22:41
Jane Frost
Thank
00:22:44
Liz Pizarro
But having, you know, Simon to to guide that because of his expertise was really important, I think.
00:22:51
Simon
Yeah, so like some tweaks will sometimes inevitably require some technical expertise. One example being, so these large language models, chat GPT, etc. You may have heard out of the box, these systems are very sycophantic or can be sycophantic and overly compliant.
00:23:08
Simon
But if you want to simulate a mental health client, you want some resistance and some complications. On the other hand, you can't just simply put in the prompt hard code, be more resistant, because then it will be overly resistant.
00:23:14
Liz Pizarro
Yeah. Yeah. Hmm.
00:23:17
Jenny
Thank you.
00:23:19
Simon
They tend to go towards extremes. So to get some sweet spot in the middle and have a reasonable degree of maybe resistance that then reduces over time, you need to do some technical things in the background to achieve that.
00:23:22
Liz Pizarro
yeah
00:23:31
Simon
And we're still experimenting with that.
00:23:35
Liz Pizarro
no
00:23:35
Jane Frost
So is there a way to get the but prompt to have some consistency about the conversations it's having and make sure that it's clinically safe?
00:23:48
Jane Frost
Because if you put something into chat GPT, as a skeptic, I'm saying it could come out with anything.
00:23:49
Jenny
as trying to measure, that's just a little bit of sense. So, obviously, that was a think a good
00:23:56
Liz Pizarro
Yeah, I think that's an interesting question because the when you, you know, working with real life clients, they're not like, they're not clinically safe, right?
00:23:56
Jane Frost
No?
00:23:56
Jenny
I think was a one. I think was good one. I think it was good one. think it was a good one.
00:24:06
Liz Pizarro
They, they share things that might trigger a student that a student might actually relate to.
00:24:07
Jenny
I think was good I it was I think it a good one. I think it was one.
00:24:15
Liz Pizarro
It might genuinely distress them.
00:24:16
Jenny
I think it was a good one.
00:24:18
Liz Pizarro
If they're, if a client is sharing about an experience, a trauma that,
00:24:24
Jenny
the last matches the record.
00:24:24
Liz Pizarro
a student can relate to that's realistic.
00:24:25
Jenny
That's what it
00:24:27
Liz Pizarro
So I think it's really different if you're developing therapy bots. I think that's when that's really important.
00:24:33
Simon
yeah
00:24:36
Liz Pizarro
But for for this, I guess we we give it the prompt and Simon can talk a little bit more to the technicalities of it, but it's not so when the student is actually playing the therapist and the client and the chatbot is playing a client, it is genuine when a client responds in a way that is distressing to the student.
00:25:03
Liz Pizarro
I'm not sure if that makes sense.
00:25:06
Jane Frost
Yeah, i guess I guess we all come from a nursing background where perhaps if we were developing a chatbot, we may need some more consistency or some specific things that remain the same with the person.
00:25:25
Jane Frost
Noting that you've just said to us that these are fifth year, already trained, so the risk is is less perhaps.
00:25:35
Jane Frost
giving those scenarios to an untrained person may be more stressful and I think that's where I'm thinking of the parameters.
00:25:42
Liz Pizarro
Yes.
00:25:43
Simon
Yeah, that's good point.
00:25:46
Jane Frost
And, you know, our audience will be b looking at implementing things for a wide range of people.
00:25:47
Simon
Consistency. in the earlier stages, you're right, consistency and training wheels are more important.
00:26:01
Simon
I think that type of thing can be implemented. Firstly, by just having a prompt of a certain nature and ensuring that it's consistently used.
00:26:10
Jane Frost
was trying to be kind
00:26:12
Jenny
you.
00:26:13
Simon
And secondly, you can do some extra things on top, some wrappers around what you're doing with a large language model to just ensure that things are consistent and safe. But also, as Liz said, unlike car using chatbots for say virtual therapy where safety is paramount and you know you want to have safety guard valves, at this stage having a chatbot that is perhaps a bit unpredictable or you know open, it's not that much of an issue because that is essentially what students are going to have to encounter in their real practice.
00:26:50
Jane Frost
Yeah, that's a really interesting perspective that you're you're actually wanting that unpredictability and you don't mind that there's different responses from the chatbot.
00:26:56
Simon
Hmm. Hmm.
00:27:04
Jane Frost
Jenny, Kerry, if you've got any other questions that you'd like to ask?
00:27:09
Jenny
I've got one. I'm super curious. Where to from here? What happens next for your project or how else are you going to develop the chat box?
Future Research and Closing Thoughts
00:27:19
Jenny
You said you that your students got improvement.
00:27:23
Jenny
Is there a next step now?
00:27:26
Liz Pizarro
this We do have a current project underway where we are going to be getting experienced psychologists to score the transcripts.
00:27:41
Jenny
sure was going to start with the program.
00:27:41
Liz Pizarro
So when we had the the the practices, the pilot that we ran last year, part of what with the data that we generated were the transcripts, the the text-based transcripts.
00:27:43
Jenny
I'm going to start with the program. I'm going to start with the program. I'm going to start with the program. I'm going to start with the program. I'm going to start with the program. I'm going to start with the program.
00:27:53
Liz Pizarro
so we will be engaging three psychologists to use some validated tools to assess whether there was improvement from practice to practice or across the four practices in the core counselling skills and also the suicide risk assessment skills that students attempted during, during their, their practice.
00:27:54
Jenny
I'm going to start with the program. I'm going to start with the program. I'm going to start with the program.
00:28:18
Liz Pizarro
So that's, that's one of the things that we're going to be, uh, doing this year. I mean, Simon can talk a little bit more to the sort of technical, analysis of some of of the the language data.
00:28:26
Simon
Sure.
00:28:30
Jenny
Thank you.
00:28:30
Simon
Yeah, so it's part of a broader endeavor you using chatbots to simulate practice opportunities, counselling, psychotherapy scenarios is one half, the first half of things.
00:28:44
Simon
I see the second half has been using natural language processing to provide opportunities to give automated feedback to students and educators. So, say in the psychology, the therapy world,
00:28:56
Simon
There are various scales that exist to rate the quality of how a practitioner is doing.
00:28:59
Jenny
Thank you.
00:29:02
Simon
So you get a transcript, say for cognitive behavioral therapy, there's a scale called the CTRS, by which a session can be scored for how well it is in adhering its fidelity to CBT practices.
00:29:16
Simon
This is traditionally a very laborious process. A human who knows their stuff has to go over the transcripts and annotate it manually. okay Could we develop machine learning models that can just get the transcript of a session and provide an automated evaluation of it?
00:29:32
Simon
And that's what we're also looking into as a research area. And this is one way we can do it with the data we generated with Liz last year from our study. We have about 60 transcripts to begin with. We're going to annotate them and then use a machine learning model to explore some training prediction models.
00:29:50
Jenny
That's pretty exciting. are you going to comparison between the live people and the chat bot analysis? Or is that two separate projects?
00:29:59
Jane Frost
Thank
00:30:01
Simon
So the human annotators that Liz alluded to will do their so their own scoring and then we'll get some inter-rater reliability to make sure everything's okay there. And then, you know, as you standard it in machine learning, you get some percentage of your labeled transcripts as your training data set, and then you leave some aside as your test data set to make sure that your model is doing a decent job of predicting outside of what it's being trained on.
00:30:28
Jenny
Oh, thanks, Simon.
00:30:31
Liz Pizarro
It is really exciting and it's one of the things that our academic staff here asked. It's one of the first things they ask. So do students get feedback when they are practising with the chatbot or are they just practising?
00:30:44
Liz Pizarro
And at the moment they were just practising. So the idea is to be able to develop, as Simon explained, some automated feedback that is based on expert feedback ratings so that students can get some guidance as they go and some immediate feedback.
00:31:08
Jane Frost
Well, thank you. That's been amazing. I think I can hand over to Jenny for our final question, which is the important one, isn't it, Jenny?
00:31:18
Jenny
So at each session we like to i ask our guests, if you could be having a drink anywhere in the world, what would that drink be and where would you be, Liz?
00:31:29
Liz Pizarro
Oh my gosh.
00:31:31
Jenny
Simon, do you have one?
00:31:33
Simon
I do probably.
00:31:34
Jenny
Oh, good. Well, what? Oh, good.
00:31:36
Simon
I mean, I have more than one actually.
00:31:38
Jenny
Where?
00:31:40
Liz Pizarro
You came prepared.
00:31:41
Simon
So one would be an Aperol Spritz somewhere on the Southern Italian coast.
00:31:48
Jenny
Nice.
00:31:49
Jane Frost
Fair enough.
00:31:49
Jenny
And anyone else would you like to be in drinking?
00:31:52
Simon
Maybe in wintertime, some mulled wine somewhere in a bit further north in Europe.
00:31:58
Jenny
nice.
00:32:02
Liz Pizarro
Now that I've had a moment to think about it, don't know, I'd probably be having Chilean red wine from the Valle del Elqui with my family in Chile.
00:32:18
Jenny
Beautiful. Sounds nice. I'd be happy to join you on both of those occasions.
00:32:28
Jane Frost
Thank you so much.
00:32:30
Liz Pizarro
Thank you.
00:32:33
Jane Frost
Now we do have Mel online