Transcript
Jane Frost: Hi everyone, name is Jane.
Jenny: Hi, I'm Jenny.
Jane Frost: And today we're going to be talking with Liz Pizarro-Campagna and Simon D'Alfonso. And the topic of today a research
Jane Frost: Would you like to tell us a little bit about yourself? I'm going to start with Liz, is that okay?
Jenny: Thank you.
Liz Pizarro: Oh, hi Jan, thanks for having us. So yeah, I'm Liz and I am an academic at Australian Catholic University based in Melbourne, i'm in Fitzroy.
Liz Pizarro: And I teach postgraduate psychology students here. I coordinate one of the postgrad psych programs, the Master of Professional Psychology.
Liz Pizarro: And as part of my broader academic role, I also do
Jenny: okay
Liz Pizarro: to research and some of the research that I've been doing lately has been collaborating in collaboration with Simon and we've been looking at tools, simulation tools, particularly the use of AI in training psychologists.
Jane Frost: Great. Thank you, Simon.
Simon: And I'm Simon. I'm a senior lecturer in digital health from the School of Computing and Information Systems at the University of Melbourne. So I'm a computing guy and my research explores the application of AI to mental health and psychology more generally.
Jane Frost: Great. Thank you. So I'm fascinated to talk to you. But before we go any further, want to welcome our co host Kerry, who has just joined. Joined as hi, Kerry.
Kerry Anne ReidSearl: Hi, Jane, and welcome to Women and Liz as well.
Jenny: and Tara, we are
Jane Frost: OK, so my first question is what prompted you to explore AI chatbots as a simulation modality for suicide risk assessment?
Liz Pizarro: Well, in training of postgraduate psychology students, one of the limitations that we have is that we don't really have very many opportunities for practice of the skills that students are learning.
Liz Pizarro: Main reason for that is because it's really expensive to pay actors to play the role of clients. essentially we rely
Liz Pizarro: on role plays with their peers, which, you know, as educators, we know that it's, you know, peers don't really simulate clients very well.
Jane Frost: Thank you.
Liz Pizarro: they They giggle, they don't stay in role. So it's not very true to life, genuine, realistic practice for students.
Jenny: Yeah.
Liz Pizarro: So I guess we've been trying to think about watch what can we do to
Liz Pizarro: and I guess with advancements in technology, we have some new tools available, to us, and LLM chat pots is, is one of those potential, you know, tools that has a lot of potential, for being, affordable and scalable, realistic, which is, I guess, really important or more realistic than, than role playing with your peers.
Jenny: and continue to work with humans, and of course, the world's been really doing to And we are going to talk about what we're doing to us because we're going to do this. And we're going to talk about what we're doing to do.
Jenny: And we're going to talk about what we're doing to do. And we're going to talk about what we're doing to do. And we're going to talk about what we're doing to
Jane Frost: Do you mind if I just go back a step and say, so we're talking about AI chatbots. What do you mean when we're talking about an AI chatbot? For our listeners, what are we talking about here?
Jenny: So I'm just going to ask you a question.
Liz Pizarro: Thank you.
Jenny: I'm just going to ask you a question.
Simon: So a chatbot, we've heard of ChatGPT these days, or I think most people have.
Jenny: I'm just going to ask you question. I'm just going to ask you a question.
Simon: was around 22, 2023, I think that but they couldn't large language models started to become more prominent via systems such as OpenAI's ChatGPT and Google's Gemini platforms.
Jenny: I'm just going to you a question. just going to ask you a question. I'm just going to ask you a question. I'm just going to ask you a question. I'm just going to ask a question.
Simon: And these frameworks and these systems make it easier these days to just craft your own prompt, send that prompt to the system and tell it to act in a certain way and have a conversational engagement.
Simon: So even 10 years ago, there was considerably more work involved in trying to get one of these chatbots off the ground to create your own chatbot.
Simon: It's become much easier over the last few years with these large language model systems that are readily available.
Jenny: Thank you.
Simon: So if you've used ChatGPT, that's what a chatbot is essentially.
Jane Frost: Okay, I'm going to follow that up with that. So is this a chatbot? So it's purely text? Or are they verbally responding?
Simon: Good question. So we use... these LLMs for our own platform, which we call Client 101. And Client 101 is a web platform that houses various chatbots that simulate mental health clients.
Simon: The first modality is textual interaction. We are currently adding voice interaction to that. And perhaps a future possibility might be some visual mode or an avatar.
Jenny: Thank you.
Jane Frost: Okay, great. Thank you. So did you want to tell us a bit about your research?
Liz Pizarro: Oh, so I guess to extend on that sort of broader challenge of being trying to think of of ways to provide better scaffold in more realistic simulation, practice for post-grad psychology students.
Liz Pizarro: One of the areas that, is really important and pressing and in need of further practice is, where there's risk to clients.
Jenny: Thank you.
Liz Pizarro: So, looking at using chatbots to allow students to practice suicide risk assessment skills we felt was really important and the literature shows that graduates and early career psychologists don't feel competent, confident in undertaking suicide risk assessments and in fact often will avoid it
Jenny: Yes.
Liz Pizarro: which obviously puts clients at risk.
Liz Pizarro: would simulate a client presenting with depressive symptoms, but particularly who would present with a a medium suicide risk that would allow the students to practice a suicide risk assessment.
Liz Pizarro: So that's kind of the beginning of the project is the idea of developing the chatbot for students to practice suicide risk assessment.
Jenny: All right.
Liz Pizarro: So from that, suppose we ran a pilot last year. And we were wanting to ask, I guess, two main questions that we explored quantitatively and also qualitatively.
Liz Pizarro: And this was collaboration. There was contribution from a number of people, including two of my students. So I had an honours student and also Hannah Giddy was my honours student and I had Cassandra Amount who was a master's student who was completing a clinical master's degree.
Liz Pizarro: And so they looked at qualitative and quantitative aspects of the research question, which was, is it useful?
Jane Frost: Great.
Liz Pizarro: Do students, do post-prite psychology students find it useful, beneficial to practice suicide risk assessments with a chat bot? And does it improve their self-efficacy in undertaking suicide risk assessments? So that's that was our research question.
Jane Frost: And what did you find?
Liz Pizarro: What did we find? So we found that, yes, they did really enjoy interacting with the chatbot and practicing suicidal risk assessment skills.
Liz Pizarro: They found that it, we found that it improved their self-efficacy. So we did, we got them to practice four times.
Liz Pizarro: So it was very structured and time limited studies. So we invited them to participate. immediately after they had completed training in suicide risk assessment.
Liz Pizarro: So they completed a unit in the postgraduate program that teaches them how to do suicide risk assessment. And then following that, they were invited to participate in the chatbot.
Liz Pizarro: So we wanted to make sure that they'd had the training. This wasn't a training tool, but rather a practice tool. And so we... invited them to practice four times over a period of four weeks, so once a week over four weeks and this was scheduled and it was done online remotely.
Liz Pizarro: They didn't have to come in so it was very practical and accessible for them and we measured So they completed very brief online questionnaires at baseline, so before any practice and then after each of the four practices as well.
Liz Pizarro: And what we measured on each occasion was their self-efficacy. So was very brief self-efficacy and risk assessment tool. And at the very end, we also asked them some questions about acceptability.
Liz Pizarro: So how useful, beneficial and acceptable they found the tool. And yeah, we found that they enjoyed interacting with it.
Liz Pizarro: They found it beneficial, they found it useful and it was quite consistent across the quantitative and qualitative findings. We did find that practice, the number of practices mattered. So the benefit was in the quantitative data, we saw benefit at week three and week four compared to the baseline level of self-efficacy.
Liz Pizarro: So the first practice, we didn't find a significant difference. So it took a few practices, which I guess is consistent with learning theory that says that you know practice is really important for increasing skill and self-efficacy.
Liz Pizarro: um'm We did find though that there were some limitations to that. So you know it wasn't perfect and the students did. So some of the in the qualitative data, some of what came up in that was really interesting and helpful in terms of you know future development of these types of tools that
Jenny: yes
Liz Pizarro: one of the challenges was obviously that, what we used was only text-based and so the students weren't able to see a client. And when you, you know, working with clients, you really rely on those, nonverbal cues.
Liz Pizarro: You rely on, body language, you rely on, uh, the tone, of the voice and other, Such cues and so obviously you don't have that when you're interacting with text So that was a limitation and the other one was repetition so Because we it was a pilot we only had one client we had Kyle So there was between session repetition because they practiced on each occasion with Kyle so obviously practicing with some clients repetitious but I guess
Jenny: and the other one. What's your reputation to be able to create? That was a good idea. Yeah. And I was going say, well, reputation when you come to the education community, and you can see the benefit to the education community.
Jenny: And I'm going to say, it's going to be a good reputation for the community. And I'm going to say, well, it's going to be good reputation for the community. I'm going to say, well, it's going to be reputation for the community.
Liz Pizarro: more interesting was that within session repetition where sometimes the student would try to ask a question in a different way to try and undertake a more thorough assessment, but the chatbot would respond with almost an identical response, which is one of the limitations and Simon could probably talk a little bit more to that and what work we're doing and thinking of.
Jenny: to the next few years to the next few years.
Simon: Sure. So client evolution is an active research and development problem, trying to get the chatbots to organically or inorganically develop, especially if a student will be interacting with the same chatbot on more than one occasion.
Simon: In this case, it was this caveat that we actually created four instances of Kyle, Kyle 1, 2, 3, and 4, and we deliberately made each instance a fresh slate when the students started with that instance for the week.
Jenny: Great. So what's the light?
Simon: So there was bound to be repetition there just because of the structure of the study.
Jenny: So what's the light? So what's the light?
Simon: But at any rate, yeah, this idea of trying to have a chatbot somewhat evolve as a student comes back to it the second week, the third week.
Jenny: Just the light? So what's the light? So what's the light? So here is the light? So what's the light?
Simon: So there is some type of something developing, I suppose.
Jenny: So what's the light? So what's the light?
Simon: And we have some basic basic mechanisms at play.
Jenny: So what's the light? So what's the light?
Simon: We now have a memory system.
Jenny: So what's the light? So what's the light?
Simon: functionality whereby say a summary of previous sessions will be appended to the current prompt that we send to the large language model.
Jenny: So what's the light? So what's the light?
Jenny: This is a very interesting question.
Simon: And you will just have an instruction like in formulating your, you know, next response, you can consider all these bits of previous information that have occurred in the sessions.
Simon: And then you could even do possible things like, you know, if there's something out in the real world that's happened, you can drag that bit of news into the context and tell the chatbot about that. So like, just pretend like that the chatbot is this entity in the real world that is learning some new things throughout the week prior to having the next session with the with the therapist.
Jane Frost: Can you talk me through a session, as in they've got four sessions, so the student would come in and use or be allowed to use the chat bot, just the words and have a conversation?
Jenny: OK. OK.
Jane Frost: Is that what you're saying?
Liz Pizarro: The students had access to the chat bot for the four week period and they could access it at any time, but to prevent that we created a schedule.
Jane Frost: Yep.
Liz Pizarro: So the students who were working on the project collaborated with the participant to create a schedule of practice so that each participant was practicing only once per week over that four week period.
Liz Pizarro: And when we look at the data, we can confirm that that is what happened, that participants were practicing once a week over the four week period.
Jane Frost: So they just had one conversation with the chatbot, is that what they're doing?
Simon: Although
Simon: one per week over four weeks, though some students like sometimes doubled up.
Jane Frost: Yeah. Yeah, OK.
Simon: like They were only supposed to interact with the first instance of COAL once in the first week, but they might have done it twice when they weren't supposed to, but that's okay.
Jane Frost: OK.
Liz Pizarro: Yeah.
Jane Frost: Yeah.
Liz Pizarro: Yeah. Hmm.
Jenny: I've got a question please, Jane. I'm really curious.
Jenny: Suicide risk assessment, what safety did you put in around, what were your safety considerations in building these scenarios and how did you manage student stress?
Liz Pizarro: If I could flip that question around actually, because one of the things that we're trying to do here is in some ways expose the students to the stress that they will encounter when they're undertaking suicide risk assessment.
Liz Pizarro: So in training psychologists, one of, you know, we're preparing them to work with clients and people in distress, people who are unpredictable.
Jenny: Thank you.
Liz Pizarro: there's, they don't come with a script and you never know what a client is going share with and one of those things that clients share with you is suicidal risk.
Liz Pizarro: and they, you know, don't necessarily come out with it immediately. It's something that you have to kind of be attuned to and know what you're listening for, looking out for, knowing how to ask the questions in in a way, in a sensitive way, but that is also,
Liz Pizarro: able to elicit that information from a client so that you can support them so guess we're not trying to protect the students from distress we are wanting them to experience as realistic as we can what it is to come in contact with a distressed client who might be presenting with suicidal risk and And, you know, I guess if we're sort of thinking of guardrails, because this was a pilot, we involved, we invited students who had already received suicide risk assessment training. And these are provisionally registered psychologists in their fifth year of training.
Liz Pizarro: So that's really the main guardrail, that they've got a fair bit of training under their belt already. They're provisionally registered psychologists and and had had suicide risk assessment training, but the actual practice with the chatbot itself, we weren't trying to protect them from, from distress, I guess.
Jenny: Thank you.
Simon: Although you did provide provide some reminders and links to mental health service resources if they just wanted to debrief afterwards, or were concerned.
Liz Pizarro: Yes.
Liz Pizarro: Yes. Yeah, we did. So we provided the Lifeline number and also within the university we offer free, there's free counselling that is separate from psych training discipline and we, know, reminded students that they have access to that.
Jenny: Lovely, thank you.
Jane Frost: just going ask Kerry if you've got any questions.
Kerry Anne ReidSearl: Yeah, look, sorry, there's a bit of echo here, but I'm just wondering if you had a takeaway for educators curiously experimenting with AI, what would you
Liz Pizarro: I didn't quite catch that, sorry.
Jenny: I think what Kerry was asking Liz was if you had a takeaway message for educators.
Jane Frost: So I think
Jenny: Is that it Kerry? Yeah. What would be your takeaway message for educators in working with chat bots and AI?
Simon: So I think the first important point to make now is that the possibilities and opportunities are ripe now.
Liz Pizarro: Oh, yeah, Simon's done heaps, he'll work here.
Simon: The entry barriers are pretty low to just trying to find you do a Google search if you're interested in this and you will probably find some option to just set up something relatively easy and just start experimenting. You can craft your own prompts that can influence what you want to generate and just see how you go.
Jenny: So do I have to have any knowledge, Sam? I've got no experience.
Simon: Yeah, so to have a really, really basic system, you don't need to have any knowledge. You can try out your own, go into ChatGPT and prompt it to behave in a certain way.
Simon: So write a prompt, go, you are this type of client. These are some of your characteristics. Now respond to me as i as if I am a therapist and we are having a conversation and it would go back and forth as a starting point.
Jenny: Thank you.
Simon: But then to use these technologies and develop your own platform, obviously you need to have some web development experience and know how to call what are called application programming interfaces and all this stuff.
Simon: But there individuals like myself who know how to do the computing things and you know we're open to collaborations and such and there are others like me out there probably so just find some computing people I suppose who'd be interested in working together if you're say from the mental health disciplines or the allied health services and try it out.
Jenny: so
Liz Pizarro: I think if I could add to that a little bit, I wouldn't have been able to develop this on my own. working with Simon, really helped guide how you develop a prompt for a client.
Liz Pizarro: trialed it, obviously we tested it out before, getting the student, you know, before running the pilot, we tested it with, with staff, you know, we played around with it, we challenged it, we did all sorts of things to to make sure that it was acting like a client.
Jenny: So
Jane Frost: Thank
Liz Pizarro: But having, you know, Simon to to guide that because of his expertise was really important, I think.
Simon: Yeah, so like some tweaks will sometimes inevitably require some technical expertise. One example being, so these large language models, chat GPT, etc. You may have heard out of the box, these systems are very sycophantic or can be sycophantic and overly compliant.
Simon: But if you want to simulate a mental health client, you want some resistance and some complications. On the other hand, you can't just simply put in the prompt hard code, be more resistant, because then it will be overly resistant.
Liz Pizarro: Yeah. Yeah. Hmm.
Jenny: Thank you.
Simon: They tend to go towards extremes. So to get some sweet spot in the middle and have a reasonable degree of maybe resistance that then reduces over time, you need to do some technical things in the background to achieve that.
Liz Pizarro: yeah
Simon: And we're still experimenting with that.
Liz Pizarro: no
Jane Frost: So is there a way to get the but prompt to have some consistency about the conversations it's having and make sure that it's clinically safe?
Jane Frost: Because if you put something into chat GPT, as a skeptic, I'm saying it could come out with anything.
Jenny: as trying to measure, that's just a little bit of sense. So, obviously, that was a think a good
Jane Frost: No?
Liz Pizarro: Yeah, I think that's an interesting question because the when you, you know, working with real life clients, they're not like, they're not clinically safe, right?
Jenny: I think was a one. I think was good one. I think it was good one. think it was a good one.
Liz Pizarro: They, they share things that might trigger a student that a student might actually relate to.
Jenny: I think was good I it was I think it a good one. I think it was one.
Liz Pizarro: It might genuinely distress them.
Jenny: I think it was a good one.
Liz Pizarro: If they're, if a client is sharing about an experience, a trauma that,
Jenny: the last matches the record.
Liz Pizarro: a student can relate to that's realistic.
Jenny: That's what it
Liz Pizarro: So I think it's really different if you're developing therapy bots. I think that's when that's really important.
Simon: yeah
Liz Pizarro: But for for this, I guess we we give it the prompt and Simon can talk a little bit more to the technicalities of it, but it's not so when the student is actually playing the therapist and the client and the chatbot is playing a client, it is genuine when a client responds in a way that is distressing to the student.
Liz Pizarro: I'm not sure if that makes sense.
Jane Frost: Yeah, i guess I guess we all come from a nursing background where perhaps if we were developing a chatbot, we may need some more consistency or some specific things that remain the same with the person.
Jane Frost: Noting that you've just said to us that these are fifth year, already trained, so the risk is is less perhaps.
Jane Frost: giving those scenarios to an untrained person may be more stressful and I think that's where I'm thinking of the parameters.
Liz Pizarro: Yes.
Simon: Yeah, that's good point.
Jane Frost: And, you know, our audience will be b looking at implementing things for a wide range of people.
Simon: Consistency. in the earlier stages, you're right, consistency and training wheels are more important.
Simon: I think that type of thing can be implemented. Firstly, by just having a prompt of a certain nature and ensuring that it's consistently used.
Jane Frost: was trying to be kind
Jenny: you.
Simon: And secondly, you can do some extra things on top, some wrappers around what you're doing with a large language model to just ensure that things are consistent and safe. But also, as Liz said, unlike car using chatbots for say virtual therapy where safety is paramount and you know you want to have safety guard valves, at this stage having a chatbot that is perhaps a bit unpredictable or you know open, it's not that much of an issue because that is essentially what students are going to have to encounter in their real practice.
Jane Frost: Yeah, that's a really interesting perspective that you're you're actually wanting that unpredictability and you don't mind that there's different responses from the chatbot.
Simon: Hmm. Hmm.
Jane Frost: Jenny, Kerry, if you've got any other questions that you'd like to ask?
Jenny: I've got one. I'm super curious. Where to from here? What happens next for your project or how else are you going to develop the chat box? You said you that your students got improvement.
Jenny: Is there a next step now?
Liz Pizarro: this We do have a current project underway where we are going to be getting experienced psychologists to score the transcripts.
Jenny: sure was going to start with the program.
Liz Pizarro: So when we had the the the practices, the pilot that we ran last year, part of what with the data that we generated were the transcripts, the the text-based transcripts.
Jenny: I'm going to start with the program. I'm going to start with the program. I'm going to start with the program. I'm going to start with the program. I'm going to start with the program. I'm going to start with the program.
Liz Pizarro: so we will be engaging three psychologists to use some validated tools to assess whether there was improvement from practice to practice or across the four practices in the core counselling skills and also the suicide risk assessment skills that students attempted during, during their, their practice.
Jenny: I'm going to start with the program. I'm going to start with the program. I'm going to start with the program.
Liz Pizarro: So that's, that's one of the things that we're going to be, uh, doing this year. I mean, Simon can talk a little bit more to the sort of technical, analysis of some of of the the language data.
Simon: Sure.
Jenny: Thank you.
Simon: Yeah, so it's part of a broader endeavor you using chatbots to simulate practice opportunities, counselling, psychotherapy scenarios is one half, the first half of things.
Simon: I see the second half has been using natural language processing to provide opportunities to give automated feedback to students and educators. So, say in the psychology, the therapy world,
Simon: There are various scales that exist to rate the quality of how a practitioner is doing.
Jenny: Thank you.
Simon: So you get a transcript, say for cognitive behavioral therapy, there's a scale called the CTRS, by which a session can be scored for how well it is in adhering its fidelity to CBT practices.
Simon: This is traditionally a very laborious process. A human who knows their stuff has to go over the transcripts and annotate it manually. okay Could we develop machine learning models that can just get the transcript of a session and provide an automated evaluation of it?
Simon: And that's what we're also looking into as a research area. And this is one way we can do it with the data we generated with Liz last year from our study. We have about 60 transcripts to begin with. We're going to annotate them and then use a machine learning model to explore some training prediction models.
Jenny: That's pretty exciting. are you going to comparison between the live people and the chat bot analysis? Or is that two separate projects?
Jane Frost: Thank
Simon: So the human annotators that Liz alluded to will do their so their own scoring and then we'll get some inter-rater reliability to make sure everything's okay there. And then, you know, as you standard it in machine learning, you get some percentage of your labeled transcripts as your training data set, and then you leave some aside as your test data set to make sure that your model is doing a decent job of predicting outside of what it's being trained on.
Jenny: Oh, thanks, Simon.
Liz Pizarro: It is really exciting and it's one of the things that our academic staff here asked. It's one of the first things they ask. So do students get feedback when they are practising with the chatbot or are they just practising?
Liz Pizarro: And at the moment they were just practising. So the idea is to be able to develop, as Simon explained, some automated feedback that is based on expert feedback ratings so that students can get some guidance as they go and some immediate feedback.
Jane Frost: Well, thank you. That's been amazing. I think I can hand over to Jenny for our final question, which is the important one, isn't it, Jenny?
Jenny: So at each session we like to i ask our guests, if you could be having a drink anywhere in the world, what would that drink be and where would you be, Liz?
Liz Pizarro: Oh my gosh.
Jenny: Simon, do you have one?
Simon: I do probably.
Jenny: Oh, good. Well, what? Oh, good.
Simon: I mean, I have more than one actually.
Jenny: Where?
Liz Pizarro: You came prepared.
Simon: So one would be an Aperol Spritz somewhere on the Southern Italian coast.
Jenny: Nice.
Jane Frost: Fair enough.
Jenny: And anyone else would you like to be in drinking?
Simon: Maybe in wintertime, some mulled wine somewhere in a bit further north in Europe.
Jenny: nice.
Liz Pizarro: Now that I've had a moment to think about it, don't know, I'd probably be having Chilean red wine from the Valle del Elqui with my family in Chile.
Jenny: Beautiful. Sounds nice. I'd be happy to join you on both of those occasions.
Jane Frost: Thank you so much.
Liz Pizarro: Thank you.
Jane Frost: Now we do have Mel online
