Transcript
Melanie Barlow: All right, welcome to another episode. It's just me today and I'm joined by the fabulous Katie Walker. How are you, Katie?
Katie : I'm well, thanks, Mel. Okay.
Melanie Barlow: So I'm just going to do a little bio on you. Katie is an initially internationally recognised leader in healthcare simulation with more than two decades of experience.
Melanie Barlow: in simulation-based education, workforce development and patient safety. Originally from Australia, Katie led the development of Health Workforce Australia's national simulation program, which everyone in Australia benefit benefited from, one of the largest healthcare simulation initiatives globally, supporting simulation integration across health professions, education and workforce training.
Melanie Barlow: She's held senior leadership roles in Australia and internationally, including the Director of Simulation at New York City Health and Hospitals. Katie has been an active contributor to the Society of Simulation in Healthcare, serving in a range of leadership positions and supporting the advancement of simulation practice worldwide. Her interests include translational simulation, healthcare team performance, patient safety, organisational improvement,
Melanie Barlow: and the strategic use of simulation to enhance health professional education and healthcare care outcomes. So Katie, it's fabulous to have you back in Australia.
Melanie Barlow: You're doing lots of research work in simulation. You're about to defend your PhD
Katie : Yes, I am.
Melanie Barlow: and yeah, a dear friend. So welcome.
Katie : Thank you, Mel. Thank you, Mel. It sounded like a lot, but I'm very grateful. Thank you.
Melanie Barlow: So today we're going to talk about your paper, in which you co-wrote with a number of fabulous people, on the value-based simulation in healthcare, a new model for metrics reporting in advances in simulation.
Katie : Yes.
Katie : Yep.
Katie : Yep. Okay, great.
Melanie Barlow: Fabulous. So we're not going to go through the details of the paper because Simulcast have done a fabulous job on that, which you were on.
Katie : Yeah, they have. Yep. Yep.
Melanie Barlow: Today, we really want to just have a conversation with you about the framework and some really tangible, I guess, ways we can apply it.
Katie : Yep.
Katie : Yeah.
Katie : Yeah, absolutely. Let's get into it.
Melanie Barlow: All right.
Katie : <unk>
Melanie Barlow: So, Katie, I know the model is really close to your heart. Do you want to tell me a bit about it?
Katie : Oh, sure. Okay, so it's definitely been a bit of a baby for me, Mel. It's grown out of years of me, our simulation teams, hospital leaders, and colleagues who became our author group, trying to figure out the best way of demonstrating the value that simulation brings to healthcare.
Katie : So as we know these days, we all drown in data. And so we think the answer is knowing which data makes the most sense for each organization and for each simulation service.
Katie : And our catchphrase is, our value is local.
Melanie Barlow: you
Katie : So the VBSH is our attempt to give simulation services a better vocabulary for that conversation. So instead of climbing a pyramid toward ROI, like other models, we've created this taxonomy with six value categories. So things like, you know, how well a program is running, what people learned, what changed when people returned to their workplace. So that a sim service and their organizational leaders can pick what's actually relevant to their situation rather than chasing the highest metric.
Melanie Barlow: I love that because we've all been there trying to chase that metric and not ever succeeded.
Katie : Yeah. Yeah.
Katie : That's right.
Melanie Barlow: So where did it actually come from? Like, was there a moment where you just went, aha, like, this is it?
Katie : Yeah. Yep. Yep. No, there definitely was. so So it just came from us literally trying to force simulation into existing value models and just not being able to make it fit.
Katie : So we kept circling back to this idea that there really isn't a hierarchy of importance. Every level matters depending on who's asking the question. And then 2023, Lara Vapio, of whom I'm a fan, and Jonathan Shabino published a paper arguing that patient outcomes had basically become an unquestionable idol in health professions education research, and that treating it as beyond critique was actually a threat to the field.
Katie : So that was that really piqued my interest so much.
Melanie Barlow: Oh.
Katie : And it gave us academic permission to formalize what we'd already been arguing, that there was there's no doubt that patient outcomes are a goal.
Katie : I mean, that's obviously why we're here. But there are many other value propositions that are equally as important and lead to, you know, better patient outcomes.
Katie : Things such as improving team skills and culture and unit efficiencies. So we just thought there's got to be a way that we can present this and, you know, in some ways focus more on this.
Katie : Yeah.
Melanie Barlow: I love that. I love that. So there are six categories. Can you walk me through them so we can understand?
Katie : Yeah.
Melanie Barlow: And there's a great diagram in the paper of them.
Katie : Yes, yes. So it's table eight.
Melanie Barlow: Yeah.
Katie : So in in the paper, there's table eight, which actually has, you know, all the different categories and and the elements of each category.
Melanie Barlow: Okay.
Melanie Barlow: Yeah.
Katie : But we actually think of it more of a pie chart than actually a ladder. so So the service products dimension is the what, who and how.
Katie : what type of courses are run, how many hours are logged. It's about participant data, resource data and activity data. So the operational inputs and the operational outputs of all the activities of the simulation program.
Katie : And then we move on to program perceptions, which is how people feel about it. And it's not just the learners, it's some actually staff, the funders and the executives. So anyone who actually touches the simulation program, they are really interested in what the perception is. i think In the past, we've kind of done the happy sheets, you know, what do you know the learners think about the program. But in all reality, yeah, the people funding it and the the unit leaders, they want to know that it works as well. So so it's the perception of both participants and leadership.
Katie : And then in a acquired expertise, we only count in this one verified skill gain, not someone's self-rating. So I think often we ask about learners' increased competence and confidence, and that's good and that's important to know, but that actually fits into program perceptions.
Melanie Barlow: Mm-hmm.
Katie : So in acquired expertise, it's actually more what universities do actually by having, you know, assessment rather than summative kind of performance, you know.
Katie : So it's how it's actually assessing the expertise to make, to ensure that it's improved. And then in workplace performance, it looks at actually what changes on the floor after you've done the training.
Katie : So it's impact, simulations impact upon the workplace. And it's also the workplace insights that we gain from simulation. So yeah, there's many insights about the processes and what's happening within units that we actually learn from running simulations.
Melanie Barlow: Amen.
Katie : So it's it's it's also that. And system benefit is more the big picture impact. So things like patient safety, equity, staff retention.
Katie : And so this is both the tangible and the intangible benefits to healthcare by using simulation. And finally, there is the value analyses, which is the category that's a that's really about the cost and about the money.
Katie : And the question is, you know, was it worth it? So for all that we invested, you know, are we getting the outcomes that we what we want in in a fiscal kind of way, which which is important, but it's not the only thing.
Katie : Yeah. So that's, yeah, but it but definitely is important. and And I think the thing with that, you know, I think we've gone into the depths of how you actually calculate it, but I think as a simulation leader, really, it's working with the finance department,
Katie : wherever you are they are the absolute experts in finance so use their expertise and honestly you become friends with them and they believe in what you're doing it's way more important than you you know trying to struggle yourself to work out know what the what the fiscal gain or loss is so you know yeah that's just a little point there
Melanie Barlow: Yeah, really great point. Yeah, and I love this model. As you said, it's not a pyramid, it's circular.
Katie : Yeah.
Melanie Barlow: And in saying that, what you do kind of isn't every categorically is equally valid. Is that just a nice way of saying services get to pick whichever number looks the best?
Katie : So that's a really good point now. So the model is designed so that organizational leaders and simulation leaders come together to work out what value means to them.
Katie : So I think I said before that value is local. And it's about that conversation and about both the organisation and the simulation people coming together and thinking, okay, how are we adding value here?
Melanie Barlow: Thank you.
Katie : And the idea is to have these conversations early between funders and leaders with the model being presented with the six different categories and working together to decide the data that matters to each unique organisation.
Katie : So it's not the service hiding behind numbers that may look great. It's a shared conversation about the relevance of the service to the stakeholders.
Katie : And we're trying to make a shift from metric power to metric relevance. So it's all about what is most relevant to each organisation.
Melanie Barlow: hmm
Katie : And if anything, I think it actually brings the data to life because you can't just default to we ran 40 sessions, for instance. If your funder actually wants to know whether staff feel safer speaking up, that's what's important to them. And I know that's your point of expertise, Mel, speaking up.
Katie : And if, you know, if your organisation wants to know that's the metric that they're measuring the success of your program on, they don't care how many people, or they might care a bit how many people are trained, but mostly, you know, they're looking at that dimension of, you know, did this actually, you know, cause a change in what people are doing? so yeah.
Melanie Barlow: I actually really love that. When you were just talking then, i don't know that I knew it, but it didn't really hit me how this model is really contextualised to
Katie : right
Katie : yes yes
Melanie Barlow: each organisation and what is most relevant, which is fabulous because then you don't have to try and struggle to get up those levels in the pyramid.
Katie : yeah yeah yes
Melanie Barlow: It's really choosing what it is you want to focus on and measuring that, which is just so much more pleasant. Yeah.
Katie : Yes, that's for sure. Yes. And, and honestly, you might be, you know, wasting money and resources, trying to achieve something that, you know, the the people around you are just kind of not interested in, right?
Katie : You know,
Melanie Barlow: Yes.
Katie : Because you don't know unless you have the conversations and you ask them. So this is like, you know, this is a vehicle to open those conversations and you know, give you the kind of expertise to talk about it, you know.
Melanie Barlow: Yeah.
Katie : Yeah.
Melanie Barlow: Yeah. And it's a really good reason to set up that meeting with someone that you wouldn't normally
Katie : Exactly.
Melanie Barlow: you know, converse with.
Katie : Yeah.
Melanie Barlow: it So in the paper, you give us a case study, which I think is fictional.
Katie : Yeah.
Katie : Yeah.
Melanie Barlow: Can you talk about like, how you've applied this in a real situation?
Katie : Yeah, yeah. So one time, i guess, when it really hit home to me was when I was in New York City. And so the simulation program was in the New York City Health and Hospital System. And we were funded by New York City to run large scale program. simulation program aimed at decreasing maternal mortality among women of colour.
Katie : So New York City Council, who was the funder, wanted to improve this statistic where women of colour were eight times more likely to die from maternal mortality than white women.
Melanie Barlow: Amen.
Katie : The metric that they were concerned with and regularly wanted reports on was training numbers because they were concerned with because because the value to them was that every single person on every professional team that cared for these women had been trained.
Katie : And that's not a lesser metric. just because it sits low on Phillips pyramid, it was literally the thing that mattered most to them as the funder. So if we had put extra effort into measure fiscal return on investment, for example, it would have used precious resources that we were able to use more effectively to measure the data that they actually wanted.
Katie : So, yeah, so what they actually wanted was the first service products.
Melanie Barlow: Oh.
Katie : Service products was the the element that was absolutely most important to them. And they they just wanted to know that we had trained at least in every obstetric unit 85% of staff, and that was literally all they were interested in.
Katie : And we were able, you know, obviously to say that, you know, it did improve things, which was fantastic. But yeah, but it's just that we didn't have to put extra resources into understanding that that that happened by way of training this many people.
Katie : And so it it was very interesting, I think, you know, yeah, that that you can actually waste, you know, brainpower and resources just trying to, you know, produce data on something that people aren't really interested in.
Melanie Barlow: Yeah.
Katie : So, Yeah.
Melanie Barlow: How often does that happen?
Katie : Yeah, right.
Melanie Barlow: lot, I would say.
Katie : Yeah. Yeah.
Melanie Barlow: Yeah, yeah.
Katie : Yeah.
Melanie Barlow: That's a really great, really great example. So i know this was really written with hospital based simulation in mind, because all the authors are hospital based simulationists.
Katie : yeah
Melanie Barlow: A lot of us work now in higher education. Can this model be applied in that context?
Katie : Yeah, that is a really great question, Mel. And as we were working through it, we were in the back of our minds and actually two of our authors did have crossover with higher ed as well. So that was really useful. But honestly, think... I think it translates better than we actually might expect. So the six categories are built around, as we've said before, who's asking, who's asking the question and what they value, not specifically around hospital funding structures.
Katie : And now where we get a bit of a shift is workplace performance.
Melanie Barlow: Mm-hmm. Mm-hmm.
Katie : And probably that's something that we might think more about because obviously in a hospital system, It's about bedside behaviour change. And in higher education, the equivalent is really performance on clinical placement, right?
Melanie Barlow: Thank
Katie : And system benefit shifts as well. So instead of hospital system level outcomes, you're looking to higher ed outcomes, which You would definitely know more about those than me, but, you know, they may be things like, I don't know, accreditation standards or professional body requirements or maybe program reputation, you know, probably a few different things.
Katie : But definitely service products, program perceptions, acquired expertise and value analyses mapped directly across from this model.
Katie : And as you know, a university sim lab has the same inputs and outputs.
Melanie Barlow: Yeah.
Katie : It has the same monitoring of perceptions of courses. It verifies real skill gain, i would argue, much more than probably hospital systems do, rather than, you know, self-reports.
Katie : And it also has to justify its budget to the faculty, which I'm sure you are very aware of. Yeah, and I think I mentioned in our own author group, we do have some colleagues that span clinical and academic positions.
Katie : So even though it's been developed for predominantly for hospital systems, it definitely would map to educational institutions as well. Yeah, yeah.
Katie : That's a great question. Yeah, it's good thinking about it.
Melanie Barlow: Yeah, that's a, yeah, that is good.
Katie : oh
Melanie Barlow: Yeah. It does map quite well, doesn't it?
Katie : Yeah, I think so. yeah no, it does. Yeah.
Melanie Barlow: Excellent.
Katie : And then, you know, if you wanted to take it further, Mel, if you wanted to build on it, you could, you know, do an adaption for ed.
Katie : Yeah. higher read
Melanie Barlow: That is true. yeah
Katie : then
Melanie Barlow: Let's chat.
Katie : and yeah
Melanie Barlow: So thank you, Katie. For those that want to read the paper, we'll put a link in the notes of the show. For anyone who wants to reach out and find more information, have a chat about maybe learning more about it and how to apply it in their setting, maybe reach out to the show and then we can get you in contact with Katie.
Katie : Yeah, yeah, no, definitely. And we could, we are giving a workshop.
Melanie Barlow: Yeah.
Katie : Yeah.
Melanie Barlow: good plug.
Katie : Yeah. Yeah.
Melanie Barlow: Yes, there is a national simulation conference on the Gold Coast in Australia in October, the 21st, 22nd.
Katie : yes
Katie : yeah and
Melanie Barlow: And yes, registrations are open now and Katie and the team will be running a workshop on this evaluation.
Katie : yeah
Melanie Barlow: on the Wednesday pre-conference workshop. So if you're super keen, come on down. It'll be a great event.
Katie : Yeah. Yeah. Be great.
Melanie Barlow: Very good.
Katie : Yep. Yep. Very good.
Melanie Barlow: So, Kay, before we let you go there is one question we ask everybody.
Katie : Yeah. Okay.
Melanie Barlow: because we like to live vicariously.
Katie : I love that.
Melanie Barlow: of if i know it's in the morning and we're sipping on coffee, but if you could be anywhere in the world for a happy hour, where would you be?
Katie : Yeah.
Katie : Ooh. Ooh.
Melanie Barlow: would you be sipping?
Katie : And after that, when I can finally relax, hopefully, fingers crossed, a I am then going to the Greek islands, which I'm very excited about.
Melanie Barlow: I'm sure.
Melanie Barlow: Oh, get out.
Katie : Yes, yes, yes, yes. And just to three little islands, which look totally little.
Melanie Barlow: Just three, just three of the Greek islands.
Katie : Yeah, yeah. Not the big ones. We're just doing the little ones. But they look gorgeous. And I did read that they have some small batch wines there in couple of the others.
Melanie Barlow: How fabulous.
Katie : So I'm thinking, hmm.
Katie : Looking at the AGMC while I'm sitting on my, probably, probably my dream.
Melanie Barlow: Oh my gosh. Oh my gosh.
Melanie Barlow: How wonderful.
Katie : Yeah,
Melanie Barlow: All right. Well, send us a photo.
Katie : that was, yeah.
Melanie Barlow: Excellent. Well, thank you t for taking the time to have a chat with us.
Katie : that was yeah
Melanie Barlow: Katie, I think it's a a fabulous model. And I know people were here rumblings around the world at different conferences that people are starting to apply it and use it and so I'm sure we'll be seeing a lot more of it out there.
Katie : Great. Yeah.
Katie : Very exciting. Great. Yep. Yep. Yep. All right.
Melanie Barlow: thanks Katie, thanks everyone for listening.
Katie : Okay. Thanks Mel.
