Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Research, Assessment & Evaluation: a CoP Spotlight
95 Plays10 months ago
This episode our on-air host Adam Kuhn is joined by Dr. Jason Laker, Dr. Lisa Endersby, Tim Tang, Heather Doyle, and Andrea Pape as we explore the topic of Research, Assessment & Evaluation: a CoP Spotlight. We hope that you enjoy and choose to follow along as we release a number of episodes geared towards celebrating our past, present, and future as an organization.
The CACUSS50 Oral History Project is an initiative of the Canadian Association of College and University Student Services in recognition of our organizations 50 years of engaging student affairs professionals in Canada. The series of podcasts is recorded and produced by: Sean Fast, Adam Kuhn, Nicholas Fast, Rachel Barreca, Stephanie Muehlethaler, Noah Arney, Sally Chen, Estefania Toledo, Paula Jean Broderick, Jennifer Brown, Margaret de Leon, and Becca Gray. This podcast is recorded, produced, and published on the traditional territories of hundreds of Indigenous nations from across the northern half of Turtle Island, also known by its settler-colonial name, Canada. We are grateful for the opportunity to live, work, and learn on this land. For more information on the territories you may reside on, visit: https://native-land.ca/
Music: Expanding the Limits | Performed by Audiorezout & Written by Oleksii Striapchyi | Stock Media provided by Audiorezout / Pond5
Podcast Cover Art by: Ravi Gabble (UTM)
Want to be a guest? Drop us a line:
Twitter: @cacusstweets
Instagram: @cacussphotos
Website: https://cacuss.ca/Insert Description
Recommended
Transcript
Caucus 50th Anniversary Oral History Project Introduction
00:00:19
Speaker
Hello and welcome to the caucus 50th anniversary oral history project. Today's episode, we are going to be speaking with leaders past, present, future of one of the very important community practice in caucus, which is the research and research and education research and assessment community practice. And we've got five esteemed colleagues who are joining us from different parts of Canada and the United States. So this is an evaluation. Thank you, Jason.
00:00:47
Speaker
Research Assessment and Evaluation, Community of Practice.
Meet the Guests
00:00:50
Speaker
So first, I'd like to introduce Lisa Endersby, who currently works as an educational developer at York University. We've got Jason Laker is a professor of higher education, student affairs and community development at San Jose State University. Heather Doyle is currently the director of research, assessment and planning at Dalhousie University. And we've got Andrea Pape, who is an independent education specialist, researcher and consultant.
00:01:15
Speaker
And then last but not least, Tim Tang is currently AVP student experience at the University of Alberta. So the research assessment and evaluation of community practice has a pretty special history.
Early Goals and Inclusivity in COP
00:01:29
Speaker
And so I was wondering maybe we could just start with the very, very basics of what is the research and assessment community of practice. And I feel like, Lisa, were you one of the first co-chairs of the COP?
00:01:42
Speaker
Yeah, I think so. That's so strange. I was just saying, as we were getting started, I went back through my emails to attempt to refresh my memory. And I was looking at the the dates, which is always a bad idea. And it said 2016 next to a bunch of those emails, which feels like a lifetime ago and likely was. But yeah, I think for for me anyway, at its early iteration, there were a couple of goals that I think were central to the community and central to me anyway, one was
00:02:11
Speaker
recognizing that we don't often get to celebrate and showcase the great work that happens in student affairs, whether that is through different opportunities for gratitude or appreciation or more formally, as it were, through research and evaluation. This is a field that sometimes we feel like we have a bit of an identity crisis. We don't know who we are or where we fit.
00:02:32
Speaker
Um, so trying to encourage and showcase that scholarly work was important, but then also wanting to use that work for the spirit of continuous improvement. And now that I'm, I don't know, on the other side, I guess, as it were, I'm no longer in student affairs, working in the academic realm, as it were, we see the same thing on, on this side of the house. There isn't always a space to not just collect data, but make use of it in a meaningful way.
00:03:01
Speaker
So wanting opportunities to do both of those things I think was really central to our early work. Awesome. Thanks, Lisa. And Heather, how would you describe to, you know, a new caucus member who just joined and they're curious about the COPs and how would you describe what the Research Assessment and Evaluation COP is about?
00:03:20
Speaker
Well, for sure you should join because it's definitely one of the best ones. But as someone that I think I'm probably the newest in the assessment, research and evaluation field with my colleagues on this panel and my background has was advising and kind of coming into this field. And I think that one of the really great things about the COP is that unlike some of the others, it's really for everyone because we really do want to ensure that we're embedding this work
00:03:49
Speaker
across all of our different areas. And so I think that it's for those that are doing this work full time like many of us are, but it's also for those that are just like, I have no idea and I'm terrified about research assessment and evaluation. I was one of those people not that long ago. Somehow I ended up doing it full time. I don't know how, but that it's for them too. And Jason and I actually just came from an event where we had a book club
00:04:19
Speaker
and talking about kind of some of the issues around equity and assessment in the field. And so it's an opportunity for folks to engage with others who are doing this work, to engage in some kind of conversations around what are some of the things happening on our campuses as well as in our communities and really just building up our community of practice and making us better practitioners and being able to really understand the student experience and the student voice. And so,
00:04:46
Speaker
I think really if you don't think it's for you, it is. If you think it's for you, it is. And so really kind of I think that we're all encompassing, which I think is really powerful. And I think that it's one of the things that I really love about this community of practice.
00:05:03
Speaker
Awesome. I love that framing of thinking that it's not just for folks who might have research assessment or evaluation in their title. It really is for folks across different functional areas and roles. And I liked hearing about the book club. Maybe we can go over to Jason.
Demystifying Research and COP Activities
00:05:16
Speaker
Jason, can you tell us a little bit more about what are the activities that the COP engages with in terms of learning, community development? What is the COP up to?
00:05:26
Speaker
Well, I first have to just note that my dear colleague and co-chair Heather was the inspiration and the driver of the book club. And so I'll defer to her to talk about that activity, because it's wonderful. Beyond that, we have, at least in the last few years that I've been involved, done a number of meetup events. We're centering around particular problems of practice. I think relating to what others have talked about that
00:05:53
Speaker
It is one of those things where words like research and assessment and evaluation all can be very intimidating, not only to new, not only to new professionals. And I think that has a lot to do with perhaps failures in our professional socialization, where we get these ideas that just really in the end, when I used to be at Queens, for example, overseeing the Division of Student Affairs,
00:06:16
Speaker
talking with the staff, I would just try to reframe it and say, you know, really, in the end, this is a question of how do you know, you know, what, what lets you know that it worked? You know, how do you know it was, it was good, you know, like, what are the practices? So try, so I think a lot, a through line of our efforts is to demystify
00:06:36
Speaker
and to really invite colleagues from across Canada in particular, of course, to revisit these internal ideas that it's such a scary thing and recognize that in the end this is because we care about what we do and that it makes perfect common sense to want to develop ways to determine whether what we hoped would happen
00:06:58
Speaker
did happen and whether what we hoped would happen was all that should happen, did we miss anything, this kind of thing. And then so the meetups, the sometimes sponsoring of sessions at the conference, we have another great idea that I think Heather and I
00:07:17
Speaker
co-developed, I think, has to do with the consultation thing, where we are also trying to, given that we're a community of people that include both newcomers and experts in these areas, to provide consultation to our fellow members, to be collegial that way and give people a chance to talk to someone who's perhaps not in the ecosystem or
00:07:37
Speaker
the drama of their particular institution or department where they can talk to some friendly expert colleagues about whatever they're trying to sort out and have some good thought partners for that. So that's another thing.
00:07:52
Speaker
So, you know, the variety, I don't want to and then meetups at the conferences, but I also just want to end with that. For me, my attraction to even joining the in the first place research assessment and evaluation is and it relates to something Heather was talking about. I was a student affairs professional in the beginning of my career, eventually head of student affairs in both Queens and then here I'm in actually in California now.
00:08:17
Speaker
at the University I'm at, but also now I'm a professor and a researcher. And what I love about research assessment evaluation, just even by those names, it tells us that the range of inquiries, regardless of what hat you're wearing, that this is in the end a meta
00:08:34
Speaker
group and I agree with Heather's point that it's relevant to every single member because in the end we all want to do a good job and we want to find the best ways and good ways to find out that we're doing right for our students and for our organizations and for ourselves. And I just think the RA is a fantastic venue for that.
00:08:55
Speaker
Awesome. Thanks, Jason. And I think that the comment that you made around demystifying all of this work is really landing with me in a significant way. Because I feel like a lot of practitioners feel like intimidated because they're not doing robust statistical analysis for their just to analyze the workshop that they did. And so I think kind of how do we normalize different ways of talking about it? And there's different
00:09:19
Speaker
tactics that we can take for different different types of interventions.
Qualitative Research and Personal Journeys
00:09:23
Speaker
And I'll add personally, I'm exclusively a qualitative researcher. And so even when I was overseeing the Division of Student Affairs, I really we can do surveys off. We can buy them or we can create them. But but in the end, the stories really tell us
00:09:38
Speaker
And that's not the purpose of this particular session to get into ways that small numbers of people that you can make broader knowledge claims. But let me just say as a qualitative researcher that there are absolutely ways in which you can take a relatively limited number of stories when they're collected intentionally with people you select intentionally.
00:09:59
Speaker
to really get it not just into whether it was good or not good or so forth, but into the contours and the nuances, which is key for sophisticated and reflective practice. So this is a community where you can be in community with other people who build a respective capacity to do that. Yeah, and asking these big questions. That's fabulous. Thank you. Now, Andrea, could you tell us a little bit about your involvement with the COP and what value you get out of being involved?
00:10:30
Speaker
Absolutely. So for me, I originally joined the COP when I was a master's student and just trying to get into, I would say, more of the research side of things. I would say research felt very intimidating as a budding
00:10:49
Speaker
Student Affairs practitioner. And so I was hoping to just try to, you know, put myself out there, understand how to be a part of things. Again, very intimidating when I felt like everyone else had this research assessment evaluation experience, understanding whether that's from an academic background or enough years within. So I joined the COP for that reason and then eventually became more and more involved of putting myself out there.
00:11:19
Speaker
multiple COPs in this one in particular. So I would say that my encouragement to other folks, whether you're engaging in the COP or any other COP is it's okay to be new to something and not really know everything in order to take a leadership role or a community role.
00:11:37
Speaker
And now I'm surprised, for whatever reason, that folks come up to me, whether virtually or in person, asking for insight and advice and ideas and really what all the community is, whether you're in a leadership role or just in a community membership role. So that would be my two cents on it.
00:11:56
Speaker
That's fabulous. And thank you for speaking to your experience about, you know, getting involved and kind of pursuing other leadership opportunities within the COP. Because I think for younger members and newer members who are getting involved with caucus, I think it can be really intimidating if you're the new person, you're showing up to a community and it doesn't take long, a few times just raising your hand and signing up and then all of a sudden you're the person that people are looking to for advice and leadership. So thanks for sharing that story.
00:12:22
Speaker
I would like to just also point out that there are plenty of colleagues who've been in the field a long time and are still, you know, nervous about these things. So it's true that newcomers, whether they're young or new to the field, it's important to engage them. We also embrace our colleagues who've been around the block a few times.
00:12:43
Speaker
have thus far not had the need or desire to dig into some of these practices, but for which it could really energize their practice and really be kind of a renaissance for them at that point in their career.
00:12:57
Speaker
Yeah, I think the vibe that I'm getting is that there is something for everyone regardless of where you are at in your career. They didn't either say that for Jesse. Yeah, so I'm just emphasizing this, yeah, come one, come all. So Jason, you also kind of mentioned this idea around research assessment and evaluation and the intersections of these practices. And I'm just curious, I feel like every
00:13:20
Speaker
assessment workshop I've ever gone to starts with, what is research? What is assessment? What is evaluation? And maybe for folks who are curious about how this COP might connect with their work, Tim, I'm wondering if, and I'm not looking for like textbook definitions, but perhaps in terms of how the COP might address the needs of folks who are engaging with these different practices at different times, how might you explain that to them?
00:13:44
Speaker
Yeah, well, first of all, Adam, I wanted to start by thanking you for bringing us together today to talk about the role of research assessment and evaluation to the Canadian Student Affairs profession.
00:13:55
Speaker
I've really been enjoying listening to the Caucus 50 podcast and I think it's particularly meaningful that we have co-chairs here past and present to talk about the growth and evolution of the COP and I'm just thrilled to be able to be here to talk about something that I think is both timeless and timely in our profession. You know, something that we can do to help improve how we serve our students.
00:14:19
Speaker
Maybe I'll start by just saying, you know, I was one of the initial co-chairs of this COP along with Lisa. And so, you know, have been following the growth and evolution of this COP over the years. And when we first engaged in conversation about establishing the COP over seven or eight years ago now, it was really in the context of wanting to create a space
00:14:41
Speaker
for Canadian student affairs professionals to be able to not only develop knowledge, skills and capacity related to research, assessment and evaluation, but also to come towards a shared understanding of what we mean by these terms and how we see these come to the forefront of our work as practitioners. And so, you know, we were guided a lot in those early days in terms of this terminology
00:15:05
Speaker
you know, in terms of, you know, some of the work of caucus's own statement of guiding principles in terms of how we enhance the quality and effectiveness of the co-curricular learning environment. We were guided a lot by the work of ACPA in the early 2000s talking a lot about
00:15:22
Speaker
how we develop appropriate methodological approaches to assess learning outcomes and impact and to evaluate the effectiveness of various programs and services that facilitate student learning and development. So we were really informed by some of these foundational documents, but very much informed by how
00:15:44
Speaker
our members conceptualize these terms. I recall in the early days, you know, Lisa, you and I kind of took a terms, a draft terms of reference with some example definitions around this to help, you know,
00:15:56
Speaker
our members really understand what we mean and how we might come to a shared vision and around this. And so, you know, we really saw, you know, research assessment and evaluation to be complementary, you know, in terms of their role in developing theory and informing practice. And, you know, in some cases, you know, helping us to be able to use evaluative inquiry to make decisions about the merit worth and significance of our programs and services.
00:16:24
Speaker
Each of these goals had a role and a place in terms of how we understand our work and how we bring different aspects of systematic inquiry to the profession. And one of the things, Jason, you mentioned that really resonated with me is that notion of demystifying. Because I think at its core, what we're talking about here is how we as
00:16:50
Speaker
professionals, practitioners can embrace the spirit of inquiry around our programs and really seek to understand our outcomes, our theories of change, what we're trying to achieve in terms of our interventions and using that to inform our practice, to inform theory and to inform decisions about
00:17:08
Speaker
how we go forward with our programs and services. And so really that's my conceptualization of the early days of this work, the evolution and why we chose to include, be inclusive of all of these three different terms and to really reflect the breadth of inquiry that we can use in our profession.
00:17:28
Speaker
And I have to build on what Tim said, which I feel like is the majority of what we used to do. Tim would speak and then I would be inspired and then I would say something else. So this feels very much like early days of our coach Eric experience.
00:17:40
Speaker
One of the things that Tim said that I think is just important to highlight is what we're doing, not just sort of within the profession, but frankly for the profession. Now that I'm in, I keep saying the other side and I really need to change the way that I use that wording, but working far more closely with faculty now rather than with students directly in that more academic space.
00:18:01
Speaker
Sometimes it goes without saying when I talk to faculty, when I use words like research. Like for many of the faculty I work with, they are so well versed in research. It's often what they were doing before teaching. It might be what they're doing more so than what they're doing in the classroom. But it's always interesting to have those conversations and then hear words like data informed and research driven.
00:18:23
Speaker
being applied both to that work in the classroom, but also the work that is happening outside of the classroom. There was always, again, I use the term identity crisis because I'm thinking about the caucus identity project. And when we were talking about big questions like what does it mean to be a student affairs professional and what is student affairs in Canada? Really small philosophical. Is it a profession? Yeah. Is it a profession? Who gets to be a professional?
00:18:46
Speaker
And this is also coming out of my dissertation research where I talked to early career student affairs professionals who were asking the same questions. What am I doing? Who am I? Just small questions that we got to muddle through together in my dissertation interviews. But I think more and more I'm just so excited to see that the early work that we did and the great work that's continuing
00:19:06
Speaker
makes the case for the fact that this work is research-informed and data-driven, that while perhaps we talk about running events or running workshops, it's not just the fun stuff. It's not the other stuff that students do. There's lots of research we could all cite off the tops of our heads or pull the different textbooks off of our shelves.
00:19:28
Speaker
So it's so important to hear and reiterate, as Tim was saying, the work we're doing, I think, for the profession is just as important as the work that we're doing individually that's inspired by the community. Love it. And Tim, I feel like you've got another gem to add into there.
00:19:47
Speaker
Yeah, no, I'll just add too that, you know, at the time that we envisioned establishing this COP, there weren't a lot of positions that were focused on research assessment and evaluation as part of their full-time job or even as a majority of their position. And so, you know, I think one of the things that was really important to us, you know, at that time was just to have a space to invite and encourage, you know, all student affairs professionals who might have been looking to
00:20:14
Speaker
increase and develop their knowledge of and skills and research assessment and evaluation just to have a place to be able to gather at a national level to encourage that spirit of inquiry and to really have an opportunity to discuss and profile the tools and methodology that can be used to guide theory development and form practice and evaluate programs for decision making. And so that was really one of the pieces that we felt was really important in those early days too, to have that space for us
00:20:45
Speaker
I love that. And I'm having these recollections of the COP kind of getting up and running and appreciate the significance of the labor it took to kind of build those conversations and be really deliberate around how that community was coming together. So thank you for all the work that has taken place then and since.
00:21:03
Speaker
And moving through that temporality from then to today, let's talk about some of the concerns that are top of mind for folks in the research assessment and evaluation space in 2023.
Current Concerns: Equity and Data Usage
00:21:15
Speaker
So Heather, as one of our current co-chairs, what's on your mind, what's on the mind of COP members, what is dominating the conversation for the COP these days? Sure. And I just wanted to kind of quickly go back to Tim's point, because I think because I'm going to weave in some data, ironically.
00:21:31
Speaker
talking about, you know, institutions in Canada who might or might not have kind of, you know, dedicated positions, especially in student affairs. And I think probably as most folks are aware, caucus recently did a survey of institutions across Canada to kind of get, you know, an understanding about student affairs. And one of those questions was around whether they have a dedicated either person and or office. And just over 30% of institutions in Canada have
00:21:57
Speaker
an assessment position located in Student Affairs. But a lot more of the institutions were looking at embedding something. And just to comment with that number is that obviously it's still kind of on the lower end, but you don't need to wait for those positions to be doing this work. And then conversely, even if you have those positions,
00:22:19
Speaker
it doesn't mean that as a practitioner in advising, say, or whatever it looks like, that that means that I don't need to do assessment because there's someone here. And so it really is all of our work to be doing. And so I just thought that was an interesting, I think that that's a great stat that caucus is able to kind of share. So I'm not going to, I'm like, I have to limit myself and how many challenges that, you know, I hear from folks I'm going to talk about too. And
00:22:43
Speaker
And I have some privilege of being able to kind of travel across Canada and the United States and talking to folks about assessment and assessment in specific fields in particular, advising still near and dear to my heart. But I think one of the big pieces, and Lisa, you mentioned kind of being data-informed.
00:23:01
Speaker
Data and form means different things to different people. And I think that there's inherently a concern that our data is being used in ways that maybe is not comfortable to us. And so I think about, again, evaluation, pulling in that term, that it's going to be used against us in some ways. Like if students don't like an event, there goes any money for that. Or if they don't like an experience in an office, that's going to come back to you. And so I think sometimes
00:23:29
Speaker
how we've used data has affected kind of how people will interact with it now. And then also a feeling that it dehumanizes our students. And we in Student Affairs, I think, you know, the center of what we do is students, their voices and their stories.
00:23:48
Speaker
And I think that there's often a fear that like the numbers are going to take away from that student voice and that student narrative. And I think that we do need to be careful with that, but I would also argue that it's a way for us to share the student story in different ways. And so as someone, this is my entire position now at Dell, I think that I hold a lot of privilege to be able to
00:24:13
Speaker
hear from students constantly, like the good, the bad, the ugly. Right. And, you know, and I read their comments and then having to translate that in a way that's compelling to different audiences and making sure that I'm accurately capturing what those students were looking at. I think there's there's so much privilege. And I think that we can share the student voice in a different way. And it's both. And so it's the student stories like Jason talked about in the qualitative research.
00:24:42
Speaker
and also some of those numbers. But with that piece I also think which is I think a really big topic in Student Affairs in Canada in particular and it was actually the focus of our book club is around equity and assessment and really making sure that we're embedding different ways of knowing into the assessment process that we are looking at
00:25:05
Speaker
what students are, what types of students are being represented and who are not, how we're sharing data, what data we are sharing, what we're not sharing, making sure that our data isn't having self-fulfilling prophecies for certain groups of students. And there's a really great quote that talks about how assessment done without equity in mind creates inequities. And so I think that there's a lot of conversation happening about
00:25:31
Speaker
We don't need to take these colonialized, westernized standards of what it means to be a researcher all of the time. We do have the freedom and the flexibility to be able to bring in these different ways of knowing what does that look like in practice? How do we do that meaningfully? How do we unpack our own biases?
00:25:50
Speaker
How do we think about kind of how we're interpreting these things? And I think that that's a really big conversation and a really, really important one. And then the other thing that seems to be coming up more often, and I'm actually meeting with the case managers community of practice in January is around what if you just don't have big numbers, right? And so it's one thing if you're doing a big orientation event and you've got hundreds of students and you can do all of these different things and you,
00:26:16
Speaker
You know, you can create great graphics and it looks really great. But we also have positions in Student Affairs that their work is through the depth of the relationship with students and the impact that they're having on those students, which can't be translated always with just an N. And so really thinking meaningfully about how we are sharing their story in a really meaningful way, that's talking about the breadth and depth of that work.
00:26:43
Speaker
And then the true impact, and I would say real impact that that's having on students, their development, their learning. And so thinking about how we can represent that. And so I think that as we start to embed assessment, evaluation, research, whatever you're calling it, tends to be used interchangeably in Canada, was a study on that as well, which I think is why it's great that
00:27:04
Speaker
Tim and Lisa were really intentional about wanting to use that language meaningfully in the beginning, but is really kind of thinking about how do we do this in real and meaningful ways that actually capture the true extent of the work that we do. And then the impact that we're having on students learning and development that goes beyond just they liked the pizza and they came to our event. And that requires real ongoing work conversation, unpacking,
00:27:32
Speaker
But I think that that's where the real power and impact is. And I think that that's really where we want to be headed as a profession in Canada. And I think it's going to be really great to see. I love that. That was such a good answer. And what I can say from someone who has kind of been observing the activities of the COP is just really wanting to acknowledge the work that the COP and the folks involved with COP have done in advancing this discussion for us as a profession, because I feel like
00:27:59
Speaker
Every time we're talking about it, we're getting more nuanced. We're getting more thoughtful into what it is we're trying to do and how we are not serving those inequities. And I think a lot of that dialogue has been emerging from this group. So I just want to acknowledge that. Andrea, from your perspective, oh, I feel like I was going to go over to you just as I'm seeing you're having a sneezing or coughing fit. So I'm just going to take a long time to ask this question.
00:28:24
Speaker
for you to have a sip of water. But as we're talking about kind of what are some key themes and trends in the research evaluation and assessment space these days, what are you noticing? Oh, here we go. We can come back to
Incorporating Diverse Perspectives
00:28:39
Speaker
you. Does anyone else have any thoughts about what they're noticing in this space? I'll tag on to what Heather was saying.
00:28:47
Speaker
and that it has been very heartening to see the emergence of interest and efforts to learn and apply practices like anti-racist lenses, decolonial work, indigenous, feminist, queer, the critical lenses and bringing them into the center of how we examine things. And I think related to that is seeing that actually that's good for everybody.
00:29:13
Speaker
For example, I do a lot of work around men and masculinity issues and using feminist work to engage men in critical reflection of their lives and their development and their learning opens them up to being more human. That's good for men to engage in feminism. That's just one example.
00:29:32
Speaker
one could say similar things about white people looking at their own racial identity and the lived experience and noticing the differences in how we might understand our lived experience and the salience of it compared to the way other people who are different races speak about it and so on. Like it really fosters some incredible both self-awareness and recognition of the variety of people, temperaments and ways of being and thinking. And that opens up
00:30:01
Speaker
And that's expansive for everybody. And I think that research assessment evaluation at its best really offers that, not just to see how well things are working and so forth, whether they have the impact, which are both very important, but also it's a social exercise, really. It's one that brings human beings together to
00:30:23
Speaker
examine whether we're on the same page of what we understand things to be, what's true, what's not true, what we care about, what we don't care about as much. All of this, that this in the end is a very human endeavor and that's something that I like about because it's often thought of as just sort of this nerdy abstract thing. But it can be deeply personal and the tools of research assessment evaluation can really liberate people. And I think that's really important and I see that more in conversation now.
00:30:52
Speaker
Yeah, thank you. And thank you for pulling those threads together in such an eloquent way. Thanks. Thanks, Jason. Before we get into our question around the future of research assessment and evaluation, I was just curious from Lisa and Tim, who were part of the original band that got the COP together and then seeing the discussion, you know, today in 2023. Is there anything that you're noticing about how the discussion and the discourse has changed, evolved, grown? What are you noticing?
00:31:22
Speaker
I'm happy to kick us off, but first I do want to say I cannot agree more with comments that Jason and Heather have made about the importance of bringing more diverse set of worldviews into research assessment and evaluation practice and to be able to adopt a more culturally relevant, culturally responsive approaches, terminology, methods,
00:31:45
Speaker
that incorporate indigenous perspectives and the perspectives of other equity-deserving, equity-denied experiences and folks. And so I think in a lot of ways it starts with how we grow representation of BIPOC folks engaging in research assessments and evaluation.
00:32:05
Speaker
requires us to think differently about our terminology and methods and challenging frameworks that we hold to be able to bring some deep reflection and contemplation about how we do this work and so I do think that's a really important piece even in terms of the evolution of the COP over the last seven or eight years that I think you know really centers the work of in this field in our profession.
00:32:32
Speaker
The other piece I'm hearing a lot in terms of the evolution over the last number of years is in terms of capacity building. And I think Heather alluded to this earlier on, not just continuing to have a space for discussion and engagement nationally as it relates to this work, but also working to continually increase individuals' motivation, knowledge, skills, and to enhance
00:32:55
Speaker
our profession's ability to conduct and to use research assessment and evaluation in our work and in our own institutional context and to be able to do that in a way that is sustainable, that we bring sustainable practices into our work and where we are able to continually ask questions
00:33:12
Speaker
where we're able to continually collect, analyze and interpret data and to use findings to develop theory, inform practice and make decisions about our programs and services. You know, I think, you know, in a lot of ways, one of the evolutions of the last number of years is that we've seen potential for and greater movement towards, as Heather has alluded to, building capacity across
00:33:35
Speaker
our divisions using an embedded approach, you know, where the responsibility of research assessment and evaluation isn't necessarily outside of our area, but really that it rests within student affairs, maybe even at the unit or departmental level, and that
00:33:51
Speaker
program managers, program coordinators might play a role and have a research assessment and evaluation function to be able to strengthen programs and services by using various tools and methods related to research assessment and evaluation. And so I would say that being able to work actively in an ongoing way to build capacity to engage in this work, which is something I think that has really been
00:34:20
Speaker
amazing about this COP as a space for our field, you know, really allows us to engage in appreciative inquiry in a way that supports organizational learning and development and allows us to clarify our goals, our theories of change, and to be able to question the assumptions and practices we have embedded into our programs and to measure outcomes in a meaningful way. And so I would say that has been, you know, a piece that has really evolved over the number of years is ongoing work to build capacity for this work.
00:34:50
Speaker
So well said. Thank you. And yeah, I agree. Heather just said this in the chat, that Tim, I could listen to you talk all day. And I was about to say the exact same thing. So thank you for validating that, Heather. Lisa, did you have anything else to add as someone who was one of the original co-chairs and seen kind of the evolution of the discussion around research assessment and evaluation? I mean, I don't know if I can. How do I follow Tim? It's the same question I asked in 2016. It's the same question I'm asking in 2023. So that's one thing that hasn't changed since the early days of the new practice.
00:35:19
Speaker
To add, with the benefit of hindsight, of course, and experience, I really like that Heather was talking about, you know, what sometimes I refer to as the smiley face surveys. Like, what did you like? What did you not like? And we would have the the Likert, Likert, maybe that's the big debate in student affairs, whether it's Likert or Likert scale. But the scales that would sometimes literally have the smiley faces on it, you know, right on a scale of frowny face to smiley face, how you felt about this.
00:35:44
Speaker
And I really appreciate that Heather's talking about going beyond that and really using language like impact. I think that's a positive trend that I've been seeing a lot of, particularly again, I keep referring to my current position because I came out of student affairs working in leadership development and career services into this field of educational development, where I work directly with faculty to support their teaching and learning. And I think that language and that idea of teaching and learning
00:36:10
Speaker
is to me one of the biggest positive trends in the field and particularly around research assessment and evaluation. Even though our classrooms are not quote unquote traditional, if you will, or what's typically thought up to be a classroom space, this field is a teaching field and the students who engage with our work, who engage with our services are learning. So to see the data, to see the conversations about impact center around things like learning outcomes, to me,
00:36:39
Speaker
When I started in the field, the idea of having a learning outcome or a set of learning outcomes for orientation was so far beyond me. It was, we're going to run an event and we're going to give them this information and there's probably going to be pizza. And then we're just going to make sure that everybody's safe and that'll be the end of it. And while those things are still important,
00:36:57
Speaker
the addition of conversations about learning outcomes and assessing learning along with if you will evaluating the experience itself because we used to talk about you might still need the smiley face survey because even though you're anticipating great learning will happen doesn't matter if students can access the event and i use access very broadly.
00:37:17
Speaker
It doesn't matter if you've got a great set of learning outcomes, but the event just truly is not accessible or not in a way that allows students to engage. So seeing the trend where there are learning outcomes at all as an educational developer who works with faculty is super exciting. That's the first question I ask in every consultation, what are your learning outcomes?
00:37:38
Speaker
But to see that those learning outcomes and that goal of meaningful learning is tied to student affairs and services in all its forms. I think that's a really exciting positive trend. And this COP now gets to work with folks who are wanting to tell those stories, as Jason mentioned, who want to engage in those conversations where we can demonstrate or articulate and showcase, as it were, that really meaningful learning is happening.
00:38:06
Speaker
Yeah, that's fabulous. And I think in terms of also stepping into those conversations...
00:38:12
Speaker
and being okay with a little bit of the murkiness of it all. Because I think what I'm noticing is that in addition to learning outcomes, folks are looking also to more of the affective and the spiritual and the other outcomes around the things we're aiming to do on our campuses. Confidence, self-efficacy, belonging, these things are not easy things to kind of wrap our head around. And I feel like folks who are leading this are in our assessment and evaluation spaces around.
00:38:41
Speaker
What do we really mean by these? Who is, who are we centering in this? What does that discussion look like? And so that has been really exciting to watch from, from my perspective. So we've talked a little bit about what's happened in the past, what's happening in the present. So let's look on the horizon. What is on the minds and in the future for folks who are thinking about research assessment and evaluation. Andrew, if you're okay, I can go to you first. What are, what are, what's on the horizon in this space?
Future Trends and Opportunities in COP
00:39:09
Speaker
Great question. I mean, I think it's always a bit of an unknown as we are both proactive and responsive to trends that are outside of student affairs too, I will say. And being responsive to our student body and our community, generally speaking, I don't know that we can always foresee what will be the next response that is equitable or supportive or responsive to our student population. So that's always
00:39:36
Speaker
a question mark for me, and there's a bit of a chicken and egg there, I think, of being responsive but also trying to work ahead. I will say overall something that I'm really enjoying seeing, and folks have mentioned it, but that capacity building from folks are relevant of what their role is.
00:39:56
Speaker
at the institution around research assessment and evaluation is something that I think we're going to continue to see as folks building capacity around that, both out of interest and also out of need. So I would say that nearly everyone that I'm interacting with irrelevant of how long they've been in student affairs is being asked questions around assessment and evaluation for their roles, even if they're not formally engaged in it. It's being built into
00:40:25
Speaker
job descriptions, it's being asked at all levels. And that's something that I think a decade ago, I wasn't seeing baked in to focus expectations. So I do think that's a trend. And then being responsive, I think, to the question marks around our students and our student body and their needs.
00:40:45
Speaker
More generally speaking, I'm seeing a lot of more openness around moving away from that Likert. I'm going to use Likert. Likert scale. But more towards qualitative or nontraditional unorthodox, I would say, you know, not the standard basic templates of how you conduct assessment and evaluation institution to truly understand from students what their needs challenges.
00:41:13
Speaker
are and so I'm seeing that I'm seeing an openness not just for students to better understand some of our student populations that their perspectives and needs aren't being captured in rating the programs in design but also I would say engaging with students in the design of research is something that we're seeing and that I've been engaged in I would say more deeply in the past two years and I'm seeing many other folks
00:41:41
Speaker
engaging that as well. So that's a big one, not just for our students, but with our students.
00:41:48
Speaker
Absolutely, thanks Andrea. I feel like this term that is borrowed from a lot of social movements is like nothing about us without us and I feel like we're applying that a lot more in our research and evaluation work with students is doing it with, not for, or at, or to our students. Jason, what's on your mind in terms of the future for this COP and the idea of research assessment and evaluation?
00:42:13
Speaker
Well, we've been very optimistic and happy in our conversation, and I think it is important, and it relates a lot to that question, that the future being happy or unhappy, I think there's a fork in the road in that regard, that there's more and more pressure
00:42:32
Speaker
And the way I'm observing it is more and more pressure that really pushes toward old school ways of assessment, like these traditional ideas where we continue to make these connections between assessment and funding, for example. And I've often remarked in professional spaces that
00:42:53
Speaker
I'm under no illusion that merit and money go together, but I think that there's still this gaslighting in our socialization, our professional socialization that suggests that those are connected. And so it leaves people feeling really bewildered when they go through all the effort to come up with really elaborate assessment or evaluation processes and then see their program cut or something like that. And I think that
00:43:20
Speaker
So the fork in the road is this. On the happy side, we are coming up with more indifferent and more sophisticated ways to examine things, and we should. And that's been remarked upon in this conversation. And that is in the interest of being good at what we're doing. I think that whether the future is bright or not has also a lot to do with the extent to which we can see and understand the connections between or the differences between these endeavors of assessment and evaluation.
00:43:50
Speaker
on the one hand and the way politics actually work. And this is true between faculty and staff often don't have a lot of
00:44:01
Speaker
understanding about the political processes, that's also intimidating, just as assessments. And so to the extent that we can form communities sort of where the different expertise is present, that we have some people who are good methodologists around these practices, and then we have people who are politically savvy, we need to be in conversation.
00:44:20
Speaker
Because there is a need, I think, to, if you will, protect or open space or maintain space for doing this work in the ethical, authentic, effective way that it should be done, and that we need political cover. And there are ways to do that. And this, obviously, one podcast, one hour is hard to get into the details of how to do that. But since you asked in terms of sort of future themes,
00:44:45
Speaker
That's what I think, is that whether our efforts to do good assessment for impact and learning and all of that, and to be able to do that well, in order for the future to include all of that, we also need to amp up our political savvy and our networking and connections and being
00:45:11
Speaker
I'm trying to find the right way, but like partners with or being comrades, maybe is the word that's starting to come to mind. We need to merge those things if we want to have the future be consistent with what we know to be the right things, if that makes sense. So we need to get more politically smart, is what I'm saying, in order to achieve the things that we've talked about today.
00:45:37
Speaker
Yeah, and I think that's an important conversation that I don't think we are often having when we're talking about research assessment evaluation is the politics of it all. Even telling bad news, you know, because the truthful data might have uncomfortable news in it and you need to understand the politics of that. Yeah, I feel like we have a bias towards good news stories. And so sometimes we need to be open to some of the challenging news and insights as well. Tim,
00:46:06
Speaker
What's on your mind looking ahead? Lots on my mind around research and assessment and evaluation always, but to pick up on what Jason was saying, I do agree about the importance of being just creative and innovative around how we think about this work.
00:46:22
Speaker
In an era of greater accountability and fiscal responsibility, we're seeing all sorts of things emerge in our field around differentiation, around performance metrics, around new funding models. And in this context, I do think it's also important to think about how we find the right tools and methodology to answer the questions we have. And I know we tend to lean in some ways quite heavily on the quantitative by virtue of our need to
00:46:48
Speaker
to evaluate effectiveness and engagement and to demonstrate progress along key indicators. But, you know, I do see an important role too for, as we've alluded to earlier, the qualitative, the narrative inquiry and the work that helps us to understand the nature and the scope of our outcomes and impact of our programs and interventions.
00:47:09
Speaker
You know, I don't have to say to all of us, you know, that we all really value the significant impact we make on the student experience. And that requires us to give context to numbers and context, you know, and storytelling, you know, and ways that help us share, you know, the story of the whole student. And, you know, finally, I'll just say, you know, in terms of future engagement, you know, I think there's ongoing work we can do around advocacy, you know, continuing to elevate the profile and awareness of inquiry.
00:47:38
Speaker
in our field, continuing to engage our members and maybe even external organizations, other organizations and partnerships. I think about a role for education and training, to have more opportunities for training and skills development.
00:47:51
Speaker
consolidating and sharing resources for capacity building. I see a role for more focus, additional focus on research and scholarship and how we as a profession contribute to generating and disseminating research and maybe working with scholarly communities like the Canadian Society for the Study of Higher Education to be able to promote scholarship related to post-secondary education. And so lots of opportunities I think to
00:48:20
Speaker
be engaged and to be creative and innovative in this space as a profession. I love it. So a call to action for creativity and innovation in this space. Fabulous. Thank you, Tim. Lisa, what about you? What's on your mind?
00:48:35
Speaker
I like that I keep following Tim. This is perfect. He keeps taking me back. Two things. One, just picking up on and ideally reinforcing some of the points that were just made. Thinking about who isn't at the table or who isn't yet at the table when we have these conversations. I think that's a broader conversation about this field, but frankly, lots of fields in higher education or lots of areas in higher education.
00:49:02
Speaker
There's also, to me, this sense of, as you said, Adam, nothing about us without us, but also this idea of partners and partnership. We tend to use that word in a lot of different ways, like we use a lot of words in a lot of different ways in higher education. But I'm doing a lot of work right now around students as partners. There's a lot of great research emerging and otherwise being done.
00:49:24
Speaker
around engaging students as partners in this work. And so we often talk about the impact on or for students, but as Andrea so rightly put it, often that impact goes in many different directions.
00:49:36
Speaker
and can be engaged with in many different ways. So I really see a future where this work is done in partnership, particularly with students, but again, recognizing that student success is everybody's business and everybody's responsibility. So how do we engage faculty colleagues? How do we engage other administrators? I've heard external partners. We use the term in student affairs a lot, holistic student experience. And I laugh only because it's sometimes overused or was overused, I think, in some early work.
00:50:05
Speaker
But holistic means that students don't always see the orientation office or the advising office. They just see a student experience. And they're having an experience that we touch in many different ways. So how do we capture those many different data points? And the only other thing I wanted to bring up was also in this realm of research and evaluation and assessment. There's also, to me, notions of feedback and reflection. So I spent a lot of time working in experiential education as pedagogy.
00:50:33
Speaker
And we've done some really deep work around reflection and feedback, which includes self-reflection. And how do we become more reflective practitioners? So I'm building on Shon's early work perhaps, but thinking a lot about, we do a lot of assessment and evaluation and reflection maybe outward. And I think there's also a call now for inner reflective work as well.
00:50:57
Speaker
and what it looks like for us to be reflective practitioners and whatever that might mean or look like because the data that we get perhaps informs lots of different things but to me it also informs my own practice in my own learning which is exciting uncomfortable because it's supposed to be but also a way to I think better the profession and the the small but hopefully mighty way I can do that.
00:51:21
Speaker
Lovely. Thank you. Thank you, Lisa. And Heather, last but not least, what's on your mind thinking ahead to what's on the horizon for research assessment and evaluation? Yeah, and I think everyone summed it up perfectly. And so, but I just wanted to add one thing and it kind of really relates to what Tim was talking about, too. And it's really kind of a call to action is
00:51:44
Speaker
is that there are some really amazing things happening in Canada in student affairs with the work that we're doing with students. And we need to be in the literature. And so much of what we look at is US-centric, sorry, Jason, and US-focused. And we really need to kind of, we have a different system, we have a different structure, a different climate, and really kind of contributing to that research and that body of knowledge, I think,
00:52:13
Speaker
is important for us as a field, as a discipline. And so I would say just as a kind of call for action, what I hope the future of this kind of community of practice, but of the field in general will be is that we see more research and literature coming out of Canada around the unique experiences of our students and our post-secondary education institutions.
00:52:43
Speaker
I love that. That's such a good place to end because we could always use more and then we could also use ways to mobilize what's currently out there because sometimes it's hidden in a repository somewhere or it's in an article or a journal somewhere. But there is good stuff out there. And so how do we get it out there into the world so we can use it and apply it in our practice?
00:53:05
Speaker
I have one last question, which is on a scale of one to five, how much did you enjoy this interview? Okay, just kidding. I just felt like I had to end on a cheesy, Likert scale joke. But I wanted to say thank you so much for you wonderful five humans for your time and your brilliance. I learned a lot chatting with you today. And I'm so excited that we've got this artifact of this discussion as a podcast episode for caucus members today and in the future to engage with. So thank you so much for your time.
00:53:34
Speaker
Thank you for having us. Thank you, Adam. Thanks everyone.
00:53:58
Speaker
The Caucus 50 Oral History Project is an initiative of the Canadian Association of College and University Student Services in recognition of our organization's 50 years of engaging student affairs professionals in Canada. The series of podcasts is recorded and produced by Sean Fast, Adam Kuehn, Nicholas Fast, Rachel Barreca, Stephanie Mulettoller, Noah Arney, Sally Chen, Estefania Toledo, Paula Jean Broderick, Jennifer Brown, Margaret De Leon,
00:54:28
Speaker
and Becca Gray. Intro and outro music is courtesy of Alexei Stryapji. This podcast is recorded, produced, and published on the traditional territories of hundreds of Indigenous nations from across the northern half of Turtle Island, also known by its settler colonial name, Canada. We are grateful for the opportunity to live, work, and learn on this land. Miigwetch.