Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
It will test your head, and your mind, and your brain, too
186 Plays8 months ago
Phil, Neil, and Morgan discuss generative AI and its implications for assessments. What changes are we already seeing in testing policies, or in rethinking how to assess student learning? Will Gen AI be the trigger for institutions to finally make changes in assessment design that have been discussed for years?
Recommended
Transcript
Introductions and Weather Chat
00:00:11
Speaker
Hello and welcome back to Online Education Across the Atlantic. Joined as always by Neil and Morgan and it's great to see you all. I see Neil you're in the nice short sleeve shirts. Are you guys having a warm spell or you just have the heater up?
00:00:28
Speaker
I just have the heater up, to be honest, Phil. It's dreary here. I'm generally one of those people that are kind of more on the kind of warm end of the spectrum. So yeah, no, we're not having any kind of heat wave, that's for sure, in Wales, sadly. But yeah, pretty grim here.
Generative AI's Role in Education
00:00:45
Speaker
How about you? Oh, well, I do have the legitimate short sleeve shirt because it's a good 70 degrees and sunny today. So what's that, 27 or something?
00:00:55
Speaker
So it is a nice day, not necessarily in the office, but outside. And I should have practiced this ahead of time, but tomorrow is St. David's Day. So happy St. David's Day, Neil. I should have practiced it so I could have said it to you in Welsh. Thank you, Morgan. Yes, my children are preparing for some kind of fancy dress extravaganza at school. I think my boy keeps asking me if he can wear his dragon hat tomorrow.
00:01:25
Speaker
I say yes. I say yes. Yeah. Yeah. Yeah. And my daughter's asking me if she can wear a dragon costume, which I'm not sure is as appropriate, but we'll see what happens in the morning as, as is the case most days. Well, it's too bad. We're not recording this on St. David's day. We'd have you put on a dragon outfit. Yeah. Yeah. Although our listeners wouldn't benefit from that. And I think it'd just be, you know, just be for you to you guys to just laugh along at me.
00:01:54
Speaker
Well, today we're going into a new topic. We finished our little mini series on taking stock of online education. And today the big topic we're going to cover is generative AI, but not in general, but specifically how it's affecting assessments. So that's going to be the main topic that we
Online Degrees and Governance Issues
00:02:13
Speaker
get today. But before we do that, we should take a run through of some of the news and sounded from your email, Neil, like you're actually enjoying some of the US news you're reading.
00:02:23
Speaker
Well, yeah, I was thinking of a segue, you know, on the weather chat, thinking of somewhere that's kind of nice and warm. I was interested in, you know, the kind of drama miniseries that seems to be the University of California's saga around online degrees. And it sounds like that band's been overturned.
00:02:42
Speaker
Well, yeah, and it's turning into a governance story as much as anything, and just for people who aren't following. So there was a study that was commissioned several years ago about online education, and should the University of California system, how should they treat it?
00:02:59
Speaker
And the system faculty senate a year ago this month, as a matter of fact, clearly decided the smart move was let's ban all undergraduate fully online degrees. It's not the same quality as on campus. Let's stick our head in the sand and ignore it. And then what Neil, what you're referring to is the Board of Regents just said, that's not your decision to make. And so they overturned the faculty senate.
00:03:26
Speaker
And now there's a fight teed up on governance around that. Yeah. I mean, I think we touched on the fact that this was, you know, as much becoming as much a governance issue as it was about online education. But as I was reading the Inside Higher Ed piece, it just it made me think of, A, this feels a bit like dysfunctional families at the dinner table falling out over stuff.
00:03:55
Speaker
And B, I was thinking what would, if someone kind of say didn't have any idea of how higher education governance functions, I wonder what they'd think about you know the way that these two bodies in terms of governance kind of interface and the way it all works. Would they be scratching their heads and thinking this seems very bizarre the way things are done?
00:04:15
Speaker
Well, I'm trying to figure out if we're using the dysfunctional family metaphor, who's the drunk uncle in this case? Yeah, I mean, I don't think I want to go there, but it's interesting. I mean, another interesting facet of the story is that research piece, because I know that in the piece they were talking about the fact that they felt that the kind of research around the efficacy of online education was focused really narrowly in on private providers, and they didn't think that was kind of reflective of their context.
00:04:44
Speaker
I mean, interesting layers to this story, but that's my favorite US story of the moment.
Higher Ed Financial Controversies
00:04:51
Speaker
So I thought it would be good to touch on that. And I don't think it's over either. I mean, you're going to be reading more about this, I suspect. Well, you know, I feel like just in terms of like a box set, I'm looking forward to the next installment and seeing what comes in twists and turns. Well, the University of Arizona says, yeah, you want a dysfunctional family. Hold my beer.
00:05:14
Speaker
There are all kinds of lawsuits flying and because the chairman of the Board of Regents or Board of Governors is looking into legal action against the head of Faculty Senate and the governor of Arizona is really, really mad. And my new hero, I'd like to point out, because she slapped them down really hard. Now she needs to back that up. Somebody's got to go if you ask me. But it's
00:05:43
Speaker
This can't be a coincidence, can it? Actually, let's go ahead and mention it. University of Idaho buying the University of Phoenix was supposed to be approved in the next month or two, and news came out today that Morgan shared that the president of the University of Idaho, the one who was leading the deal,
00:06:03
Speaker
that they paid his former law firm 7.3 million dollars in consulting fees on the University of Phoenix deal. So on the specific deal which leads into its strategic online education and it's a dysfunctional family and governance issue. I mean all three of these. At least the
00:06:31
Speaker
seeming semblance of mispropriety or it doesn't look good. Were you picking your words so that we don't get sued? Either that or I'm getting old and I'm losing my words. Anybody can be the judge there.
00:06:50
Speaker
You know, we don't have, I don't think we have at the moment any such privilege of kind of internal wrangling soap operas. We have very much an extenuation of the extension, sorry, of the kind of existential crisis in higher education. So one of the things that's been really big in the last couple of days is just, I guess it ties in to a certain extent with some of the things that we talked about in previous podcasts around signals for
00:07:17
Speaker
demand for higher education so we've just had you know a bunch of data whether it's been surveys from business schools over here or data on kind of visa applications but the signals aren't good around our UK universities recruitment of international students which if you're not familiar with the kind of travails of UK higher education is where
00:07:42
Speaker
where institutions have had to go because the money that they get from domestic students through undergraduate fees isn't bolstering them. So universities over here have had to increase international student recruitment where they get higher fees. But because of what our government's doing,
00:08:00
Speaker
around you know just generally negative narrative around international students but also measures like a ban on dependence being able to come to the UK with those that are studying that they know you know that's kind of creating a big decline in international students so it seems particularly at postgraduate levels so
00:08:19
Speaker
extension of kind of challenges. So this is a really kind of, yeah, worrying sign for the financial sustainability of university. So maybe we would wish for some kind of internal drama rather than, you know, these really negative negative signals that we're getting. Yeah, if we go the American way, you need to focus more on lawsuits and getting it deeply personal. I think that would help you guys out get through this crisis.
00:08:47
Speaker
Yeah, yeah. Well, I mean, I think our main source of wrangling will be between universities and, you know, maybe the government. So, you know, I don't know how that would work in terms of lawsuits, but, you know, those are the they're the kind of persona non grata at the moment, I think. I guess another piece
Trends in Online Education
00:09:05
Speaker
of news that definitely chimes with things that we've covered recently, thinking especially around that kind of little mini series on where things were at was I don't know if you guys saw the key path results.
00:09:17
Speaker
So that seemed to, to a certain extent, back up what we'd said around signals for growth in online education. I think they've not seen, I don't think it's massive growth, but their growth in enrollments over the last half year of their financial.
00:09:34
Speaker
year and I just I think it's I think one of the things with key path that's quite interesting is their focus towards health care and APAC region I just wondered you know whether that's going to be a sign of maybe where some companies might think to go in terms of maybe differentiating a bit more directly.
00:09:52
Speaker
Well, I think that they have a strategy. I mean, part of the way I take it, well, the first thing is, despite the problems with 2U, RevShare or OPM is not dead. Coursera is seeing increases. Grand Canyon, KeyPath is seeing increases. And that's one of the ways I interpret that. The other one is, I think they're being rewarded that
00:10:14
Speaker
key path has more of a strategic focus. We fit here, healthcare. We're growing here in Asia. They're being pretty selective and smart business wise. So I know there's more to the story, but you're right. It's increasing demand. Revshare, OPM is not dead. And I think it's a company that's being strategic, being rewarded for it.
00:10:36
Speaker
Morgan, did you actually listen to the call or just? I didn't know. I didn't. I read the report and I did have a quick look at some of the data just on U.S. universities that work with KeyPath. And one of the things that's striking about universities and OPMs in general is that there's like a bimodal distribution. There's ones that earn a boatload of money, like in the 20, 30 million dollar kind of
00:11:04
Speaker
range which they pay as a rev share to the to the OPM and then and then at the other end of the spectrum they're pretty small you know like a million dollars but but but key path is is somewhat different than that you know we're not talking a large n here but that there's sort of more in that five million dollar range so their enrollments per institution at least in that very small sample that i that that i can i can get access to is
00:11:30
Speaker
is pretty solid, which is sort of interesting given that they're focusing on things like health care. You know, they're not broad spectrum. Let's go in and start 23 programs or whatever.
Challenges of AI in Assessments
00:11:40
Speaker
To me, it's smart management. I wouldn't be surprised to find out that sort of this, we're not bimodal. We're going for these mid-sized clients. That might be part of their differentiation as well.
00:11:51
Speaker
Yeah, I think one of the things I always enjoy when I'm reading key path reports is the way that they refer to their programs as kind of vintages. I think that's a, yeah, I don't know. It's an interesting, interesting touch. But yeah, I mean, I agree with you, Phil. It's interesting to, it feels, you know, a good rationale for their strategy to kind of focus in on those kinds of things. And, you know, it's kind of positive signals in terms of the success of that kind of
00:12:19
Speaker
Playing out so we'll we shall see we shall see Outside of you know, the university of california you mentioned university of arizona. What else has been happening over your side of the pond? Oh my gosh. Well I'm my head is spinning. I'm actually Morgan and I haven't talked about this i'm wondering if we need to do another podcast just cover the news there is just so much news happening left and right that every time you
00:12:45
Speaker
want to write about something, there's a new development. So obviously, the University of Arizona and that shit show is getting worse and worse. University of Idaho, maybe they're not going to be able to buy the University of Phoenix and there was a
00:13:02
Speaker
Something came out just yesterday, Morgan, right, about that there's a legal analysis saying that the University of Idaho board was not allowed to create a corporation to do this in the first place. Oh, I didn't see that, but yeah, another wrinkle. I mean, there just seems to be wave after wave after wave, like the folks in Idaho who weren't in the room to strike the deal or sort of like, hey, you know, this is not above board.
00:13:26
Speaker
Yeah, so we have that going on. We have the Negreg negotiated rulemaking that's looking to not up and online education as Morgan wrote, it's not going away, but they're certainly trying to kill innovation and trying to hold it back. All in the interests of reducing costs, they're making people jump through lots and lots and lots of hoops, including taking attendance in online courses.
00:13:52
Speaker
Yeah. Yeah. So it's, we have all that stuff going on and we're going to recover the generative AI right now. It's just, we're, it's coming at us from all angles over here is relevant ed tech and online education news. I suppose I should be happy for that. Cause we're never bored for a topic, but my head is spinning lately. The author Tolkien's, you know, a lot of the rings.
00:14:21
Speaker
once described the news as being nothing but murders and football scores. So I'm trying to think of an equivalent in terms of higher ed news. It's murders of universities and personal scores or something like that. I think one other piece of news which came out in the UK which is
00:14:41
Speaker
you know very relevant segue into our topic was the University of Glasgow who basically for their life sciences students what seems like a quite an abrupt announcement that they were moving from
00:14:53
Speaker
online open book exams to close book in-person handwritten exam that would be the worry for me handwriting for anything longer than 10 minutes and I get an ache now but you know students are kind of complaining about that partly because of a short notice but partly because of the kind of sense in which that's a kind of backwards step but
00:15:15
Speaker
I think that probably nicely frames what we wanted to talk about today. Before we go in the gen AI aspect of it, let's cover the other part of the story. That was stunning to me. What did they get? They gave students two weeks or maybe they gave them a full three weeks notice.
00:15:33
Speaker
Yeah, I think it was around that. It might have been a bit longer. I wasn't sure if it's two months. I have to check. Okay, but it was within the middle of the term. Yeah. And so students, so for example, one student had a great point saying, listen, I haven't been used to and I haven't been planning to recall facts. I haven't been studying so that I can recall facts because it's been open book and I focus on how to build an argument.
00:15:57
Speaker
And now you're throwing this at me, and I have to take the exam. And I can't go back and change how I was studying for this course. So I mean, there's a very strong non-AI component to the story as well that's really stunning, to be quite honest. And somebody had a great point. If this is about AI, why such a rush to make a draconian change with very little notice? They're really shooting themselves in the foot.
00:16:23
Speaker
Yeah, and I guess sometimes through these pieces, because they're looking for the voices of students, sometimes there's a mixture in terms of the balance, but similar to you, Phil, when I read it, these are all very legitimate things. Short notice. Gen. II isn't that new. You haven't prepared us for it. Yeah, it seemed a bit of a worrying kind of story. And students flagging things like mental health concerns, which that is a big
00:16:52
Speaker
topic and a big issue over here as well so yeah an odd one an odd one worth saying that it was life sciences and i don't think it was a policy for the whole of the university but nevertheless i guess it kind of represents the tis that universities are getting themselves in around assessment but i was going to kind of throw it over to you guys to say you know this is one example of how universities over here are responding you know what's what are you seeing in terms of
00:17:22
Speaker
how universities are changing and responding in terms of assessment on the back of what's happening with Gen AI. They're forming committees? Yes. Yeah, we haven't seen or at least I haven't seen that hard and concrete of an immediate tangible change is what's happening. In part because most of them didn't go the open book
00:17:49
Speaker
route to start with, at least not for most exams at the undergraduate level. And I must confess, when I read that story, I immediately went back to my days in grad school studying for my PhD. I remember we still always have open board exams one time.
00:18:03
Speaker
the faculty member, a guy called Phil Shively handed out, and this is a good graduate program, he handed out our exams, we had 24 hours to write something and get it back to him. He said when he was in grad school at the University of North Carolina, they had the same system. And so he and a friend picked up his exam paper one noon and said, oh, we've got a lot of time, let's go to the bar. Anyway, they got home at six in the morning.
00:18:29
Speaker
with a terrible hangover and they were not having slept and they had to write the exam in six hours so we
AI Ethics and Bias Concerns
00:18:37
Speaker
took that as guidance so we did we we tended to do the same sort of thing roughly with our open book exams so I was immediately just mourning their ability to drink on the job
00:18:49
Speaker
Yeah. But you are right that, yeah, we haven't, you're right, we haven't seen that type of shift to primarily online open book exams either. Yeah, it seems like you guys are swinging the pendulum a bit more than what I've seen over here. I think there's a lot of studying, but not much concrete detail is the way I would summarize what's happening over here. I think there will be changes, but right now it's being debated.
00:19:18
Speaker
Interesting. Yeah, I mean, I think one of the one of the sort of interesting things over here is that there was that move to, you know, open book.
00:19:27
Speaker
online, you know, people having, you know, not less time bound assessments on the back of the pandemic. And I think broadly speaking, most people over here thought that that was a good move, and it was moving away from the traditional exam. And so the pandemic was a really good catalyst for that. But along comes Gen AI. And suddenly, you know, that move
00:19:49
Speaker
potentially presents a different challenge because of what's happening. But similar to you guys, there's definitely lots of committees. I'm sure I was on a call recently and someone was telling you about the working group or the committee that had been set up and that's pretty standard kind of practice.
00:20:11
Speaker
I'm just interested as well in terms of what guidance is coming out for universities. So we have bodies like the Quality Assurance Agency that put out guidance. But if there's not much happening or coming to the surface in terms of what's happening over there, generally, where is the guidance leading universities? What is it saying in terms of assessments and what they might do as a response?
00:20:35
Speaker
I would I would say most places are sort of just coming up with their own. I haven't seen centralized guidance from accrediting bodies or or other kinds of things. So places are trying to develop their own, their own guidelines for how to deal with AI. But mostly it, it is dealing with sort of more of the panic and not so and
00:20:58
Speaker
not sort of dealing with sort of the central question of how does this get to the heart of the matter? And what do we need to change? That's my sense of it. I would add, I think that we have there's the whole DEI angle, I think is where you get a lot of stuff is without getting to the heart of the matter, understanding the implications. We don't know what's happening there, but we're darn sure we're going to lecture everybody about make sure we do this fairly to every demographic group and think about access and
00:21:28
Speaker
So it's getting framed very much in that aspect. And I see concrete, I think I've seen concrete guidance coming out on that angle. But again, it's like Morgan said, we're not actually, I'm not seeing anything concrete on the heart of the matter. And so for example, how does this change what we're assessing students on? Should it be on their thought process, constructing an argument or recall of facts?
00:21:54
Speaker
those core elements that's sitting right there. I'm not saying much in that angle. Yeah, that's interesting. Like, I think our guidance has been, you know, on the whole kind of helpful, but I think it's the guidance that I've seen, the kind of general feeling that I'd have towards it is it's kind of a guidance for a period in which people are not really sure what to do. And so there's guidance like talk to your students about Gen AI,
00:22:24
Speaker
think about...
00:22:26
Speaker
students kind of declarations when they submit assignments and kind of you know that kind of honor code type side of things things like review your policies but I guess one of the one of the interesting things has been maybe a nudge towards things like observed assessments and face-to-face kind of presentations but one of the kickbacks over here and I think it's completely legitimate kickback is you know
00:22:56
Speaker
particularly programs that are scaled, that's not the model. We can't support moving assessments to live kind of real time. A lot of the debate is that the essay is a really cheap and well scalable form of assessment. And actually, if we break that and everything becomes a presentation or an observed assessment,
00:23:21
Speaker
then our model doesn't support that. And I just wondered if those kinds of things have been discussed over there and what you thought about how that might relate to the US model of higher education too. For our American listeners, I just want to intervene at this point to say that when
00:23:37
Speaker
When Neil says kickback, he means pushback. In the US or what you get in the state of Illinois. That's a good point, yeah. There might be that happening as well, Morgan. I don't know.
00:23:53
Speaker
Yeah, it does it does break the model, you know, like, whenever I think about assessment and the challenge of technology and both online learning and AI technology, it reminds me of the light bulb joke, you know, how many it's how many sociologists does it take to change your light bulb? None, because it's not the bulb, it's the system that needs changing.
00:24:14
Speaker
Yeah. I think that the scale, you mentioned scale. I think that's where a lot of the challenge is because I've seen individual courses, departments, disciplines, who even before AI have started to figure this stuff out. So for example, in music education, you know, the way that you actually get feedback and get assessed, I've seen courses where students are doing it on Zoom or, you know, some video conferencing.
00:24:42
Speaker
And it, whereas it used to be, okay, everybody show up, sign up for a time on Thursday or Friday in this room. And, you know, you have all the logistics around it by simply going to quick zoom things where you can get video, seeing what students are doing.
00:24:59
Speaker
and video feedback of it. And then that worked great. So some of these scaling things, they're very solvable. The problem is it's scaling within a discipline. We haven't had broad based change, I guess is what I'm saying. And I don't even hear a whole lot being talked about it. But I personally think that there's going to be a long period of exploration of observation based assessments and people trying out different models. I guess this is
00:25:29
Speaker
the Pollyanna or positive view that I have, that I think that you're going to see a lot more exploration of these assessment types over time, but not at the, oh, we can broad base apply this across a university type of lesson. Yeah. And, you know, on that kind of more positive slant, do you think that this is an area that online education can lead the way in because, you know, you're talking about a virtual method of, you know, live assessment
00:25:58
Speaker
Because I have to confess, I haven't seen much movement either in the online space. But that is something that the Zoom assessments equate to that a little bit. Well, yes, I do think so. And if you look at Southern New Hampshire, I know everybody talks about it, but typically about their massive enrollment. But when they were also leading, they had the College for America, the competency-based education. And they do build that into a lot of what they do.
00:26:28
Speaker
And one thing they did is their assessments were very much project-based assessments. You need to submit a paper. You need to do a research project. You need to do a video assignment. And only occasionally was it multiple choice basis. So if you look at that study, I think that online education can be a leader in this area if people are willing to pay attention to the right part of the story.
00:26:55
Speaker
And Open University of Catalonia, they were pretty much completely project based, weren't they? And fully online. Yeah. So, because I remember talking with them about their LMS and that was part of the challenge. Oh, that's part of why they customize the LMS for that as well.
00:27:11
Speaker
Yeah, so I guess we're saying not only, or I'll put it in my words, not only yes, can online education pave the way, we already have seen a lot of historical examples that are available today to show that you don't just have to do the same type of static assessments you've been doing in other places forever. So we already have case studies to get started on this path.
00:27:37
Speaker
I don't know about you guys, but I'm frustrated at the moment by the way people are using some of the newer technologies, not to solve the problem, but to just double down on a bad model in some ways. So, you know, for example, seeing people using AI, not to come up with some different ways of doing assessment, but just to, okay, here's some content, come up with some multiple choice questions about it, which are probably going to be bad.
00:28:03
Speaker
or using AI to grade essays. I just saw grade in the UK got some additional funding and I'm sure they do a great job and things like that but you know it's like doing that.
00:28:16
Speaker
I think, you know, and I was struck, I recently sort of went through 20 to 30 of the top venture capital firms funding in tech. And there's very little about assessment in there in terms of what they're funding in their portfolios right now. And one of the big, exciting ones, Imbalos from a few years ago was actually acquired by Roblox and has disappeared. So, you know, it's a bit disappointing to me. Do you think that speaks to
00:28:46
Speaker
you know, a product aiming at what a university might want rather than offering something more innovative. I think that is a very key thing and I was just thinking about that in terms of a different thing yesterday, you know, about listening to a report of an organization that's helping universities go online and they said, you know, they're going to local
00:29:07
Speaker
local businesses and asking them what they want and then building programs around that. And I think there's a lot to be said for that, but there is the sort of faster horse problem. They're not necessarily going to be that insightful in terms of seeing the future. And I think you need to get input, but you also need to really think in a complicated way about what's coming.
00:29:28
Speaker
Going back to your frustration on the way AI is getting used, to me, there's a difference if we're talking summative or formative assessment. Yeah, I would exclude the formative there, because I think that's a nice usage. Yeah, OK. Well, this is bad. I was trying to get an argument going, but it turns out we agree. No, anything that gets more formative usage or more formative assessment in there, because I think that's where too often,
00:29:57
Speaker
too much assessment is summative and it's about, yeah, okay, spit out everything you know and then forget it. Whereas I think formative is much more pedagogically rich in sound. And AI fits right in and you can do more of it, which fits
00:30:13
Speaker
Well, it's disappointing. We agree on that topic. I thought I had a good argument there. Agreement has broken out in the podcast. I'm so sorry. I'll try to think of something provocative to ask. I think it's a good point. I kind of feel like almost there's a little bit of consensus around, well,
00:30:33
Speaker
the way i would describe it and kind of my thoughts on it is more of that kind of assessment as a progressive piece throughout a course and too often assessment is the kind of full stop at the end of the end of a course and i think you know higher education has lived on that midterm end of term kind of model for assessment and not everyone weaves those things throughout the the course and i think that's one of the i think we're all in this kind of state where
00:31:01
Speaker
you know, we're still trying to figure out what this means and I think that's why, you know, to your point Morgan around.
00:31:09
Speaker
what Ed Tech's doing, it's still hard to think beyond our reference points around what we do, but I think that's kind of the compelling thing for me. And I think certainly over here, that would not necessarily break the model, but it would change the model to kind of move to something like that, or even to just move to some of the more in-person and kind of live assessments. But the one other kind of aspect of this, and maybe this is getting a bit
00:31:39
Speaker
esoteric though is just thinking about the way in which this challenges the way that we think about stuff so I read a really interesting paper recently from Jess Liu from the University of Hong Kong and what she'd done is basically she'd analyzed a bunch of top universities policies around assessment and AI and the lens that she looked at it through was you know what is the problem that they're trying to solve here and a lot of it was framed around originality
00:32:08
Speaker
and AI not somehow subverting students submitting original work. And I just wondered what you thought about that. Have we got a slightly odd view about originality? Because we're all drawing from different ideas. Yes, we totally have an odd view of originality as all the ugly fights going on in the US right now.
00:32:34
Speaker
The way I look at it to pull back in the metaphor, this feels like, okay, right now we're in the period where the drunk aunt and the drunk uncle are over yelling and arguing about plagiarism and cheating and academic integrity. And we have to actually wait for that to die down so that we can have a nice dinner and talk about planning a family vacation and where should we be going and what would be a good path forward. So I feel that we're in that period. So the optimistic view
00:33:04
Speaker
is the drunk aunt and the drunk uncle are going to go fall asleep or give up at some point and we can have real conversations. The pessimistic view is going back to Morgan's, the light bulb, the pessimistic view is I'm hoping for a beautiful light bulb when the system itself is not going to change.
00:33:24
Speaker
Just going back to the originality thing, I always, and I'm probably getting into dangerous ground here, because both of you know a lot more about music than I do, but I think we tend to think of academic work as pristine sprung from nothing, whereas it's much more like jazz or hip hop in that it builds on things and it's fluid and that sort of thing. So I think, yeah, a lot of the paradigm is wrong.
00:33:51
Speaker
I'm so glad that you mentioned music because when I think about this I've thought about music and it's slightly different in terms of building on stuff but it's more about the idea that in a way you kind of have to you have to sort of ape an idiom to be able to get to a stage of originality so I kind of think about you know look the Beatles had to do Please Please Me before they could do Revolver and
00:34:13
Speaker
you know the Beach Boys had to do you know fun fun fun before they could do pet sounds and those two earlier albums are really influenced by you know they copy things like you know Chuck Berry's Sweet Little 16 I think one of the Beach Boys ones so I'm going off on my my hobby horse here but that's another aspect of it that actually learning is also not just about originality it's kind of being able to embody the idioms as well so you can get to that stage I think
00:34:44
Speaker
So we're facing a real opportunity, in general, in higher ed anyway, about how do you think about assessment. So if we had this conversation two years ago, pre-open AI type of issue, I think we could have had the same discussions
00:35:01
Speaker
the difference being generative AI has brought the topic to the fore. And so you get the Glasgow, you know, and you get the news story, a lot of it's overreaction, but there's also a real new opportunity that just didn't exist before. And we're at this point of inflection around how we do assessments. So we already know the things that need to change, right? The,
00:35:25
Speaker
one time or two time summative assessments instead of ongoing assessments and learning process. Not thinking about just plagiarism but how do you learn the idioms and move forward. So we've known a lot of these things where I think we're at is the inflection point driven by AI that I'm not saying it's a magic bullet that's going to solve them but it
00:35:47
Speaker
could force us to think about them differently and do things with a lot less overhead than we would have had to do before if we're going to try to solve this. So I guess I'm being optimistic again. Yeah, I feel like there's similarities between the pandemic in that something has happened that's forced higher education to confront something, that it kind of knew that maybe it wasn't
00:36:12
Speaker
wasn't necessarily on board when or it needed to kind of engage with more seriously and yeah I guess that's difficult in the moment and you get the kind of the responses like you know we're gonna ban chat GPT or we're gonna go back to in-person exams and you know that's the
00:36:28
Speaker
I'm going to take this Phil. That's the kind of drunk uncle and anti-phase maybe. This is, you know, maybe there needs to be a revision of the Gartner hype cycle to kind of tie in with, you know, drunk, sorry Morgan. We just knew somebody who was involved with Gartner. To, you know, to add that kind of colloquial Phil Hill flavor maybe. The other thought that I had that it's kind of a little bit more
00:36:56
Speaker
little bit more bigger picture is I guess in terms of you know Phil you shared something around Google Gemini because I think another aspect of this debate is also about how we think about bias and that was a really interesting example of that. I've seen little snippets but it seems like a kind of a politically motivated perhaps. Fusion.
00:37:18
Speaker
I would write, okay, but it's deep seated. So, you know, my view of what's happened is once Gemini came out, the image generation came out as somebody had a great description. Nate Silver said something about it felt like this is what you would think about if you're politics, if you're in the middle of, you know, Occidental College and a sophomore seminar.
00:37:40
Speaker
But it's not just image generation where it could not do any white male generation, even if you tried. Nothing was historically accurate, but it's affecting their text. It gets down to the whole core of who Google is because they're building Gemini into the whole company.
00:37:59
Speaker
And what's happening, it's not the politics have infused. It's clearly the typical engineer or person in Google. They all they have groupthink like out of control groupthink. And that's where it got injected. But the question is sort of broader than that. It's if we're going to use on our topic today, if we're going to use generative AI as part of assessment,
00:38:25
Speaker
and use Gemini today with all of the things. Now you have a true bias coming from the algorithms that somebody else has injected in there and you have no idea, like here it was sort of predictable, but somebody else injected these ideas. You don't use the educator, you the course designer have no control over what the right answers are that are coming in and they're all hidden for you.
00:38:51
Speaker
It almost feels like, I hate to say this, it feels like a very strong argument for open source AI. And the reason I hate saying that is sort of backing up meta and their approach. But I think there's a real, do we even know the biases built in? And that's the question that I think is really key from my standpoint. I think it's an argument against black box things of any kind here, which we see a lot in student success as well. Great point.
00:39:21
Speaker
Yeah, I mean, I think the thing for me also as well is that the way that we've tricked ourselves a little bit about bias that
00:39:29
Speaker
that this is purely a problem that's written through Gen AI. I think you see bias in research papers, you see bias all over and actually, arguably higher education's role is to provide the basis of knowledge that you can navigate all of that. And I think that's one of my frustrations around the debate on assessment or anything like that is that
00:39:57
Speaker
We've got this sort of slightly naive view of things like originality and bias. Yeah, we kind of need to get engaged with that. But the Google Gemini thing was a kind of a big example of that. And I think if I'm right, the thing that you sent Phil kind of seemed to point that that's having a negative impact on
00:40:16
Speaker
Google in terms of stock price and things like that as well. Yeah, I'd like to believe that you would love to hear from my perspective I would love to hear that there was so much pushback on this that people realize this isn't just around Gemini this gets to a Lack of trust of what Google does and Google is everywhere think of all the Chromebooks in education Think of everything that they're building this into
00:40:39
Speaker
So I would love to believe that it's going to hurt them, but I don't have any confidence there. I mean, typically when markets deal with this type of thing, it starts out and you say like, yeah, this will teach them to do things more open. And then you check back a month later and you're going to say, whoa, wait, their stock is back above where it was before. They're getting rewarded for it. So I'm actually negative.
00:41:01
Speaker
I'm pessimistic that market pressures are going to force a change, but I hope that it happens. I wouldn't say universal. There is a huge pushback on this, so hopefully it gets enough attention, but I'm just pessimistic that financial performance of the company is going to be it. I'd love to be wrong there.
00:41:23
Speaker
Yeah, and I guess the salutary tale that probably circles back to online education and ed tech is don't rush something out when it's not ready. Or, my gosh, didn't somebody have an idea of testing some of these features ahead of time? So it might involve not just don't rush it out, but maybe do a little bit of real world testing.
00:41:47
Speaker
And I think that if you look at the piece that Nate Silver wrote, but then also Ben Thompson, I've seen some others that are quite good. It's like the people who were testing this had the same mindset. So the way they thought about testing this before they went, the real world was completely divorced from people having different ideas. So they didn't know to test this. So, you know, there are a lot of lessons here. I'm hoping there's real change because of it, but I think it's a real subject. Take politics out.
00:42:17
Speaker
It's what should we be doing as higher education around assessment? We better have some control over the validity of the assessment and where that's going. And that will be a problem we need to address with Gen AI. But as you point out, it's not the first time we've had this challenge, but hopefully it's a wake up call that this needs to be part of the committee discussions that we're having.
00:42:41
Speaker
While we're on the committee discussions, Morgan, put you on the spot. Are you going to go to San Antonio based on the email I forwarded to you? Yeah, I was going to talk to you about it afterwards. Not at all. Talk it on the podcast. There's an AI-based system discussion at the University of Texas and it's happening next week and we've been invited to go and attend.
00:43:03
Speaker
listening to that. So I'll find out if Morgan's going to actually attend. I was just bulking at the last minute airline prices. And just for our listeners, Phil is the only person in the world who thinks I'm cheap. Everybody else thinks that I have no sense about money. Listen, don't worry about the airline price. It's an important topic for the podcast. Then I'm there. We have an excellent budget on this podcast.
00:43:30
Speaker
Actually, we don't have any budget. But in any case, we're soliciting topics for us to research and we're now soliciting invites to AI committees and working groups in university.
00:43:42
Speaker
Yes. Well, what I was going to suggest is this is almost like we talked about reader request, or listener request. Morgan, you can report back to us next time we record or the time after about what were the discussions. Are we starting to hear people talk about the real core issues of AI and how it's going to impact education? Are we reading it right? And I think it's a great chance to get a good view of that and to get some great barbecue.
00:44:11
Speaker
Yes. I was going to say, I'll report back on the AI and the podcast and the barbecue in the newsletter. So, uh, yes. Texas barbecue, by the way, is the best note. Don't come at me with your North Carolina and certainly not with Kansas city and stuff. Texas is by far the best. I feel like you've got the, you've got the fight you've been spoiling for that Phil. I think she's going, she's going to go to the conference. She's going to do well. We agree on the barbecue though.
00:44:39
Speaker
Yeah, we agree on little, but on barbecue and on formative assessment. Yeah. Okay. Well, with that in mind here, we did some real world planning on this podcast. This has been a great topic, and I think that we're going to need to keep tracing this as we move forward, because I like talking about Gen AI in the context of assessments. I think there are other topics we could do the same thing on Gen AI in this context.
00:45:06
Speaker
So I enjoyed the conversation. I appreciate you leading this, Neil. And for all of our listeners out there, obviously send in the request, and we are actively soliciting. Oh, final thing before we hang up. Neil, put you on the spot. Have you fixed the leads conference so Morgan and I can fly over for that?
00:45:26
Speaker
Oh no. You're adopting a continuous assessment model with me and I keep failing every hurdle. I haven't, I haven't, I haven't. You've, you've shamed me twice now. I'm going to have to just get in touch. Okay. So I'm going to, before the next one, I'll have an update. I promise. Okay. Hey, well, we hope this happens, but it's great talking to you all and we will see you again next week.