Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
95: Deciphering "Learning Loss" w/ Akil Bello
12 Plays4 years ago
Today, we are joined by Akil Bello. Akil is a supplemental education and test preparation expert. He's launched two companies, developed test preparation programs, and trained hundreds of instructors. He was the founding partner and CEO of Bell Curves, a test prep company based on community partnerships, worked for The Princeton Review, and now is the Senior Director of Advocacy and Advancement at FairTest.
Akil and I talk about the advent of "learning loss" after pandemic schooling, the way that testing companies are using this term to generate more tests and test prep software, what was lost in the pandemic, and what we can do as teachers to build back better.
GUESTS
Akil Bello, Senior Director of Advocacy and Advancement at Fairtest, founding partner and former CEO of Bell Curves, and contributor on test equitability, learning loss, and much more
RESOURCES
Recommended
Transcript
Introduction & Acknowledgments
00:00:06
Speaker
Hello, and welcome to episode 95 of our podcast at Human Restoration Project.
00:00:10
Speaker
My name is Chris McNutt, and I'm a high school digital media instructor from Ohio.
00:00:14
Speaker
Before we get started, I wanted to let you know that this is brought to you by our supporters, three of whom are Emma, Steve Peterson, and Alicia Houselak.
00:00:21
Speaker
Thank you for your ongoing support.
00:00:23
Speaker
You can learn more about the Human Restoration Project on our website, humanrestorationproject.org, or find us on Twitter, Instagram, or Facebook.
Guest Introduction: Akhil Bello
00:00:45
Speaker
Today, we are joined by Akhil Bello.
00:00:47
Speaker
Akhil is a supplemental education and test preparation expert.
00:00:51
Speaker
He's launched two companies, developed test preparation programs, and trained hundreds of instructors.
00:00:56
Speaker
He was the founding partner and CEO of Bell Curves, a test prep company based on community partnerships, worked for the Princeton Review,
00:01:02
Speaker
and now is the Senior Director of Advocacy and Advancement at Fairtest.
Learning Loss Post-Pandemic
00:01:06
Speaker
Akhil and I talk about the advent of learning loss after pandemic schooling, the way that testing companies are now using that term to generate more tests and test prep software, what was lost in the pandemic, and what we can do as teachers to build back better.
00:01:21
Speaker
So the learning loss narrative is fascinating.
00:01:26
Speaker
So the best I can track it is you actually have to go back to Summerslide, if you remember those phrases, right?
00:01:33
Speaker
Which, again, I tried to track the history and where these things all came from, right?
00:01:40
Speaker
Somewhere along the way, they tried to measure...
00:01:44
Speaker
how much students lose during the summer.
00:01:47
Speaker
That was driven largely by using test scores from spring to fall.
00:01:53
Speaker
And so what they're measuring is the spring to fall test score prediction of score change.
00:02:00
Speaker
And that's where things get really weird, right?
00:02:02
Speaker
Is it an actual score dip or was it a prediction?
00:02:05
Speaker
So they're using predictions of what's happening and then they're trying to turn it into months of loss.
00:02:12
Speaker
So it started as summer slide, it became summer learning loss.
00:02:15
Speaker
And then during the pandemic, they picked that narrative up and then just continued it and became learning loss and then it was turned into unfinished learning.
00:02:25
Speaker
So it's just, it's,
Validity of Test Predictions
00:02:26
Speaker
and I think that if we look at it from a non-cynical point of view, essentially it's an attempt to quantify what happens when the normal cycle and schedule of school is disrupted.
00:02:41
Speaker
That makes sense and that's cool.
00:02:43
Speaker
It makes sense.
00:02:44
Speaker
Let's see if we can quantify and identify places where and how much impact it has by disrupting the normal cycle of school.
00:02:54
Speaker
However, and in all things education, it seems to come back to bubbles.
00:03:02
Speaker
However, the measurement of it was a test prediction.
00:03:06
Speaker
It was largely driven by test publishers.
00:03:11
Speaker
based on their test results.
00:03:14
Speaker
So either they're saying, we are predicting, which was the worst study I saw, where I think was NWEA, and that's actually early in the pandemic, like June of NWEA, and June of 2020, NWEA released a report that was like, oh, we expect this to happen, which is like, whoa, that's where, like,
00:03:36
Speaker
We just started the pandemic.
00:03:37
Speaker
And you're like, oh, yeah, kids are going to lose this much learning.
00:03:40
Speaker
And so that's bizarre.
00:03:43
Speaker
And then also the term has been defined multiple ways.
00:03:46
Speaker
So there's the prediction of basically changes of test scores.
00:03:50
Speaker
There is the narrative definition, which is not really based on research of learning loss, which is when crap is messed up, students won't learn as much.
00:04:00
Speaker
I'm like, okay, cool.
00:04:01
Speaker
I don't know that we need to measure it.
00:04:03
Speaker
We know this.
Critique of Testing Reports
00:04:04
Speaker
We all knew during the pandemic, things weren't good.
00:04:07
Speaker
Lots of places were struggling.
00:04:08
Speaker
We don't need to put a number and dates and months to that.
00:04:12
Speaker
And then there's the use of the term, which almost suggested students were unlearning things.
00:04:18
Speaker
Things they had previously known, they don't learn it.
00:04:21
Speaker
And that's also tied to testing in terms of like, they were scoring this level on the test, but in the future, they're going to score at this other level on the test.
00:04:29
Speaker
which means that they've lost some learning.
00:04:32
Speaker
And to me, that just made me laugh because I've done test prep for 30 years, you know, literally coaching.
00:04:37
Speaker
If they fall out of practice, they'll probably dip a little bit.
00:04:39
Speaker
That's like not hard.
00:04:42
Speaker
That's not learning loss so much as it is lack of practice of the temporary skills required to do well on your multiple choice test.
00:04:52
Speaker
Is that the report that basically also said that it implied that teachers didn't do anything?
00:04:59
Speaker
Like once the pandemic started, it was just like, if no learning at all occurred from March until now, then this is what's going to happen.
00:05:09
Speaker
Our scores are going to drop by 30 points or whatever.
00:05:11
Speaker
Yeah, I've looked at so many of those reports.
00:05:13
Speaker
They all blur to me.
00:05:14
Speaker
But there's definitely some that are like, if no learning, if no teaching happens at all, here's what the result is.
00:05:21
Speaker
If hybrid teaching, I think they also use.
00:05:24
Speaker
And then if fully virtual.
00:05:26
Speaker
So like they created this weird...
00:05:30
Speaker
split of like, you know, there's virtual teaching, there's hybrid teaching, which excludes things like, you know, when you're going back and forth to in-person and not.
00:05:40
Speaker
So there's a lot of nuance that's left out in these reports.
00:05:44
Speaker
I don't remember specifically one that was like, yeah, we're just going to assume no teaching whatsoever, which I'm just like, okay, that's bizarre.
00:05:51
Speaker
Right.
00:05:51
Speaker
And the whole phrasing is problematic.
00:05:53
Speaker
Is it learning loss or is it teaching loss?
00:05:55
Speaker
Yeah.
00:05:56
Speaker
It's also kind of surprising to me that, at least in a lot of the reports that we were looking at, the scores really don't drop by that much, which I don't know, as a teacher who doesn't really concern myself much with standardized tests, it kind of questions the validity
Interpreting Test Scores
00:06:12
Speaker
of the measurement.
00:06:12
Speaker
Because if you're saying that, like, this is absolutely terrible, kids haven't been learning in months, and then the score dropped by three points,
00:06:20
Speaker
Or sometimes stay the same.
00:06:21
Speaker
I'm just like, well, then what is the validity of the test to begin with if we're going to take it by their dictation?
00:06:28
Speaker
And that's a lot of the problems with the narratives around this is that they aren't digestible by most people.
00:06:38
Speaker
And I worry about how policymakers interpret these things.
00:06:44
Speaker
Right.
00:06:44
Speaker
Because often policymakers are politicians and not educators and not researchers and not psychometricians.
00:06:51
Speaker
So when you say a three point drop, does that mean three percentage points?
00:06:55
Speaker
Does that mean three percentile points?
00:06:57
Speaker
Does that mean three scaled score points?
00:06:59
Speaker
Is that three raw score points?
00:07:01
Speaker
Because if it's a five point scale for the test and you drop three points, that's huge.
00:07:09
Speaker
But if it's a percentile scale and you drop three points, that's irrelevant.
00:07:13
Speaker
or not irrelevant, but it is not very relevant.
00:07:17
Speaker
Now, it could also be a huge change from what's been happening from year to year.
00:07:23
Speaker
If a test is fairly stable from year to year, and a one-point drop is almost never seen in that test, then three-point drop is significant in that context.
00:07:35
Speaker
I'm not sure if it's meaningful.
00:07:37
Speaker
What would I expect as a drop out of the pandemic?
00:07:41
Speaker
big.
00:07:41
Speaker
Because I also wonder, you know, if you're making kids tests during a pandemic, how many of them are mailing it in?
00:07:48
Speaker
I absolutely would have.
00:07:50
Speaker
There's no way on God's green earth, my mom's going to say, today you're not learning with your classmates and teachers in school.
00:08:00
Speaker
You're going to log on to virtual school and take a bubble test.
00:08:04
Speaker
That doesn't matter for you at all.
00:08:08
Speaker
oh, I'm getting out of that real fast and
Financial Incentives & Government Role
00:08:12
Speaker
going to play Nintendo.
00:08:13
Speaker
Yeah.
00:08:14
Speaker
Yeah.
00:08:15
Speaker
I mean, we've totally been there at the end of last school year.
00:08:18
Speaker
Our students took 17 tests in a two week period.
00:08:23
Speaker
And the last group of tests were the, so yeah, like state tests, CTE tests, some other one, and then map testing, the one that is being reported out on for this, this narrative.
00:08:35
Speaker
And yeah,
00:08:35
Speaker
From what I could tell, pretty much every kid, if possible, they give you like an alert if they start just randomly hitting buttons, like it pops up on your computer.
00:08:44
Speaker
And the number of students in my room alone that I had to go like, you got to slow down because it's going to lock you out of the test.
00:08:49
Speaker
We're just hitting random buttons because it doesn't matter.
00:08:51
Speaker
It's irrelevant.
00:08:52
Speaker
And I'm sure there's a percentage of kids who just slowed down enough so they didn't get locked out.
00:08:58
Speaker
I've done that.
00:08:59
Speaker
I've done like, you know, whatever assessment is online or something, they're making you wait and read it.
00:09:03
Speaker
It's like, cool, I'll just wait and do something else.
00:09:05
Speaker
And then click the button randomly and then wait some more.
00:09:07
Speaker
It's like, cause I'm not going to give this my attention.
00:09:10
Speaker
Let's turn our attention now to, I guess, the more cynical side of this, which is the financial side of this.
00:09:15
Speaker
You know, what's the concern here with the fact that the testing companies that are selling this narrative, I mean, this is a multi-billion dollar industry.
00:09:24
Speaker
What should we be worried about knowing that and knowing what this learning loss narrative is trying to do?
Quality of Educational Interventions
00:09:30
Speaker
So let me try to go Pollyanna on this first.
00:09:34
Speaker
I'm going to be positive in my outlook.
00:09:37
Speaker
I'm turning over New Leaf here.
00:09:40
Speaker
So let's say that we know that the normal process of schooling was disrupted because of the pandemic.
00:09:50
Speaker
So if we want students on the normal schedule
00:09:58
Speaker
we have had for years, then we have to do something to try to get back on track for the normal process of schooling.
00:10:05
Speaker
Fine.
00:10:06
Speaker
So there's going to need to be supplemental things to what we normally would have done.
00:10:10
Speaker
We can't just do eight to three in the classroom and make up for a year that was disrupted, potentially two years that's disrupted, right?
00:10:20
Speaker
So there needs to be external things and add-ons to catch up, so to speak.
00:10:27
Speaker
Totally makes sense.
00:10:28
Speaker
I'm totally on board.
00:10:29
Speaker
What are those new things?
00:10:32
Speaker
So the government has provided money to do the add-ons.
00:10:36
Speaker
Cool.
00:10:37
Speaker
Let's get some good add-ons in there.
00:10:40
Speaker
So the question becomes, what are the add-ons that are going to make up for the disruptions?
00:10:44
Speaker
Who's going to provide them?
00:10:45
Speaker
Are they of quality?
00:10:46
Speaker
Are they aligned with what students need and want and will participate in?
00:10:51
Speaker
All of those sort of things.
00:10:53
Speaker
So I think that
00:10:54
Speaker
From a theoretical point of view, from a high level, it makes sense.
00:10:58
Speaker
Things were disrupted.
00:10:59
Speaker
In order to catch up, we're going to add on more.
00:11:02
Speaker
Let's choose some things to
No Child Left Behind: Challenges
00:11:03
Speaker
add on.
00:11:03
Speaker
That's sort of the theory that was behind No Child Left Behind as well.
00:11:09
Speaker
We have to catch up.
00:11:10
Speaker
We have to do some add-ons.
00:11:11
Speaker
I ran a tutoring company at the time of No Child Left Behind.
00:11:14
Speaker
The processes that were put in place for me as an entrepreneur to be paid for those programs almost required me to cheat.
00:11:25
Speaker
So, and that's a lot of my concern, like all of the scandals that came out of No Child Left Behind, all of the ed tech miracle working programs that are currently being sold, although they've never been proven or tested or this and that, those are my concerns, right?
00:11:41
Speaker
Is as a tutoring company with No Child Left Behind, the program I remember looking at and thinking about applying to and deciding not to was providing after school tutoring,
00:11:51
Speaker
specifically focused on testing, which was my area, but paid based on attendance.
00:11:57
Speaker
It's like, wait a minute.
00:11:59
Speaker
So this was like, and I think the metric, like in paid based on attendance immediately says to me, how am I going to prove that they were there?
00:12:05
Speaker
Right now everybody has to keep track of attendance, but what if the kids don't show?
00:12:08
Speaker
What if they, what if I send 10 teachers to,
00:12:12
Speaker
And there's a PTA meeting and the kids don't show, can't show, or the kids decide to go to a football game.
00:12:18
Speaker
So we haven't signed the attendance sheet anyway, which happened.
00:12:21
Speaker
Or the attendance sheet is signed.
00:12:24
Speaker
So I never participated in the No Child Left High.
00:12:26
Speaker
I just thought it would, you know,
00:12:28
Speaker
I would be taking too big of a risk to hope to get paid at the end of that, right?
00:12:32
Speaker
Because paying me based on attendance is crazy because I'm going to have to put out all these resources to get people to show up and to have programming in place and materials on the hope that there aren't conflicts after school that prevents the students from showing up.
00:12:47
Speaker
How are these programs implemented is almost as important as are the vendors...
External Factors Affecting Education
00:12:55
Speaker
legitimate, so to speak.
00:12:56
Speaker
And then there's still the question of, do you really want an army of external vendors coming in and working with your kids?
00:13:03
Speaker
Yeah, there's also the motivation component and the engagement component, because the people that come in, especially ed tech, that come in to solve these problems, although
00:13:14
Speaker
I'm sure they probably do increase test scores in some cases.
00:13:18
Speaker
It's like at what cost?
00:13:20
Speaker
Because it's so boring.
00:13:23
Speaker
We've been in so many different circumstances where our test scores at our school are horrendous.
00:13:27
Speaker
They always have been.
00:13:30
Speaker
The solution, quote unquote, has been, well, let's pay $20,000 for this ed tech software.
00:13:36
Speaker
They promise in two weeks, test scores will go up.
00:13:37
Speaker
Sometimes they do.
00:13:39
Speaker
But the kids are bored to death and they hate it.
00:13:41
Speaker
They no longer love math or reading or whatever the thing is that they're testing.
00:13:45
Speaker
I mean, that's part of the challenge, right, is that the external struggles to education impact the performance in the classroom.
00:13:54
Speaker
The underfunding of a lot of schools impact the performance of the classroom.
00:13:57
Speaker
And the solutions that are often developed don't alleviate those core problems.
00:14:05
Speaker
Right.
00:14:06
Speaker
So it's like if you say that there are five external things that are impacting classroom performance and therefore also impacting test scores to put something in place to improve test scores ignores all the other things.
00:14:22
Speaker
So I'm not sure that's going to solve the problem.
00:14:27
Speaker
And also, if you think of standardized testing as impacted by two separate things, one is permanent skills that students were supposed to acquire in their regular school classroom and temporary skills that are unique to timed standardized testing environments.
00:14:46
Speaker
test prep predominantly addresses the latter.
00:14:50
Speaker
Test prep can most immediately, short-term test intentional prep, most immediately addresses the temporary test specific skills.
00:15:00
Speaker
If the problem is predominantly the long-term permanent skills,
00:15:05
Speaker
And Tessper isn't doing so much to address that.
Testing Industry Influence
00:15:08
Speaker
It seems like, sadly, a lot of the funding that we're seeing, we have the Operation Reverse the Laws, which I think is a very funny name, from the Department of Education.
00:15:19
Speaker
I'll put it in the show notes.
00:15:20
Speaker
Future Ed put together this summary of where all the spending is going in every single state.
00:15:25
Speaker
And a lot of it's pretty generic, broad terms.
00:15:28
Speaker
But you see, for many of them, sadly, the number one thing that they're addressing is some kind of a
00:15:34
Speaker
I would call like a test score buzzword, like accelerating learning, academic recovery, learning loss, these different words that basically mean increased test scores.
00:15:43
Speaker
And some of them do put on like funding counselors or SEL or putting in money for local communities, like free lunch, that kind of stuff.
00:15:52
Speaker
Things that are very important that we definitely saw from the pandemic that were exacerbated because of the pandemic.
00:15:59
Speaker
But at the end of the day, kind of what you're alluding to here, and you've written about this is that
00:16:05
Speaker
The way that we're really going to solve a lot of these problems is by fixing all of the other things, like funding local communities, ensuring people have stable whole lives, investing money into, I don't know, community organizations.
00:16:17
Speaker
Because if a kid's coming in and they're struggling, the last thing that they're worried about is performing well on the map at the end of the year.
00:16:24
Speaker
It's just not relevant.
00:16:26
Speaker
Which, A, is not relevant, and B, there's no penalty to it.
00:16:30
Speaker
So what if you don't perform well?
00:16:32
Speaker
Right.
00:16:32
Speaker
So if you're going to graduate anyway, if you're going to, you know, like, so who cares?
00:16:38
Speaker
And I think that's that's good and bad.
00:16:41
Speaker
You know, I don't think graduation should be held up by the performance on one of these standardized tests or moving to the next grade.
00:16:49
Speaker
Right.
00:16:49
Speaker
But then I do understand the concerns of like, then how do you get students to take them seriously?
00:16:55
Speaker
So it's like, yeah, I get it.
00:16:56
Speaker
But there's I think one of the struggles that are existing right now in education, right, is like,
00:17:03
Speaker
It's hard for policymakers to fund stable communities.
00:17:10
Speaker
Let me say that better.
00:17:10
Speaker
It's hard for education policymakers in education law to fund and address things that impact education but aren't directly education.
Simplifying Educational Narratives
00:17:24
Speaker
It almost feels like the infrastructure bill, where people are like, well, healthcare is not infrastructure, and I don't know, internet is not infrastructure.
00:17:33
Speaker
It's like, yeah, what it is.
00:17:36
Speaker
But you can see those who are trying to push back on funding anything,
00:17:41
Speaker
There is a legitimate argument there.
00:17:44
Speaker
I don't know if the motivation for the argument is honest, but I can see the argument that says these things aren't what we traditionally define as infrastructure.
00:17:55
Speaker
That's what often happens in education as well as as we're looking more clearly at the impacts on education are all the social emotional things going on at home.
00:18:04
Speaker
So we have to solve those.
00:18:05
Speaker
You're going to get pushed back on.
00:18:06
Speaker
Well, you know, I can't pay for somebody's home life to be better.
00:18:11
Speaker
Is there a place there for educators to help solve that?
00:18:14
Speaker
Because to me, a lot of this issue just comes from the fact that the testing industry has convinced policymakers that this is the best way forward.
00:18:23
Speaker
And they have a lot of money and their ability to lobby.
00:18:26
Speaker
For better or for worse, I don't think that all politicians are corrupt, but even the ones that aren't corrupt probably think that this is the way forward because that's the way things are done.
00:18:35
Speaker
It's just like this is the thing that people do.
00:18:38
Speaker
Is there a way for educators to disrupt that narrative to try to come up for a fix for that?
00:18:44
Speaker
I think so.
00:18:45
Speaker
I think one, I don't know that it's corruption or incompetence or any of those words.
00:18:53
Speaker
I don't know that it is those things.
00:18:56
Speaker
I have personal opinions that sometimes I express on Twitter about my belief, how connected those things are.
00:19:02
Speaker
Right.
00:19:04
Speaker
I think the piece that we can address is the information.
00:19:07
Speaker
And that's that's part of why I like learning loss just boggles my mind, because what often happens is there's research and to make the research digestible to policymakers and the public.
00:19:17
Speaker
catchphrases emerge.
00:19:19
Speaker
Nobody is going to read the report that says the quantified numeric, the time quantified implications of disrupted school learning over the past 18 months.
00:19:34
Speaker
No one's paying any attention to that, right?
00:19:36
Speaker
But that's really kind of what we're saying, right?
00:19:39
Speaker
Measuring using time, the disruption of tests, the changes in test
00:19:47
Speaker
that becomes learning loss, that becomes unfinished learning, that becomes unfinished teaching and all of those other phrases that are put out there,
Clarifying Educational Terms
00:19:55
Speaker
right?
00:19:55
Speaker
So I think it's important for educators to make sure the conversation is being had better beyond those phrases.
00:20:04
Speaker
I think that shifting from learning loss to unfinished learning is an improvement at least, right?
00:20:15
Speaker
It gets rid of the idea that something was lost because nothing, because nowhere are we saying something was lost.
00:20:21
Speaker
It's really a question of not yet done, right?
00:20:25
Speaker
Which is fascinating to me because
00:20:27
Speaker
Somebody suggested it, and I don't remember where, but somebody suggested it like after the pandemic, we just start over.
00:20:32
Speaker
Everybody starts back to where they were.
00:20:35
Speaker
Which is a great idea.
00:20:37
Speaker
It's like, yeah, we just didn't do that year right.
00:20:39
Speaker
Let's start again.
00:20:40
Speaker
Except we have an age-driven, time-driven situation.
00:20:44
Speaker
requirement in education, right?
00:20:46
Speaker
It actually would really be hard to have everyone pause because there's a whole crop of kids that were born and were supposed to be starting school.
00:20:55
Speaker
So what grade do you double up, right?
00:20:57
Speaker
So there is that problem, right?
00:21:00
Speaker
But to a certain extent, that's almost what we needed to do.
00:21:02
Speaker
That year was done sporadic, if at all.
00:21:05
Speaker
So for a lot of people, we just need a do-over.
00:21:08
Speaker
That's really hard to get driven into policy.
00:21:12
Speaker
So I think it's important for educators to make sure that parents, policymakers, whoever you're talking to is clear what these things actually mean.
00:21:22
Speaker
I'm a parent of a 10th grader and an 8th grader.
00:21:26
Speaker
And being in test prep, I've always, any test they're taking, my question is like, can I see it?
00:21:33
Speaker
Can I see the test?
00:21:34
Speaker
Can I have an actual detailed report?
00:21:37
Speaker
Because I know what's on these tests.
00:21:38
Speaker
I know that.
00:21:39
Speaker
And I also know my kids.
00:21:41
Speaker
Right.
00:21:41
Speaker
And so I know that sometimes like one of my students, one of my kids will get pretty much every two part question wrong ever in his entire life.
00:21:50
Speaker
If you ask him to do two parts, he'll do the first part and he's done.
00:21:53
Speaker
So if the test is entirely, you know, find this, oh yeah, and also that, I know he's not going to do well because that's just not how he does it.
00:22:01
Speaker
It's like, I'm going to do the first thing and then move the hell on, right?
00:22:04
Speaker
It's also how he does dishes in the house, right?
00:22:08
Speaker
So I want to know if the test is entirely two-part or if it's all one part and he actually did know some stuff, right?
00:22:14
Speaker
Which I think testing agencies don't give that information to parents.
00:22:19
Speaker
Right?
00:22:20
Speaker
Right.
00:22:20
Speaker
So I think tests are hard for parents to use because the level of detail provided is almost nothing.
00:22:26
Speaker
Right.
00:22:26
Speaker
You get these vague categorizations of, you know, math, arithmetic skills using numeric digits.
00:22:33
Speaker
And it's like,
00:22:34
Speaker
great, maybe that means something.
00:22:37
Speaker
And who knows what it is.
00:22:39
Speaker
Yeah, I mean, diving into, we were reading the different reports just for trying to understand the learning loss narrative.
00:22:46
Speaker
I feel like you need a doctorate just to dissect some of the paragraphs.
Alternative Educational Approaches
00:22:51
Speaker
And I started to wonder, honestly, like, is it just made up?
00:22:54
Speaker
Like, there was like some things in there.
00:22:56
Speaker
I remember reading, I think it was Illuminate Education had something that was like, they're losing 0.01 standard deviation of learning every day.
00:23:03
Speaker
What in the world does that mean?
00:23:05
Speaker
Like, what is 0.01 standard deviation?
00:23:08
Speaker
How do you lose that?
00:23:09
Speaker
Yes.
00:23:10
Speaker
I mean, that's just fascinating, right?
00:23:12
Speaker
And to a certain extent, it is made up, right?
00:23:14
Speaker
Because basically, so here's the, like, and I actually read the reports and like ignore the math.
00:23:19
Speaker
I accept their math, the calculus and crap that they put in there that like, I don't really know because I'm not a PhD or researcher.
00:23:26
Speaker
But I've learned the language of these things.
00:23:29
Speaker
Right.
00:23:30
Speaker
And so that to me, the point zero one is fascinating because that says they've looked at test scores.
00:23:35
Speaker
They've taken the test score scale and the standard deviation of the test score scale, and they've divided it by the time between tests so they can turn the score into a number so they can quantify it in terms of days.
00:23:51
Speaker
Right.
00:23:52
Speaker
Because somehow it's important to that person to say, you've lost three weeks and a Wednesday.
00:23:59
Speaker
Like, I'm not sure what I would do with that.
00:24:01
Speaker
Like, let's pretend it's true.
00:24:03
Speaker
How is that meaningful?
00:24:06
Speaker
You've lost two full weeks of school and a Monday and a Wednesday afternoon.
00:24:12
Speaker
That's not how teaching works.
00:24:15
Speaker
I never quantify by teaching based on Wednesday afternoon versus Monday morning versus three weeks or two.
00:24:22
Speaker
Like, so it's a weird number to put in front of people.
00:24:29
Speaker
Right.
00:24:29
Speaker
But from their research end of things, I think what they're trying to do, if we look at it from the most benevolent way possible, is...
00:24:39
Speaker
take their test score scale and put it in context that a school-based person will understand, right?
00:24:46
Speaker
It's not meaningful to say you've lost 10 points on the map, right?
00:24:50
Speaker
But it might be meaningful to say this person is three months behind where we thought he should be.
00:24:55
Speaker
And the piece that often gets left out is the end of that, where he thought he should be.
00:25:00
Speaker
So where we thought he should be, where we predicted him to be.
00:25:03
Speaker
It's like, I'm not sure that's meaningful.
00:25:05
Speaker
And are your predictions meaningful when you're in a pandemic?
00:25:09
Speaker
Is there an alternative to the way that we look at what is now framed as acceleration, but also seen as remediation, where instead of looking at this as, hey, throw a kid in a summer school for two weeks, or hey, get them involved on this specific online program to catch up their reading scores.
00:25:30
Speaker
Is there a way that we can look at that from an alternative lens where we do something that's more, I don't know, student friendly, something that students actually want to do that will keep them involved?
00:25:42
Speaker
Because my worry is, is that, I mean, if I were to go ask the students in my class next year, hey, next summer, I guess it'd be this summer at this point.
00:25:52
Speaker
you're coming in for two weeks is we got to make sure your reading scores go up.
00:25:55
Speaker
I'm pretty sure like there would be an actual revolt.
00:25:58
Speaker
Or if I told these kids that have used MimePlay in the past that this year they're going to use a MimePlay again, because it's the adaptive learning suite that we purchased for this upcoming year, there's going to be groans, they're going to hate it.
00:26:10
Speaker
And I'm not entirely confident that they actually are gonna be learning anything.
00:26:14
Speaker
I think that they're just going to skip through that just like they skipped through the other stuff before.
00:26:17
Speaker
Is there an alternative means of engagement knowing that there was disruption, but also recognizing that that disruption isn't going to be solved by just cramming a bunch of new stuff in there?
00:26:29
Speaker
So my immediate response to that is almost at a parent level, which I'm going to, I say that carefully because I know what I'm saying, right?
00:26:40
Speaker
I know I'm asking a crap ton of parents, many of whom won't have the resources, time, blah, blah, blah, to do it.
00:26:46
Speaker
But when I think about reading, I think about there's technical skills to learning words, decoding words, definitions, blah, blah, blah.
00:26:57
Speaker
There is engagement and enjoyment.
00:27:01
Speaker
And then there's the meeting of the two.
00:27:04
Speaker
You're speaking, it seems, more to how do I create more engagement in order to allow practice of reading to be easier?
00:27:14
Speaker
in order to allow their skill with reading, their fluency with reading, to grow through the practice of reading.
00:27:21
Speaker
One of my immediate thoughts was I really like New ZLA.
00:27:26
Speaker
I really like New ZLA because I feel like it's relevant, it's easier, but they also do the multiple choice piece of it so you can get the testing thing in there.
00:27:33
Speaker
I like that sort of theory of take something loose and relevant and
00:27:39
Speaker
mutate it into test-ish so that we can connect these things a little bit better without making it heavy and overwhelming and just rote practice.
00:27:49
Speaker
So as you're talking about things like that, I'm thinking about, is there a summer program where we can just, I don't know, hate read news articles?
00:27:57
Speaker
One of my Twitter pastimes is...
00:28:00
Speaker
hate reading an article where point by point I show what's missing, what they've left out, things like that.
00:28:07
Speaker
And I think that that's a, you know, I started it because somebody just got so many things wrong.
00:28:12
Speaker
It annoyed me.
00:28:14
Speaker
But I also could see kids loving it, right?
00:28:16
Speaker
It's like, wait, wait, wait, this says this about Jay-Z.
00:28:18
Speaker
I know that's not right.
00:28:19
Speaker
And like getting it into something they're more engaged with, right?
00:28:23
Speaker
And it teaches sort of the decoding and to pay attention.
00:28:26
Speaker
So you could create a program around that.
00:28:29
Speaker
I've just turned into ed tech.
00:28:31
Speaker
And that's the problem with it.
00:28:34
Speaker
Also, it's really hard to do with your class of 50 kids where their interests are so
Teacher-Student Relationships
00:28:40
Speaker
varied.
00:28:40
Speaker
So that's why it comes back to me, okay, now this has to be a parent level, an individual level, but it's hard to fund do something different for every kid.
00:28:50
Speaker
It's hard to fund trust the teachers.
00:28:53
Speaker
Because to a certain extent, the answer is trust you.
00:28:57
Speaker
give you funding for two weeks to do something good for the students you work with because you know them.
00:29:03
Speaker
One of the challenges I always had as a test prep person is, if you're bringing me after school, I don't know these kids.
00:29:08
Speaker
I actually don't really have any pull or penalties.
00:29:11
Speaker
So what if you don't do the homework for the after school test prep class?
00:29:16
Speaker
It's like kids will figure out pretty quickly, oh, he can't give me a detention.
00:29:19
Speaker
He's not going to tell my mama.
00:29:21
Speaker
So that's the problem with all of these external folks.
00:29:26
Speaker
Do they have either the confidence, or not confidence, but the goodwill and connection with the student to get them engaged and to do the work?
00:29:35
Speaker
Or do they have the stick of penalties and parents and all of that sort of stuff?
00:29:41
Speaker
And generally speaking, outside vendors don't have either.
00:29:45
Speaker
We haven't had time to develop the relationship to motivate and encourage the students.
00:29:49
Speaker
And you're an outside vendor, so you don't have access to the parents or the penalties or any of that sort of stuff.
00:29:54
Speaker
Even the best intended external vendors are challenged if the students don't have the intrinsic motivation to do extra stuff during their vacation to make up for something they had no control over and disrupted their lives and made things miserable.
Future of Testing
00:30:15
Speaker
It's hard for adults to have the motivation to do extra, you know, given what we just came out of.
00:30:22
Speaker
It's also, I mean, it's not, it's definitely not good for morale either to constantly feel like you're behind or you did something wrong.
00:30:29
Speaker
And then starting off a school year with that frame, I feel it'd be so detrimental, especially recognizing as we were talking about before, it's not that
00:30:41
Speaker
Kids didn't do anything for an entire year.
00:30:44
Speaker
It was just different types of stuff.
00:30:46
Speaker
And those different types of things are valuable as well.
00:30:48
Speaker
And we can highlight those things.
00:30:49
Speaker
The cynical side of me is like the world's falling apart.
00:30:51
Speaker
The world's in flames is terrible.
00:30:53
Speaker
I also don't want to communicate a message every single day when I walk into a group of 120 kids over the course of the day, like, hey, the world sucks.
00:31:00
Speaker
You're here.
00:31:00
Speaker
Because that's not going to be an effective teaching strategy.
00:31:05
Speaker
So this is kind of an interesting question because a lot of your work has been in test prep and fair test and looking at tests in a variety of different ways to make them more equitable.
00:31:15
Speaker
Is there a place for moving away from the test in general?
00:31:20
Speaker
You've written about like test optional universities and how that can be good and some issues with that as well.
00:31:28
Speaker
What, if anything, could there be a future that just doesn't have the test?
00:31:34
Speaker
Is that a possibility?
00:31:37
Speaker
I mean, I don't know.
00:31:40
Speaker
One can dream.
00:31:41
Speaker
One can work for it.
00:31:42
Speaker
I think that the hopeful Pollyanna side of me says yes.
00:31:48
Speaker
The more realistic side of me says yes.
00:31:52
Speaker
In all likelihood, we can minimize and diminish the harmful impacts of testing.
00:31:57
Speaker
We can get rid of that 17 test week for your students.
00:32:01
Speaker
We can return to the 70s and 80s where it was testing every two years rather than testing every year.
00:32:09
Speaker
I don't know that we can get rid of it, but I think we can minimize the harm and the damage.
00:32:15
Speaker
To me, what's interesting is I kind of believe in testing.
00:32:18
Speaker
I actually don't disbelieve in testing.
00:32:21
Speaker
I do believe tests show some things.
00:32:23
Speaker
I don't actually agree with the narrative that tests show nothing.
00:32:26
Speaker
I disagree with the narrative that tests show everything kind of is almost that's that's almost how we've been using them.
00:32:33
Speaker
Right.
00:32:33
Speaker
That these actually.
00:32:35
Speaker
So a lot of my work is in admissions test.
00:32:38
Speaker
Like the SAT, which is scored on a 400 to 1600 scale in 10-point increments, where a lot of noise is made around the difference between a score of 1,000 versus a score of 10, 10, which is an absolutely irrelevant difference according to the test makers.
00:32:55
Speaker
In K-12, testing is often a little bit better than that, where you have these proficiency levels.
00:33:03
Speaker
I actually do think there is some meaning to going, okay, that person's a one, not a five.
00:33:08
Speaker
Okay, that means that we expected them to be able to do a certain level of work and they didn't do it.
00:33:11
Speaker
And if we don't turn that into too much, right, does a one mean they didn't try?
00:33:17
Speaker
Or does a one mean they don't know how to add?
00:33:21
Speaker
And we expected them to know how to add.
00:33:23
Speaker
Right.
00:33:24
Speaker
And what are we telling the student who got the one?
00:33:26
Speaker
Often the problem becomes, how do we interpret those scores?
00:33:29
Speaker
How do we convey those scores?
00:33:30
Speaker
And what do we say to people who get those scores?
00:33:32
Speaker
Right.
00:33:33
Speaker
To look at somebody who got a one and say, oh, you can't do first grade work.
00:33:39
Speaker
As opposed to you blew that test.
00:33:42
Speaker
Like we don't do it anywhere else.
00:33:44
Speaker
We don't take a kid and give them a piano test and be like, oh, my God, you'll never be a musician.
00:33:48
Speaker
Right.
00:33:49
Speaker
No, you sucked at the piano.
00:33:50
Speaker
So I was like, okay.
00:33:53
Speaker
And if you value piano playing, you better work on that.
00:33:57
Speaker
My kids can play piano and I can't.
00:34:00
Speaker
I'm actually literally okay with that.
00:34:04
Speaker
I have no aspirations to, like, I have a dream to play piano, but I will never actually work at it.
00:34:09
Speaker
So I'm okay with having no ability to do it because I will never work at it.
00:34:15
Speaker
So that's what I think is lost in educational.
00:34:18
Speaker
testing the nuance of like this skill you displayed that this is not a skill you currently possess we think you should possess it so you probably should work at it right and i think if that were more the conversation it would be healthier
00:34:34
Speaker
That makes a lot of sense to me.
Broader Educational Assessments
00:34:35
Speaker
And it makes me think of, and I have to be very careful in how I word this because this could go haywire really poorly.
00:34:40
Speaker
But Susan Engel used to be on our board.
00:34:42
Speaker
She's an author and researcher.
00:34:43
Speaker
And she convinced me that we should be actually testing more things, but in less increments.
00:34:52
Speaker
So for example, testing, do you feel comfortable at school?
00:34:56
Speaker
Testing, do you feel safe at school?
00:34:59
Speaker
These other things, in addition to academics to get a better read of that,
00:35:03
Speaker
And I was like, well, how do you even measure that?
00:35:06
Speaker
And then she's like, well, you pick up a Time magazine every year or whatever, and it has some new study about how kids are more engaged at home than ever.
00:35:14
Speaker
And you read that, you don't really question it.
00:35:16
Speaker
Well, someone did a study of that, and someone figured that out, and you're going based off of that research, and why can that not be applied to school?
00:35:23
Speaker
Again, I have to be very careful to say that, because that could also lead to manipulation of that.
00:35:27
Speaker
It could lead to a billion more tests, and the way that it's funded, there's a lot of different issues that could go into that.
00:35:33
Speaker
But it could also force schools to pay attention to some other things instead of just being gauged off their academic ability.
00:35:42
Speaker
Off of, not their academic ability, a narrow swath of academic performance.
00:35:48
Speaker
Because we're not sitting here talking about history tests.
00:35:51
Speaker
We're not sitting here talking about science performance.
00:35:54
Speaker
For the most part, we're talking about reading, maybe grammar, and then math.
00:36:03
Speaker
Right?
00:36:04
Speaker
So that's part of the problem is that often
00:36:08
Speaker
We are using a narrow swath of curriculum and drawing broad conclusions.
00:36:14
Speaker
We're using a narrow swath of curriculum in order to develop an estimated ability performance and then drawing conclusions about students, school systems, whatever, based on this narrow swath of a sampling of the curriculum and an estimated score.
00:36:32
Speaker
And also just the ability to understand how to take a test.
00:36:36
Speaker
Even if you don't, especially as you get older, even if you don't necessarily know the answer, figuring out how to answer it in a proper way.
00:36:44
Speaker
The reason why a rich kid can go to some crazy tutoring camp and all of a sudden have an immaculate score.
00:36:50
Speaker
Which, you know, I mean, it ain't hard to find me on the internet listed for $400 an hour.
Socio-Economic Factors & Testing
00:36:55
Speaker
So people pay money to do, I mean, these tests aren't
00:37:01
Speaker
If it can be tested, it can be taught.
00:37:04
Speaker
And someone who spends all their time and energy decoding the test will know things about the test that others won't.
00:37:11
Speaker
Test prep will always work to some extent.
00:37:15
Speaker
The question is, how much can a test score be impacted in X amount of time?
00:37:24
Speaker
That's the only real question.
00:37:26
Speaker
I can't take a 12th grader who's never been
00:37:29
Speaker
in any kind of school whatsoever or not a 12th grader.
00:37:32
Speaker
At that point, we're really talking about a 17-year-old, right?
00:37:34
Speaker
Because if you haven't been through the grades, you're not a 12th grader.
00:37:37
Speaker
You took a 17-year-old who's never been through any schooling, you can't give me six weeks to make that up and get him a X score on a test, right?
00:37:46
Speaker
But if you give me six years one-on-one, I could probably make it in your gym pack.
00:37:51
Speaker
Give me two years.
00:37:52
Speaker
I can probably make an impact.
00:37:53
Speaker
Right.
00:37:53
Speaker
So anything that can be tested can be taught.
00:37:55
Speaker
It's really just a question of time.
00:37:57
Speaker
And I think that's the one big part of this narrative that's left out.
00:38:00
Speaker
There is a time driven.
00:38:03
Speaker
There is a dedication to precociousness.
00:38:06
Speaker
Right.
00:38:07
Speaker
And an age related performance that is built into our educational system that COVID has disrupted.
00:38:15
Speaker
Right.
00:38:15
Speaker
But that's really often what we're measuring, right?
00:38:17
Speaker
Think about like gifted and talented.
00:38:19
Speaker
That really just means precocious.
00:38:23
Speaker
But we've all had kids who walked faster, who talked faster, who did all those sort of things, and they didn't get tracked for life.
00:38:32
Speaker
So the kid who tests better faster or reads faster gets tracked for life in the accelerated program.
00:38:39
Speaker
So there's a lot of these time-driven, age-driven things that are baked into what we're doing that are problematic.
Future Testing Policies
00:38:47
Speaker
Because I'm not sure that they mean much more than
00:38:49
Speaker
okay, did that faster.
00:38:51
Speaker
Yeah, yeah.
00:38:51
Speaker
And the labeling as well and the things I can carry.
00:38:56
Speaker
I guess the only other question I have for you then would just be, what's next?
00:39:00
Speaker
What are you looking at in your work, both through fair tests and just your own personal interests?
00:39:05
Speaker
What are you pushing for?
00:39:06
Speaker
So these days, the work work is the same as the work has always been.
00:39:13
Speaker
Let's try to influence policy and practice around testing to make sure it is as least harmful as possible.
00:39:22
Speaker
We were working hard to try to get opt outs to happen around this year's testing and to get the Fed to grant broader waivers for testing.
00:39:33
Speaker
Now the next phase is going to be how are those test scores going to be used?
00:39:37
Speaker
And making sure that there is thought and care and nuance around use of scores if they're used at all.
00:39:46
Speaker
You know, there's one article I read from a psychometrician who gave, I want to say it was an eight point checklist for the things that need to be done.
00:39:57
Speaker
to use test scores appropriately.
00:40:00
Speaker
It was actually Andrew Ho out of Harvard, which was funny to me because his same article was like, here are the eight things that you need to do in order to use test scores properly this year.
00:40:07
Speaker
So let's test and do these eight things.
00:40:08
Speaker
And I was like, because you have a list of eight things, let's not test at all.
00:40:12
Speaker
So he's one of those type of interests I actually agree with.
00:40:15
Speaker
He has nuance and thoughtfulness around the use of testing.
00:40:19
Speaker
His conclusion was, yes, go ahead, test.
00:40:21
Speaker
And I'm going to trust you to do these careful use of test scores.
00:40:25
Speaker
I'm a little bit more cynical, like, yeah, y'all will never do those things.
00:40:29
Speaker
If you did them, I'm probably okay with it, but you'll, you know, I've seen too many school districts and they will never use the scores that carefully and nuanced.
00:40:40
Speaker
So let's just not test, right?
00:40:44
Speaker
That ship has sailed.
00:40:46
Speaker
So the question is now holding districts and the federal government in different places to account around the use of test scores.
00:40:56
Speaker
Right.
00:40:56
Speaker
And what are we trying to say with the use of test scores?
00:41:00
Speaker
I feel like it was the IELTS that just came out, maybe it's Illinois, just published their test score data.
00:41:06
Speaker
I haven't dug into the report yet, but like, oh, surprise, surprise, there was a drop during the pandemic.
00:41:14
Speaker
And, you know, and I'm just going to go in and look for what are the caveats, right?
00:41:18
Speaker
Like 90% of the students in our study didn't actually finish the test.
00:41:22
Speaker
Like there's going to be all of these weird caveats.
00:41:24
Speaker
It's like, so why are you even reporting out this data if 80% of the students did it in 12 seconds?
00:41:31
Speaker
And I'm not saying that's actually in the report, but I expect to find things like that, that like, okay, this data is weird.
00:41:39
Speaker
It's influenced by the pandemic.
00:41:40
Speaker
Some of the kids did it at home.
00:41:41
Speaker
Some of them did it in the building, wearing masks, social distance, did like, I'm not sure this is meaningful data.
00:41:47
Speaker
And then when you take all of that into account, a five point drop,
00:41:52
Speaker
That's not bad at all.
Conclusion & Call to Action
00:41:53
Speaker
So that's going to be the work.
00:41:54
Speaker
The work is going to be, since they have tested in a lot of places, how do we show what the scores really mean beyond the big headline of there was a drop?
00:42:12
Speaker
Thank you again for listening to Human Restoration Projects podcast.
00:42:15
Speaker
I hope this conversation leaves you inspired and ready to push the progressive envelope of education.
00:42:19
Speaker
You can learn more about progressive education, support our cause, and stay tuned to this podcast and other updates on our website at humanrestorationproject.org.