Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Bonus: Elevating the Conversation on NAEP Scores w/ John Warner
11 Plays2 years ago
The results of the National Assessment of Educational Progress, or NAEP, were released yesterday, September 1st, prompting a New York Times headline that read “The Pandemic Erased Two Decades of Progress in Math and Reading”, the 74 headline added “Two Decades of Growth WIPED OUT by Two years of Pandemic”. Peter Greene, an education policy watcher, called it NAEP Pearl Clutching Day. I myself had tweeted out “With the release of pandemic NAEP scores, we're about to have the worst cycle of education discourse imaginable”, and man did that ring true. Everyone was running to their corners to abolish teacher’s unions, attack remote & hybrid learning and mask mandates - just relitigating every pandemic issue imaginable - and the results brought out the usual resident experts in everything, like Matt Yglesias, who called the scores “A Short-term L for the left that was more supportive of closure”.
While everyone online is jumping to conclusions, we thought it would be important to help provide some context, to step back and take inventory of the data, claims, headlines, and provide context and forecast next steps: what, if anything, could or should we do in response to this report? So I reached out to author and educator John Warner, whose intuition I tend to trust on this kind of thing. John is the author of several books, Why They Can’t Write, The Writer’s Practice and Sustainable. Resilient. Free.: The Future of Public Higher Education, released in 2020. Thanks John for taking the time to talk with me today. Let’s start with what the NAEP results say and what they mean, and then we’ll compare that to the headlines. So what do the results say and what should we make of them? Why does the framing matter? What context is missing? How could we meaningfully report on these results? What’s missing in the discourse?
GUEST
John Warner, author of Why They Can't Write, The Writer's Practice, and Sustainable. Resilient. Free: The Future of Public Higher Education. He serves on Human Restoration Project's Board of Directors.
RESOURCES
Recommended
Transcript
Introduction to NAEP Results Discussion
00:00:11
Speaker
Hi there.
00:00:12
Speaker
So my name is Nick Covington.
00:00:13
Speaker
I'm from the Human Restoration Project.
00:00:15
Speaker
And we're here to talk today about the NAEP results.
00:00:18
Speaker
So the results of the National Assessment of Educational Progress, or NAEP, were released yesterday, September 1st, prompting a New York Times headline that read, The Pandemic Erased Two Decades of Progress in Math and Reading.
Sensationalism and Misinterpretation of NAEP Results
00:00:33
Speaker
The 74 headline added two decades of growth wiped out by two years of pandemic.
00:00:39
Speaker
Peter Green, an education policy watcher, called it NAEP pearl clutching day.
00:00:45
Speaker
And I myself even tweeted a lot about this.
00:00:48
Speaker
You know, with the release of the pandemic NAEP scores, we're about to have the worst cycle of education discourse imaginable.
00:00:55
Speaker
And man, did that ring true.
00:00:57
Speaker
So yeah.
00:00:57
Speaker
Everyone seems to run to their corners to abolish teachers unions, attack the remote and hybrid learning precautions that we took, mask mandates, and just relitigating every pandemic issue imaginable.
00:01:12
Speaker
The results even brought out the usual resident experts in everything, like Matt Iglesias, who called the scores immediately.
00:01:19
Speaker
a short-term L for the left that was more supportive of closure.
Contextualizing NAEP Results with John Warner
00:01:23
Speaker
So while everyone online is jumping to conclusions, we thought it would be important to help provide some context, step back and take
00:01:30
Speaker
take inventory of the data, the claims, the headlines, and provide some context and forecast the next steps.
00:01:36
Speaker
So what, if anything, could or should we do in response to that report?
00:01:40
Speaker
So I reached out to author and educator John Warner, whose intuition I tend to trust on this kind of thing.
00:01:47
Speaker
John's the author of several books, Why They Can't Write, The Writer's Practice, and Sustainable Resilient Free, The Future of Public Higher Education, released in 2020.
00:01:56
Speaker
So thanks, John, for making the time to turn around and talk with me about that today.
00:02:01
Speaker
Oh, my pleasure.
00:02:01
Speaker
Good to see you and try to make some sense of this stuff that's going on.
00:02:07
Speaker
Yeah, let's get right into it.
00:02:09
Speaker
So let's just start with what the results say and what do they mean?
00:02:13
Speaker
And maybe we can compare that to the headlines as well.
00:02:16
Speaker
What do the results say?
00:02:17
Speaker
What should we make of them?
00:02:18
Speaker
So in contrast to the headlines, which frame this around this idea that somehow learning the nine-year-olds, again, we've got to remember what we're talking about.
00:02:28
Speaker
We're talking about nine-year-olds
00:02:29
Speaker
who took a single test for reading and math back sometime in the early part of the year, have scored at a level that is consistent with student scores on average from 20 years ago.
00:02:46
Speaker
The history of the NAEP test has been a kind of relatively steady growth and increase in scores.
00:02:55
Speaker
So when we say it's equivalent to 20 years ago, it's
00:03:00
Speaker
significantly lower than students who took it the last time it was administered, which was 2020, and therefore on par with students who took it in, more like students who took it in 2002.
00:03:15
Speaker
The headlines erasing two decades of progress or two decades of growth are wiped out are absurd.
00:03:25
Speaker
that these were allowed to be published in major national publications should be to their everlasting shame because that's not what this data says.
Cultural and Educational Shifts due to Pandemic
00:03:35
Speaker
What this data says is at one particular time and place, nine-year-olds scored this on this exam for reading and math.
00:03:47
Speaker
We can now take a step back and ask, so what are the factors that have gone into this score that is significantly lower than the last time it was taken, that is on par with 20 years ago prior to what has been a sort of gradual increase?
00:04:02
Speaker
But the idea that suddenly our nation's nine-year-olds are 20 years behind where they should be, it's nuts.
00:04:13
Speaker
And even just thinking...
00:04:15
Speaker
Let's assume that's true, right?
00:04:18
Speaker
That somehow our nation's nine-year-olds have been getting steadily smarter over the last 20 years.
00:04:26
Speaker
We're learning steadily more over the last 20 years.
00:04:30
Speaker
How are our 28, 29, 30-year-olds doing today?
00:04:34
Speaker
Just fine, right?
00:04:37
Speaker
The notion that these scores are dispositive, that they...
00:04:41
Speaker
carry some sort of inevitable predictive weight for a future 20 years from now, 10 years from now, five years from now, is a total abuse and misuse of what a single test and a single data point means.
00:04:57
Speaker
It doesn't mean it's meaningless, but it does not mean what these screaming headlines say.
00:05:04
Speaker
And I think absent from those singular pieces of data that are going around now is all of the context of the last two years to go alongside that too.
Critique of Quick-Fix Responses to NAEP Data
00:05:14
Speaker
Something that you've commented on and I've commented on as well is, I mean, missing from reducing the experience of the last two years into two singular points, and particularly just for reading and math, disregarding every other
00:05:29
Speaker
subject that kids would study in school, but kind of just disregarding the project of the last two years, which might not have been to keep raising math and reading scores, but instead, you know, to turn schools into safer learning environments that
00:05:45
Speaker
adults and students alike can inhabit without getting each other sick, risking bringing infection home to others, you know, scrambling for schools to be able to provide the same level of services that schools have provided in terms of free meals to students and, and, um,
00:06:03
Speaker
And obviously, you know, child care then for working parents as well.
00:06:06
Speaker
So there was just a seismic cultural shift across the entire board.
00:06:11
Speaker
And one of the reasons why I think, I mean, the NAEP is called, you know, the nation's report card.
00:06:16
Speaker
It's the gold standard for these standardized assessments is even though the world might change around it, you know, NAEP always stays the same.
00:06:23
Speaker
And yet here we have this assessment result that seems incongruous with our experience of the last two years.
00:06:30
Speaker
I was just looking back at a Gallup survey from
00:06:33
Speaker
earlier this summer that said K-12 workers are the most burned out employees in the United States.
00:06:39
Speaker
Higher burnout rates than medical workers, than anybody else.
00:06:42
Speaker
College faculty are on that list as well.
00:06:45
Speaker
So when you look at a five-point decline in math or reading scores,
00:06:50
Speaker
alongside teachers are the most burned out employees in the entire country, you say, well, what were we doing?
00:06:57
Speaker
You know, if we were just sitting at home doing nothing, surely we wouldn't be as burned out as we were.
00:07:03
Speaker
So why does that framing matter?
00:07:05
Speaker
You know, like how could, what could a more meaningful, I would even say like conscientious, you know, conscientious reporting of these results do what's missing from this discourse that we're having online about this?
00:07:19
Speaker
Well, one of the things we should recognize is as the nation's report card, as the snapshot of something that, of a test on a single day in a single context of a single time, is disruption has happened.
00:07:34
Speaker
And this is now reflected in the data.
00:07:37
Speaker
Whatever schooling was in whatever form it took under whatever circumstances was disrupted by a global pandemic.
00:07:48
Speaker
And so not only school was disrupted, the entire lives of everybody in the country and the world was disrupted.
Factors Influencing NAEP Scores Beyond the Pandemic
00:07:55
Speaker
So the notion that we could have avoided this with some kind of different school or different choices, going back in person sooner or anything like this, it's not reflected in any of the data.
00:08:12
Speaker
In fact, in some of the underlying data they release, there's, there's, uh,
00:08:19
Speaker
indications that none of that mattered, right?
00:08:21
Speaker
Like in reading scores, students who are in city districts, their reading scores were actually flat, the only metric that didn't go down.
00:08:32
Speaker
And these were schools that were much more likely to be closed for longer.
00:08:38
Speaker
Regional differences don't tell us much of anything in terms of schools in the South, which were more likely to go back to school in person sooner.
00:08:50
Speaker
had decreases equivalent to, and in some cases greater than, schools in the Northeast, which were likely to have gone back in person later.
00:09:01
Speaker
So what we have really is a picture like a big thing happened.
00:09:05
Speaker
And this big thing has affected how students perform on this test.
00:09:09
Speaker
And it could be any number of reasons.
00:09:12
Speaker
It could be something like actual specific test prep is the thing that was sacrificed.
00:09:19
Speaker
And if that was the case, that would be a really interesting finding.
00:09:22
Speaker
It would tell us that these sorts of tests, when they're prepped for, maybe don't tell us all that much about what students are learning.
00:09:31
Speaker
I don't know how big a factor that is or if it's a factor at all.
00:09:34
Speaker
It could be that students were grieving the loss of a caretaker.
00:09:40
Speaker
It could mean that students who were, and there is some data to suggest this, students who are on the lower end of the scale in terms of
00:09:49
Speaker
of performance were more significantly affected by these disruptions.
00:09:56
Speaker
So their scores decreased by a larger percentage.
00:10:00
Speaker
So it's like these marginal students often in situations where they don't have sufficient resources for their educations or even their lives
00:10:15
Speaker
Their scores fell off a cliff while others were more modest.
00:10:21
Speaker
These are all things that we can dig in deeper and try to figure out over time, not necessarily from the NAEP score, but other tests and other indicators.
00:10:34
Speaker
But we don't know what many people are claiming to know, that in-person school
00:10:42
Speaker
would have made a difference or removing masks would have made a difference or that remote learning cannot work under any circumstance.
00:10:51
Speaker
We do know that students who tend to have less access to good educational resources did even worse than those with better access.
00:11:03
Speaker
But beyond that, we just don't know all that much.
00:11:06
Speaker
We know something really big happened and it's going to take time to recover from it.
00:11:11
Speaker
I used this analogy yesterday in a piece I'm writing and working on.
00:11:17
Speaker
Imagine we're talking about domestic air travel, right?
00:11:21
Speaker
Where over 800,000 people took a flight in 2019 and that dropped to 300,000 in 2020.
00:11:23
Speaker
And it's up over 650,000 or something like that in 2021.
00:11:26
Speaker
If you compare 2021 to 2019, you would say that the
00:11:37
Speaker
Air travel industry is collapsing because we're down from 800,000 to 650,000.
00:11:45
Speaker
The reality is it's in the midst of a recovery after a large shock that has caused significant change.
00:11:53
Speaker
I think that's a much more plausible explanation of what's happening with students.
Misuse of Educational Data in Discussions
00:11:58
Speaker
And the idea that these students will be permanently harmed or that they cannot recover from this
00:12:06
Speaker
I think is totally unfounded, counterproductive.
00:12:11
Speaker
And really anybody who wants to say that is pressing an agenda that does not have much to do with trying to understand at a deeper level, at a more practical and actionable level, what can we do to help students in schools today, tomorrow, next year,
00:12:29
Speaker
We're going to continue to experience some level of disruption from this pandemic who are going to continue to have their own issues around burnout, just as their teachers are and are not out of the woods in terms of the effects of this disruption.
00:12:47
Speaker
Certainly.
00:12:48
Speaker
And I think what is so interesting, right, is this just is such a microcosm of
00:12:57
Speaker
what it says about how we use data in education, what data counts, and kind of how we communicate the success of schools and school systems in the general public.
00:13:06
Speaker
A couple of points on this.
00:13:07
Speaker
I think one of my concerns then is just understanding how reactive our political systems are to something like the NAEP score to a New York Times headline.
00:13:18
Speaker
We're going to be making some really
00:13:21
Speaker
dramatic changes to education in response to, in the short term, in response to the singular piece of data that is just going to serve that data.
00:13:30
Speaker
So the conversations that I've seen are around accelerated learning, extending school hours or the school year, even though the United States already has students in schools and American teachers already work more hours and teach, have more contact time with students than their counterparts in the rest of the world.
00:13:48
Speaker
You know, all of these potential solutions.
00:13:52
Speaker
And one thing that I think is particularly interesting, too, is that I don't think it was these results are a surprise.
00:13:59
Speaker
Right.
00:13:59
Speaker
I don't mean to sound callous at all in this kind of sense, too, to your point.
00:14:04
Speaker
We faced a global disruption.
00:14:06
Speaker
We have a million dead Americans.
00:14:07
Speaker
We have hundreds of thousands of orphaned children.
00:14:10
Speaker
And to expect that schools were going to be immune from those changes, we really were just going to be... It was a question of what is the drop going to be in these test scores, right?
00:14:20
Speaker
Because schools were focused on other priorities.
00:14:24
Speaker
Now, the other part of this that I think is interesting and again, not surprising, is that the score drops tracked alongside the groups in society who would have been most heavily impacted by the various waves of the pandemic.
00:14:40
Speaker
You said people in the Northeast, the South, poor communities,
00:14:46
Speaker
black and brown communities that are underserved in so many of the other aspects of public life and access to health care and all of that as well.
00:14:56
Speaker
And I think what is particularly frustrating is our response is not really going to meet the needs of those communities in the sense that we know that it's going to take more resources to sort of, you know,
00:15:09
Speaker
Not just resolve the tensions that are simmering socially, but there's also this leap into the pedagogical side of things, too.
00:15:18
Speaker
So we want to try and solve these systemic social issues of lack of access to health care and poverty, etc.
00:15:25
Speaker
with these pedagogical programs.
00:15:27
Speaker
They're going to try to solve
Addressing Systemic Issues in Education
00:15:28
Speaker
it when they get to the schoolhouse door.
00:15:29
Speaker
Yeah.
00:15:31
Speaker
So kind of to respond to this, I'm curious what your response is here, but what do we say to those people who are going to argue that this data is important, that it's helpful for us in order to address those equity concerns?
00:15:45
Speaker
So it pretty demonstrably shows that people in underserved communities or hit harder than people with more resources.
00:15:55
Speaker
That's not a surprise to anyone who's been in education for longer than five minutes.
00:15:59
Speaker
You know, they would argue we have the data.
00:16:02
Speaker
Here's the data.
00:16:03
Speaker
Here's how it shows how kids who were behind fell behind even further.
00:16:08
Speaker
Again, probably referring to poor kids, black and brown kids, et cetera, that this data is going to help us target those interventions, accelerations, et cetera.
00:16:15
Speaker
Right.
00:16:15
Speaker
There's a lot of sincere people who believe that that's going to be doing what is best for kids.
00:16:22
Speaker
And what what is the response to that?
00:16:23
Speaker
You know, is there is there a response?
00:16:25
Speaker
What's the conversation?
00:16:26
Speaker
Well,
00:16:27
Speaker
I think if there's any clear response to this data, and really this is a response to the NAEP data every single year, it's that under-resourced schools need more resources.
00:16:40
Speaker
And schools by themselves cannot fix larger social ills and lack of access to stable facilities
00:16:57
Speaker
healthy existences.
Rethinking Education Post-Pandemic
00:17:00
Speaker
The data says, as we've been talking, if you were in a worse situation in terms of access to high-speed internet,
00:17:16
Speaker
or your own computer, or a quiet place to study, or some adult willing to help, your score dropped more.
00:17:22
Speaker
These are all things that are still in place, even if we go back to school.
00:17:27
Speaker
And school really, you know, this test and schooling in general, is really often a byproduct of the larger societal atmosphere and the things that are going on around students.
00:17:42
Speaker
The notion that
00:17:48
Speaker
this is going to be solved by a kind of accelerated schooling or sometimes what sounds just like a harder cracking of the whip over the heads of students as we try to spur them on to achievement seems misguided to me because as we know, the loss has not only been to their academics, it's been a loss to their overall experience of, of,
00:18:15
Speaker
their humanity, you know, as it has been for all of us, right?
00:18:20
Speaker
Teachers and others aren't burned out only because they've been working under difficult circumstances, either through remote learning or the challenges of trying to teach in person while mitigating the virus.
00:18:32
Speaker
It's that there has been a lot of other stuff going on in our lives around the virus, not the least loved ones getting sick,
00:18:42
Speaker
And potentially dying.
00:18:45
Speaker
So there is a larger recovery to this and a longer recovery to this.
00:18:51
Speaker
One of the points I try to make with people is a year from now, we're going to get next year's NAEP scores and they're going to go up.
00:19:02
Speaker
And people are going to be asking about the innate miracle.
00:19:04
Speaker
How did we return to the previous status quo or close to it in just a year's time when we were 20 years behind last year?
00:19:16
Speaker
And my answer right now, and it'll be the same next year, is time.
00:19:22
Speaker
We need time to process.
00:19:26
Speaker
We need time to quite literally heal.
00:19:28
Speaker
We need time also.
00:19:30
Speaker
And this is the thing.
00:19:32
Speaker
that I wish more of these conversations were about, to figure out what school can and should be going forward.
00:19:41
Speaker
The tremendous hunger to return to the status quo among many corners of the discussion baffles me because there were a lot of criticisms, justified criticisms of the status quo from various parts of the spectrum.
00:20:02
Speaker
from people who I would fundamentally disagree with, they were also critical of status quo.
00:20:06
Speaker
I was critical of status quo.
00:20:09
Speaker
Why are we rushing to go back to something that nobody thought was ideal?
00:20:16
Speaker
Just because it's more comfortable, just because it seems better than what's going on currently?
00:20:22
Speaker
I think those are all natural responses, but I think it's a shame if we don't use this as an opportunity to have
00:20:30
Speaker
better, deeper conversations about what students would benefit from in our schools.
00:20:35
Speaker
So we can use these scores as a way to have those conversations.
00:20:40
Speaker
Unfortunately, that's not what's really been going on, at least not in the first 36 hours since the score's release.
00:20:49
Speaker
Yeah, this is definitely the instant react kind of idea.
00:20:53
Speaker
But I'd
00:20:55
Speaker
really doubt that the discourse is going to get a lot more, a lot healthier after that.
00:21:00
Speaker
It's become this tug of war, right?
00:21:02
Speaker
Of the various forces that want to bend schools to their, to their idea of what schools should be.
Future of Education and Continued Conversations
00:21:10
Speaker
And so seething on this data point in order to do that was wholly predictable.
00:21:15
Speaker
It's, it's dismaying.
00:21:18
Speaker
It's very difficult to counteract because the first impulse is,
00:21:22
Speaker
is to simply grab your end of the rope and start tugging it in the direction you want.
00:21:31
Speaker
But if we can at least advocate for a smarter conversation, maybe we can make some collective progress.
00:21:40
Speaker
I take the
00:21:42
Speaker
I try to take the good faith of those who see these things differently, seriously, that they also want students to thrive and do well in school.
00:21:51
Speaker
So if that's the case, we should be able to have a better, deeper conversation rather than litigating the past, rather than trying to get in our time machine and decide if students should have been in person sooner or should have been masked or whatever.
00:22:09
Speaker
All that is done.
00:22:12
Speaker
people tried to make the best decision they could have based on what we knew at the time.
00:22:17
Speaker
That's all we can do right now going forward.
00:22:20
Speaker
So let's have a decent look at what's happening so we can make some better decisions.
00:22:27
Speaker
I think that's a great place to end it, John.
00:22:29
Speaker
So thank you so much for spending some time unpacking that with me today.
00:22:32
Speaker
My pleasure.
00:22:33
Speaker
Thanks.
00:22:36
Speaker
Thanks again to John Warner for joining me for that conversation.
00:22:38
Speaker
You can follow him on Twitter at Bibli Oracle.
00:22:41
Speaker
You can also keep the conversation going by following us at HumeResPro and by visiting our website, humanrestorationproject.org.
00:22:50
Speaker
We have an entire Learning Loss Handbook.
00:22:52
Speaker
that is over 50 pages of history and insight into these tests and the learning loss narrative, particularly surrounding the last two years of pandemic schooling.
00:23:02
Speaker
In addition, we have a podcast on the topic of learning loss with Akil Bello and highlights of that same episode available on our YouTube channel if you search for Human Restoration Project.
00:23:13
Speaker
Thanks for listening.