Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Re:Teaching, EP 3: Why What Works Won't Work and Why What Works May Hurt
63 Plays3 years ago
Long time listeners may recognize the title to this podcast: Re:Teaching. Last year, Nick released a series of short podcasts on a separate channel, focused on current events and short-form highlights of writing. We soon realized that by splitting our podcast into two segments, most folks didn't hear any of this series! Enjoy, and there will be even more new Re:Teaching episodes on the way.
This episode is a feature read of Why What Works Won't Work and Why What Works May Hurt by Nick Covington, published in March 2020. Learn more about Human Restoration Project and find a wealth of free resources to create human-centered classrooms at humanrestorationproject.org, and follow us on social media, @HumResPro.
Recommended
Transcript
Adoption of Experimental Methodologies in Various Fields
00:00:06
Speaker
What do fields like medicine, physics, agriculture, and education have in common?
00:00:11
Speaker
Well, over the last several decades, there has been increasing pressure on teachers and researchers alike to adopt the experimental methodologies and random controlled trials of the laboratory to find out what works in education and to elevate teaching to the level of an evidence-based profession.
Criticisms of Evidence-Based Educational Research
00:00:28
Speaker
But what does it mean to be evidence-based?
00:00:31
Speaker
What should be the relationship of the laboratory to the classroom?
00:00:34
Speaker
And what are the consequences or adverse side effects of adopting such practices?
00:00:39
Speaker
In this episode of Reteaching, I'll unpack the criticisms and limits of what works in what we call evidence-based educational research and practice and try to understand why, in the words of Dutch researcher Geert Biesta, the dynamics of education are fundamentally different from the dynamics of, say, potato growing or chemistry.
Evidence-Based Practice in Healthcare
00:01:07
Speaker
What does it mean to be evidence-based?
00:01:11
Speaker
The top results of a general Google search for evidence-based practice yields a number of topics, for example, from the American Journal of Nursing on the seven steps of evidence-based practice, a page from the American Speech Language Hearing Association on EBP, or
00:01:27
Speaker
or evidence-based practice, as the, quote, integration of clinical expertise and expert opinion, external scientific evidence, and client-patient caregiver perspectives, and a link to a 1998 article from the National Institute of Health declaring that, quote, evidence-based practice is spreading in popularity in many healthcare disciplines, end quote.
00:01:48
Speaker
So it's not surprising that Geert Biesta opens his 2007 article, Why What Works Won't Work, with, quote, the idea that education should be or become an evidence-based practice and that teaching should be or become an evidence-based profession has recently come to prominence in several countries around the world, end quote.
Double Transformation in Educational Practice
00:02:08
Speaker
Borrowing the language of the medical field, the goal of evidence-based practice and education is what Biesta calls a double transformation of both educational research and practice, and ultimately an alignment of research, practice, and policy led by public and private organizations like the Office for Standards in Education, Research Ed, and the Center for Evaluation and Monitoring in the UK,
00:02:33
Speaker
and the What Works Clearinghouse in the U.S., which is part of the Institute of Education Sciences within the U.S. Department of Education.
00:02:42
Speaker
The homepage of the What Works Clearinghouse, for example, implores visitors to find what works based on the evidence and provides links to categories like literacy, mathematics, behavior, teacher excellence, and charter schools.
Evaluating Educational Interventions with What Works Clearinghouse
00:02:56
Speaker
For some reason, arts and humanities, social studies, foreign language, and physical and mental wellness are not available options.
00:03:03
Speaker
Within each category is a list of interventions defined as, quote, an educational program, product, practice, or policy aimed at improving student outcomes, unquote.
00:03:13
Speaker
And within each intervention is listed a six-point effectiveness rating
00:03:18
Speaker
from negative to positive and mixed results in between, as well as an improvement index, defined as, quote, the expected change in percentile rank for an average comparison group student if that student had received the intervention, end quote.
00:03:34
Speaker
It all seems very simple.
00:03:36
Speaker
To improve educational outcomes, choose the area you wish to see improved.
00:03:40
Speaker
Select from the menu of interventions the ones with the highest effectiveness rating and improvement index score, and overnight we will have had an educational revolution.
00:03:49
Speaker
Or as one research ed leader lists in their Twitter bio, quote, teaching is simple.
00:03:54
Speaker
Retrieve, explain, practice, repeat, end quote.
Critique of Causal Models in Education
00:03:59
Speaker
This is what Biesta refers to as a causal model of professional action, which presumes that educators administer classroom interventions to students in the same way healthcare providers administer medical treatment to patients, that there is a cause in the intervention or treatment and effect in the outcome or results, that a student learns or the patient's health improves.
00:04:22
Speaker
It is also a technological model of professional action, which, in Biesta's words, quote, assumes that the ends of professional action are given and that the only relevant professional and research questions to be asked are about the most effective and efficient ways of achieving those ends, end quote.
00:04:40
Speaker
that the goal of teaching science, for example, is to improve state assessment scores, and the role of the educational professional is simply to choose from the menu of approved interventions.
00:04:50
Speaker
When scores go up, learning is improved.
00:04:53
Speaker
Here is where Biesta offers his criticism of education as a causal and technological practice.
Education vs. Medicine: Symbolic Interactions and Open Systems
00:05:00
Speaker
First, that the causal assumptions of medicine do not translate into education because education is not a physical process, as he points out, quote, Being a student is quite different from that of being a patient.
00:05:13
Speaker
Being a student is not an illness, just as teaching is not a cure, end quote.
00:05:17
Speaker
He continues, quote, Education is a process of symbolic or symbolically mediated interactions.
00:05:23
Speaker
End quote.
00:05:38
Speaker
Education does not operate through physical forces, unlike, say, the closed deterministic systems that mediate the biological and chemical pathways relied upon in medicine.
00:05:48
Speaker
Education is an open, recursive system that depends upon mutual interpretation and exchange, as students make sense of new information and teachers respond to student sense-making.
Complexity Reduction in Education
00:06:00
Speaker
So what should be the relationship of the laboratory to the classroom?
00:06:04
Speaker
In this way, Biesta argues that evidence-based practice can't tell us about what works, but only about what has worked.
00:06:12
Speaker
To make evidence-based practices work, to create the kind of order which is needed to replicate the experimental conditions of the random controlled trial, would mean transforming learning from an open, recursive system of symbolic sense-making to
00:06:26
Speaker
into the closed deterministic system of inputs and outputs valued in the clinical laboratory, a process Biesta refers to as complexity reduction.
00:06:36
Speaker
Systems use complexity reduction to limit the number of possible actions and options that are available to make processes more efficient and to control for outcomes.
00:06:46
Speaker
Schools are an example of complexity reduction in education.
00:06:51
Speaker
Schooling takes place in a particular building away from other kinds of work.
00:06:55
Speaker
A school day takes place across a number of distinct periods of definite length and disciplinary purpose guided by standards and curriculum.
00:07:03
Speaker
And learning is assessed and measured by a particular set of assessments.
00:07:07
Speaker
Quote, from the vast number of possible outcomes of schools, Biesta writes, only those are selected that are considered to be valuable.
00:07:15
Speaker
End quote.
00:07:17
Speaker
In both a theoretical and a practical sense, each of the complexity reductions students pass through in a day at school are the result of the collective social construction of value, drawing from a range of relevant and appropriate considerations not isolated to instructional effectiveness.
00:07:34
Speaker
How long a day should be?
00:07:36
Speaker
When it should start and end?
00:07:37
Speaker
Who should attend school?
00:07:39
Speaker
How should they get there?
00:07:40
Speaker
How large should class sizes be?
00:07:43
Speaker
What will be taught?
00:07:44
Speaker
How will learning be measured?
00:07:46
Speaker
What kind of teacher will teach?
Moral and Non-Causal Nature of Education
00:07:49
Speaker
What grading system will be used?
00:07:51
Speaker
How evidence of learning will be collected?
00:07:54
Speaker
How behavior will be managed?
00:07:56
Speaker
All of it.
00:07:57
Speaker
If designing the complexity controls of school is more than managing factual statements about what is possible or what works, more than selecting from a menu of interventions, if it is ultimately about implementing educationally desirable policies and practices, then, quote, education should be understood as a moral, non-causal practice, which means that professional judgments in education are ultimately value judgments, not simply technical judgments, end quote.
00:08:27
Speaker
We also understand that governance of schools is inherently political.
00:08:32
Speaker
School boards in America are elected, policies are voted upon, superintendents are appointed, teachers are hired, and curriculum is designed.
00:08:41
Speaker
As such, we should also understand the implementation of socially constructed value-based complexity controls as the exercise of power.
00:08:50
Speaker
Making decisions about which interventions are implemented when, at what levels, to what extent, for whom, who is excluded, and even deciding who makes these decisions are all exercises in power.
00:09:04
Speaker
As Piesta writes, quote, Since any attempt to reduce the number of available options for action for the elements within a system is about the exertion of power, complexity reduction should therefore be understood as a political act.
Political Nature of Implementing Evidence-Based Practices
00:09:19
Speaker
None of this is to say that evidence-based practices should not be considered and adopted, only that their implementation should be rightfully understood for what it is, not objectively removed from values, power, and policy, but precisely as an exercise of professional judgment and democratic decision-making about what is desirable in education.
00:09:42
Speaker
To suggest that research about what works can replace normative professional judgment is not only to make an unwarranted leap from is to ought, it is also to deny educational practitioners the right not to act according to evidence about what works, if they judge that such a line of action would be educationally undesirable.
00:10:05
Speaker
The What Works Clearinghouse, funded by the U.S. Department of Education, imploring teachers to find what works by focusing academic outcomes exclusively on literacy, math, and science, while excluding the arts and humanities, can be best understood not as a scientific consensus, but as a political act.
00:10:26
Speaker
The inclusion of charter schools and SAT-ACT test prep as evidence-based practice is a political act.
00:10:34
Speaker
The exclusion from their website of any mention of responsible citizenship, tolerance, cooperation, and curiosity is a political act.
00:10:44
Speaker
This becomes deeply problematic in those cases in which it is argued that professionals should only be allowed to do those things for which there is positive research evidence available.
00:10:55
Speaker
An approach which has been, in my view, correctly identified as a form of totalitarianism
00:11:05
Speaker
Where there are unintended side effects to this kind of educational policy and practice is where the lens developed by Professor Jung Jo comes in handy.
Studying Side Effects in Educational Research: A Comparison to Medicine
00:11:14
Speaker
I last included Jo's work on this podcast via his critical analysis of PISA, in which, much like Biesta argues that evidence-based education presumes a causal and technological model of professional action,
00:11:27
Speaker
Zhou outlines the ways in which the use of PISA as an international measuring stick to rank educational systems, quote, imposes a monolithic and West-centric view of societies on the rest of the world, end quote, and, quote, distorts the purpose of education, end quote.
00:11:44
Speaker
And where Biesta describes the desire to align educational and clinical health practice along the lines of what works in the narrow context of effective intervention, Zhou makes the important case for understanding what he calls the side effects of what works, to help us understand that what works may hurt, from his 2017 article by the same name.
00:12:07
Speaker
Medical research is held as a field for education to emulate.
00:12:11
Speaker
Education researchers have been urged to adopt randomized controlled trials, a more scientific research method believed to have resulted in the advances in medicine.
00:12:21
Speaker
But a much more important lesson education needs to borrow from medicine has been ignored.
00:12:27
Speaker
That is the study of side effects.
00:12:29
Speaker
Medical research is required to investigate both the intended effects of any medical interventions and their unintended adverse effects or side effects.
00:12:39
Speaker
In contrast, educational research tends to focus only on providing the effectiveness of practices and policies in pursuit of what works.
00:12:47
Speaker
It has generally ignored the potential harms that can result from what works.
00:12:54
Speaker
Understanding and weighing the risks versus the benefits of any medical treatment is a complicated accounting of a patient's medical history, presenting conditions, and a provider's professional judgment, where a patient ultimately consents to treatment or not, based on the same kind of complex calculations.
00:13:12
Speaker
For example, in 2017, nearly 13% of Americans over the age of 12 reported taking some kind of antidepressant in the last month.
00:13:21
Speaker
Selective serotonin reuptake inhibitors, or SSRIs, are among the most commonly prescribed drug treatments for depression and anxiety, and they work by preventing the body from reabsorbing the feel-good neurotransmitter serotonin, causing the body to use it more efficiently.
00:13:40
Speaker
However, some people report side effects, tiredness and insomnia, nausea and dizziness, weight gain and sexual dysfunction, even increased suicidal ideation that may escalate if their dosage increases, causing them to stop taking the SSRIs entirely.
Questioning Universal Application of Educational Interventions
00:13:56
Speaker
This can lead to an additional series of unpleasant but brief side effects as the brain readjusts to the changes referred to as antidepressant discontinuation syndrome.
00:14:06
Speaker
SSRIs have undoubtedly helped millions of people effectively manage their depression and anxiety, but we recognize that the benefits of what works may not outweigh the side effects for everyone, and that some patients and their providers may decide to stop treatment in favor of other interventions.
00:14:24
Speaker
Understanding how to manage the adverse consequences of what works in medicine helps patients and professionals alike make better choices to improve the quality of treatment and quality of life.
00:14:37
Speaker
So what if we knew a particular instructional intervention was effective for knowledge transmission, yet stifled creativity and problem solving, constrained exploration and discovery, and inhibited curiosity?
00:14:49
Speaker
Would we prescribe that intervention in every classroom for every child?
00:14:54
Speaker
Zhou documents a number of side effects associated with both upper and lower case direct instruction models and argues that direct instruction proponents have not failed to convince critics, quote, because of their lack of data or rigorous research method, end quote,
00:15:10
Speaker
or even doubting the effectiveness of DI, but rather that most of the opposition, quote, stems from a different set of concerns, such as the rigidity and prescriptiveness of the approach, inconsistency with developmental theories, inappropriateness for certain groups of children and contexts, sustainability of the effects over time, suppression of learner autonomy and development of creativity, and other potential damaging side effects, end quote.
00:15:36
Speaker
Joe then goes on to catalog a number of studies finding, for example,
00:15:47
Speaker
A 1979 meta-analysis found that, quote, "...with direct or traditional teaching, students tend to do slightly better on achievement tests, but they do slightly worse on tests of abstract thinking," end quote, "...and that while with open teaching, students may perform slightly worse on tests of achievement," quote, "...they do somewhat better on creativity and problem-solving," end quote.
00:16:09
Speaker
and that, quote, open approaches excel direct or traditional approaches in increasing students' attitudes toward school and toward the teacher and in improving students' independence and curiosity, end quote.
00:16:23
Speaker
A 2011 study based on children's play with toys found that, quote, children who were taught a function of a toy performed fewer kinds of actions of the toy and discovered fewer of its other functions than children who did not receive a pedagogical demonstration, end quote.
00:16:39
Speaker
Their study demonstrated that, quote, direct instruction can be effective in promoting rapid and efficient learning of target material, end quote, but it can negatively impact creativity if, quote, instruction necessarily limits the range of hypotheses children consider, end quote.
00:16:57
Speaker
A separate 2011 study involving toys and groups of preschool children found that children who were given instructions and directed how to play with toys were less likely to explore and come up with novel solutions and were more likely to imitate the instructor.
00:17:13
Speaker
A 2014 study of math students found that, quote, students generated more solutions to the problems before instruction than after, end quote, and that students who received instruction first, quote, tended to produce only the correct solutions they were told, end quote.
00:17:30
Speaker
And a 2012 study used the phrase, unproductive success, to describe the performance of a group of students under direct instruction conditions who were initially more successful in solving well-structured problems.
00:17:43
Speaker
Yet on later tasks that required deeper conceptual understandings, their performance was inferior compared to a group of students who were encouraged to solve complex problems first before receiving any instruction in potential solutions.
00:18:02
Speaker
Zhou's point is not to label direct instruction a failed pedagogy any more than we would call SSRIs a failed drug intervention because their side effects overwhelm some patients.
00:18:13
Speaker
But how can we make a professional decision about what works without also understanding the potential for unintended consequences?
00:18:21
Speaker
Zhou argues that we need, quote, both effective ways to transmit knowledge and foster creativity, end quote, and that by presenting side effects alongside measures of effectiveness, we can begin to bridge the gap between proponents and opponents of direct instruction as we work together to build a pedagogy that balances our desire for effective instruction without damaging curiosity, motivation, engagement, and creativity.
00:18:47
Speaker
As Zhou writes,
00:18:49
Speaker
A one-time treatment of direct instruction is unlikely to inhibit children's curiosity and creativity for life.
00:18:56
Speaker
But what if children are exposed to only direct instruction for 12 years or longer?
00:19:01
Speaker
Would it cause them to become less creative?
00:19:05
Speaker
It turns out there is some evidence that curiosity has already been such a casualty as direct instruction approaches dominate the evidence-based research guiding educational policy and decision-making.
Student Engagement and the Need for Curiosity
00:19:18
Speaker
Based on self-reported survey data from more than 900,000 5th through 12th grade students,
00:19:25
Speaker
A 2015 Gallup report famously found that the number of students engaged at school declines from 75% in 5th grade to only 34% by senior year, bottoming out in 11th grade as only 32% report being engaged.
00:19:41
Speaker
Gallup scientists categorized fully 1 in 10 American students as both disengaged and discouraged and reported that older students, quote, feel less cared for by adults and see less value in their own work, end quote.
00:19:57
Speaker
Mirroring this decline in engagement, children ask fewer questions as they proceed through school.
00:20:02
Speaker
Children aged 14 months to 5 years old have been found to ask an average of 107 questions an hour.
00:20:10
Speaker
As a parent of two kids 5 and under, I can anecdotally confirm this finding.
00:20:14
Speaker
But by the time those children reach elementary school, they may only ask 2 to 5 questions over 2 hours.
00:20:21
Speaker
According to research from developmental psychologist Dr. Susan Engel, fifth graders her team observed in the same two-hour time frame failed to ask their teacher a single question.
00:20:32
Speaker
And if we are really concerned with outcomes and closing achievement gaps, cultivating curiosity might be just what the doctor ordered.
00:20:40
Speaker
As a January 2020 headline in The Guardian reads, quote, schools are killing curiosity.
00:20:46
Speaker
Why we need to learn to stop telling children to shut up and learn.
00:20:49
Speaker
Pupils who ask lots of questions get better results, especially those from poorer homes, end quote.
00:20:56
Speaker
The article is based in part on the work of a research team from the University of Michigan, led by researcher Prachi Shah, which performed a longitudinal study of 6,200 children in 2018 and found a strong connection between school performance and curiosity, especially among children from poorer backgrounds.
00:21:15
Speaker
Dr. Shaw found that, quote, promoting curiosity in children, especially those from environments of economic disadvantage, may be an important, under-recognized way to address the achievement gap.
00:21:28
Speaker
Promoting curiosity is a foundation for early learning that we should be emphasizing more when we look at academic achievement, end quote.
00:21:36
Speaker
The article also quotes Paul Howard Jones, a professor of neuroscience and education at Bristol University, who affirms the importance of making space for curious questioning in school.
00:21:49
Speaker
Children should be prompted and encouraged to ask questions, even though that can be challenging for the teacher.
00:21:54
Speaker
We do need to find some time for questions during the day.
00:21:58
Speaker
There is not enough time in schools for creativity and following up on curiosity.
00:22:15
Speaker
The intersection of values, power, and policy at the center of education systems means that no decision made about practice is free from ideology.
Role of Evidence in Professional Action: A Broader Educational Experience Design
00:22:25
Speaker
As Biesta writes, quote, The important question, therefore, is not whether or not there should be a role for evidence in professional action, but what kind of role it should play, end quote.
00:22:37
Speaker
So it's as important for proponents of evidence-based education to turn the same critical lens on their own beliefs that they wield so quickly against other theories of learning and other ways of doing school.
00:22:50
Speaker
Our goal should be to design an educational experience for children that can serve a number of worthy and suitable educational goals, which may mean drawing from wells of professional practice, experience, and community engagement other than just those with the highest effect size.
00:23:06
Speaker
As educators, we have the responsibility to use our professional expertise to assess, judge, and rationalize the best possible methods.
00:23:15
Speaker
Understanding what best is, as in, is best improving test scores or repeating information, or is best motivating, inspiring, connecting with the community, reaching all students, and building a better future.