When the claims and the numbers in education research don’t add up, it’s really worth understanding why.
I’ve thought about this a lot over the last two weeks as we traveled through Portugal and Spain. As a little light reading for Lisbon and Madrid cafes, I’d brought along the two major meta-analyses of ethnic studies research along with my graphing calculator. This put my husband into the awkward position of having to explain to other cafe patrons – in an endearing combination of broken Portuguese and interpretative dance — what was causing his wife to double over with laughter, gripping her graphing calculator and pointing at the display. Ultimately, each time, he surrendered to the truth and shrugged. “She’s doing math.”
As we wandered through the galleries of the Museo Reina Sofía in Madrid, I was enjoying the collection of works by Picasso, Juan Gris, Marc Chagall, and the rest of the Cubist and Cubism-adjacent gang from the 1937 Paris International Exhibition when I spotted a large number of familiar-looking mobiles and engineering sculptures – works by Alexander Calder, Lowell Class of 1915, whose works I have always loved and taught my geometry students about.
That got me thinking about Lowell and about teaching math and starting school again in several weeks. I thought back to the embarrassing research papers I’d been reading. It’s disheartening to discover that yet another field of education research operates with such low standards of academic rigor and empirical research that it actually threatens, rather than supports, student outcomes.
A mischievous irony flashed across my mind. Lowell has been teaching math continuously since its founding in 1856, producing three Nobel Prize winners (so far), as well as renowned artists like Calder, plus entrepreneurs, medical researchers, politicians, lawyers, actors, comedians, and educators.
Using the illogic of the ethnic studies, one could argue that an SFUSD student who takes math at Lowell rather than at any other high school is 11,478 times more likely to earn a Nobel Prize or become a world-renowned artist whose works hang alongside those of Picasso … but that would be a claim we could safely and technically call B.S.
As a matter of fact, it’s the essence of what business psychology and decision-science researchers – who use rigorous, standards-based experimental designs and methods – would call pseudo-profound B.S.: a set of “seemingly impressive assertions that are presented as true and meaningful but are actually vacuous.”
That is a very good description of too much of the research coming out of leading graduate schools of education right now.
Back in 2021, when our team at Families for San Francisco was working on the math report that started the turnaround of math education in San Francisco and California, my coauthors (Paul Gardiner, Maya Keshavan, Seeyew Mo, Patrick Wolff, Rex Ridgeway, Tita Bell) and I looked critically at both the mathematics and the research behind the bold and revolutionary claims being made by a well-known Stanford Graduate School of Education researcher about SFUSD’s radical math program. We evaluated these claims in light of standard statistical methods, common sense, and public records requests, and sure enough, these claims turned out to be, in fact, pseudo-profound B.S.
But as a result of our research – plus three successful recalls, one ballot initiative, and a number of lawsuits – SFUSD finally revised its math program to conform to the laws of both mathematics and reality-based research.
So when I read the latest interview on SFGate with Nikhil Laud, co-coordinator of SFUSD’s ethnic studies program, my B.S. detectors went off again. Speaking about the two Stanford research studies they use to justify the program, he burbled, “I’m optimistic that the district has reaffirmed a commitment to a program that’s been wildly successful, probably maybe the most successful academic program in SFUSD history…. I think you don’t have a program that’s this wildly successful for students if you’re not continuously trying to improve.”
This Trumpian rhetoric motivated me to dive into the research on ethnic studies and the high-profile mathematicians’ critiques of this research. My theory was that if this is the basis on which SFUSD’s Board of Education is claiming we should spend $2 million a year and tie up two semesters of every ninth grader’s course schedule, it had to be worth understanding better, right?
Leading researchers have been sounding the alarm on Ethnic Studies’ claims of miraculous outcomes for students. When will SFUSD policymakers be ready to start listening?
What I discovered, unfortunately, was … wait for it – an awful lot of B.S.
If you want to understand the landscape of ethnic studies research on effectiveness, then there are two key pieces of writing by experts you need to read. The first is a 2021 letter written by 35 leading California mathematicians that raises serious questions about the review of the literature used by the California State Board of Education to justify its Ethnic Studies Model Curriculum (ESMC). The second is from 2022, and is a withering and often hilarious, step-by-step demolition derby by two social sciences statisticians of the rather epic statistical errors and methodological mistakes in the landmark study of SFUSD’s ethnic studies outcomes by another well-regarded Stanford Graduate School of Education researcher.
Neither of these left me with any confidence that the claims being made for SFUSD’s ethnic studies course make it worth having as a two-semester graduation requirement.
But I was willing to read on anyway.
As much as I loved the step-by-step demolition derby piece, I’m taking the high road and focusing on the evidence base for ethnic studies. And I’m going to let you in on a secret, friends. It’s pretty thin. Also unprofessional – enough that other scholars have been calling it out in public since 2021.
Let’s start with the letter by the 35 diverse and world-recognized quantitative researchers, most of whom are in California.
The authors list reads like a who’s who of California’s quantitative research brain trust. They’re the perfect guides to the skills and practices one should be able to expect from other quantitative research scholars.
In any doctoral research program, you develop the ability to organize an overview of all existing research on a given topic. This is called a review of the literature. As a scholar, you are expected to be able to accurately, insightfully, and effectively summarize, assess, and answer questions about this background, including process and methods, and it’s pretty standard across fields of study. That’s what makes it a broadly transferable skill.
In their letter to state Superintendent Tony Thurmond, the 35 scholars sounded the alarm about the four foundational articles cited in the State Board of Education’s Ethnic Studies Model Curriculum as “proof” of the benefits Ethnic Studies benefits for students. And hoo-boy, what they report falls shockingly short of what I remember as the minimum standards for published research.
Their three main conclusions speak for themselves:
“[G]iven that the model curriculum was presumably drafted by expert educators in the field of ethnic studies, the embarrassing carelessness with which this section was written – inaccurate and mistaken attribution to empirical research, or no attribution at all – offers a very poor impression of the academic rigor of this ethnic studies model curriculum, and of the field of ethnic studies more generally.” (p. 10)
“It is unconscionable that with so much at stake, the State Board of Education would mislead California citizens into believing that bold claims about the benefits of ethnic studies courses for K-12 students are supported by considerable and robust empirical evidence, when this is simply untrue.” (p. 10)
“Therefore, in the name of honesty and academic integrity, we urge you to remove from the ESMC the entire section entitled ‘The Benefits of Ethnic Studies,’ or at least all of the overarching and specific claims that are made in it.” (p. 10)
These assessments raise red flags about the entire enterprise of ethnic studies as an evidence-based field; this is not merely nit-picking about punctuation or formatting.
The errors fall into three categories: (1) errors of experimental design; (2) errors in reporting and citing of results and key information; and (3) egregious errors of interpretation of results. This pretty much covers the full spectrum of what constitutes rigor and method in all forms of empirical research.
None of these are subtle problems. Researchers failed to adhere to basic design standards for treatment and control groups, including the failure to control for self-selection and/or bias. They mixed experimental and nonexperimental studies and results so severely that these limit the field’s ability to generalize their results or support the bold claims being put forward on their basis. Some reported results actually suggest the opposite of what is being claimed. Harms are labeled as positive effects. “Theoretical arguments and public enthusiasm” are summoned as empirical evidence where quantitative evidence would be expected as proof of the efficacy of the treatment under study. Critical success factors are not defined before being celebrated as having been met or exceeded. The small scale of many studies renders them unusable for broad generalization.
These flaws are matched note for note by the careless mathematical and statistical errors cited in the more mathematical meta-analysis by Sander and Wyner, which I leave for the interested reader to devour with a cold beer.
And that’s just the evidence base for the curriculum. We haven’t even talked about issues of instructional practices or about rogue teachers contaminating the set treatment group lessons by introducing their own false and bias-driven materials.
Is it any wonder that parent dissatisfaction is finally boiling over into public awareness after so many years of promising miraculous results if SFUSD parents and voters would just agree to spend $2 million a year of PEEF funding on ethnic studies without any measurement or oversight?
What will it take for Dr. Su and the Board of Education to wake up?
