In August 2022, three researchers at Khan Academy, a popular math practice website, published the results of a massive, 99-district study of students. It showed an effect size of 0.26 standard deviations (SD)—equivalent to several months of additional schooling—for students who used the program as recommended.

....

Those gains, and many others like them reported each year, are impressive. Since use of these tools is widespread, one could be forgiven for asking why American students are not making impressive gains in math achievement. John Gabrieli, an MIT neuroscientist, declares himself “impressed how education technology has had no effect on . . . outcomes.” He was talking about reading but could equally have called out mathematics, the other big area in which education technology is widely used but growth in achievement has not followed.

A clue is in those wiggle words “students who used the program as recommended.” Just how many students do use these programs as recommended—at least 30 minutes per week in the case of Khan Academy? The answer is usually buried in a footnote, if it’s reported at all. In the case of the Khan study, it is 4.7 percent of students. The percentage of students using the other products as prescribed is similarly low.

Imagine a doctor prescribing a sophisticated new drug to 100 patients and finding 95 of them didn’t take it as prescribed. That is the situation with many online math interventions in K–12 education today. They are a solution for the 5 percent. The other 95 percent see minimal gains, if any.

...

It’s not at all clear that the program vendors are at fault, any more than you would blame a pharmaceutical company for the failure to see results among patients who didn’t take their drug. Indeed, the vendors point to data that students who use their program more show higher performance. But that is a correlation. As Hilary Yamtich, a fourth grade math teacher at a school in Oakland, California, who conducted a study of her own, points out, “students who are more motivated to learn are more likely to choose to use Khan.”