A University Trial Just Found AI Tutoring Made Students Worse Off

For the past three years, nearly every pitch for artificial intelligence in education has rested on a single borrowed statistic. In 1984, the educational psychologist Benjamin Bloom reported that students who received one-to-one tutoring combined with mastery learning outperformed their conventionally taught peers by roughly two standard deviations — a difference so large it moved an average student to the ninety-eighth percentile. Bloom called it the "two sigma problem," and he meant it as a problem: individual tutoring works beautifully and no society can afford to give it to everyone.

Then came large language models, and the problem looked solved. Commercial claims now circulate freely that an AI tutor can deliver Bloom's effect at the marginal cost of a server. Infinite patience. Instant availability. No scheduling, no fees, no human unpredictability. It is a genuinely moving vision, and we have been as drawn to it as anyone.

A new randomized trial should give all of us pause.

What the study found

In a working paper released through Brown University's Annenberg Institute, Liu and colleagues report results from a randomized trial of 2,379 undergraduates given access to a course-integrated AI tutor. This was not a side experiment with a chatbot bolted onto a syllabus as an afterthought; the tool was built into the course environment, available to students as a standard part of their work.

Students randomly assigned access earned final grades 0.37 standard deviations lower than their peers without it. Their participation on the course platform fell by 0.90 standard deviations.

Those numbers deserve a moment. An effect of 0.37 SD in the wrong direction is not noise and not a rounding error — in education research, it is a large effect, the kind that usually gets celebrated when it points the other way. And the participation drop is nearly a full standard deviation. Whatever the AI tutor was doing, it was not supplementing students' engagement with the course. It was replacing it.

The distributional finding is the one that should sting most. The largest losses fell on first-generation college students — precisely the population that "democratized tutoring" is always invoked to serve. The tool that was supposed to close a gap appears, in this trial, to have widened one.

What the study does not show

We want to be careful here, because the temptation to overclaim runs in both directions.

This is one trial, of one tool, in one course, at one institution. It is a working paper — circulated for comment, not yet through peer review. A different AI tutor, differently designed and differently introduced, might produce different results. It is entirely possible that the specific implementation mattered enormously: how the tool was framed, what it was permitted to give away, whether it scaffolded thinking or short-circuited it.

Nor does this study tell us that AI has no place in learning. It tells us that this configuration — an AI handed to the students as a replacement for an actual, human tutor or teacher — did harm in a setting where almost everyone expected it to help. That is a narrower claim than "AI doesn't work," and a more useful one.

But it is also the first large, well-powered randomized test of the exact product category that has been sold hardest: AI replacing teachers.

The finding that should worry parents most

A 0.37 SD grade drop is bad. But grades are a lagging indicator, and they can be explained away — maybe students leaned on the tutor for assignments and were caught short on exams. The 0.90 SD fall in course-platform participation is something else. It describes students withdrawing from the course itself.

This is the mechanism worth sitting with: A sufficiently helpful machine can make the hard parts of learning feel optional. Why wrestle with the problem set discussion, why post the half-formed question, why show up to the slow communal work of figuring something out, when a patient oracle will hand you a clean answer at two in the morning? Why risk failure?

The AI did not make students lazy; it made effort feel unnecessary, which is a subtler and more corrosive thing. Productive struggle is not an unfortunate side effect of learning. It is most of the mechanism.

And the thing we're stuck on — perhaps because, at Cicero, we offer the opposite in real, human, one-to-one learning — is a chatbot cannot notice that a student has stopped trying.

It can notice that a student has stopped typing, in the sense that the conversation ended. It cannot notice that the questions have gotten thinner. It cannot hear the difference between a student who is stuck and a student who has quietly decided the course is beyond them. It does not know that this is the third week in a row the work has arrived to their session late or haggard, or that something has changed at home, or that a bright student has started hedging every answer because they are afraid of being wrong in front of people. The robot has no stake in the student's future, no memory of who they were in September, and no capacity to be disappointed in a way the student would care about.

That last item is not a sentimental flourish. It's teaching. Ask any teacher to describe their craft and they speak in deeply human ways, not about grades and achievement, but about connection and the "ah ha" journey that every learner goes through. A great deal of what makes one-to-one instruction work is the simple, slightly uncomfortable fact of being known by someone whose good opinion you want. Responsiveness — the ability to answer instantly and correctly — is the part of tutoring that machines have genuinely mastered. Relationship is the part of teaching that they'll never approach. The trial suggests we may have been confusing the two.

Where AI actually earns its place

None of this makes AI useless in education.

Education Week's survey of the emerging research — drawing on a review from Stanford's SCALE initiative and the National Student Support Accelerator — reaches a conclusion that fits this trial uncomfortably well: the strongest evidence is for AI used by tutors, not by students alone. The promising results cluster around tools that make a human instructor better — coaching them on questioning technique, surfacing what a student misunderstood, handling the clerical overhead that eats into instructional time.

That is a far less exciting story, and a far more plausible one; and one that we see in practice here at Cicero. Much of our admin is handled either by AI or by tools that AI has built.

For parents, at back-to-school night

The 74 recently published a wry observation about how differently adults are reacting to all this — anxious parents, nervous teachers, and coaches who seem entirely unbothered. The coaches are worth paying attention to. Nobody has yet suggested that a swim team can be run by an app, because everyone understands intuitively what a coach does: watches, notices, corrects, insists, and cares whether you showed up. The reps matter, but the person watching the reps is why you do them.

Academic learning works more like this than we usually admit. The teacher's real job is not information delivery. It is attention — noticing the misconception before it solidifies, noticing the disengagement before it becomes a grade, noticing the student before they become a case.

Here's what we take from this study: an AI tutor can answer every question a child asks, and still miss the moment they stopped asking.The two-sigma result was never really about information. It was about someone being there.

---

Further reading

• The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial — Liu et al., EdWorkingPaper 26-1598, Annenberg Institute at Brown University
• When Does AI Help Most With Tutoring? What Emerging Research Says — Education Week
• AI & Back-to-School Night: Anxious Parents, Nervous Teachers and Chill Coaches — The 74