Efficient ≠ Good

Generated by ChatGPT

You’ve seen and heard it before. We’re now being told, usually with great confidence and little to no evidence, that Generative AI (GenAI) will “transform” education. This is the umpteenth instance of the same old claim that some technological tool will revolutionise teaching and learning^[1]. The newest claim is that GenAI will save teachers time, personalise instruction for the student, produce lesson plans, generate exercises, differentiate materials, and write feedback. The problem is that producing text quickly isn’t the same as teaching well, and saving teacher time isn’t the same as improving student learning.

A randomised field experiment by Alp Sungu, Benjamin Lira, and Angela Duckworth^[2] (Generative AI Can Harm Teaching) provides a much needed reality check. Across fourteen middle and secondary schools in Turkey, teachers were randomly assigned either to continue their teaching as usual or to receive access to a GPT-4o-based teaching tool, with some also receiving reminders^[3]. The analysis covered 193 teachers, 2,816 students and more than 14,000 student-course observations; not bad for a RCT in a real school under real learning conditions. Note: this wasn’t a self-report survey nor a lab study using an artificial task. It examined what happened when teachers used GenAI in real schools.

Students whose teachers received access to AI reported lower intrinsic motivation. They found their courses less enjoyable, less interesting and less important. The overall decline was 0.11 standard deviations, and was larger when teachers already were heavier AI users. Student confidence also declined, though the average effect was less statistically certain. On average, examination performance didn’t change significantly, which will no doubt allow enthusiasts to say that AI “did no harm”. But averages are often the nooks and crannies where educationally important effects go to hide.

Among students taught by lower-performing teachers, the use of GenAI reduced academic achievement by 0.129 standard deviations and their confidence by 0.183 standard deviations. Among students with stronger teachers, there was no statistically reliable harm and perhaps even a small positive effect. In other words, the technology sold as a way to compensate for differences in teacher quality appears, in this study, to have widened them. The Matthew Effect strikes again! Why? Stronger teachers may have been better able to judge, reject, adapt, and integrate what Chat produced. Weaker teachers, on the other hand, seemed to be more likely to let the tool substitute for the thinking that good teaching requires.

This distinction is essential. A lesson plan isn’t teaching, a worksheet isn’t instruction, and a set of questions isn’t formative assessment let alone retrieval practice. These products of GenAI only become educationally useful when a knowledgeable teacher selects examples, anticipates misconceptions, sequences explanations, decides what their students already know, judges where support is needed, and adapts what happens next. That work isn’t simply ‘clerical preparation’ surrounding teaching. It’s a/the central part of teaching! When we outsource it to GenAI or wherever (the teacher edition of a teacher-proof learning method, a colleague who has already done all the groundwork…), we are removing the cognitive activity through which we as teachers understand the content and the students we are about to teach. In Donald Schön’s words: our reflection on action and our reflection in action.

The usage data are also revealing. Roughly two-thirds of teachers’ conversations with the GenAI concerned producing teaching materials, such as lecture notes, homework, examinations, and syllabi (i.e., make this for me). Only a much smaller share involved instructional support such as addressing misconceptions or differentiation. The median conversation contained just two teacher prompts. In other words, this median of just 2 prompts means that “teachers typically accepted AI-generated outputs with minimal iteration, treating the tool as a generator of finished products rather than as an iterative collaborator”. This doesn’t look like (pardon my sarcasm) a thoughtful dialogue where a professional interrogates suggestions, tests alternatives, and refines a pedagogical approach. It looks much more like delegation or work and responsibility: Ask, Receive, Copy, Use. Hey, a new acronym ARCU 😊

And that may explain the motivational result. Students don’t experience teaching as the sterile transfer of technically adequate content from one container (the teacher) to another (the learner). They experience (and expect) a teacher; their examples, humour, stories, expectations and ways of explaining difficult ideas. GenAI tends towards being as smooth, generic, and plausible as possible. It can effectively and efficiently generate a lesson that looks like a lesson, just as it can generate an explanation that looks like an explanation. But generic instructional prose may remove precisely the personal and intellectual signature that makes a course coherent and a teacher credible. When students sense that materials haven’t really been thought through by the person standing in front of them, why should they invest themselves in them?

IMHO the equity implications of this research deserve particular attention. This study didn’t measure family income or socioeconomic disadvantage, so we won’t pretend that it directly proves greater harm for poorer pupils. What it does show is that the damage was concentrated among students taught by less effective teachers and research has taught us that lower-achieving students and those from poorer neighbourhoods and schools have poorer (i.e., lower-achieving) teachers^[4]. In addition, more advantaged students often have more resources: better educated parents, access to tutoring, a quiet study space, books, and the confidence to ask for help. Students with fewer resources depend more heavily on the quality of what school and their teachers provide. When AI weakens the teaching of those who most need support in exercising professional judgement, the pupils least able to compensate outside school are likely to carry the greatest risk.

There are limitations to this study. The intervention lasted just one semester, the examinations showed ceiling effects, and the tool was one particular implementation. Perhaps better training, stronger guardrails or better-designed systems would produce different results. But notice here how quickly the burden of proof shifts. Before evidence appears, AI is promoted as an obvious improvement. When evidence of harm appears or when it is shown to be worthless, we’re told that the implementation was imperfect or that future versions will be better. Education has seen this film before; Back to the Future XXI.

The conclusion here isn’t that teachers shouldn’t use GenAI. As a tool, it may be useful for low-stakes administrative tasks, locating resources, or generating possibilities that a teacher then evaluates critically. It can also be used as a sparring partner to stimulate one’s thinking or comment on proposals. But it should not be used to replace the intellectual preparation through which teachers build pedagogical content knowledge and make instruction their own. A tool that saves time by removing the thinking needed for good teaching isn’t efficient, it’s educationally expensive.

The question schools should ask isn’t, “How much time can AI save teachers?” It’s, “Which parts of teachers’ work must remain cognitively and professionally theirs because pupils’ learning depends on them?” Until we answer that question with evidence rather than enthusiasm or advertorials, scaling GenAI in classrooms isn’t innovation. It’s an experiment conducted on children which IMHO is morally and ethically unacceptable, with the greatest risks borne by those who can least afford another educational disadvantage.

---

^[1] In actuality, I know of only two technological revolutions in education, namely the printing press and the blackboard The printing press was a revolution as it made access to and spreading of information possible by mass producing textbooks and other scholarly works. The blackboard was a revolution because it shifted teaching from individual, oral recitation to collective visual instruction, thereby enabling one teacher to guide an entire room at the same time.

^[2] Yes, Professor Grit, and as you know not one of my favourites so you can safely assume that I was critical of the article.

^[3] Teachers were provided with the same tool plus weekly reminders containing individualised usage statistics to encourage continued engagement.

^[4] Babu and Mendro report that low-achieving students are twice as likely to be assigned to ineffective teachers than high-achieving students.
Babu, S., & Mendro, R. (2003, April). Teacher accountability: HLM-based teacher effectiveness indices in the investigation of teacher effects on student achievement in a state assessment program [Conference presentation]. AERA Annual Meeting, Chicago, IL