What Are They Teaching Them In Those Schools?

We receive news: Brown Professor Suspects Majority of His Class Used AI to Cheat:

For the first time since he started teaching Welfare Economics and Social Choice Theory nearly two decades ago, Brown University economics professor Roberto Serrano gave his students a take-home midterm this spring. Quite a few students had expressed anxiety about being in a classroom after a gunman killed two students and injured nine in a December mass shooting at Brown, and so “it was appropriate,” he said, to allow students to take their exams at home.

But by the end of the semester, Serrano regretted the decision. Dozens of students in the class likely used artificial intelligence to cheat and earn perfect or near-perfect scores on their midterm, he said. Serrano in turn made the final exam in-person, which led more than a dozen students to drop the course and even more to fail it.

It’s worth clicking through just for the plot showing scores in that take-home midterm compared to the final exam score. I bet you can pick out the students who did not use AI on that midterm test, but I definitely have my suspicions about students 57 and 58!

It is abundantly clear at this point that AI has killed the take-home test. A significant proportion of students will simply delegate the whole of the work to AI. They will not use it for research, or to test their arguments, or to help them with grammar. They will simply let the AI do the work, and go do something else.

The obvious problem is that, even for students who attend class and pay some level of attention, that limited engagement with the material is not remotely sufficient. After the change to an in-person test, eighteen students dropped the class entirely, and nine students did not take the exam — and even so, three students scored zero in the exam. Two of those had scored a full hundred points in the take-home test! What are we even doing here?

Things were better in my day

I’ll put my own prejudices out there: I went to school in Italy all the way through the end of high school, and my own kids are at various stages of the Italian school system. There are many idiosyncrasies of the Italian education system, but one big thing that it gets right is that a big chunk of students’ evaluation]¹ is based on in-person exposition (interrogazione): you stand up at the teacher’s desk, the dreaded cattedra, and answer their questions until they are satisfied. This could be an exercise at the blackboard for a maths problem or a Greek declension, or it could be a structured dialogue about a piece of literature or a period in history.

This sort of thing is very resistant against getting an AI to do your homework. It doesn’t matter if your exercise book is perfect; at some point you have to stand up on your hind legs and be able to answer questions about the material. You may be able to snow the teachers some of the time, but they’ve been at this game for many years, and can generally sniff out weakness and expose it mercilessly.

Notably, the positive uses of AI that apologists trot out, helping students prepare themselves and understand the material, are not impacted by this style of evaluation. If you know the material, there is little difference between writing it out at home and in class. Sure, some teachers are sticklers for exact dates or whatever, but that is known going in: crusty old prof So-And-So wants the exact year for everything, so I’ll make sure I drill on at least the key dates. But most are more interested in making sure that students have the shape of the material, what comes before and after, how events influence each other, or whatever the appropriate evaluation is.

But students who simply prompt the AI and submit its output not only do not understand the material — after all, it has been parsed by the bot, not by them — but they apparently lose the ability to read the AI output, even to detect obvious tells so they can be removed before submission, whether in an academic context or a professional one, such as writing an article for a magazine:

Maybe this is why AI is taking over jobs that would previously have been done by hard-working humans, such as creating provocative social media posts to influence elections?

---

1. Most of the rest is based on in-class written tests: composition, translation, longer maths problems, and so on. Homework is graded in the moment, but it’s mostly not part of the final score at the end of the year, except that it may be taken into account if a student falls between two marks. Basically, a diligent (and punctual) student will have a better chance of getting their mark rounded up. I did not get marks rounded up very often. ↩