Large Language Models (LLMs) like ChatGPT are becoming so good, in fact, that they are starting to make traditional assignments like essays and conceptual questions almost obsolete.
“The case for no AI no longer exists,” says Robert Brayassociate professor of operations at Kellogg who teaches a course on how to effectively use generative AI. “If it’s homework, students will use AI.”
From Bray’s perspective, resisting this trend is fighting a losing battle. But if it could adapt AI models to guide students through their assignments – and not just give answers like standard LLMs do – then students could rely on AI as a helping hand rather than a crutch. This would give students the opportunity to not only develop their AI skills but also, perhaps for the first time, feel excited about doing their homework.
“If students still need to use AI, it gives teachers a way to strategically customize the AI and have some control over operations,” he says. “But the question is, ‘Can teachers actually create an AI homework experience that outperforms the already incredibly good default models?’ »
Bray met this challenge head on, focusing on one of his data analytics courses for MBA students.
The course included 20 teaching sessions and 19 quizzes that tested students’ ability to analyze operational data using code, with the help of ChatGPT. Each quiz had corresponding optional assignments that helped prepare students for the quiz.
Bray collaborated with Sébastien Martinalso Associate Professor of Operations at Kellogg, to create a custom AI agent specifically for this course. They fed the existing ChatGPT-4o base model with a set of instructions explaining each assignment and how the AI should help students complete them. The AI’s imperative was to interact with students as a sort of virtual tutor.
For example, teachers prompted the model to “help students solve each question, but not blurt out the answer. Try to encourage the student to solve as many problems as possible; give small hints when possible.”
“While ChatGPT thinks its job is to give you the correct answer, our AI agent actually knew its job was to teach students,” says Bray.
Before each quiz, Bray randomly asked half of the students to do their homework using this AI tutor. He asked the other half to do it using standard ChatGPT. It then compared the experience of students using the AI tutor to that of ChatGPT across a total of 17,946 homework questions.
Overall, students not only preferred using the AI Tutor over ChatGPT, but also found the AI Tutor more helpful.
Nearly twice as many students reported having a “very positive experience” using AI Tutor for homework (47% of students), compared to using ChatGPT (26%). And 40% said the AI tutor was “very helpful,” compared to 30% for ChatGPT. Both of these differences were statistically significant. The professors also found that students’ preference for the AI tutor grew stronger as the tasks became more complex.
“Students tend to like that homework is more like a video game, where you chat with an intuitive AI agent,” says Bray.
Although students preferred the AI tutor, its use ultimately did not change the amount of time they spent on homework or the number of questions they attempted to solve. There was also no significant difference in average quiz scores when students did their homework with the AI tutor compared to ChatGPT.
But this isn’t necessarily a blow to AI. “It is well established in the literature that it is very, very difficult for almost any technological innovation in the classroom to significantly influence exam scores,” says Bray. “Actual quiz performance depends more on students’ awareness: how much concentration they individually put in and how motivated they are to study. »
Indeed, the fact that the AI tutor helped improve students’ homework experience is a worthwhile achievement in itself. “Homework is something students have to do,” Bray says, “so if we can make it a better experience, that’s a win.” »
Additionally, the study’s findings can be applied to all kinds of teams and businesses beyond the classroom.
“Every company offers some form of training, and people learn more on the job than in school,” says Bray. “So if you just put a little effort into teaching an A.I. [model] how he should behave – and training him to become a sort of tutor – this can actually provide workers with a nicer and better experience.
