In this blog article, we give you a quick rundown of three recent studies that have looked at how AI tools can help in the classroom. Each of these studies used randomized control trials, where one group of educators was given access to an AI tool, and another group was not. With the randomized design, we can have high confidence in these studies’ findings. First, teachers were able to save about 25 minutes a week using ChatGPT to help prepare their lessons. Second, a real-time teacher feedback tool increased the number of focusing questions teachers asked in the classroom. Finally, tutors working with AI assistants were able to help more students reach mastery on mathematics topics.
All three of the studies used rigorous designs, and while two of them are still going through the peer review process, they provide some of the best information we have to date about the cutting edge of AI application in education. Here at Colleague.AI, we are closely following all the latest developments (and have more than a few of our own papers in the works!) so we can make the best tools for you.
ChatGPT in Lesson Preparation
A major goal for AI tools is to save teachers time and make their jobs easier. Now, we have some strong evidence showing this can really happen. In this randomized trial, teachers at 34 schools were randomly chosen to use ChatGPT in teacher lesson planning, while teachers at another 34 schools were asked to refrain from using generative AI tools. The study focused on 7th and 8th grade science classes. Teachers participating in the study kept a weekly diary logging their working hours for ten weeks. The teachers using ChatGPT reported spending about 56 minutes per week on lesson preparation, 25 minutes less than the teachers who didn’t use ChatGPT.
What did they use ChatGPT to do? Mostly, the teachers created questions and quizzes for their students. They didn’t use it across the board in their lesson planning. The researchers also had an expert panel review the created materials and didn’t find any difference in quality for those made with AI help. It’s important to know that in this study, the teachers were just using regular ChatGPT, and not an AI tool specifically designed to support teachers. This finding suggests a likely minimum for the amount of time teachers can save in lesson planning.
Automated Feedback for Teachers
General purpose AI Tools aren’t just being used to help teachers save some time. Researchers are also testing tools to give teachers automated feedback to help improve their classroom practice. In this study, teachers used a phone app from TeachFX that records audio in their classroom and provides analysis of the classroom instruction taking place. 523 Utah teachers participated in the study. All the teachers recorded at least one class session and all of them had access to general reports about their classroom instruction. Half of the teachers were also randomly given additional feedback about their use of focusing questions (questions that ask students to explain their thinking or share their reasoning) in the classroom. Focusing questions have been found to be a highly effective practice for improving student learning.
Teachers’ receiving additional feedback about focusing questions did substantially increase their use of this pedagogical practice. However, it’s unclear if the increased number of focusing questions turned into broader gains in student engagement or learning. In general, teachers in the study had trouble trusting the tool and the automated feedback it provided. The researchers suggest that future development should embed the automated recording analysis with existing, trusted, professional learning programs and coaches.
Tutor Copilot
Finally, we turn to a study that found real improvements in student learning. This study looked at using AI to support tutors. Novice tutors often struggle to give the right feedback to their students – it’s hard to know what questions to ask and when and how to ask them. The study found that AI support can help tutors, and especially less skilled tutors, better reach their students.
Here, the researchers studied an online tutoring program where human tutors worked with 3rd- 8th-grade students who had previously failed to reach proficiency on their state math test. 386 tutors were given access to an integrated AI assistant that provided advice to the tutor about how to best help the student,t and another 396 tutors continued to work as they normally did – without AI. The AI assistant provided suggestions like encouraging the tutors to ask questions of the student, provide hints, or explain concepts, depending on how the student was engaging.
The researchers found that when tutors were supported by an AI, 66 percent of students successfully answered an end of session exit ticket, compared to only 62 percent of students whose tutors didn’t have an AI assistant. This 4 percentage point difference is statistically significant. Interestingly, this effect was much stronger for the least skilled tutors (as evaluated by the agency overseeing the tutoring program before the experiment took place).
So, what did the AI do? The researchers looked into the advice it was giving and found that the tutors with the AI assistant were more likely to ask students to explain their thinking and more likely to ask guiding questions. The AI assisted tutors were also much less likely to directly give out answers in the tutoring session.
AI Tools Are Helping Teachers
Taken together, these three randomized trials provide strong evidence that AI tools can positively support educators in the classroom. The impacts found aren’t enormous, but with the rigorous randomized trials, we can have high trust that these studies represent real steps forward. Building on this research we can continue learning how to effectively integrate AI tools into learning environments as a helpful colleague for teachers.
1Roy, P., Poet, H., Staunton, R., Aston, K., & Thomas, D. (2024). ChatGPT in lesson preparation: A teacher choices trial – Evaluation report. Education Endowment Foundation.
2Demszky, D., Liu, J., Hill, H. C., Sanghi, S., & Chung, A. (2023). Improving Teachers’ Questioning Quality through Automated Feedback: A Mixed-Methods Randomized Controlled Trial in Brick-and-Mortar Classrooms. EdWorkingPaper No. 23-875. Annenberg Institute for School Reform at Brown University.
3Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024). Tutor CoPilot: A human-AI approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017.
