AI-powered tools are increasingly being integrated into K-12 instruction, offering transformative possibilities for educators [3]. Recent research provides valuable insights into how mathematics educators interact with conversational AI during lesson preparation, highlighting both promising applications and areas needing improvement.
In this blog, we share findings from a comprehensive qualitative analysis of over 3,400 conversations between mathematics educators and AI agents, examining how effectively AI meets educators’ instructional needs across various educational contexts.
Educators Seek Context-Specific AI Guidance
Educators frequently turn to AI tools for assistance tailored to their specific teaching environments and student needs. Approximately 42% of educator-AI conversations explicitly referenced distinct educational contexts, with mixed-ability classrooms being the most common scenario (14.58%). Other notable contexts include classrooms with reluctant participants, below-grade-level students, gifted learners, students with special education needs, and English Language Learners (ELLs). Educators clearly expect AI responses that address these specific scenarios rather than generic recommendations.
Instructional Practices in AI Conversations
When educators engage with AI tools, their inquiries center around several core instructional practices, notably differentiated instruction, real-world connections, assessment strategies, and high-level cognitive engagement. Within differentiated instruction, teachers predominantly seek strategies for tiered scaffolding (20.6%), emphasizing the need to address varying mastery levels effectively. In terms of engagement, educators often request AI-generated real-world problems or project-based activities (16.8%), highlighting their intent to make mathematics relevant and meaningful for students. Besides, educators also utilize AI’s support for modeling problem-solving process (23.9%) and explain mathematical concepts and languages (15.2%) to cultivate high-level critical thinking.
Educators teaching student groups with special needs, such as ELL or special education students, particularly emphasize adaptive, context-specific guidance. This reflects AI’s capability to support instruction and provide specialized instructional materials and actionable strategies that align with specified learning and teaching needs.
Evaluating AI’s Responsiveness: Accuracy, Relevance, and Usefulness
To determine how effectively AI tools meet educators’ needs, AI responses were evaluated based on accuracy, relevance, and usefulness [1, 2, 4, 5]. Educators consistently rejected AI responses scoring lower in these dimensions, especially if answers were irrelevant or lacked actionable guidance. Conversely, AI responses rated highly for relevance and usefulness were explicitly accepted, underscoring educators’ preference for practical, context-aligned support.
Interestingly, AI generated content maintained high accuracy even without explicit contextual information. However, relevance and usefulness significantly increased when educators provided detailed context and instructional practices. This suggests the importance of educators providing clear and specific prompts, as AI tools could greatly benefit from enhancements that enable proactive anticipation and deeper contextual understanding.
The Adaptive Potential of AI—And Its Current Limits
The research confirms that AI tool exhibits strong adaptability in addressing certain educational needs, particularly differentiation strategies for mixed-ability classrooms and engagement techniques for fostering positive classroom climates. AI’s effectiveness in education varies across different contexts. While generally helpful, it shows limitations when requiring accommodating special needs for advanced learners or when addressing behavioral and emotional needs, AI’s responses sometimes lacked accuracy, leading to potential misunderstandings that could adversely affect the educational process. These challenges become more apparent for nuanced instructional practices and educational context combinations, such as creating visual aids for a specified educational context or promoting critical thinking in less engaged classrooms, indicating a gap between AI capabilities and various contextualized educational needs.
Additionally, proactive AI behaviors, such as initiating questions to gather more contextual information, occasionally reduced educator engagement. This highlights the delicate balance AI must strike—being proactive yet user-friendly to encourage consistent engagement.
Recommendations for Enhancing AI Integration in Education
These findings underscore several actionable recommendations for educators and developers:
- For Educators: Clearly specify your instructional context and needs when interacting with AI. This significantly enhances AI’s ability to provide relevant, actionable guidance.
- For Developers: Improve AI’s capability to infer context, proactively anticipating educators’ instructional challenges without compromising usability. Tools should offer adaptive, relevant, and practical solutions tailored to various educational scenarios.
The Road Ahead
While AI shows considerable promise in supporting educators with context-sensitive instructional practices, substantial room remains for growth. Addressing AI’s adaptability limitations will require concerted efforts by researchers, educators, and technology developers alike. Through continued collaboration and innovation, AI can become an indispensable partner, empowering educators to meet the needs of all students.
At Colleague.AI, we remain committed to exploring and refining the educational applications of AI. Stay tuned for more insights and innovations as we continue supporting educators in delivering exceptional instruction.
- Boyatzis, R.E. 1998. Transforming Qualitative Information: Thematic Analysis and Code Development. SAGE Publications, Inc.
- Brooke, J. 1986. SUS – A quick and dirty usability scale. (1986).\
- Mills, A., Bali, M. and Eaton, L. 2023. How do we respond to generative AI in education? Open educational practices give us a framework for an ongoing process. Journal of Applied Learning & Teaching. 6, 1 (Jun. 2023). DOI:https://doi.org/10.37074/jalt.2023.6.1.34.
- Vlachogianni, P. and Tselios, N. 2022. Perceived usability evaluation of educational technology using the System Usability Scale (SUS): A systematic review. Journal of Research on Technology in Education. 54, 3 (May 2022), 392–409. DOI:https://doi.org/10.1080/15391523.2020.1867938.
- Yousefzadeh, R. and Cao, X. 2023. Large Language Models’ Understanding of Math: Source Criticism and Extrapolation. arXiv.
