Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions

TL;DR

This paper evaluates pre-trained models for assessing AI-assisted educational questions, finding that Large Language models outperform traditional ML and transformer models. Strategic feature enhancements, such as text splicing and model retraining, improve performance on out-of-distribution datasets. The study highlights the trade-off between using pre-trained models and achieving robust performance.

Why It Matters

The ability to accurately assess AI-assisted educational materials is crucial for ensuring their pedagogical quality. This research has practical significance for educators and developers of AI-assisted educational tools, as it provides insights into improving the evaluation of these materials. Effective assessment of AI-generated educational content can enhance student learning outcomes.

Key Findings

• LLMs outperform traditional ML and transformer models on out-of-distribution datasets
• Text splicing improves model performance
• Model retraining provides the largest improvement across models and datasets