Evaluating the impact of large language models on training mathematical modelling skills

Project introduction and background information

In our previous project, Teaching mathematical modelling in the age of generative AI (https://www.4tu.nl/cee/innovation/project/10431/teaching-mathematical-modelling-in-the-age-of-generative-ai), we tested the performance of large language models (LLMs) on questions from our BSc and MSc mathematics courses. For different free-to-use and publicly available LLM models, we compared solutions provided by LLMs to our own model solutions.  Our objective was to observe how well LLMs perform on these problems such that, if the performance was good, we could consider whether and how LLMs should be integrated into our courses as a calculation tool, or consider rewriting the problems such that students could not use LLMs as a means of passing courses without learning required skills. A key takeaway from this work was that LLMs were more likely to provide correct answers to modelling problems so long as the posed problem did not contain any hidden implicit assumptions that a teacher would assume students to be aware of. 

LLMs have developed at a high pace since we did our test, but this has not resulted in LLM accuracy improving to such a degree that students can use LLMs carefree. In fact, the burden has shifted more towards the students. While LLM output can considerably improve with good prompting, LLMs do not have 100% accuracy, are still susceptible to bias and misinterpretation, and still produce hallucinations. The natural follow up question to this observation is, therefore, if student’s use LLMs to assist their education, will the use of an LLM have an impact on their ability to develop, recall, and transfer important mathematical skills?

To test this question, we took two student group’s where one group was allowed to use LLMs during the training of modelling skills and the other group was prohibited from LLM use. Both groups then had to solve a novel modelling problem without access to LLMs: this novel problem was similar in style to those completed in the training phase. To correct for a student’s so-called baseline mathematical level, pre-tests and post-tests around these tasks were included so that performance in the training and test phases was considered relative to this baseline level and we could also assess variability in a student’s performance.

Based on our results, we find that the use of LLMs during training has, at best, no impact on student performance in new LLM-prohibited situations or, at worst, student performance dropped between LLM-allowed and LLM-prohibited situations relative to their baseline level. These results strongly suggest that the use of LLM tools during student training should be critically evaluated and courses should make use of teaching or assessment methods to ensure a student has developed required skills.

Objective and expected outcomes

• Objective 1: to evaluate whether the use of LLMs during student training influences whether students can solve related problems when LLMs are prohibited.
• Objective 2: to quantify the relationship between student performance during and after training, corrected for a student’s baseline knowledge level.

Results and learnings

From our results, we observed statistically significant differences between students that were allowed to use LLMs, and performed better, compared to those prohibited from LLMs during training (baseline-corrected ANCOVA test, p=0.018). This statistical difference disappeared when students had to then solve a novel problem without the use of LLMs during the testing phase of our study (p=0.84). Such alterations in statistical differences between the two groups at different stages likely highlights how LLM usage allows students to produce improved results compared to LLM-prohibited students. In this way LLM usage allowed students to obtain statistically better results than the prohibited group during training, but this advantage was lost in the resulting LLM-free test stage.

To quantify the impact that LLM usage may have on student’s skill development and performance when LLMs are prohibited, we calculated an LLM-impact score. This score takes the grades during training and test phases, with a correction for baseline level. As such, if the impact score is positive then student performance on the LLM-prohibited test problem was better than on the training problem and, if the score is negative, student performance decreased between training and LLM-prohibited test phases. We observed here that, consistently, students not allowed to use LLMs during their training phase obtained positive impact scores. This suggests that all of these students improved their performance from training to test phase. For students that were allowed to use LLMs during the training phase, we observed two subgroups. In the first subgroup, students performed equally on both the training and test phases. The second subgroup showed negative values for 3 students which suggests that these students scored lower on the LLM-prohibited test phase than the training phase when LLM usage was allowed, and that this drop in performance was related to their baseline level. By only looking at students with positive or negative impact scores, we see that if students are prohibited from LLM usage during training then there is a 45.4% relative improvement in performance during later test phases. Whilst for students that are allowed LLMs during training, we observe a 102.5% drop in relative performance during later LLM-prohibited test phases (p-value < 0.005 when comparing I_LLM for students who did not use LLMs during training and those that were negatively impacted by LLM use with a two-tailed Welch test).

A technical report of our results can be shared upon request from Rob Smith: robert1.smith@wur.nl

Recommendations

• Recommendation 1: we have shown that LLMs can often (confidently) provide incorrect answers and make mistakes when solving modelling problems. Here, we show that relying on LLMs during training means that there is a risk that students will not learn the skills required to model without LLMs or recognise errors provided by LLMs (which is in line with the 'cognitive surrender'). Based on the results of our studies, we believe that, in courses, students need to be made aware of both the content and psychological pitfalls of relying on LLMs during their studies.
• Recommendation 2: we are aware of limitations to our study. This includes the small number of participants and the lack of extra control groups in our study amongst others. To confirm our conclusions and more concretely quantify the effect of LLM usage on students’ skill development, we would encourage that a larger study is undertaken with extra study groups that help differentiate between different forms of LLM usage.
• Recommendation 3: we do not advocate for a complete ban on LLMs in courses (which appears practically impossible). Rather the opposite: LLMs will likely remain and probably have to be integrated into education in an effective way, which implies their use should be accompanied by proper instruction and training of students and teachers as well as a wider implementation of AI-resistant assessment methods to evaluate if students have really acquired skills after a course. Ultimately, all users should be able to justify why they used an LLM tool, for what purpose, and how they evaluate the output of the LLM in relation to their study goal.

Practical outcomes

Our work helps to highlight the potential impact of long-term reliance on LLMs and a student’s ability to learn new skills. Whilst we have focussed on mathematical modelling skills in this study, our approach, metrics, and conclusions are likely applicable to a range of other subjects.