๐๐ก๐ฒ๐ ๐ฝ๐ฟ๐ฒ๐ฝ๐ฟ๐ถ๐ป๐๐ Grading feels like a Turing test. Here's something we can do about it!
It often feels like we're evaluatingย students' prompting skills rather than their understanding. As AI tools become common, our research explores a constructive path forward beyond controlling AI use. Our core idea is shifting focus from final product to learning process by searching for learning in student-AI conversations.
๐ข๐๐ฟ ๐ฎ๐ฝ๐ฝ๐ฟ๐ผ๐ฎ๐ฐ๐ต:ย We developed a taxonomy to classify student-GenAI interactions, then annotated 1450 interactions (i.e., prompts) from 70 graded essays (2 writing-intensive courses: BSc and MSc). We connected interactions with learning indicators via essay grades and interaction quality evaluation.
To identify learning in GenAI interactions, we developed the ๐๐ฅ๐๐ฉ๐ย framework. It identifies two critical patterns: students actively steering the conversation (๐๐ถ๐ฟ๐ฒ๐ฐ๐๐ถ๐๐ฒ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด ๐๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป) and demonstrating knowledge throughout dialogue (๐ฉ๐ถ๐๐ถ๐ฏ๐น๐ฒ ๐๐ ๐ฝ๐ฒ๐ฟ๐๐ถ๐๐ฒ).
Our research led to two main findings:
๐น ๐๐ผ๐ ๐๐๐๐ฑ๐ฒ๐ป๐๐ ๐๐ฎ๐น๐ธ ๐๐ผ ๐๐ ๐ฟ๐ฒ๐ณ๐น๐ฒ๐ฐ๐๐ ๐๐ต๐ฒ๐ถ๐ฟ ๐น๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด.ย Our data showed strong positive correlation between high-quality GenAI interactions and higher essay grades (see image). This means analyzing chat logs can reliably identify learning.
๐น ๐๐ณ๐ณ๐ฒ๐ฐ๐๐ถ๐๐ฒ ๐๐ ๐๐๐ฒ ๐ถ๐ ๐ฝ๐ฎ๐ฟ๐๐ป๐ฒ๐ฟ๐๐ต๐ถ๐ฝ, ๐ป๐ผ๐ ๐ฑ๐ฒ๐น๐ฒ๐ด๐ฎ๐๐ถ๐ผ๐ป. High-quality GenAI interactions involved treating AI as a "partner" to co-develop ideas. Lower-quality interactions involved task delegation or basic search use.
Although our findings apply primarily to academic argumentative writing, teachers can use ๐๐ฅ๐๐ฉ๐ย as a guiding framework to tailor the assessment criteria for student-GenAI interactions to their (writing) course-specific learning objectives. Our paper discusses some key considerations related to this approach.
This approach allows us to move beyond the assessment dilemma and focus on what truly matters: helping students navigate AI technology thoughtfully, including when to use it and when not to.
๐ฃ๐ฟ๐ฒ๐ฝ๐ฟ๐ถ๐ป๐ ๐ต๐ฒ๐ฟ๐ฒ๐: https://lnkd.in/eEfRMXD5
Authors: Manuel Oliveira, Carlos Zednik, Gunter Bombaerts, Bert Sadowski, Rianne Conijn