Generative AI and the assessment of academic writing

Project introduction and background information

In its Strategy 2030 document (Executive Board TU/e, 2018: 31), TU/e stresses the importance of digitization, to allow learning at any place, at any time and to support adaptive and personalized instruction and feedback. Intelligent systems could be used to fulfil this need, by providing personalized, automated and timely feedback.

The power, sophistication, and societal prominence of generative AI systems such as ChatGPT has only grown. As is evidenced by discussions in the academic, professional, and popular media, these systems pose an unprecedented challenge to established structures and institutions in many different domains, including higher education. The particular problem facing higher education is that generative AI undermines the principle of constructive alignment between learning objectives and assessment methods, impacting students and teachers alike.

For students, the advent of generative AI is poised to change the skills they require to succeed after leaving university and entering the workforce. Given their general-purpose nature, systems such as ChatGPT are likely to be used in many different domains, from software engineering to journalism and marketing. Graduating students entering these domains must possess relevant skills, which are likely to differ from the ones higher education is traditionally designed to promote. In particular, when performing writing tasks such as conducting literature review or compiling reports, students will no longer need to do so on their own, but will instead be expected to collaborate with AI technology to enhance the speed and quality of their writing. Among others, this will require mastery of skills such as prompt-engineering and machine summarizing, as well as a critical engagement with AI-generated content. Higher education generally, and TU/e specifically, should equip students with these skills, and will therefore need to identify and articulate “future-oriented” learning objectives to be achieved in writing-based university courses.

For teachers, the increased power and availability of generative AI challenges their ability to assess student learning. Because tools such as ChatGPT can be used to produce deliverables such as essays, reports, diagrams, and code, the provenance of these deliverables can no longer be traced to individual students as opposed to sophisticated machines. As a consequence, it is unclear that students are actually satisfying the stated learning objectives, and teachers will require “AI-proof” assessment methods that allow them to measure the extent to which students have mastered the ability to write clearly and effectively, either in collaboration with relevant AI systems, or on their own.

If TU/e is to stay at the forefront of societally relevant engineering education, it will have to promote the implementation of new education methods to promote effective writing. Crucially, this action will have to occur sooner rather than later: as generative AI systems become even more powerful, available, and easy to use, it is in the university’s interest to stay ahead of developments and be proactive rather than reactive.

Objective and expected outcomes

The main objectives and expected outcomes of the project are manifold, consisting of the following working packages:

WP1: Mapping the literature on future-oriented and GenAI compatible higher education

This work package will study the impact of Generative AI (GenAI) on aligning learning objectives, activities, and assessments in higher education. The main focus will be on identifying future-oriented learning goals and GenAI-compatible pedagogical and assessment methods based on literature searches. 

Expected outcomes:

• Report 1a: Report summarizing key insights about future-oriented learning objectives, and lists of learning activities and assessment methods useful to accommodate GenAI in education (Dec 2024). Closely related, research insights related to this WP will be presented at the 4TUCEE End of Year event (November 2024).
• Report 1b: An updated version of report 1a incorporating any relevant updates in light of AI technology advancements (June 2025).

WP2: Designing a framework for learning assessment through AI interaction analysis

This work package will study interactions between students and GenAI chatbots like ChatGPT and their relation with  learning outcomes. Collaborating with course teachers, education experts, and learning analytics specialists, we will develop a framework to analyze these interactions (e.g., logs) for multiple courses at TU/e that involve writing assignments. The focus will be on identifying patterns that indicate learning progress, GenAI literacy, critical thinking, and knowledge construction.

This analysis framework includes:

• Collecting student-GenAI interaction logs (user prompts and chatbot responses).
• Applying qualitative coding schemes for dialogue content to assess patterns of interaction, including critical thinking evidence, question types, and iteration patterns.
• Developing quantitative metrics for interaction analysis (e.g., rubric scores, input frequency). Conducting rubric-based assessments comparing traditional evaluations with GenAI-interaction assessments.
• Automating chat log analysis using natural language processing to identify learning patterns.
• Surveying student characteristics and perceptions through open-ended questions about their use of GenAI tools in learning.
• Collaborating with students to critically assess and optimize evaluation criteria for student-AI interactions (in alignment with TU/e BOOST objectives)

This working package will take place in AY 2024-2025 across various Bachelor’s and Master’s courses from different departments.  

Expected outcomes:

• Report 2a: A description of the methodology and framework for assessing learning outcomes through AI interaction log analysis as well as the outcomes from a set of pilot courses (April 2025). Closely related, research insights related to this WP will be presented at the 4TUCEE End of Year event (November 2024).
• Report 2b: Tutorial for teachers on practical implementations of the assessment framework.(April 2025)

• WP3: Develop pedagogical activities powered by Generative AI

Based on the insights obtained from the literature review in WP1, as well as the insights gained from the development of the AI compatible assessment method from WP2, we aim to develop pedagogical activities to improve constructive alignment by teaching students how to effectively use AI to improve their performance on student-GenAI interaction assessments. The design of these GenAI tutors will be informed by insights gained in the previous work packages, and tailored to the learning objectives of the course and the specificities of the course assignments. These tools will be tested in pilot studies, in at least two courses with different types of assignments (e.g., argumentative essay vs. statistical programming and data analysis). For example:

• In a TU/e statistics course like “Advanced research methods and research ethics” where the objective for students is to learn how to think about their analytical approach to a given problem, the GenAI tutor can be designed to steer the student towards the assignment solution / desirable outcomes through questions posed to the student (scaffolding), as opposed to directly providing the solution. Simultaneously, a second GenAI tutor available to the same student could assist with another task required by the assignment, such as statistical programming, by providing explanations of the code or helping with code debugging. The behavior of the AI chatbot can be configured through prompt engineering techniques such as “Chain-of-Thought” combined with “Flipped interaction” (White et al., 2023).
• For writing-oriented courses such as “Data Science Ethics”, a GenAI tutor tool can be designed to assist with academic writing through scaffolding, such as guiding essay outlines, prompting critical thinking in the student, and offering instant feedback on their writing in a way that is aligned with the course learning objectives (embedded in the tutor’s underlying model or knowledge context). Likewise, the behavior of the AI chatbot is configured through prompt engineering techniques such as “Chain-of-Thought” combined with “Flipped interaction” (White et al., 2023).

By designing these GenAI tutor prototypes and incorporating them as tools in a course we will investigate:

• Whether using GenAI tutor tools lead to higher grades in student-GenAI interaction assessments, thereby providing insight into quality of constructive alignment in the course
• The impact of using these tools on self-perceptions of skill mastery (e.g., self-efficacy), thereby providing insights on alignment with students’ psychological
•  Students experiences using these tools (using open-ended questions in a post-assignment survey)

Expected outcomes:

• Tutorial on how to design a GenAI tutor and tailor it to a specific use case (March 2025)
• Report 3: Report describing the results of the GenAI tutor studies (Nov 2025).

WP4: Workshops on GenAI tools for educational activities

AI literacy is a crucial step towards the responsible use of GenAI technology in educational practices (Kasneci et al., 2023; Redecker, 2017). In this work package, we will develop a workshop aimed at either or both students (ranging from Bachelor to PhD candidates) and staff where we teach how to properly design and implement a GenAI based assistants using the most relevant and/or accessible AI chatbot tool(s) at the time they take place. The workshop will focus on teaching the essential steps to configure the behavior of AI chatbot assistants through existing techniques (e.g. prompt engineering, fine-tuning options), and provide examples on how to tailor it to more specific use cases (e.g., drafting teaching materials, academic writing assistance, literature summarization, data analysis assistance). A second type of workshop will place a higher weight on practical tips to employ GenAI tools in academic writing activities. A third type of workshop, aimed for the ALT community (Academy for Learning and Teaching, TU/e) will focus on topics of AI literacy. Content (tentative) may encapsulate and chain the following topics: evidence-based utility of GenAI tools, building GenAI chatbots to augment teaching practice, ethical use of GenAI, plus any practical recommendations derived from insights from pilot studies. Dates and times of these workshops will be arranged with project members and project coordinators from either 4TUCEE (SEFI) and BOOST (ALT).

Expected outcomes:

• Ongoing workshops throughout

References

• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
• Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu. In Y. Punie (Ed.), Technical report. Joint Research Centre (Seville site). https://data.europa.eu/doi/10.2760/159770
• White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (arXiv:2302.11382). arXiv. https://doi.org/10.48550/arXiv.2302.1138

Results and learnings

WP1: Mapping the literature on future-oriented and GenAI compatible higher education

Context

Generative AI (GenAI) technologies like ChatGPT and similar chatbots are transforming higher education. As these tools become more sophisticated, they pose significant challenges and opportunities for teaching, learning, and assessment. In this summary, we provide an overview of the key insights from our more extensive report (available below), highlighting how our ongoing review of the state-of-the-art knowledge on the challenges and opportunities of GenAI in Higher Education can be translated into actionable information relevant to teachers and students, highlighting strategic directions, future research, and practical recommendations. The emphasis of this summary is primarily on practical recommendations, given the urgency to respond to the ongoing transformative impact of GenAI.

challenges identified

• Disruption of traditional learning and assessment: GenAI’s humanlike content generation challenges conventional teaching methods, assessment integrity, and the unique value of human instruction (Kolade et al., 2024; Rathi et al., 2024).
• Need for skill reorientation: Both educators and students must shift focus toward skills that AI cannot easily replicate—critical thinking, creativity, ethical reasoning, adaptability, and AI literacy (Bower et al., 2024; Chauncey & McKenna, 2024; Kolade et al., 2024).
• Assessment uncertainty: With AI’s capacity to generate content, verifying authorship and evaluating genuine understanding become more complex, necessitating a redesign of assessment methodologies (Fleckenstein et al., 2024; Jakesch et al., 2019).

Strategic responses and recommendations

Redefining learning objectives

• Emphasize higher-order skills: critical thinking, problem-solving, creative ideation, ethical reasoning, and adaptability (Kasneci et al., 2023; Zhai, 2022).
• Integrate AI literacy: Ensure students and teachers understand AI’s capabilities, limitations, biases, and ethical considerations (Bower et al., 2024; Chiu, 2024; Kolade et al., 2024).

Transforming assessment practices

• Move from product-focused to process-oriented assessment: Evaluate reasoning processes, metacognitive skills, and real-world application (Cheng et al., 2024; Kolade et al., 2024).
• Adopt diverse, authentic evaluation methods: Use live presentations, peer assessments, project-based assignments, and frequent low-stakes assessments to mitigate AI’s advantages in generating generic responses (Kolade et al., 2024; Xia et al., 2024).
• Leverage prompt analytics: Analyze student interactions with AI to gain insights into learning processes and provide personalized feedback (Cheng et al., 2024; Kim et al., 2024).

Faculty development and policy updates

• Invest in professional development to equip educators with skills for integrating AI responsibly into pedagogy and assessment (Chan & Tsi, 2024; Lim et al., 2023).
• Update assessment policies to establish clear guidelines on AI use, maintaining academic integrity while embracing AI-enabled learning enhancements (Mollick & Mollick, 2022; Xia et al., 2024).

Ongoing research and pilot studies

• Supporting and monitoring pilot projects (e.g., at TU/e) that assess the impact of GenAI on learning outcomes, teacher effectiveness, and student motivation.
• Continuous multidisciplinary research to refine teaching and assessment strategies, strioving for an alignment with evolving (Gen)AI capabilities and future-oriented educational goals (e.g., Deng & Joshi, 2024; Mollick & Mollick, 2024; Rowland, 2023).

Value for teachers and students

For teachers:

• Empowerment through professional development and clear guidelines on AI integration, enabling them to design engaging, authentic assessments that emphasize unique human skills.
• Improved assessment tools and strategies that provide more accurate measures of student understanding and skill acquisition.

For students:

• Development of relevant, future-oriented skills that enhance employability and adaptability in an AI-driven landscape (Chiu, 2024; Zhai, 2022).
• Learning experiences that promote creativity, critical thinking, and ethical reasoning—areas where human judgment remains indispensable (Bower et al., 2024).

Conclusion and next steps

The integration of GenAI in higher education calls for a strategic, research-backed approach to curriculum design, assessment methods, and faculty development. Through focusing on uniquely human skills and transforming assessment practices, institutions can benefit from AI’s potential while preserving academic integrity and enhancing learning outcomes. Stakeholders are encouraged to support ongoing research, pilot projects, and policy updates that inform best practices. This proactive approach prepares both teachers and students for an increasingly AI-integrated educational environment.

References

• Bower, M., Torrington, J., Lai, J. W. M., Petocz, P., & Alfano, M. (2024). How should we change teaching and assessment in response to increasingly powerful generative Artificial Intelligence? Outcomes of the ChatGPT teacher survey. Education and Information Technologies, 29(12), 15403–15439. https://doi.org/10.1007/s10639-023-12405-0
• Chan, C. K. Y., & Tsi, L. H. Y. (2024). Will generative AI replace teachers in higher education? A study of teacher and student perceptions. Studies in Educational Evaluation, 83, 101395. https://doi.org/10.1016/j.stueduc.2024.101395
• Chauncey, S. A., & McKenna, H. P. (2024). Exploring the Potential of Cognitive Flexibility and Elaboration in Support of Curiosity, Interest, and Engagement in Designing AI-Rich Learning Spaces, Extensible to Urban Environments. In N. A. Streitz & S. Konomi (Eds.), Distributed, Ambient and Pervasive Interactions (pp. 209–230). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-60012-8_13
• Cheng, Y., Lyons, K., Chen, G., Gašević, D., & Swiecki, Z. (2024). Evidence-centered assessment for writing with generative AI. Proceedings of the 14th Learning Analytics and Knowledge Conference, 178–188. https://doi.org/10.1145/3636555.3636866
• Chiu, T. K. F. (2024). Future research recommendations for transforming higher education with generative AI. Computers and Education: Artificial Intelligence, 6, 100197. https://doi.org/10.1016/j.caeai.2023.100197
• Deng, X., & Joshi, K. D. (2024). Promoting ethical use of generative ai in education. SIGMIS Database, 55(3), 6–11. https://doi.org/10.1145/3685235.3685237
• Fleckenstein, J., Meyer, J., Jansen, T., Keller, S. D., Köller, O., & Möller, J. (2024). Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays. Computers and Education: Artificial Intelligence, 6, 100209. https://doi.org/10.1016/j.caeai.2024.100209
• Jakesch, M., French, M., Ma, X., Hancock, J. T., & Naaman, M. (2019). AI-Mediated Communication: How the Perception that Profile Text was Written by AI Affects Trustworthiness. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3290605.3300469
• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
• Kim, M., Kim, S., Lee, S., Yoon, Y., Myung, J., Yoo, H., Lim, H., Han, J., Kim, Y., Ahn, S.-Y., Kim, J., Oh, A., Hong, H., & Lee, T. Y. (2024). Designing Prompt Analytics Dashboards to Analyze Student-ChatGPT Interactions in EFL Writing (arXiv:2405.19691). arXiv. https://doi.org/10.48550/arXiv.2405.19691
• Kolade, O., Owoseni, A., & Egbetokun, A. (2024). Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework. Heliyon, 10(4), e25953. https://doi.org/10.1016/j.heliyon.2024.e25953
• Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790
• Mollick, E. R., & Mollick, L. (2022). New Modes of Learning Enabled by AI Chatbots: Three Methods and Assignments (SSRN Scholarly Paper 4300783). https://doi.org/10.2139/ssrn.4300783
• Mollick, E. R., & Mollick, L. (2024). Instructors as innovators: A future-focused approach to new AI learning opportunities, with prompts (SSRN Scholarly Paper 4802463). https://doi.org/10.2139/ssrn.4802463
• Rathi, I., Taylor, S., Bergen, B. K., & Jones, C. R. (2024). GPT-4 is judged more human than humans in displaced and inverted Turing tests (arXiv:2407.08853). arXiv. https://doi.org/10.48550/arXiv.2407.08853
• Rowland, D. R. (2023). Two frameworks to guide discussions around levels of acceptable use of generative AI in student academic research and writing. Journal of Academic Language and Learning, 17(1), Article 1.
• Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21(1), 40. https://doi.org/10.1186/s41239-024-00468-z
• Zhai, X. (2022). ChatGPT user experience: Implications for education. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4312418

WP2: Designing a framework for learning assessment through AI interaction analysis

In WP2, we developed the DRIVE framework (Directive Reasoning Interaction and Visible Expertise) to assess evidence of learning through the analysis of how students interact with Generative AI (GenAI) tools. This approach shifts the focus away from the mere grading of the product (essay) towards the examination of the learning process, specifically evaluating how students critically steer the AI (Directive Reasoning Interaction) and articulate domain knowledge within the dialogue (Visible Expertise). To operationalize this framework within the specific context of philosophical argumentative writing, a domain-specific taxonomy was designed and tested. The framework's validity was established in a multi-methods study involving graded coursework across three writing-intensive university courses.

Analysis utilizing this approach revealed distinct patterns in student engagement with GenAI and their correlation with academic performance. Specifically, students who adopted a "collaborative intellectual partnership" profile (using GenAI for higher-order tasks such as idea development and critical refinement) demonstrated superior academic outcomes. On the other hand, students who predominantly engaged in "passive task delegation" or basic information retrieval were associated with comparatively lower academic performance. These results indicate that although the specific taxonomy may require adaptation for different disciplines, the DRIVE framework serves as a transferable resource that educators can choose to use to design process-focused assessments that capture the nuances of learning in AI-integrated classrooms.

WP3 Pilot Study: Evaluating RAG Chatbots in Exam Preparation

Context

To investigate the impact of course-specific AI tools in a naturalistic setting, we conducted an observational pilot study in a Bachelor-level Cognitive Psychology course. Students (N=116) were provided voluntary access to two Retrieval-Augmented Generation (RAG) chatbots (Alexandria.cx and Tilburg.ai) grounded in course materials for three weeks prior to their final exam. The study analyzed interaction logs, survey responses, and final exam grades to determine usage patterns and learning outcomes.

Main results

• Usage patterns: Engagement was highly variable and predominantly strategic rather than deep. Usage was heavily concentrated in the final four days before the exam (accounting for 85% of interactions). The majority of interactions (65%) involved lower-order cognitive tasks, such as requesting explanations of specific course topics, rather than complex study strategies.
• Student perceptions: Students who used the tools reported positive perceptions regarding usability and usefulness, indicating a high level of general AI literacy.
• Impact on performance: While students who used the chatbot achieved slightly higher average exam grades (M=5.68) compared to non-users (M=5.17), this difference was not statistically significant (p=.068). Furthermore, regression analysis showed that neither the frequency of interaction nor specific usage types significantly predicted exam performance.

Recommendations

WP1

Recommendations based on the literature review from WP1

Higher education institutions should focus on redesigning their curricula and assessment methods to emphasize skills that AI cannot easily replicate, and possibly design novel courses that tackle the increasing need for AI literacy, critical thinking, and human-technology interaction ethics. This implies shifting from content-focused instruction to developing higher-order thinking skills through activities that are less easily offloaded to AI systems. It should be noted, however, that what constitutes desirable or undesirable use of AI ultimately depends on the intended learning objectives (ILOs) of a course. Investing in increasing AI literacy of teachers should facilitate the design of courses that are more harmoniously coexisting with the technology. In practice, this means a better alignment between ILOs, pedagogical activities and assessment approaches. One example could be a course where some activities involve learning how to responsibly co-write essays with generative AI followed by the assessment of the interaction between the student and the AI throughout the writing process (i.e., prompt analytics). If the ILOs emphasize the development of core competencies that AI systems have already demonstrated a high degree of capability in executing, yet simultaneously enable students to more effectively scrutinize and assess the outputs generated by AI, teachers should consider facilitating the teaching and evaluation of such skills within an AI-free pedagogical environment.

The majority of assessed student output in higher education is in verbal format, such as essays, reports, and presentations. This type of output is directly threatened by the extremely high capability of large language models (LLMs) and other generative AI tools to easily produce and manipulate verbal content. This requires a rethinking of how teachers assess learning.  With the increasing capabilities of AI to take over otherwise hard-earned skills involved in thinking and writing tasks, assessment strategies should move away from traditional essays and exams toward performance-based evaluation methods that demonstrate authentic learning and application of knowledge. This includes implementing more real-time assessments like presentations, group projects, and case studies that require students to demonstrate critical thinking, problem-solving, and creativity in real-time while applying their knowledge in a given context.

WP2. How to assess learning through the analysis of student-GenAI interactions

To assess learning in contexts where GenAI is permitted, instructors must shift their focus from the final product to the learning process itself. This work package proposes the DRIVE framework as a method to evaluate this process by analyzing student-GenAI interaction logs. The framework assesses two core components: Directive Reasoning Interaction (DRI), which measures how students critically steer the AI, and Visible Expertise (VE), which identifies how students articulate their acquired domain knowledge within the dialogue. While this project utilized a specific interaction taxonomy tailored to argumentative writing to operationalize these concepts, the DRIVE framework itself is designed to be adaptable across different domains and assessment types.

Empirical validation of this approach revealed a strong positive correlation ($r=0.54$) between the quality of students' GenAI interactions and their final academic outcomes. High performance was associated with a "collaborative intellectual partnership" profile, characterized by students posing original ideas, refining concepts, and critically evaluating AI outputs. In contrast, lower outcomes were correlated with "passive task delegation" or basic information retrieval, where students relied on the AI to generate content without significant steering or knowledge infusion. Based on these findings, we offer the following recommendations for practice:

• Explicitly define the learning evidence: Clearly articulate whether the assessment focuses on technical AI literacy (e.g., prompting skills) or domain-specific learning (e.g., Visible Expertise in the prompt)
• Require and evaluate interaction logs: Make the submission of complete interaction logs a requirement in your assessment guidelines. This allows to gain visibility into the student's Directive Reasoning Interaction (DRI) and their agency in the co-creation process
• Design tasks promoting partnership (but think if incentive is right...): Develop assignments that require students to use GenAI as a "thinking partner" for conceptual refinement and critique, rather than for simple production or information gathering. However, you must reflect on whether this is an acceptable use of GenAI in line with your learning objectives or pedagogical context.
• Distinguish interaction profiles: Teach students to move beyond "passive task delegation" behaviors and model "collaborative intellectual partnership" strategies to support deeper learning. Again, here you must reflect on whether this is an acceptable use of GenAI in line with your learning objectives or pedagogical context.

WP3: Pilot studies using GenAI based tutoring applications in the classroom

The preliminary findings from this pilot study can, at this point, already offer several considerations for educators. First, the results suggest that simply providing access to a course-specific RAG chatbot, even one perceived positively by students, is not a guarantee of improved learning outcomes. Instructors should not assume that students will spontaneously use these tools in pedagogically optimal ways. Our current data is suggesting that the common student may engage with these tools at a very superficial level by default (e.g., last-minute clarification). To foster the deeper, agentic engagement associated with positive learning (Smirnova, 2025; Yang et al., 2024), instructors might design specific, structured activities. For example, rather than leaving use entirely open, an educator could require students to use the chatbot to generate practice questions early in a module, or to use the chatbot to find flaws in an argument, or to critique a chatbot-generated summary of a complex topic. This approach shifts the student's role from passive consumer to an active critical evaluator. Finally, educators should remain mindful of the technological "lag" discussed in the limitations. If institutional tools are perceived as less capable than rapidly evolving commercial alternatives, students may ignore them.

References

• Smirnova, L. (2025). Developing students’ agency and voice by using generative AI in an online EAP module. Innovation in Language Learning and Teaching, 1–11. https://doi.org/10.1080/17501229.2025.2538781
• Yang, Y., Luo, J., Yang, M., Yang, R., & Chen, J. (2024). From surface to deep learning approaches with Generative AI in higher education: An analytical framework of student agency. Studies in Higher Education, 49(5), 817–830. https://doi.org/10.1080/03075079.2024.2327003

Practical outcomes

WP1

The insights gained from the systematic scoping review of the literature aiming to gather information on the state-of-the-art perspectives and interventions regarding  future-oriented learning objectives and AI-compatible assessment methods, informed the design of pilot studies at TU/e assessing the impact of using Generative AI chatbots on a range of outcomes ranging from student learning to teaching activities.

WP2. Assessing student interactions with GenAI in the context of academic writing 

A central finding from this work informing practice is that higher academic performance is strongly associated with specific interaction profiles, such as the "collaborative intellectual partnership" or "targeted improvement partnership". Strategies characterizing these profiles, including the critical evaluation of AI outputs, systematic text refinement, and the development of original concepts, were linked to superior outcomes. In contrast, low performance was consistently linked to "passive task delegation" or basic information retrieval. By analyzing interaction logs through the lens of the DRIVE framework, teachers can move beyond intuitive judgments to identify these specific patterns. This visibility allows them to provide targeted feedback that guides students away from passive outsourcing and toward the Directive Reasoning Interaction (DRI) and Visible Expertise (VE) necessary for effective learning and writing skill development.

WP3: Pilot studies using GenAI based tutoring applications in the classroom

Preliminary findings from the RAG chatbot pilot study suggest that providing access to a domain-specific tool does not automatically result in improved learning outcomes. Data indicates that students often engage with these tools at a superficial level by default, such as seeking last-minute clarifications. To counter this, instructors should design structured activities that foster agentic engagement rather than assuming spontaneous pedagogical use. Practical applications include requiring students to use the chatbot for generating practice questions early in a module, identifying argumentative flaws, or critiquing AI-generated summaries of complex topics. These pedagogical interventions are necessary to shift the student's role from a passive consumer to an active critical evaluator. Additionally, educators must address potential technological lag, as students may disengage if institutional tools are perceived as inferior to rapidly evolving commercial alternatives.