Design and evaluation of genAI chatbots in education
Project introduction and background information
In the course Food Fraud and Mitigation students have to interview stakeholders from industry to assess the fraud vulnerability of a company. In the past five years, student numbers grew to around 50-60 students per year. This meant that it was undoable for our teachers to find professionals that were willing to be interviewed by all our students. In practice, this meant that students read previously conducted interviews and analyzed this data, instead of interviewing professionals themselves. However, by doing this, students could not create and ask their own questions to find out how vulnerable companies are for food fraud.
A promising solution for this problem is generative artificial intelligence (genAI) chatbots. Examples of commercial pre-programmed chatbots are ChatGPT, LeChat or DeepSeek. These bots can generate a fitting response to the questions asked by students, if programmed well. In this project, funded by the Education Development Innovation Call from Wageningen University, we aimed to find out how we can create effective roleplay chatbots and evaluate whether these bots can help students perform the complete food fraud vulnerability assessment by providing them a personalized interview experience.
Objective and expected outcomes
Our objectives are to:
- Redesign the current assignment to include an interview step with company stakeholders.
- Design, create and evaluate roleplay genAI chatbots for the course Food Fraud and Mitigation.
- Create a manual for teachers that want to create roleplay chatbots.
Results and learnings
With our team of three student assistants en three teaching staff, we were able to design and create 4 roleplay chatbots that took on the role of a quality manager and (financial) manager in a fictive meat company.
These bots were programmed in such a way that they:
- Took on the role of either a quality manager or a financial manager.
- Had a well-defined personality that was fun or engaging to chat with.
- Only answered assignment-related questions. Questions that were not in scope (e.g. politics) were not answered by them. The content of their answers was predefined by the team, so that it could be analyzed in the next steps of the assignment.
- Referred to their colleague if they did not know the answer.
We learnt a lot from the project. It was easy to create a simple roleplay chatbot, however it took us lots of time and energy to polish and perfect the bots. We performed two pilots where we tested the interaction of these chatbots with students and teachers. Based on their feedback, we improved the type of response the bots gave. We also checked with our pilot participants how easy it was for them to hack and change the bots (make them release the hidden data file or change their personality).
During the course, we launched the new assignment where students had to interview these virtual agents. Afterwards, we conducted a survey and interviews to evaluate how students experience this interaction. Results show that more than 70% of the students (n=50) liked interacting with these bots and that they preferred the interaction over reading an already conducted interview. At the end of the assignment, we also asked students to indicate how well they have achieved their learning outcomes. More than 50% of the students (n=20) indicated that they achieved most of the learning outcomes of the assignment. There were also points of improvements. Some students indicated that the interaction felt a bit robotic sometimes. Furthermore, the bots did not ask them questions back as people do in a real interview. The full evaluation can be found in the manual (chapter 5: Final reflections).
Recommendations
It is possible to create a functioning chatbot that is engaging for students. We have some tips for teachers that want to create their own chatbot in their courses:
- Start small and define what characteristics you want the chatbot to have.
- Test your chatbot in a small group before you use it in your course. During the pilot we noticed some small interaction errors and we could change these before the chatbots were used by the students in our course.
- Creating a genAI chatbot is a continuous process. The technology changes fast, so reserve enough time for yourself to design and create such a chatbot.
Practical outcomes
We created a manual for other teachers to create roleplay chatbots. The manual can be found on this page (downloads to the left).
Table of content manual:
Chapter 1: Understanding chatbots
Chapter 2: Setting up the chatbot
Chapter 3: Testing and launching your chatbot
Chapter 4: Roleplaying chatbot in practice
Chapter 5: Final reflections
Chapter 6: Additional resources and support.