Development, implementation, and evaluation of a ChatGPT-based tool for the course Health Psychology

Project introduction and background information

The main assignment in the course Health psychology involves students designing a survey to assess determinants of health behavior.  A  major challenge for students for this course is developing high-quality surveys. They often struggle to design questions that accurately measure psychosocial concepts, distinguish between well- and poorly constructed survey items and response scales, and align survey measures with their research questions and theoretical framework. These issues can result in flawed or incomplete data, making analysis more difficult and reducing the quality of their scientific articles.

Teachers face similar challenges. During the first week of a three-week course, they must provide feedback on students’ research questions, theoretical frameworks, and surveys within a few days, creating a high workload. Much of this feedback is repetitive, as students consistently make similar mistakes, particularly in survey design and the use of validated scales.

To address these recurring challenges, we developed a chatbot based on ChatGPT to help students formulate stronger research questions and surveys while reducing teachers’ workload by providing consistent, timely feedback.

Objective and expected outcomes

To address these recurring challenges, we developed a chatbot based on ChatGPT to help students formulate stronger research questions and surveys while reducing teachers’ workload by providing consistent, timely feedback.

Outcomes and Benefits

Using the customized ChatGPT as an educational tool will help students design better-quality surveys, deepen their understanding of survey methodology and enhance their critical thinking. Ultimately, this will improve the overall quality of their scientific reports, while also equipping them with valuable (transferable) skills for using AI effectively in the research contexts.

Results and learnings

Use of the chatbot was optional for the 53 students enrolled in the course. Of these, 33 students completed a short six-question evaluation survey. Most reported using the chatbot, primarily to support survey development (n = 14, 29%), formulate or refine research questions (n = 13, 27%), and structure their assignments (n = 10, 20%). Most students indicated that the chatbot either helped only sometimes (n = 10, 30%) or only a little (n = 10, 30%).

Although students generally found the chatbot instructions clear, many reported difficulties in knowing which prompts to use and how to interpret and implement the feedback they received. Prompting was identified as the greatest challenge. Some students also found it difficult to determine when their work was sufficiently refined, as the chatbot often suggested that further improvements were possible.

Recommendations

Overall, this project showed that a chatbot can support student learning, but its successful implementation requires more teacher guidance. A brief workshop on writing effective prompts, interpreting feedback, and understanding the chatbot's capabilities and limitations could improve its use. Simply making the chatbot available and advising students to remain critical is unlikely to be sufficient. With appropriate guidance, the chatbot can become a valuable complement to student learning.

Practical outcomes

The project resulted in a practical implementation report that documents the complete development process of the chatbot. The report describes how the chatbot was designed, including the definition of its role, knowledge base, behaviour, and capabilities. It also outlines the testing phase, which involved multiple testers using different evaluation methods, and presents the results of these tests, including examples of issues that were identified and how they were resolved.

In addition, the report summarizes students' experiences with the chatbot, including the main challenges they encountered, such as prompt writing and interpreting chatbot feedback. Based on these findings, the report provides practical recommendations for educators who are considering implementing similar AI-supported learning tools, covering both technical development and educational integration.