CurlyArrows - From Drawing to Understanding Organic Reaction Mechanisms

Project introduction and background information

Understanding reaction mechanisms is a fundamental yet challenging aspect of organic chemistry education. While students often learn the procedural steps required to draw mechanisms, many struggle to explain the underlying chemical concepts that drive these reactions. Personalized feedback can help address these misconceptions, but providing individual tutoring opportunities is often limited by time and staff availability.

The CurlyArrows project aimed to develop and evaluate an AI-supported learning environment that provides students with immediate, personalized feedback on both the drawing of their reaction mechanism and their written explanations of key reaction steps. Students complete exercises by drawing mechanisms and explaining key reaction steps. The system combines automated analysis with AI-generated tutor feedback and creates a digital learning environment that stimulates both mechanistic and conceptual understanding.

Objective and expected outcomes

• Stimulate students to practice organic chemistry by letting them draw reaction mechanisms on an open-ended canvas and explain their reasoning on key reaction steps
• Further develop the CurlyArrows program for the analysis of student drawn reaction mechanisms on an open-ended canvas
• Test and implement a two-stage AI workflow for evaluating student reasoning and integrate with CurlyArrows for generating tutor feedback
• Develop a WUR-hosted web application for students to practice mechanisms of organic chemistry, explain them and provide immediate and personalized tutor feedback to support a deeper understanding
• Evaluate the educational value and usability of the tool in undergraduate organic chemistry courses

Results and learnings

The project successfully delivered an AI-supported learning environment for organic chemistry education. A WUR-hosted website was developed that allows students to log in using their WUR credentials and practice 30 reaction mechanism exercises. For each exercise, the student draws (part of) a reaction mechanism and explains one or two key reaction steps. CurlyArrows was successfully integrated with GPT-4o-mini to provide personalized tutor-style feedback. The platform was tested with 25 students in two organic ohemistry courses.

The project generated valuable insights into the design of AI-supported feedback systems in education. We found that high-quality AI feedback requires more than selecting a capable language model. It requires translating learning objectives into clearly defined concepts and implementing a structured AI workflow. During classroom testing, direct interpretation of student explanations by the AI occasionally led to incorrect assessments. To improve reliability, the layered analysis approach was improved. First, student explanations are screened against predefined concept tags representing the key concepts of each exercise. The student answer is not only labelled as correct or incorrect, but also the specific errors are reported. Second, this analysis is combined with the mechanistic analysis generated by CurlyArrows to produce personalized feedback. This significantly improved the consistency and transparency of the feedback.

The project also demonstrated practical solutions to common challenges associated with LLM’s. Hallucinations were minimized through deterministic model settings, and privacy concerns and data management were controlled via the backend architecture. The resulting framework is transferable to many educational settings that use open-ended questions and formative feedback.

Project website:          curlyarrows.wur.nl

GitHub repository:     https://github.com/peervd/CurlyArrows

Recommendations

The CurlyArrows project provided valuable insights into the opportunities and challenges of using AI-generated feedback to support student learning:

• Start with clear learning objectives and define the key concepts students should demonstrate in each of the answers
• Use a layered AI architecture: first analyze student responses against predefined concepts, then separately generate personalized feedback
• Test the AI tool with real student responses early in the development process to identify unexpected errors and misconceptions
• Design AI systems with privacy, security, and learning goals in mind from the start, including careful management of student data and AI access

Practical outcomes

The CurlyArrows platform has been implemented as a web-based learning environment for organic chemistry education at WUR. Students can access the system using their WUR accounts and receive immediate feedback on reaction mechanism exercises during self-study. The project established a reusable framework for combining automated assessment with AI-generated tutor feedback. This has the potential to be adapted for other educational contexts. The underlying codebase has been documented and is made available through a public GitHub repository.