Tool support for structured drawings in automated feedback and assessment

Project introduction and background information

Many fundamental concepts, design methods, and transformational techniques in Computer Science and related disciplines are based on discrete combinatorial structures. Examples of these include graphs and finite-state machines in Computer Science, Markov chains and polyhedrons in Mathematics, business process languages and entity-relation diagrams in Economics, or even concepts like evolutionary trees in Biology.

The key idea of this project is twofold. First, in order to offer an intuitive way to test the understanding of structured drawings the ability to physically manipulate them as drawings is required. Second, due to the structured nature of the drawings itself it is possible, as we claim, to create open-ended exercises with the automated grading accuracy of closed-format questions such as multiple-choice questions. More concretely, in the project we aim to provide a graphical user interface within the learning management system where learners can directly draw their solutions. The structure of the drawings is captured behind the scenes and the answers of the learner can be encoded for subsequent automated processing. Particularly we focus on the case of finite state machines as taught in the course Automata, Language Theory & Complexity in the Bachelor Computer Science & Engineering at TU/e.

Objective and expected outcomes

The project deliverables will include:

An intuitive drawing environment within OnCourse that (i) allows the drawing of finite automata, (ii) can encode deterministic and non-deterministic finite automata for automated processing, and (iii) can cooperate with the developed plugin to enable automated feedback and grading.
Next to this a user manual for teachers will be developed, as well as documentation for programmers to further enhance the feedback or grading capabilities of the plugin.
The specification and implementation of a software package that is able to (initially formative) assess the correctness of answers to a variety of standard questions in the context of so-called regular languages, roughly covered by a quarter of the 2IT90 course.
A submission of a poster to a national engineering education or STEM conference or a paper to a practical journal on teaching in higher education.

Results and learnings

The end product offers options for automated testing in both the formative and summative setting. Compared to more traditional methods such as multiple-choice exercises the development of open-ended exercises based on structured drawings requires lecturers to be less concerned with the effects of guessing and weak distractors, and requires a higher level of cognitive understanding of students.

The plugin will be maintained by the Education Innovation group (EdIn) of the Department of Mathematics & Computer Science, who will maintain the plugin for structured drawings in OnCourse, thus assuring sustained availability. Within the scope of the project the system will be used in the 2IT90 course and used for learning and assessment by students and staff involved. Outside of project scope there is interest in the system from several courses, including but not limited to 2IL50, 2IT80, 2IHA10, 2WAB0/2WBB0/2WCB0. If the project is successful, adoption of the setup in these courses is likely.

Recommendations

The user interface is one of the most important parts of a newly developed tool. Smoother intuitive tools have a far lower threshold of acceptance. The strong focus on usability from the start paid off in the fast acceptance by students and allows for more easy adoption by lecturers.
Quickly moving to concrete results makes it much easier to swiftly develop a tool as it streamlines communication between the developers and the end-users and it allows for a concrete system to critique.
The ability for authentic testing is highly appreciated by students. Giving students the opportunity to answer (even online) in the way that directly matches the topic being studied may increase engagement.
Specific for the GraphChecker experiment in the 2IT90 course, students report positively on the instantaneous feedback from the system and the possibility to iteratively improve their answer. The instantaneous feedback is also provided in summative setting yielding higher quality answers without giving a hint at the global level at all.