Part of the 4TU.Federation
4TU.
Centre for
Engineering Education

Project introduction and background information

WUR has created a new procedure for innovating education via Educational Technology (EdTech), the so called Community for Education Innovation with EdTech. It aims to support teachers that want to experiment with new IT tools (EdTech) in their education in the best possible way. This community tries to prepare for a possible scale-up of the new IT tool within WUR at a later stage. The education innovation with EdTech procedure totally entails 11 steps beginning with an idea about educational innovation with an IT tool that is not in use at WUR (step 1) and ending with a decision on either further implementation of the IT tool or giving up using that tool (step 11) (see Figure 1). All WUR teachers are welcome to apply for the education innovation with a new IT tool (EdTech) experiment.

Figure 1. The education innovation with EdTech procedure at WUR

As part of education innovation with the EdTech procedure at WUR, one of the important and required steps is the “evaluation” (step 9) which basically means that at the end of the IT tool experimentation, evaluation will be conducted in terms of different factors such as general satisfaction, ease of use, added value, strength and weakness, and usefulness of the experimented IT tool to decide whether the it should be supported for further implementation on a large scale or not. The Education and Learning Sciences chair group (ELS) is in principle1 responsible to conduct these evaluations, in cooperation with the teachers/course coordinators involved, the IT support department and ESA. For EdTech evaluation, Dr. Omid Noroozi from ELS is the project coordinator.

Types of the EdTech evaluation varies from qualitative evaluation including analysis of the documents, semi-structured interviews and focus group discussions with teachers and also students who are experiencing with the IT tool to quantitative evaluation including surveys which are being collected through PACE evaluation. Although based on the nature and goals of the IT tool, evaluation factors might differ, generally, it includes collecting and analysis of data regarding the IT tool’s weaknesses and strengths, ease of use, added values, effectiveness and usefulness, time-saving for both teachers and students, contribution to educational goals and processes, and teachers and students general satisfaction with the IT tool.

[1]. If the courses involved are ELS courses, it is ensured that the researching staff are not also involved as teachers. Potentially, in such a case a researcher from another chair group or someone from ESA might be involved. In case teachers want to investigate innovations with EdTech as part of a Senior Qualification in Education, this is also possible; in such cases they will be supervised by ELS staff.

Objective and expected outcomes

The main goal of EdTech experimentation at WUR is to increase the quality of education and learning. Suggested IT tools by teachers are experimented on a small scale to see whether they support teachers’ training and students’ learning or not. In this regard, evaluation (step 9) plays a key role to provide an overall and detailed view of the effectiveness and usefulness of the experimented IT tool in achieving respected goals. Thus, the goal of the EdTech evaluation is to provide a report on the experimented IT tool’s effectiveness for WUR courses. If the results for the IT tool are positive then WUR will consider further implementation of the IT tool.

Results and learnings

Currently, the EdTech evaluation is responsible for evaluating four IT tools at WUR including Comproved, TrainTool, CodeGrade, and Grasple. For each tool, the evaluation reports will be shared in the downloads section on this page.

Comproved

Comproved is an assessment tool that works based on comparative judgments. This tool structures comparisons of students’ work while aiming to be a valid, reliable and user-friendly instrument.  The goal of this tool is to reduce the workload of teachers and improve the quality of assessment. The main users of Comproved are teachers. At WUR, Comproved was used in one of the courses for assessing and grading students. After experimenting with the tool, data was collected in both quantitative and qualitative formats from teachers and students through focus group discussions and surveys and analysed. The overall conclusion of the tool shows that the assessment is more valid (different teachers) and less biased and that the tool is technically easy to use by teachers, but not pleasant emotionally. The teachers’ satisfaction rate was 5.8 out of 10. Students’ satisfaction rate was 6.2 out of 10. Overall, teachers and students were dissatisfied with the tool. They found that the feedback is more general and superficial, it increased the workload of teachers and they had to invest a lot of time in it. The tool is not successful in attaining its goals.

The full results and recommendations of evaluating Comproved can be found here.

TrainTool

TrainTool is an online video training tool that focuses on practicing and testing students' communication skills. The main goal of this tool is to improve students' communication skills.  In this tool, students can record their speech, watch it in private for self-practice or they can ask for feedback from their peers or teachers. To evaluate this tool, varied types of data from teachers and students were collected through focus group discussions, in-depth interviews, online surveys, and reflection reports. The findings showed that TrainTool is well-structured, user-friendly, easy to use, and it helps teachers and students with achieving the learning goal of the tool which is for practicing communication skills. Our general impression is that TrainTool can contribute to the quality of teaching and learning by having features such as (a) self-practice and self-reflection, (b) safe self-pace, active and engaging learning environment, (c) structure, user-friendliness, and clearance (d) peer and teacher feedback functionalities. However, It would be great to make improvements in terms of interactivity, in-depth learning and assessment, decreasing workload, and group activities.

The full results and recommendations of evaluating TrainTool can be found here.

CodeGrade

CodeGrade is a grading and feedback platform for computer science. The tool helps both teachers and students in computer science courses that deal with scripting and coding. CodeGrade was used by different courses at WUR in the 2020-2021 academic year. To evaluate CodeGrade, different kinds of data from teachers and students were collected through focus group discussions, online surveys, and demo recordings on the tool. Our general impression is that the benefits of CodeGrade slightly outweigh its drawbacks. The tool contributes to the quality of teaching and learning by having features such as auto-grading, plagiarism check, self-assessment, structure, and feedback. However, further improvements are suggested in terms of reliability, flexibility, interactivity, interface, and intelligence.

The full results and recommendations of evaluating CodeGrade can be found here.

Grasple

This tool is still under evaluation. More info will follow soon.

More

News article: EdTech Project to evaluate educational technology innovations at WUR


Header image: Steffie Sauren