Peer Assesement to Improve Reproducibility of Computational Project Work

Project introduction and background information

Abstract

Computational research requires increased reproducibility for open science practices; however, reproducibility is not yet widely taught in the geosciences. Learning and internalizing reproducible research practices are an important step towards improving scientific practices. Additionally, reproducibility in the classroom will also improve student learning by facilitating peer-review and reuse of earlier work in later courses. This project tested peer assessment to improve the reproducibility of student project reports in a course that implements elements of challenge-based learning.

Introduction

There is evidence in the literature that teaching reproducibility benefits from taking the two perspectives of a researcher aiming for reproducible research output and of a reader or user trying to reproduce published research. An effective pedagogic approach to teaching reproducibility should combine practice (tutorials) of trying to reproduce someone else’s work alternated with iterations of teacher and peer feedback on the reproducibility of one’s own work. Reproducibility was introduced as a new topic in a 15 EC MSc course, which has a challenge-based learning approach where different stakeholders face the wicked problem of human-induced earthquakes due to gas extraction.

Methods 

Students work in groups for a particular stakeholder. Self-regulated feedback is encouraged to include other stakeholders’ views. Shortly after a half-day tutorial implementing the mentioned two perspectives, student groups submitted their reproducibility plans at the end of the first half of the course, which were then peer-reviewed by the other student groups. This left sufficient time to incorporate the peer feedback into the final project reports that every group had to deliver.

Objective and expected outcomes

• Test the effectiveness of teaching reproducibility to MSc students using tutorials and peer assessment.
• Improve the reproducibility of students’ course outputs.

Results and learnings

The quality and depth of the peer feedback itself provided information on how well the topic has been understood. The outcomes of the course evaluation showed that this new approach was appreciated by students and encouraged them to improve the reproducibility of their course outputs. This improvement was confirmed by an evaluation of all final student reports’ reproducibility and comparison with the results of the previous year.

Recommendations

• Implement the tutorial and peer assessment in similar courses 
• Expand the tutorials to provide additional learning opportunities

Practical outcomes

• Course materials on reproducibility
• Reproducible student assignments