Project introduction and background information
In its Strategy 2030 document (Executive Board TU/e, 2018), Eindhoven University of Technology (TU/e) has announced to fundamentally extend personalized learning within the next few years. Blended learning, and its online components, need to play a crucial role for this objective to be reached. The consequences of the Covid-19 transition only underline the vital role of online learning and blended learning will remain more prominent in the post-Covid period.
Unfortunately, online learning suffers from several barriers that make life for students more difficult. First, many motivating elements of traditional courses are missing. Students need to develop and maintain a regular schedule, motivate themselves, and persist in their learning activities. Therefore, students’ self-regulation of their learning is much more relevant in online learning (Cho & Shen, 2013). Second, the distance between teacher and students is larger in online courses than in courses that incorporate regular face-to-face interaction (Rodriguez et al., 2017). Teachers are less aware of whether students are on track in meeting the course learning goals, and of how they experience the teaching. This lack of insight makes it hard to adjust the teaching and intervene quickly when students fall behind in their learning progress.
Objective and expected outcomes
The PAELLA (Personalized Activation in Education leveraging Learning Analytics) project develops and tests a new learning design in different Bachelor courses. Within these courses, we use click-stream data from the LMS to push forward personalized feedback and formative testing in the Bachelor program. For selected students with a backlog in their online learning, we offer appropriate (mindset) interventions that stimulate students’ self-regulation of learning. For this, we need to address the following challenges:
- How can we utilize the LMS data in a scalable way to differentiate between students who are on track and those who lag behind in their learning process?
- How can we design and apply personalized interventions in LMSs that activate students and stimulate them to better self-regulate their learning in an engineering context without increasing the teachers’ workload?
- How can we track students so that students’ privacy is guaranteed and that they do not feel threatened during online and blended learning?
Results and learnings
Detailed results can be found in the uploaded documents. Here is a short sketch:
R1: In Report R1, we describe the students' point of view on our educational approach. What do they recommend on how to make use of Learning Analytics? What do they recommend for the design of the interventions? We describe how we -re-designed the courses to have a data-rich environment in each course.
R2: In report R2, we show how to create a comprehensive set of Canvas indicators (variables) that describe students’ use of the LMS.
R3: In report R3, we describe the models that we develop to identify students at risk (nonengaged students who are likely to profit more from a mindset intervention).
R4: In report R4, we demonstrate the development of the interventions, starting with the intervention of course 1 that was executed in May and June 2022 (Q4 of the academic year 2021-2022. The last intervention took place in Q2 of the academic year 2022-2023.
R5: In report R5, we apply and test the effects of the interventions. We executed the interventions in several randomized field experiments, including a Placebo treatment, to find out whether the interventions had any effects at all and whether these were stronger for at-risk students.
We summarize the learning points on how to utilize Learning Analytics for personalized online interventions in an online tutorial: https://canvas.tue.nl/courses/27339
Recommendations
Several detailed learning points for teachers emerged that we summarize in the earlier mentioned reports. On a more general level, the use of Learning Analytics-based interventions requires a well-coordinated and intensive collaboration between the ICT department on the one hand and the teachers and researchers on the other hand. Teachers must go through several steps to prepare, execute, and evaluate a Learning Analytics-based online intervention. These include the following:
1. Take into account privacy regulations and the students’ perspective.
2. (If necessary) Re-design the course to have a data-rich online environment.
3. Scripting and pre-processing of clickstream data
4. Identification of the intervention’s target group
5. Execution of (earlier prepared) online intervention
6. Evaluation of effects
More details are offered in our (Open Access) online tutorial:
https://canvas.tue.nl/courses/27339
Practical outcomes
Report R2: how to create variables out of Canvas data tables that can be used in further analyses in standard statistical packages.
Report R4: how to create an educational intervention that strengthens students' learning engagement
Open Access online tutorial: How to set up a personalized online intervention:
https://canvas.tue.nl/courses/27339