Summative Online Exams with R/exams and OpenOlat

Overview

The development version of the R/exams package has recently been extended to support a few more strategies for helping to prevent cheating in summative online exams conducted remotely in the learning management system OpenOlat. Namely, the questions in an exam can now be presented in random order while employing a “linear” navigation, i.e., disallowing to switch back and forth between the questions. While this is certainly also a bit of a nuisance for “honest” and well-prepared students who are used to starting an exam with those exercises they feel most comfortable with, it should be much more of a hurdle for students trying to cheat. In this scenario only 1 in n students on average is working on a given (type of) exercise when there are n questions (or rather sections of question pools).

Admittedly, this is still a rather weak protection against cheating compared to state-of-the-art online proctoring methods, e.g., by means of video conferencing tools. However, with several hundred participants per exam, we felt that we would (a) exclude too many students without a suitable webcam and broadband internet connection, and (b) the risk of unexpected short-term problems with the video conferencing software for individual students would be too high. Therefore, we have decided to rely only on the following strategies:

1. Students have to fill in a sworn declaration stating that they are doing the exam alone with only the tools allowed. This is not only important for legal reasons but even more so for priming the students to be honest right before the exam.
2. In each section of exercises, there is a sufficient amount of random variation employing the “usual” randomization provided by R/exams.
3. Additionally, the individual order of exercises is fixed to a particular random sequence by OpenOlat.

The results from semi-summative mid-term exams suggest that the strategies work reasonably well albeit leading to slightly better results compared to previous semesters (suggesting that cheating might be more prevalent than for classical in-class written exams). Nevertheless, we felt that we would stick to these strategies for the summative final exams later this month - and not employ more severe strategies such as video conferencing or simply increasing the workload or making exercises more difficult.

Note that Strategy 3 requires at least R/exams version 2.4-0 which is the development version on R-Forge at the time of writing. Thus, if you want to follow the steps below, please install this version before:

install.packages("exams", repos = "http://R-Forge.R-project.org")

Video tutorial

A step-by-step video tutorial is available on YouTube at https://www.youtube.com/watch?v=hRE9SGA1F7U. This not only shows how to generate the learning resources using the strategies outlined above, it also goes through the entire OpenOlat import and configuration. Specifically, this includes visibility rules and time limits for the different components of the exam.

Sworn declaration

Both for legal reasons and for priming the students to be honest, they have to sign a sworn declaration stating that they do the exam alone using only allowed tools. Technically, this is implemented by an online test with one single-choice item where they can only say “Yes”. Only when signing the declaration in this way, students can see the actual online exam.

As the declaration is also just an online test, the corresponding learning resource for OpenOlat can be easily generated with R/exams’ exams2openolat() function. The “exercise” file with the text employed at Universität Innsbruck is available both in R/Markdown and R/LaTeX format and both in English [Rmd] [Rnw] and in German [Rmd] [Rnw].

The ZIP archive containing the learning resource in QTI 2.1 XML format (Question & Test Interoperability standard) for import into OpenOlat can be generated with:

library("exams")
exams2openolat("declaration.Rmd", n = 1, name = "R-exams-declaration",
  points = 1, maxattempts = 0, cutvalue = 1, solutionswitch = FALSE,
  schoice = list(enumerate = FALSE),
  stitle = "Sworn declaration", ititle = "Declaration form",
  adescription = "", sdescription = "")

There is only n = 1 version of the declaration because it is not random and it can be passed (cutvalue) when correctly signing the only exercise (points). Moreover, the code sets a couple above of titles, descriptions, and styles.

Randomized exam

First, we can leverage the usual R/exams toolbox to create an exam with four exercises where each exercise actually comes from a pool of different exercise templates, and each individual exercise is the result of some random shuffling, sampling, etc. To make this demo exam easily reproducible we only employ exercises that are readily available as part of the package. Hence, the resulting exam can illustrate the basic idea but clearly the exercises would be too different (both regarding form and content) for an actual exam.

exm <- list(
  "Geography"   = c("capitals.Rmd", "swisscapital.Rmd", "switzerland.Rmd"),
  "Graphics"    = c("boxplots.Rmd", "scatterplot.Rmd"),
  "Inference"   = c("ttest.Rmd", "anova.Rmd"),
  "Derivatives" = c("deriv2.Rmd", "hessian.Rmd")
)

Thus, the first exercise would be a randomly selected exercise from the three templates in geography (capitals, swisscapital, switzerland). Similarly, the subsequent topics are graphics (boxplots, scatterplot), inference (ttest, anova), and derivatives (deriv2, hessian) with some random variation within each topic/section.

Then, we can draw n = 50 random exams from this specification (which we would usually do if we had around 50 participants in the exam), specifying several further options:

set.seed(2020-05-11)
exams2openolat(exm, n = 50, name = "R-exams-OpenOlat",
  points = 1, maxattempts = 0, cutvalue = 2, solutionswitch = FALSE,
  duration = 60, shufflesections = TRUE, navigation = "linear",
  stitle = names(exm), ititle = "Question",
  adescription = "", sdescription = "")

Specifically, we fix the maximum duration to 60 minutes, tell OpenOlat to shuffle the order of the four sections/topics in the exam (shufflesections) and to fix the navigation to be linear. In order to help students to see the topics of the exercises that are still to come (without giving away too much detail) we label the section titles with the four topics from above.

What the resulting online exam looks like and how it can be configured further within OpenOlat is shown in detail in the video tutorial on YouTube already mentioned above. More information regarding R/exams for e-learning with OpenOlat (and Moodle) is provided in another tutorial.