R/exams @ eRum 2018

Keynote lecture at eRum 2018

R/exams was presented in a keynote lecture by Achim Zeileis at eRum 2018, the European R Users Meeting, this time organized by a team around Gergely Daróczi in Budapest. It was a great event with many exciting presentations, reflecting the vibrant R community in Europe (and beyond).

This blog post provides various resources accompanying the presentation which may be of interest to those who did not attend the meeting as well as those who did and who want to explore the materials in more detail.

Most importantly the presentation slides are available in PDF format (under CC-BY):

Video

The eRum organizers did a great job in making the meeting accessible to those useRs who could not make it to Budapest. All presentations were available in a livestream on YouTube where also videos of all lectures were made available after the meeting (Standard YouTube License):

E-Learning

To illustrate the e-learning capabilities supported by R/exams, the presentation started with a live quiz using the audience response system ARSnova. The original version of the quiz was hosted on the ARSnova installation at Universität Innsbruck. To encourage readers to try out ARSnova for their own purposes, a copy of the quiz was also posted on the official ARSnova server at Technische Hochschule Mittelhessen (where ARSnova is developed under the General Public License, GPL):

The presentation briefly also showed an online test generated by R/exams and imported into OpenOlat, an open-source learning management system (available under the Apache License). The online test is made available again here for anonymous guest access. (Note however, that the system only has one guest user so that when you start the test there may already be some test results from a previous guest session. In that case you can finish the test and also start it again.)

Replication code

The presentation slides show how to set up an exam using the R package and then rendering it into different output formats. In order to allow the same exam to be rendered into a wide range of different output formats, only single-choice and multiple-choice exercises were employed (see the choice list below). However, in the e-learning test shown in OpenOlat all exercises types are supported (see the elearn list below). All these exercises are readily provided in the package and also introduced online: deriv/deriv2, fruit/fruit2, ttest, boxplots, cholesky, lm, function. The code below uses the R/LaTeX (.Rnw) version but the R/Markdown version (.Rmd) could also be used instead.

## package
library("exams")

## single-choice and multiple-choice only
choice <- list("deriv2.Rnw", "fruit2.Rnw", c("ttest.Rnw", "boxplots.Rnw"))

## e-learning test (all exercise types)
elearn <- c("deriv.Rnw", "fruit.Rnw", "ttest.Rnw", "boxplots.Rnw",
  "cholesky.Rnw", "lm.Rnw", "function.Rnw")

First, the exam with the choice-based questions can be easily turned into a PDF exam in NOPS format using exams2nops, here using Hungarian language for illustration. Exams in this format can be easily scanned and evaluated within R.

set.seed(2018-05-16)
exams2nops(choice, institution = "eRum 2018", language = "hu")

Second, the choice-based exam version can be exported into the JSON format for ARSnova: Rexams-1.json. This contains an entire ARSnova session that can be directly imported into the ARSnova system as shown above. It employs a custom exercise set up just for eRum (conferences.Rmd) as well as a slightly tweaked exercise (fruit3.Rmd) that displays better in ARSnova.

set.seed(2018-05-16)
exams2arsnova(list("conferences.Rmd", choice[[1]], "fruit3.Rmd", choice[[3]]),
  name = "R/exams", abstention = FALSE, fix_choice = TRUE)

Third, the e-learning exam can be generated in QTI 1.2 format for OpenOlat, as shown above: eRum-2018.zip. The exams2openolat command below is provided starting from the current R/exams version 2.3-1. It essentially just calls exams2qti12 but slightly tweaks the MathJax output from pandoc so that it is displayed properly by OpenOlat.

set.seed(2018-05-16)
exams2openolat(elearn, name = "eRum-2018", n = 10, qti = "1.2")

What else?

In the last part of the presentation a couple of new and ongoing efforts within the R/exams project are highlighted. First, the natural language support in NOPS exams is mentioned which was recently described in more detail in this blog. Second, the relatively new “stress tester” was illustrated with the following example. (A more detailed blog post will follow soon.)

s <- stresstest_exercise("deriv2.Rnw")
plot(s)

Finally, a psychometric analysis illustrated how to examine exams regarding: Exercise difficulty, student performance, unidimensionality, fairness. The replication code for the results from the slides is included below (omitting some graphical details for simplicity, e.g., labeling or color).

## load data and exclude extreme scorers
library("psychotools")
data("MathExam14W", package = "psychotools")
mex <- subset(MathExam14W, nsolved > 0 & nsolved < 13)

## raw data
plot(mex$solved)

## Rasch model parameters
mr <- raschmodel(mex$solved)
plot(mr, type = "profile")

## points per student
MathExam14W <- transform(MathExam14W,
  points = 2 * nsolved - 0.5 * rowSums(credits == 1)
)
hist(MathExam14W$points, breaks = -4:13 * 2 + 0.5, col = "lightgray")
abline(v = 12.5, lwd = 2, col = 2)

## person-item map
plot(mr, type = "piplot")

## principal component analysis
pr <- prcomp(mex$solved, scale = TRUE)
plot(pr)
biplot(pr, col = c("transparent", "black"),
  xlim = c(-0.065, 0.005), ylim = c(-0.04, 0.065))

## differential item functioning
mr1 <- raschmodel(subset(mex, group == 1)$solved)
mr2 <- raschmodel(subset(mex, group == 2)$solved)
ma <- anchortest(mr1, mr2, adjust = "single-step")

## anchored item difficulties
plot(mr1, parg = list(ref = ma$anchor_items), ref = FALSE, ylim = c(-2, 3), pch = 19)
plot(mr2, parg = list(ref = ma$anchor_items), ref = FALSE, add = TRUE, pch = 19, border = 4)
legend("topleft", paste("Group", 1:2), pch = 19, col = c(1, 4), bty = "n")

## simultaneous Wald test for pairwise differences
plot(ma$final_tests)