After a dynamic exercise has been developed, thorough testing is recommended before administering the exercise in a student assessment. Therefore, the R/exams package provides the function `stresstest_exercise()`

to aid teachers in this process. While the teachers still have to assess themselves how sensible/intelligible/appropriate/… the exercise is, the R function can help check:

- whether the data-generating process works without errors or infinite loops,
- how long a random draw from the exercise takes,
- what range the correct results fall into,
- whether there are patterns in the results that students could leverage for guessing the correct answer.

To do so, the function takes the exercise and compiles it hundreds or thousands of times. In each of the iterations the correct solution(s) and the run times are stored in a data frame along with all variables (numeric/integer/character of length 1) created by the exercise. This data frame can then be used to examine undesirable behavior of the exercise. More specifically, for some values of the variables the solutions might become extreme in some way (e.g., very small or very large etc.) or single-choice/multiple-choice answers might not be uniformly distributed.

In our experience such patterns are not rare in practice and our students are very good at picking them up, leveraging them for solving the exercises in exams.

In order to exemplify the work flow, let’s consider a simple exercise about calculating a certain binomial probability:

According to a recent survey `100 - p` percent of all customers in grocery stores pay cash while the rest use their credit or cash card. You are currently waiting in the queue at the checkout of a grocery story with `n` customers in front of you. What is the probability (in percent) that `k` or more of the other customers pay with their card? |

In R, the correct answer for this exercise can be computed as `100 * pbinom(k - 1, size = n, prob = p/100, lower.tail = FALSE)`

which we ask for in the exercise (rounded to two digits).

In the folllowing, we illustrate typical problems of parameterizing such an exercise: For an exercise with numeric answer we only need to sample the variables `p`

, `n`

, and `k`

. For a single-choice version we also need a certain number of wrong answers (or “distractors”), comprised of typical errors and/or random numbers. For both types of exercises we first show a version that exhibits some undesirable properties and then proceed to an improved version of it.

# | Exercise templates | Type | Description |
---|---|---|---|

1 | binomial1.Rmd binomial1.Rnw |
`num` |
First attempt with poor parameter ranges. |

2 | binomial2.Rmd binomial2.Rnw |
`num` |
As in #1 but with some parameter ranges improved. |

3 | binomial3.Rmd binomial3.Rnw |
`schoice` |
Single-choice version based on #2 but with too many typical wrong solutions and poor random answers. |

4 | binomial4.Rmd binomial4.Rnw |
`schoice` |
As in #3 but with improved sampling of both typical errors and random solutions. |

If you want to replicate the illustrations below you can easily download all four exercises from within R. Here, we show how to do so for the R/Markdown (`.Rmd`

) version of the exercise but equivalently you can also use the R\LaTeX version (`.Rnw`

):

```
for(i in paste0(extra, 1:4, ".Rmd")) download.file(
paste0("http://www.R-exams.org/assets/posts/2019-07-10-stresstest/", i), i)
```

In a first attempt we generate the parameters in the exercise as:

```
## success probability in percent (= pay with card)
p <- sample(0:4, size = 1)
## number of attempts (= customers in queue)
n <- sample(6:9, size = 1)
## minimum number of successes (= customers who pay with card)
k <- sample(2:4, 1)
```

Let’s stress test this exercise:

`s1 <- stresstest_exercise("binomial1.Rmd")`

```
## 1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/91/92/93/94/95/96/97/98/99/100
```

By default this generates 100 random draws from the exercise with seeds from 1 to 100. The seeds are also printed in the R console, seperated by slashes. Therefore, it is easy to reproduce errors that might occur when running the stress test, i.e., just `set.seed(i)`

with `i`

being the problematic iteration and subsequently run something like `exams2html()`

or run the code for generating the parameters manually.

Here, no errors occurred but further examination shows that parameters have clearly been too extreme:

`plot(s1)`

The left panel shows the distribution run times which shows that this was fast without any problems. However, the distribution of solutions in the right panel shows that almost all solutions are extremely small. In fact, when accessing the solutions from the object `s1`

and summarizing them we see that a large proportion was exactly zero (due to rounding).

`summary(s1$solution)`

```
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.0000 0.0200 0.5239 0.3100 4.7800
```

You might have already noticed the source of the problems when we presented the code above: the values for `p`

are extremely small (and even include 0) but also `k`

is rather large. But even if we didn’t notice this in the code directly we could have detected it visually by plotting the solution against the parameter variables from the exercise:

`plot(s1, type = "solution")`

*Remark:* In addition to the `solution`

the object `s1`

contains the `seeds`

, the `runtime`

, and a `data.frame`

with all `objects`

of length 1 that were newly created within the exercise. Sometimes it is useful to explore these in more detail manually.

`names(s1)`

```
## [1] "seeds" "runtime" "objects" "solution"
```

`names(s1$objects)`

```
## [1] "k" "n" "p" "sol"
```

To fix the problems detected above, we increase the range for `p`

to be between 10 and 30 percent and reduce `k`

to values from 1 to 3, i.e., employ the following lines in the exercise:

```
p <- sample(10:30, size = 1)
k <- sample(1:3, 1)
```

Stress testing this modified exercise yields solutions with a much better spread:

```
s2 <- stresstest_exercise("binomial2.Rmd")
plot(s2)
```

Closer inspection shows that solutions can still become rather small:

`summary(s2$solution)`

```
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 3.31 20.31 50.08 47.41 72.52 94.80
```

For fine tuning we might be interested in finding out when the threshold of 5 percent probability is exceeded, depending on the variables `p`

and `k`

. This can be done graphically via:

`plot(s2, type = "solution", variables = c("p", "k"), threshold = 5)`

So maybe we could increase the minimum for `p`

from 10 to 15 percent. In the following single-choice version of the exercise we will do so in order leave more room for incorrect answers below the correct solution.

Building on the numeric example above we now move on to set up a corresponding single-choice exercise. This is supported by more learning management systems, e.g., for live voting or for written exams that are scanned automatically. Thus, in addition to the correct solution we simply need to set up a certain number of distractors. Our idea is to use two distractors that are based on typical errors students may make plus two random distractors.

```
ok <- FALSE
while(!ok) {
## two typical errors: 1-p vs. p, pbinom vs. dbinom
err1 <- 100 * pbinom(k - 1, size = n, prob = 1 - p/100, lower.tail = FALSE)
err2 <- 100 * dbinom(k, size = n, prob = p/100)
## two random errors
rand <- runif(2, min = 0, max = 100)
## make sure solutions and errors are unique
ans <- round(c(sol, err1, err2, rand), digits = 2)
ok <- length(unique(ans)) == 5
}
```

Additionally, the code makes sure that we really do obtain five distinct numbers. Even if the probability of two distractors coinciding is very small, it might occur eventually. Finally, because the correct solution is always the first element in `ans`

we can set `exsolution`

in the meta-information to `10000`

but then have to set `exshuffle`

to `TRUE`

so that the answers are shuffled in each random draw.

So we are quite hopeful that our exercise will do ok in the stress test:

```
s3 <- stresstest_exercise("binomial3.Rmd")
plot(s3)
```

The runtimes are still ok (indicating that the `while`

loop was rarely used) as are the numeric solutions - and due to shuffling the position of the correct solution
in the answer list is completely random. However, the *rank* of the solution is not: the correct solution is never the smallest or the largest of the
answer options. The reasons for this surely have to be our typical errors:

`plot(s3, type = "solution", variables = c("err1", "err2"))`

And indeed `err1`

is always (much) larger than the correct solution and `err2`

is always smaller. Of course, we could have realized this from the way we set up those typical errors but we didn’t have to - due to the stress test. Also, whether or not we consider this to be problem is a different matter. Possibly, if a certain error is very common it does not matter whether it is always larger or smaller than the correct solution.

Here, we continue to improve the exercise by randomly selecting only one of the two typical errors. Then, sometimes the error is smaller and sometimes larger than the correct solution.

`err <- sample(c(err1, err2), 1)`

Moreover, we leverage the function `num_to_schoice()`

(or equivalently `num2schoice()`

) to sample the random distractors.

`sc <- num_to_schoice(sol, wrong = err, range = c(2, 98), delta = 0.1)`

Again, this is run in a `while()`

loop to make sure that potential errors are caught (where `num_to_schoice()`

returns `NULL`

and issues a warning). This has a couple of desirable features:

`num_to_schoice()`

first samples how many random solutions should be to the left and to the right of the correct solution. This makes sure that even if the correct solution is not in the middle of the admissible range (in our case: either close to 0 or to 100), it is not more likely to get greater vs. smaller distractors.- We can set a minimum distance
`delta`

between all answer options (correct solution and distractors) to make sure that answers are not too close. - Shuffling is also carried out automatically so
`exshuffle`

does not have to be set.

Finally, the list of possible answers automatically gets LaTeX math markup `$...$`

for nicer formatting in certain situations. Therefore, for consistency, the `binomial4`

version of the exercise uses math markup for all numbers in the question.

The corresponding stress test looks sataisfactory now:

```
s4 <- stresstest_exercise("binomial4.Rmd")
plot(s4)
```

Of course, there is still at least one number that is either larger or smaller than the correct solution so that the correct solution takes rank 1 or 5 less often than ranks 2 to 4. But this seems to be a reasonable compromise.

When drawing many random versions from a certain exercise template, it is essential to thoroughly check that exercise before using it in real exams. Most importantly, of course, the authors should make sure that the story and setup is sensible, question and solution are carefully formulated, etc. But then the technical aspects should be checked as well. This should include checking whether the exercise can be rendered correctly into PDF via `exams2pdf(...)`

and into HTML via `exams2html(...)`

and/or possibly `exams2html(..., converter = "pandoc-mathjax")`

(to check different forms of math rendering). And when all of this has been checked, `stresstest_exercise()`

can help to find further problems and/or patterns that can be detected when making many random draws.

The problems in this tutorial could be detected with `n = 100`

random draws. But it is possible that some edge cases occur only very rarely so that, dependending on the complexity of the data-generating process, it is often useful to use much higher values of `n`

.

Note that in our `binomial*`

exercises it would also have been possible to set up all factorial combinations of input parameters, e.g., by `expand.grid(p = 15:30, n = 6:9, k = 1:3)`

. Then we could have asssessed directly which of these lead to ok solutions and which are too extreme. However, when the data-generating process becomes more complex this might not be so easy. The random exploration as done by `num_to_schoice()`

is still straightforward, though.

Today, the pre-conference tutorials kick off this year’s useR! 2019 conference. The organizers kindly invited R/exams for one of these tutorials. This blog post provides some accompanying resources which may be of interest to those who do not attend the conference/tutorial. The plan is to roughly follow the outline below.

Time | Content |
---|---|

14:00 14:30 15:00 |
Introduction (overview, installation, written exam) Dynamic exercises (numeric, single choice, multiple choice, cloze) One-for-all (plain PDF, plain HTML, Moodle XML) |

15:30 | — Break — |

16:00 16:30 17:00 |
E-Learning (Moodle, Canvas, ARSnova, …) Written exams (NOPS) Outlook |

Most importantly the presentation slides are available in PDF format (under CC-BY) along with an R script (under GPL-2 or GPL-3) for the examples:

Below, the examples, code, and links from the slides and R script are also provided which might be more convenient for some readers.

To set up your computer for this tutorial, follow Steps 1-4 in the tutorial http://www.R-exams.org/tutorials/installation/.

Subsequently, we start out with setps described in more detail in http://www.R-exams.org/tutorials/first_steps/. The package is loaded and first we create an “exams skeleton” with:

`demo-*.R`

scripts`exercises/`

folder with all`.Rmd`

/`.Rnw`

exercises`templates/`

folder with various customizable templates`nops/`

folder (empty) for`exams2nops()`

output

```
library("exams")
exams_skeleton()
```

To check whether everything works we render a single-choice question (swisscapital, knowledge quiz about the Swiss capital) into a plain HTML and PDF format, respectively.

```
exams2html("swisscapital.Rmd")
exams2pdf("swisscapital.Rmd")
```

And then we do the same for a numeric question with mathematical notation: (deriv, product rule for derivatives). By default `exams2html()`

uses MathML for the mathematical notation and expects the browser to render it which works ok in Firefox and Safari. But for Chrome it is better to use MathJax, see this post about mathematical notation in web exams for more details.

```
exams2html("deriv.Rmd")
exams2html("deriv.Rmd", converter = "pandoc-mathjax")
exams2pdf("deriv.Rmd")
```

Finally, a useful trick for checking whether the meta-information associated with an exercise is processed correctly:

```
exams_metainfo(exams2html(c("swisscapital.Rmd", "deriv.Rmd")))
```

This section gives a tour of the dynamic exercises shipped along with the package. See http://www.R-exams.org/intro/dynamic/ for the supported exercise types and http://www.R-exams.org/templates/ for a list of all example exercises.

Additionally, there is a simple knowledge quiz question about useR! conferences: conferences.Rmd, conferences.Rnw.

And the following exercises are used to illustrate how to go from a static to a dynamic numeric or single-choice question, see http://www.R-exams.org/tutorials/static_num_schoice/ for more details. While the tutorial uses a basic economic example about computing elasticities, the exercises below are basic arithmetic exercises about derivatives.

# | Exercise templates | Dynamic? | Type | Description |
---|---|---|---|---|

1 | expderiv1.Rmd expderiv1.Rnw |
No | `num` |
Fixed parameters and numeric solution. |

2 | expderiv2.Rmd expderiv2.Rnw |
No | `schoice` |
As in #1 but with single-choice solution (five answer alternatives). |

3 | expderiv3.Rmd expderiv3.Rnw |
Yes | `num` |
Randomly drawn parameters with dynamic computation of correct solution, based on #1. |

4 | expderiv4.Rmd expderiv4.Rnw |
Yes | `schoice` |
Randomly drawn parameters (as in #3) with dynamically-generated single-choice solution (as in #2), computed by `num_to_schoice()` . |

5 | expderiv5.Rmd expderiv5.Rnw |
Yes | `schoice` |
As in #4 but with the last alternative: None of the above. |

All of these extra exercise files can be downloaded from within R and stored in the skeleton set up above:

```
extra <- c("conferences", paste0("expderiv", 1:5))
extra <- c(paste0(extra, ".Rnw"), paste0(extra, ".Rmd"))
for(i in extra) download.file(paste0("http://www.R-exams.org/assets/posts/2019-07-09-user2019/", i), i)
```

To take a tour of the available exercise files, we switch to the `exercises/`

folder:

```
setwd("exercises")
dir()
```

```
exams2html(c(
"deriv.Rmd", ## product rule for derivatives
"tstat.Rmd", ## 1-sample t-test statistic
"dist.Rmd", ## distances and the Pythagorean theorem
"regression.Rmd", ## simple linear regression (by hand)
"fruit.Rmd", ## image-based systems of linear equations (numeric)
"lagrange.Rmd", ## method of Lagrange multipliers
"currency8.Rmd" ## convert currencies (UTF-8 encoding)
))
```

```
exams2html(c(
"swisscapital.Rmd", ## knowledge quiz question about the Swiss capital
"Rlogo.Rmd", ## knowledge quiz question about the R logo
"deriv2.Rmd", ## product rule for derivatives (single-choice, via num_to_schoice)
"tstat2.Rmd", ## 1-sample t-test statistic (single-choice, by hand)
"dist3.Rmd", ## distances and the Pythagorean theorem (single-choice, via num_to_schoice)
"fruit2.Rmd", ## image-based systems of linear equations (single-choice, via num_to_schoice)
"hessian.Rmd", ## 2x2 Hessian matrix (single-choice)
"logic.Rmd" ## interpretation of logic gates (using TikZ)
))
```

```
exams2html(c(
"switzerland.Rmd", ## knowledge quiz question about Switzerland
"gaussmarkov.Rmd", ## knowledge quiz question about Gauss-Markov assumptions
"anova.Rmd", ## 1-way analysis of variance
"boxplots.Rmd", ## interpretation of 2-sample boxplots
"scatterplot.Rmd", ## interpretation of a scatterplot
"ttest.Rmd", ## interpretation of 2-sample t test
"relfreq.Rmd", ## interpretation of relative frequency tables
"cholesky.Rmd", ## Cholesky decomposition
"automaton.Rmd" ## interpretation of automaton diagrams (using TikZ)
))
```

```
exams2html(c(
"function.Rmd", ## string question about R functions
"countrycodes.Rmd" ## string question about ISO country codes
))
```

These combine several num/schoice/mchoice/string parts:

```
exams2html(c(
"lm.Rmd", ## simple linear regression (with CSV data, schoice/num)
"boxhist.Rmd", ## univariate exploration of a CSV file (schoice/num)
"confint2.Rmd", ## 2-sided confidence interval (num)
"dist2.Rmd", ## three distances in a Cartesian coordinate system (num)
"fourfold.Rmd" ## fourfold table (num)
))
```

The function `stresstest_exercise()`

can help to do some basic quality control of dynamic exercise templates. It runs the exercises `n`

times (default: 100) with seeds `1`

to `n`

and stores run time, solution, and accompanying variables (of length 1). This can help to spot problems or patterns in the exercises.

A simple example for a numeric exercise is:

```
s3 <- stresstest_exercise("expderiv3.Rmd")
plot(s3)
plot(s3, type = "solution")
```

And for a single-choice exercise:

```
s4 <- stresstest_exercise("expderiv4.Rmd")
plot(s4)
plot(s4, type = "solution")
```

To focus on typical errors the plots against the variables `err1`

and `err2`

can either be created by the `plot()`

method or “by hand” with some customization:

```
plot(s4, type = "solution", variables = c("err1", "err2"))
plot(res ~ err1, data = s4$objects)
abline(0, 1)
plot(res ~ err2, data = s4$objects)
abline(0, 1)
```

Finally, we switch back to the original folder:

```
setwd("..")
```

A basic design idea for R/exams is that the same exercise can, in principle, be rendered into a wide range of different output formats. For standalone files the most important formats are PDF, HTML, and possibly also Docx:

```
demo_exam <- c("swisscapital.Rmd", "deriv.Rmd")
exams2html(demo_exam, edir = "exercises")
exams2pdf(demo_exam, edir = "exercises")
exams2pandoc(demo_exam, edir = "exercises")
```

The layout can be customized using suitable templates, e.g.,

```
exams2pdf(demo_exam, edir = "exercises", template = "templates/exam.tex")
exams2pdf(demo_exam, edir = "exercises", template = "templates/solution.tex")
```

Also, `exams2nops()`

internally sets up a template on the fly and then calls `exams2pdf()`

. And e-learning exams (Moodle XML, QTI 1.2/2.1, …) are illustrated in the next section.

Output for various e-learning formats can be generated in pretty much the same way as the standalone files above. The main difference is that not one file per exam is generated but a combined output file comprising the desired number of exam replications. For example for Moodle:

```
elearn_exam <- c("conferences.Rmd", "deriv.Rmd", "ttest.Rmd", "boxplots.Rmd",
"function.Rnw", "lm.Rmd", "fourfold2.Rmd", "confint3.Rmd")
set.seed(2019-07-09)
exams2moodle(elearn_exam, n = 30, name = "useR-2019",
dir = "output", edir = "exercises")
```

For the registered participants of the course the resulting online quiz is made available at https://eeecon.uibk.ac.at/~moodle/. You can login with your e-mail address and the numeric code sent to you.

Analogously, the numeric and single-/multiple-choice questions (but not yet cloze) can be prepared for Canvas by:

```
set.seed(2019-07-09)
exams2canvas(elearn_exam[1:5], n = 3, name = "useR-2019",
dir = "output", edir = "exercises", duration = 15)
```

For OpenOLAT all exercises except one Moodle-specific exercise can be used as well:

```
set.seed(2019-07-09)
exams2openolat(elearn_exam[1:7], n = 3, name = "useR-2019-olat",
dir = "output", edir = "exercises")
```

For ARSnova only single- and multiple-choice questions have good support:

```
choice_exam <- list(
"conferences.Rmd",
"deriv2.Rmd",
c("ttest.Rmd", "boxplots.Rmd")
)
set.seed(2019-07-09)
exams2arsnova(unlist(choice_exam), n = 1, name = "useR-2019",
dir = "output", edir = "exercises")
```

The written demo exam from the tutorial can be replicated by:

```
set.seed(2019-07-09)
exams2nops(choice_exam, n = 35, name = "useR-2019",
dir = "nops", edir = "exercises",
reglength = 8, blank = 1, date = "2019-07-09",
title = "Demo Exam", institution = "useR! 2019")
```

More details about creating and evaluating NOPS exams is provided in the tutorial at http://www.R-exams.org/tutorials/exams2nops/.

]]>Version 2.3-3 and subsequently 2.3-4 of the one-for-all exams generator R/exams have been published on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. In the next days this will propagate to other CRAN mirrors along with Windows and OS X binary packages. Version 2.3-4 was necessary in addition to 2.3-3 to avoid CRAN check problems on platforms where `pandoc`

is not available (currently, OS X and Solaris). The development version of the package is now version 2.3-5 on R-Forge at http://R-Forge.R-project.org/forum/?group_id=1337.

A new interface `exams2canvas()`

for the open-source Canvas learning management system has been added. This is the first dedicated Canvas interface after previously some indirect support was available through either `exams2blackboard()`

or `exams2qti12()`

. However, both of these had certain limitations for Canvas that the new function aims to overcome. The function has only received limited testing so far and is likely to improve in future version. Please report any problems to the package maintainer.

Essentially, `exams2canvas()`

is a wrapper function to `exams2qti12()`

along with a few Canvas-specific modifications of the QTI XML specification. Supported exercise types are: single-choice, multiple-choice, numeric, and string (but not yet cloze). Embedding mathematical notation, graphics, and data supplements also works.

The default `converter`

in `exams2moodle()`

has been changed to `"pandoc-mathjax"`

. Thus, instead of relying on the browser to render mathematical notation from MathML, the MathJax plugin is now assumed to be enabled on the Moodle server (which is the default when setting up a new Moodle server). To employ MathML as in previous versions one can use `converter = NULL`

. This either selects `"pandoc-mathml"`

(or equivalently `"pandoc"`

) or `"ttm"`

depending on the markup used in the exercises.

- Assure in
`nops_scan()`

that scanned image file names contain no spaces. - Added
`\usepackage[...]{babel}`

support to language specification of`exams2nops()`

. In addition to a new`Babel: ...`

field, the language`.dcf`

can also provide a`Header: ...`

field, e.g., for changing the font encoding or adding further options. - In
`nops_eval()`

duplicated sheets (original + replacement) were sometimes omitted even if the student ID was just 000000 (i.e., not read correctly by`nops_scan()`

). A check has been added that avoids this.

`exams_skeleton()`

has been updated. The new default is`markup = "markdown"`

as this appears to be the more common choice for newcomers to R/exams. Furthermore, the list of exercises in the`demo-*.R`

scripts has been updated to use newer exercises as well (along with links to the web page: http://www.R-exams.org/templates/).- Fixed bugs in
`exams2qti21()`

(and hence`exams2openolat()`

) for certain combinations of`maxchars`

specifications in`string`

exercises. - For convenience, when
`exams2html(..., converter = "pandoc-mathjax")`

,`mathjax = TRUE`

is added automatically (unless explicitly switched off). - The
`print()`

method for`exams_metainfo`

objects received a new`block`

argument so that`print(exams_metainfo(x), which = 1, block = 5)`

prints the metainformation (type/solution) of the first exam in the object`x`

(output from any`exams2xyz()`

function) in blocks of 5 exercises. Implemented for the blog about R/exams at Texas A&M International University. - For
`exams2openolat()`

the handling of the internal pandoc fixups was extended to replace`align="right"`

in tables with`style="text-align: right;"`

(and analogously for “left” and “center”). - Changed the default of
`fix_choice`

in`exams2arsnova()`

. Current versions of the official ARSnova server have the LaTeX rendering in the choice options switched off. Hence, by setting`fix_choice = TRUE`

by default the LaTeX math markup is removed. - Add
`\setkeys{Gin}{keepaspectratio}`

in the default`exams2pdf()`

template for pandoc (`plain8.tex`

) and in`exams2nops()`

. This is necessitated by a change in pandoc that now always sets`height=\textheight`

when`width=`

is specified, see the pandoc FAQ. - Added an
`envir`

argument in`exams2moodle()`

that is passed on to`xweave()`

.

Guest post by Nathaniel P. Graham (Texas A&M International University, Division of International Banking and Finance Studies).

While R/exams was originally written for large statistics classes, there is nothing specific to either large courses or statistics. In this article, I will describe how I use R/exams for my “Introduction to Finance (FIN 3310)” course. While occasionally I might have a class of 60 students, I generally have smaller sections of 20–25. These courses are taught and exams are given in-person (some material and homework assignments are delivered via the online learning management system, but I do not currently use R/exams to generate that).

My written exams are generated by `exams2nops()`

and have two components: a multiple-choice portion and a short-answer portion. The former are generated using R/exams’s dynamic exercise format while the latter are included from static PDF documents. Example pages from a typical exam are linked below:

Because I have a moderate number of students, I grade all my exams manually. This blog post outlines how the exercises are set up and which R scripts I use to automate the generation of the PDF exams.

The actual questions in my exams are not different from many other R/exams examples, except that they are about finance instead of statistics. An example is included below with LaTeX markup and both the R/LaTeX (.Rnw) and R/Markdown (.Rmd) versions can be downloaded: goalfinance.Rnw, goalfinance.Rmd. I have 7 possible answers, but only 5 will be randomly chosen for a given exam (always including the 1 correct answer). To do so, many of my non-numerical questions take advantage of the `exshuffle`

option here. (If you are one of my students, I *told* you this would be on the exam!)

Periodic updates to individual questions (stored as separate files) keep the exams fresh.

```
\begin{question}
The primary goal of financial management is to maximize the \dots
\begin{answerlist}
\item Present value of future operating cash flows
\item Net income, according to GAAP/IFRS
\item Market share
\item Value of the firm
\item Number of shares outstanding
\item Book value of shareholder equity
\item Firm revenue
\end{answerlist}
\end{question}
\exname{Goal of Financial Management}
\extype{schoice}
\exsolution{0001000}
\exshuffle{5}
```

Since this is for a finance class, there are numerical exercises as well. Every semester, I promise my students that this problem *will* be on the second midterm. The problem is simple: given some cash flows and a discount rate, calculate the net present value (NPV), see npvcalc.Rnw, npvcalc.Rmd. Since the cash flows, the discount rate, and the answers are generated from random numbers, I would not gain much from defining more than five possible, the way I did in the qualitative question above. As you might expect, some of the incorrect answers are common mistakes students make when approaching NPV, so I have not lost anything relative to the carefully crafted questions and answers I used in the past to find out who really learned the material.

```
<<echo=FALSE, results=hide>>=
discountrate <- round(runif(1, min = 6.0, max = 15.0), 2)
r <- discountrate / 100.0
cf0 <- sample(10:20, 1) * -100
ocf <- sample(seq(200, 500, 25), 5)
discounts <- sapply(1:5, function(i) (1 + r) ** i)
npv <- round(sum(ocf / discounts) + cf0, 2)
notvm <- round(sum(ocf) + cf0, 2)
wrongtvm <- round(sum(ocf / (1.0 + r)) + cf0, 2)
revtvm <- round(sum(ocf * (1.0 + r)) + cf0, 2)
offnpv <- round(npv + sample(c(-200.0, 200.0), 1), 2)
@
\begin{question}
Assuming the discount rate is \Sexpr{discountrate}\%, find the
net present value of a project with the following cash flows, starting
at time 0: \$\Sexpr{cf0}, \Sexpr{ocf[1]}, \Sexpr{ocf[2]}, \Sexpr{ocf[3]},
\Sexpr{ocf[4]}, \Sexpr{ocf[5]}.
\begin{answerlist}
\item \$\Sexpr{wrongtvm}
\item \$\Sexpr{notvm}
\item \$\Sexpr{npv}
\item \$\Sexpr{revtvm}
\item \$\Sexpr{offnpv}
\end{answerlist}
\end{question}
\exname{Calculating NPV}
\extype{schoice}
\exsolution{00100}
\exshuffle{5}
```

While there are other methods of generating randomized exams out there, few are as flexible as R/exams, in large part because we have full access to R. The next example also appears on my second midterm; it asks students to compute the internal rate of return for a set of cash flows, see irrcalc.Rnw, irrcalc.Rmd. Numerically, this is a simple root-finding problem, but systems that do not support more advanced mathematical operations (such as Blackboard’s “Calculated Formula” questions) can make this difficult or impossible to implement directly.

```
<<echo=FALSE, results=hide>>=
cf0 <- sample(10:16, 1) * -100
ocf <- sample(seq(225, 550, 25), 5)
npvfunc <- function(r) {
discounts <- sapply(1:5, function(i) (1 + r) ** i)
npv <- (sum(ocf / discounts) + cf0) ** 2
return(npv)
}
res <- optimize(npvfunc, interval = c(-1,1))
irr <- round(res$minimum, 4) * 100.0
wrong1 <- irr + sample(c(1.0, -1.0), 1)
wrong2 <- irr + sample(c(0.25, -0.25), 1)
wrong3 <- irr + sample(c(0.5, -0.5), 1)
wrong4 <- irr + sample(c(0.75, -0.75), 1)
@
\begin{question}
Find the internal rate of return of a project with the following cash flows,
starting at time 0: \$\Sexpr{cf0}, \Sexpr{ocf[1]}, \Sexpr{ocf[2]},
\Sexpr{ocf[3]}, \Sexpr{ocf[4]}, \Sexpr{ocf[5]}.
\begin{answerlist}
\item \Sexpr{wrong1}\%
\item \Sexpr{wrong2}\%
\item \Sexpr{irr}\%
\item \Sexpr{wrong3}\%
\item \Sexpr{wrong4}\%
\end{answerlist}
\end{question}
\exname{Calculating IRR}
\extype{schoice}
\exsolution{00100}
\exshuffle{5}
```

While `exams2nops()`

has some support for open-ended questions that are scored manually, it is very limited. I create and format those questions in the traditional manner (Word or LaTeX) and save the resulting file as a PDF. Alternatively, a custom `exams2pdf()`

could be used for this. Finally, I add this PDF file on to the end of the exam using the `pages`

option of `exams2nops()`

(as illustrated in more detail below). As an example, the short-answer questions from the final exam above are given in the following PDF file:

To facilitate automatic inclusion of these short-answer questions, a naming convention is adopted in the scripts below. The script looks for a file named `examNsa.pdf`

, where `N`

is the exam number (1 for the first midterm exam, 2 for the second, and 3 for the final exam) and sets `pages`

to it. So for the final exam, it looks for the file `exam3sa.pdf`

. If I want to update my short-answer questions, I just replace the old PDF with a new one.

I keep the questions for each exam in their own directory, which makes the script below that I use to generate exams 1 and 2 straightforward. For the moment, the script uses all the questions in an exam’s directory, but I if want to guarantee that I have exactly (for example) 20 multiple choice questions, I could set `nsamp = 20`

instead of however many Rnw (or Rmd) files it finds.

Note that `exlist`

is a list with one element (a vector of file names), so that R/exams will randomize the order of the questions as well. If I wanted to specify the order, I could make `exlist`

a list of N elements, where each element was exactly one file/question. The `nsamp`

option could then be left unset (the default is `NULL`

).

When I generate a new set of exams, I only need to update the first four lines, and the script does the rest. Note that I use a modified version of the English `en.dcf`

language configuration file in order to adapt some terms to TAMIU’s terminology, e.g., “ID Number” instead of the `exams2nops()`

default “Registration Number”. See en_tamiu.dcf for the details. Since the TAMIU student ID numbers have 8 digits, I use the `reglength = 8`

argument, which sets the number of digits in the ID Number to 8.

```
library("exams")
## configuration: change these to make a new batch of exams
exid <- 2
exnum <- 19
exdate <- "2019-04-04"
exsemester <- "SP19"
excourse <- "FIN 3310"
## options derived from configuration above
exnam <- paste0("fin3310exam", exid)
extitle <- paste(excourse, exsemester, "Exam", exid, sep = " ")
saquestion <- paste0("SA_questions/exam", exid, "sa.pdf")
## exercise directory (edir) and output directory (odir)
exedir <- paste0("fin3310_exam", exid, "_exercises")
exodir <- "nops"
if(!dir.exists(exodir)) dir.create(exodir) ## in case it was previously deleted
exodir <- paste0(exodir, "/exam", exid, "_", format(Sys.Date(), "%Y-%m-%d"))
## exercise list: one element with a vector of all available file names
exlist <- list(dir(path = exedir, pattern = "\.Rnw$"))
## generate exam
set.seed(54321) # not the actual random seed
exams2nops(file = exlist,
n = exnum, nsamp = length(exlist[[1]]), dir = exodir, edir = exedir,
name = exnam, date = exdate, course = excourse, title = extitle,
institution = "Texas A\\&M International University",
language = "en_tamiu.dcf", encoding = "UTF-8",
pages = saquestion, blank = 1, logo = NULL, reglength = 8,
samepage = TRUE)
```

My final exam is comprehensive, so I would like to include questions from the previous exams. I do not want to keep separate copies of those questions just for the final, in case I update one version and forget to update the other, so I need a script that gathers up questions from the first two midterms and adds in some questions specific to the final.

The script below uses a feature that has long been available in R/exams but was undocumented up to version 2.3-2: You can set `edir`

to a directory and all its subdirectories will be included in the search path. I have specified some required questions that I want to appear on every student’s exam; each student will also get a random draw of other questions from the first two midterms in addition to some questions that only appear on the final. How many of each is controlled by `exnsamp`

, which is passed to the `nsamp`

argument. They add up to 40 currently, so `exams2nops()`

’s 45 question limit does not affect me.

```
library("exams")
## configuration: change these to make a new batch of exams
exnum <- 19
exdate <- "2019-04-30"
exsemester <- "SP19"
excourse <- "FIN 3310"
## options derived from configuration above
extitle <- paste(excourse, exsemester, "Exam", exid, sep = " ")
## exercise directory (edir) and output directory (odir)
exedir <- getwd()
exodir <- "nops"
if(!dir.exists(exodir)) dir.create(exodir)
exodir <- paste0(exodir, "/exam3", "_", format(Sys.Date(), "%Y-%m-%d"))
## exercises: required and from previous midterms
exrequired <- c("goalfinance.Rnw", "pvcalc.Rnw", "cashcoverageortie.Rnw",
"ocfcalc.Rnw", "discountratecalc.Rnw", "npvcalc.Rnw", "npvcalc2.Rnw",
"stockpriceisabouttopaycalc.Rnw", "annuitycalc.Rnw", "irrcalc.Rnw")
exlist1 <- dir(path = "fin3310_exam1_exercises", pattern = "\.Rnw$")
exlist2 <- dir(path = "fin3310_exam2_exercises", pattern = "\.Rnw$")
exlist3 <- dir(path = "fin3310_exam3_exercises", pattern = "\.Rnw$")
## final list and corresponding number to be sampled
exlist <- list(
exrequired,
setdiff(exlist1, exrequired),
setdiff(exlist2, exrequired),
exlist3
)
exnsamp <- c(10, 5, 10, 15)
## generate exam
set.seed(54321) # not the actual random seed
exams2nops(file = exlist,
n = exnum, nsamp = exnsamp, dir = exodir, edir = exedir,
name = "fin3310exam3", date = exdate, course = excourse, title = extitle,
institution = "Texas A\\&M International University",
language = "en_tamiu.dcf", encoding = "UTF-8",
pages = "SA_questions/exam3sa.pdf", blank = 1, logo = NULL,
reglength = 8, samepage = TRUE)
```

R/exams can read scans of the answer sheet, automating grading for the multiple-choice questions. For a variety of reasons, I do not use any of those features. If I had to teach larger classes I would doubtless find a way to make it convenient, but for the foreseeable future I will continue to use the function below to produce the answer key for each exam, which I grade by hand. This is not especially onorous, since I have to grade the short-answer questions by hand anyway.

Given the serialized R data file (.rds) produced by `exam2nops()`

the corresponding `exams_metainfo()`

can be extracted. This comes with a `print()`

method that displays all correct answers for a given exam. Starting from R/exams version 2.3-3 (current R-Forge devel version) it is also possible to split the output into blocks of five for easy reading (matching the blocks on the answer sheets). As an example the first few correct answers of the third exam are extracted:

```
fin3310exam3 <- readRDS("fin3310exam3.rds")
print(exams_metainfo(fin3310exam1), which = 3, block = 5)
```

```
## 19043000003
## 1. Discount Rate: 1
## 2. Calculating IRR: 4
## 3. Present Value: 3
## 4. Calculating stock prices: 5
## 5. Calculating NPV: 3
##
## 6. Annuity PV: 3
## 7. Goal of Financial Management: 1
## 8. Finding OCF: 5
## ...
```

It can be convenient to display the correct answers in a customized format, e.g., with letters instead of numbers and omitting the exercise title text. To do so, the code below sets up a function `exam_solutions()`

, applying it to the same exam as above.

```
exam_solutions <- function(examdata, examnum) {
solutions <- LETTERS[sapply(examdata[[examnum]],
function(x) which(x$metainfo$solution))]
data.frame(Solution = solutions)
}
split(exam_solutions(fin3310exam3, examnum = 3), rep(1:8, each = 5))
```

```
## $`1`
## Solution
## 1 A
## 2 D
## 3 C
## 4 E
## 5 C
##
## $`2`
## Solution
## 6 C
## 7 A
## 8 E
## ...
```

The only downside of the manual grading approach is that I do not have students’ responses in an electronic format for easy statistical analysis, but otherwise grading is very fast.

Using the system above, R/exams works well even for small courses with in-person, paper exams. I do not need to worry about copies of my exams getting out and students memorizing it, or any of the many ways students have found to cheat over the years. By doing all the formatting work for me, R/exams helps me avoid a lot of the finicky aspects of adding, adjusting, or removing questions from an existing exam, and generally keeps exam construction focused on the important part: writing questions that gauge students’ progress.

]]>A popular use case of the R/exams package is the generation of dynamic exercises for online learning management systems in large-scale courses in mathematics, statistics, or physics. Often, these contain some mathematical notation using LaTeX markup. While LaTeX can be easily rendered into PDF for printing written exams, the options for rendering it into HTML for online exams are somewhat more diverse and the best strategies depend on a number of factors. As these involve a number of technical details this tutorial aims to give a brief overview and makes a few practical recommendations. Factors include the following:

*Original markup in exercise source files:*

The R/exams exercises may employ either Markdown or LaTeX for formatting and structuring the text (see the First steps tutorial for more details). In either case LaTeX is used for the mathematical notation. Thus, also in Markdown the mathematical content is embedded as LaTeX code.*Math markup in HTML:*

For display of the formatted text online in a browser, the original markup (Markdown/LaTeX) needs to be converted (see below) to HTML with some suitable formatting for the mathematical content. Either the LaTeX can be preserved or turned into MathML, a dedicated XML format for describing mathematical content.*Rendering of math markup:*

The most widely-used way of rendering mathematical content in web pages is MathJax (see also the corresponding Wikipedia page), an open-source JavaScript library that works in all browsers without any setup by the user (except enabling JavaScript). It can render LaTeX directly but also MathML. Moreover, some browsers (mainly Firefox and Safari but*not*Chrome) have also native support for displaying MathML, i.e, without the need for an additional display engine such as MathJax. The advantage of employing the browser for MathML rendering is that it is faster and typically blends better into the regular text. The disadvantage is obviously that it is not supported by all browsers and does not support rendering of LaTeX directly.*Conversion to HTML:*

The original markup from the exercises templates needs to be converted to HTML and the most powerful document converter for this is Pandoc (see also the corresponding Wikipedia page). It can convert both Markdown and LaTeX source files to HTML, either with LaTeX embedded for MathJax or with MathML (among various other options). Moreover, LaTeX (but not Markdown) exercises can be converted to HTML using the TeX-to-MathML converter TtM. (R/exams also provides a few further options which are typically of less interest due to lower quality, such as TtH or rendered images.)*Defaults in R/exams:*

All`exams2xyz()`

functions that produce HTML-based output offer a`converter =`

argument. By default, this is configured to produce HTML with embedded MathML because this can be rendered both by MathJax in all browsers as well as by some browsers directly. The default is is`converter = "pandoc"`

(equivalent to`"pandoc-mathml"`

) for R/Markdown exercises and`converter = "ttm"`

for R/LaTeX exercises, respectively. Whether MathJax is enabled varies across output formats or might depend on the learning management system.

As a simple illustration of the strengths and weaknesses of the different approaches, the deriv exercise template (computation of a derivative using the product rule) is converted to HTML using `exams2html()`

. Here, the R/LaTeX version of the exercise is used (`"deriv"`

or equivelently `"deriv.Rnw"`

) but using the R/Markdown version (`"deriv.Rmd"`

) yields almost the same output.

The following examples and resulting screenshots contrast the output between Firefox and Chrome. By clicking on the screenshots you can also see what the HTML pages look like in your own browser.

By default, `exams2html()`

generates HTML with MathML embedded and uses a template that does *not* enable MathJax.

```
library("exams")
set.seed(1090)
exams2html("deriv")
```

The screenshots that the native MathML support in Firefox leads to output that renders fast and smoothly. However, the lack of native MathML support in Chrome means that the exercise cannot be displayed correctly.

To easily explore the effect of MathJax rendering `exams2html()`

supports the argument `mathjax = TRUE`

which inserts the MathJax `<script>`

into the template (loaded from RStudio’s content delivery network).

```
set.seed(1090)
exams2html("deriv", mathjax = TRUE)
```

Now the math output looks good in both browsers. However, for Firefox users the version without MathJax might still be preferable as it renders faster with somewhat smoother output.

To preserve the LaTeX equations, the argument `converter = "pandoc-mathjax"`

can be used. Then, Pandoc converts the LaTeX text to HTML but preserves the LaTeX equations (in minimal HTML markup).

```
set.seed(1090)
exams2html("deriv", converter = "pandoc-mathjax", mathjax = TRUE)
```

The output is very similar to the MathML rendered MathJax above. However, note that the alignment in the equations is changed (from left to right). This is caused by Pandoc replacing the LaTeX `{eqnarray*}`

environment employed in the `"deriv"`

exercise by `{aligned}`

environments.

*Original markup in exercise source files:*

Whether to use R/Markdown (`.Rmd`

) or R/LaTeX (`.Rnw`

) markup in the exercise source files is mostly a matter of taste. The former is probably somewhat easier to learn for beginners but generally differences are small if there is only moderate text formatting. It is also good practice to keep the formatting simple to be robust across the different output formats and more advanced math constructs should be checked carefully.*MathML as default math markup:*

As of today (January 2019) it might seem more natural to use LaTeX rendered by MathJax as`rmarkdown`

does. However, when HTML conversion was added to R/exams in early 2012, MathML rendered by Firefox was the more robust choice. MathML has been preserved as the default since then because it can be rendered both by MathJax and some browsers.*Pandoc vs. TtM:*

Pandoc is the default converter for all`.Rmd`

exercises while TtM is still used by default for`.Rnw`

exercises. The latter is mostly for backward compatibility but might change to Pandoc in the future. However, differences are not very large for most exercises anyway but some mathematical LaTeX constructs are just supported by one and not the other converter.*Enabling MathJax rendering:*

Many modern learning management systems have MathJax enabled by default, e.g., in Moodle, Canvas, or OpenOLAT. There are a couple of caveats, though: First, the default MathJax configuration in Moodle and OpenOLAT switches off rendering of MathML. Second, some systems do not host their own copy of the MathJax library but employ a content delivery network (CDN). Thus, there is a small risk that the learning management system might be up and running but there are problems loading MathJax from the CDN.*Moodle:*

As noted above the default configuration for Moodle has MathJax support (via a CDN) but switches off rendering of MathML. As`exams2moodle(...)`

currently uses MathML markup by default, this necessitates Firefox or Safari for viewing the quizzes in Moodle. In contrast,`exams2moodle(..., converter = "pandoc-mathjax")`

would use LaTeX math markup and render it by MathJax (unless the Moodle configuration switched off MathJax support).

**Request:**Feedback from Moodle users would be appreciated on whether they prefer the current default or`converter = "pandoc-mathjax"`

. It would be especially useful to find out whether the latter works in their Moodle installations.*Canvas:*

We are currently working on a dedicated`exams2qti12()`

-based function`exams2canvas()`

for generating quizzes for Canvas. This will keep the MathML-based default for the math notation as this is rendered smoothly by Canvas’ own MathJax support.*OpenOLAT:*

A new dedicated interface`exams2openolat()`

for generating quizzes for OpenOLAT has been added recently to R/exams (version 2.3-1). By default, this is a wrapper to`exams2qti21(..., converter = "pandoc-mathjax")`

because OpenOLAT provides MathJax rendering of LaTeX math (but not MathML). Some additional tweaks are necessary, though, because OpenOLAT expects the LaTeX to be embedded slightly differently from standard Pandoc output.

The mathematical equation in the random draw of the deriv exercise in LaTeX is: `f(x) = x^{8} e^{3.4x}`

. Here, we highlight that all converters yield almost equivalent output when rendered by MathJax:

`"pandoc-mathjax"` |
`"ttm"` |
`"pandoc-mathml"` |
---|---|---|

\(f(x) = x^{8} e^{3.4x}\) | $f(x)={x}^{8}{e}^{3.4x}$ | $f(x) = x^{8} e^{3.4x}$ |

(**Note:** If you are viewing this on R-bloggers or another aggregator some or all of the equations will not display correctly. Refer to the R/exams site for a version with MathJax properly enabled.)

The underlying LaTeX code generated by `converter = "pandoc-mathjax"`

is simply the original LaTeX code with some minimal HTML markup:

```
<span class="math inline">\(f(x) = x^{8} e^{3.4x}\)</span>
```

The MathML code generated by `converter = "ttm"`

differs slightly from the of `converter = "pandoc"`

(or equivalently `"pandoc-mathml"`

). The former yields:

```
<math xmlns="http://www.w3.org/1998/Math/MathML">
<mrow><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo>
<msup><mrow><mi>x</mi></mrow><mrow><mn>8</mn></mrow>
</msup>
<msup><mrow><mi>e</mi></mrow><mrow><mn>3</mn><mo>.</mo><mn>4</mn><mi>x</mi></mrow>
</msup>
</mrow></math>
```

The Pandoc version is very similar but contains some more markup and annotation:

```
<math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>
<mrow><mi>f</mi>
<mo stretchy="false" form="prefix">(</mo><mi>x</mi><mo stretchy="false" form="postfix">)</mo>
<mo>=</mo>
<msup><mi>x</mi><mn>8</mn></msup><msup><mi>e</mi><mrow><mn>3.4</mn><mi>x</mi></mrow></msup>
</mrow><annotation encoding="application/x-tex">f(x) = x^{8} e^{3.4x}</annotation>
</semantics></math>
```

]]>Version 2.3-2 of the one-for-all exams generator R/exams has been published on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. In the next days this will propagate to other CRAN mirrors along with Windows and OS X binary packages. The development version of the package is now version 2.3-3 on R-Forge at http://R-Forge.R-project.org/forum/?group_id=1337.

- Support for further languages is available in
`exams2nops()`

and`nops_eval()`

: Russian (`ru`

, contributed by Boris Demeshev) and Serbian (`sr`

, contributed by Tatjana Kecojevic). Furthermore, Croatian (`hr`

) was streamlined along with Serbian by Tatjana Kecojevic. - In
`exams2nops()`

the`header`

argument can also be specified simply as`header = "\\mycommand{value}"`

rather than`header = list(mycommand = "value")`

. The former is more flexible, e.g., for passing additional options or more complex commands. Internally, the former is turned into an unnamed list which is then processed correspondingly by`exams2pdf()`

. - Scanning of written NOPS exams is enhanced and registration IDs are processed more reliably: First,
`nops_scan()`

gained a new argument`trim = 0.3`

that controls how much of the check boxes is trimmed in order to shave the borders prior to determining the average gray level. In versions up to 2.3-1 this implicitly had a value of`0.25`

hard-coded. Now the default increased to`0.3`

in order to shave box borders more reliably, e.g., in more pixelated scans. Second,`nops_scan()`

tries to process registration numbers more reliably. In case one of the registration columns contains more than one potential mark, the heuristics of determining the intended mark have been improved. `nops_eval()`

gained a new argument`labels = NULL`

that can be used to give labels for the marks that differ from the default`(length(mark) + 1):1`

.

- The new exercise confint3 (Rmd/Rnw) illustrates how to use the
`verbatim`

type for advanced processing in Moodle when asking for the computation of a 2-sided confidence interval. Rather than using a single correct value for each numeric result (along with a corresponding tolerance interval) as in`num`

exercises (see confint2), the exercise also provides a second partially correct value for each result. More precisely, the exercise yields 100% of the points for the correct solution based on t quantiles but still 50% for the solution based on normal quantiles. This is possible by setting the type to`verbatim`

and specifying the solution via the necessary Moodle XML markup directly, e.g.,`:NUMERICAL:=<value1>:<tolerance1>~%50%<value2>:<tolerance2>#<comment>`

. The exercise is a contribution by Ulrike Groemping.

- Rather than fully importing the basic dependencies
`stats`

,`graphics`

,`grDevices`

,`tools`

, and`utils`

, only the required functions from those packages are imported selectively. The main motivation for this was that otherwise the code evaluated in R/exams exercises might also use the same NAMESPACE semantics as the`exams`

package - instead of employing the setup in the user’s environment. The issue was raised in this StackOverflow thread by Florian Oswald. - In
`include_supplement()`

an argument`target = NULL`

is added to optionally include the supplement with a different file name than the original file name. - In all LaTeX templates that use Helvetica (phv) as the font for the main text, this is also used now in math mode by
`\usepackage[helvet]{sfmath}`

(rather than`\usepackage{sfmath}`

, as employed previously). In particular, this affects`exams2nops()`

and`tex2image()`

output.

- In previous versions the
`pandoc`

-based HTML`converter`

erroneously produced unbalanced`<p>`

tags in certain situations. - The QTI 2.1 XML output from
`exams2qti21()`

is somewhat cleaner, fixing some computations for correct/incorrect answers in`schoice`

/`mchoice`

exercises.

`confint3`

Computing the 2-sided confidence interval at 95% level for the mean based on a random sample. The exercise is a cloze with two numeric answers for the lower and upper bound of the confidence interval, respectively. Using the 'verbatim' clozetype for Moodle, the exercises yields 100% of the points for the correct solution based on t quantiles but still 50% for a partially correct solution based on normal quantiles.

Yes

Random numbers

Yes

No

No

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("confint3.Rnw")
set.seed(1090)
exams2pdf("confint3.Rnw")
set.seed(1090)
exams2html("confint3.Rmd")
set.seed(1090)
exams2pdf("confint3.Rmd")
```

]]>R/exams was presented in a keynote lecture by Achim Zeileis at eRum 2018, the European R Users Meeting, this time organized by a team around Gergely Daróczi in Budapest. It was a great event with many exciting presentations, reflecting the vibrant R community in Europe (and beyond).

This blog post provides various resources accompanying the presentation which may be of interest to those who did not attend the meeting as well as those who did and who want to explore the materials in more detail.

Most importantly the presentation slides are available in PDF format (under CC-BY):

The eRum organizers did a great job in making the meeting accessible to those useRs who could not make it to Budapest. All presentations were available in a livestream on YouTube where also videos of all lectures were made available after the meeting (Standard YouTube License):

To illustrate the e-learning capabilities supported by R/exams, the presentation started with a live quiz using the audience response system ARSnova. The original version of the quiz was hosted on the ARSnova installation at Universität Innsbruck. To encourage readers to try out ARSnova for their own purposes, a copy of the quiz was also posted on the official ARSnova server at Technische Hochschule Mittelhessen (where ARSnova is developed under the General Public License, GPL):

The presentation briefly also showed an online test generated by R/exams and imported into OpenOLAT, an open-source learning management system (available under the Apache License). The online test is made available again here for anonymous guest access. *(Note however, that the system only has one guest user so that when you start the test there may already be some test results from a previous guest session. In that case you can finish the test and also start it again.)*

The presentation slides show how to set up an exam using the R package and then rendering it into different output formats. In order to allow the same exam to be rendered into a wide range of different output formats, only single-choice and multiple-choice exercises were employed (see the `choice`

list below). However, in the e-learning test shown in OpenOLAT all exercises types are supported (see the `elearn`

list below). All these exercises are readily provided in the package and also introduced online: deriv/deriv2, fruit/fruit2, ttest, boxplots, cholesky, lm, function. The code below uses the R/LaTeX (.Rnw) version but the R/Markdown version (.Rmd) could also be used instead.

```
## package
library("exams")
## single-choice and multiple-choice only
choice <- list("deriv2.Rnw", "fruit2.Rnw", c("ttest.Rnw", "boxplots.Rnw"))
## e-learning test (all exercise types)
elearn <- c("deriv.Rnw", "fruit.Rnw", "ttest.Rnw", "boxplots.Rnw",
"cholesky.Rnw", "lm.Rnw", "function.Rnw")
```

First, the exam with the choice-based questions can be easily turned into a PDF exam in NOPS format using `exams2nops`

, here using Hungarian language for illustration. Exams in this format can be easily scanned and evaluated within R.

```
set.seed(2018-05-16)
exams2nops(choice, institution = "eRum 2018", language = "hu")
```

Second, the choice-based exam version can be exported into the JSON format for ARSnova: Rexams-1.json. This contains an entire ARSnova session that can be directly imported into the ARSnova system as shown above. It employs a custom exercise set up just for eRum (conferences.Rmd) as well as a slightly tweaked exercise (fruit3.Rmd) that displays better in ARSnova.

```
set.seed(2018-05-16)
exams2arsnova(list("conferences.Rmd", choice[[1]], "fruit3.Rmd", choice[[3]]),
name = "R/exams", abstention = FALSE, fix_choice = TRUE)
```

Third, the e-learning exam can be generated in QTI 1.2 format for OpenOLAT, as shown above: eRum-2018.zip. The `exams2openolat`

command below is provided starting from the current R/exams version 2.3-1. It essentially just calls `exams2qti12`

but slightly tweaks the MathJax output from pandoc so that it is displayed properly by OpenOLAT.

```
set.seed(2018-05-16)
exams2openolat(elearn, name = "eRum-2018", n = 10, qti = "1.2")
```

In the last part of the presentation a couple of new and ongoing efforts within the R/exams project are highlighted. First, the natural language support in NOPS exams is mentioned which was recently described in more detail in this blog. Second, the relatively new “stress tester” was illustrated with the following example. (A more detailed blog post will follow soon.)

```
s <- stresstest_exercise("deriv2.Rnw")
plot(s)
```

Finally, a psychometric analysis illustrated how to examine exams regarding: Exercise difficulty, student performance, unidimensionality, fairness. The replication code for the results from the slides is included below (omitting some graphical details for simplicity, e.g., labeling or color).

```
## load data and exclude extreme scorers
library("psychotools")
data("MathExam14W", package = "psychotools")
mex <- subset(MathExam14W, nsolved > 0 & nsolved < 13)
## raw data
plot(mex$solved)
## Rasch model parameters
mr <- raschmodel(mex$solved)
plot(mr, type = "profile")
## points per student
MathExam14W <- transform(MathExam14W,
points = 2 * nsolved - 0.5 * rowSums(credits == 1)
)
hist(MathExam14W$points, breaks = -4:13 * 2 + 0.5, col = "lightgray")
abline(v = 12.5, lwd = 2, col = 2)
## person-item map
plot(mr, type = "piplot")
## principal component analysis
pr <- prcomp(mex$solved, scale = TRUE)
plot(pr)
biplot(pr, col = c("transparent", "black"),
xlim = c(-0.065, 0.005), ylim = c(-0.04, 0.065))
## differential item functioning
mr1 <- raschmodel(subset(mex, group == 1)$solved)
mr2 <- raschmodel(subset(mex, group == 2)$solved)
ma <- anchortest(mr1, mr2, adjust = "single-step")
## anchored item difficulties
plot(mr1, parg = list(ref = ma$anchor_items), ref = FALSE, ylim = c(-2, 3), pch = 19)
plot(mr2, parg = list(ref = ma$anchor_items), ref = FALSE, add = TRUE, pch = 19, border = 4)
legend("topleft", paste("Group", 1:2), pch = 19, col = c(1, 4), bty = "n")
## simultaneous Wald test for pairwise differences
plot(ma$final_tests)
```

]]>`fruit2`

A system of three linear equations has to be solved and the solution has to be entered into a fourth equation. However, the system is not defined through a verbal description or mathermatical notation but through images (clip art of tropical fruits). The problem can be interpreted as prices of three fruits (banana, orange, pineapple) and corresponding fruit baskets with different combinations of fruits. Images are stored in Base64 encoding within the exercise files and embedded dynamically into the output. A set of five answer alternatives is generated based on two potential mistakes and two random solutions from a suitable range. PDFs are best generated from the Rnw version, HTML is best generated with pandoc from either the Rmd version (where pandoc is used by default) or the Rnw version (where ttm is used by default, but pandoc can be easily used as well.)

Yes

Random numbers, shuffled graphics

Yes

No

Yes

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("fruit2.Rnw")
set.seed(1090)
exams2pdf("fruit2.Rnw")
set.seed(1090)
exams2html("fruit2.Rmd")
set.seed(1090)
exams2pdf("fruit2.Rmd")
```

]]>`fruit`

A system of three linear equations has to be solved and the solution has to be entered into a fourth equation. However, the system is not defined through a verbal description or mathermatical notation but through images (clip art of tropical fruits). The problem can be interpreted as prices of three fruits (banana, orange, pineapple) and corresponding fruit baskets with different combinations of fruits. Images are stored in Base64 encoding within the exercise files and embedded dynamically into the output. PDFs are best generated from the Rnw version, HTML is best generated with pandoc from either the Rmd version (where pandoc is used by default) or the Rnw version (where ttm is used by default, but pandoc can be easily used as well.)

Yes

Random numbers, shuffled graphics

Yes

No

Yes

No

*(Note that the HTML output contains mathematical equations in MathML. It is displayed by browsers with MathML support like Firefox or Safari - but not Chrome.)*

**Demo code:**

```
library("exams")
set.seed(1090)
exams2html("fruit.Rnw")
set.seed(1090)
exams2pdf("fruit.Rnw")
set.seed(1090)
exams2html("fruit.Rmd")
set.seed(1090)
exams2pdf("fruit.Rmd")
```

]]>