The new package exams2forms, co-authored with Reto Stauffer, provides several building blocks for embedding exercises written with the R package exams (also known as R/exams) in interactive documents or quizzes written with rmarkdown or quarto.
The idea is that the dynamic exercises in R/exams’ Rmd (R/Markdown) or Rnw (R/LaTeX) format can also be reused in HTML documents, web pages, or online books. This facilitates their use for self-paced learning and self-assessment without the need for a learning management system etc. And for (summative) assessments the same dynamic exercises could then be exported to different learning management systems or employed in written exams.
All R/exams exercise types are supported:
Many of the ideas as well as the code in the package have been adapted from the webexercises package, authored by Dale Barr and Lisa DeBruine.
As quick demonstration for R/exams exercises embedded into an HTML document, the two examples from the First Steps tutorial are included below: The single-choice exercise swisscapital and the numeric exercise deriv, both in three random variations.
What is the seat of the federal authorities in Switzerland (i.e., the de facto capital)?
There is no de jure capital but the de facto capital and seat of the federal authorities is Bern.
What is the seat of the federal authorities in Switzerland (i.e., the de facto capital)?
There is no de jure capital but the de facto capital and seat of the federal authorities is Bern.
What is the seat of the federal authorities in Switzerland (i.e., the de facto capital)?
There is no de jure capital but the de facto capital and seat of the federal authorities is Bern.
What is the derivative of \(f(x) = x^{7} e^{3.9 x}\), evaluated at \(x = 0.67\)?
Using the product rule for \(f(x) = g(x) \cdot h(x)\), where \(g(x) := x^{7}\) and \(h(x) := e^{3.9 x}\), we obtain \[ \begin{aligned} f'(x) &= [g(x) \cdot h(x)]' = g'(x) \cdot h(x) + g(x) \cdot h'(x) \\ &= 7 x^{7 - 1} \cdot e^{3.9 x} + x^{7} \cdot e^{3.9 x} \cdot 3.9 \\ &= e^{3.9 x} \cdot(7 x^6 + 3.9 x^{7}) \\ &= e^{3.9 x} \cdot x^6 \cdot (7 + 3.9 x). \end{aligned} \] Evaluated at \(x = 0.67\), the answer is \[ e^{3.9 \cdot 0.67} \cdot 0.67^6 \cdot (7 + 3.9 \cdot 0.67) = 11.860944. \] Thus, rounded to two digits we have \(f'(0.67) = 11.86\).
What is the derivative of \(f(x) = x^{4} e^{3.1 x}\), evaluated at \(x = 0.53\)?
Using the product rule for \(f(x) = g(x) \cdot h(x)\), where \(g(x) := x^{4}\) and \(h(x) := e^{3.1 x}\), we obtain \[ \begin{aligned} f'(x) &= [g(x) \cdot h(x)]' = g'(x) \cdot h(x) + g(x) \cdot h'(x) \\ &= 4 x^{4 - 1} \cdot e^{3.1 x} + x^{4} \cdot e^{3.1 x} \cdot 3.1 \\ &= e^{3.1 x} \cdot(4 x^3 + 3.1 x^{4}) \\ &= e^{3.1 x} \cdot x^3 \cdot (4 + 3.1 x). \end{aligned} \] Evaluated at \(x = 0.53\), the answer is \[ e^{3.1 \cdot 0.53} \cdot 0.53^3 \cdot (4 + 3.1 \cdot 0.53) = 4.343937. \] Thus, rounded to two digits we have \(f'(0.53) = 4.34\).
What is the derivative of \(f(x) = x^{5} e^{3.6 x}\), evaluated at \(x = 0.7\)?
Using the product rule for \(f(x) = g(x) \cdot h(x)\), where \(g(x) := x^{5}\) and \(h(x) := e^{3.6 x}\), we obtain \[ \begin{aligned} f'(x) &= [g(x) \cdot h(x)]' = g'(x) \cdot h(x) + g(x) \cdot h'(x) \\ &= 5 x^{5 - 1} \cdot e^{3.6 x} + x^{5} \cdot e^{3.6 x} \cdot 3.6 \\ &= e^{3.6 x} \cdot(5 x^4 + 3.6 x^{5}) \\ &= e^{3.6 x} \cdot x^4 \cdot (5 + 3.6 x). \end{aligned} \] Evaluated at \(x = 0.7\), the answer is \[ e^{3.6 \cdot 0.7} \cdot 0.7^4 \cdot (5 + 3.6 \cdot 0.7) = 22.440478. \] Thus, rounded to two digits we have \(f'(0.7) = 22.44\).
In addition to the question and the interaction element, there are three buttons providing the following functionality.
Button | Function |
---|---|
✓ | Check the answer and display whether it is correct or not. When clicked, the symbol is toggled and ↰ is displayed, which can be clicked to hide the solution again. |
? | Display the full correct solution explanation. |
↺ | Switch to the next question. |
Inclusion of the solution explanation is optional and the next question button is only displayed if there is more than one random variation of a question. The icons and mouseover text can also be adapted (see below).
To set up a similar standalone file with these two exercises, the exams2webquiz()
interface from the exams2forms
package can be used:
library("exams2forms")
exams2webquiz(c("swisscapital.Rmd", "deriv.Rmd"), n = 3)
To showcase some more exercise types, the following examples from the R/exams package are used: capitals (multiple-choice), function (string/text), fruit (numeric with table and images), lm2 (cloze containing string, multiple-choice, numeric, and single-choice elements as well as an embedded data file).
Which of the following cities are the capital of the corresponding country?
What is the name of the R function for logistic regression?
glm
is the R function for logistic regression. See ?glm
for the corresponding manual page.
Given the following information:
\(+\) | \(+\) | = | \(508\) | |||
\(+\) | \(+\) | = | \(735\) | |||
\(+\) | \(+\) | = | \(203\) |
Compute:
\(+\) | \(+\) | = | \(\text{?}\) |
The information provided can be interpreted as the price for three fruit baskets with different combinations of the three fruits. This corresponds to a system of linear equations where the price of the three fruits is the vector of unknowns \(x\):
\(x_1 =\) | \(x_2 =\) | \(x_3 =\) |
The system of linear equations is then: \[ \begin{aligned} \left( \begin{array}{rrr} 0 & 2 & 1 \\ 1 & 0 & 2 \\ 2 & 1 & 0 \end{array} \right) \cdot \left( \begin{array}{r} x_1 \\ x_2 \\ x_3 \end{array} \right) & = & \left( \begin{array}{r} 508 \\ 735 \\ 203 \end{array} \right) \end{aligned} \] This can be solved using any solution algorithm, e.g., elimination: \[ x_1 = 59, \, x_2 = 85, \, x_3 = 338. \] Based on the three prices for the different fruits it is straightforward to compute the total price of the fourth fruit basket via:
\(+\) | \(+\) | = | ||||
\(x_1\) | \(+\) | \(x_2\) | \(+\) | \(x_3\) | = | |
\(59\) | \(+\) | \(85\) | \(+\) | \(338\) | = | \(482\) |
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that y
increases significantly with x
(at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-0.66428 -0.16666 -0.00005 0.16780 0.53977
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.008474 0.026627 0.318 0.751
x 0.637386 0.046427 13.729 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.266 on 98 degrees of freedom
Multiple R-squared: 0.6579, Adjusted R-squared: 0.6544
F-statistic: 188.5 on 1 and 98 DF, p-value: < 2.2e-16
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
Again, the exams2webquiz()
function can be used to set up a standalone file based on the same exercises:
exams2webquiz(c("capitals.Rmd", "function.Rmd", "fruit.Rmd", "lm2.Rmd"))
To accomplish the functionality demonstrated above, the package provides the following contents:
exams2forms()
: Main workhorse function from the package. Like other exams2xyz()
interfaces this takes a vector or list of exercise files and returns Markdown text, including HTML snippets, than can be included in rmarkdown
or quarto
documents. This includes questions, suitable interactions for the different types of answers (with correct solutions also embedded in the HTML), and optionally full solution explanations.webex.css
, webex.js
: CSS (Cascading Style Sheets) and Javascript files shipped within the package and providing the code necessary for the exercise and quiz display and processing the different types of interactions.webquiz()
: Small wrapper function that creates a knitr::html_document()
but includes the CSS and Javascript files above.exams2webquiz()
: Convenience interface that combines all of the above elements. It sets up a webquiz()
document in which exams2forms()
is used to embed the specified exercises, calls rmarkdown::render()
to process it, and by default displays it in the browser. This is most useful for quickly trying out how R/exams exercises can work in HTML documents.forms_num()
, forms_string()
, forms_schoice()
, forms_mchoice()
: Helper functions for just embedding the user interactions for the different types of exercises. Typically not called directly by the user.While exams2webquiz()
is convenient for quickly setting up an HTML document containing certain exercises, further customizations are typicallly needed for more elaborate documents. To demonstrate how this works the exams2forms
package provides two demo rmarkdown
files which can also be downloaded here:
The first file quiz.Rmd
renders a number of different exercises into a quiz using a single exams2forms()
call, indicating that the results
should be included "asis"
:
---
title: "R/exams quiz"
output: exams2forms::webquiz
---
```{r setup, include = FALSE}
## package and list of various exercises
library("exams2forms")
exm <- list(
c("swisscapital.Rmd", "capitals.Rmd"),
"deriv.Rnw",
"deriv2.Rnw",
"fruit.Rmd",
"boxplots.Rmd",
"ttest.Rmd",
"function.Rmd",
"lm2.Rnw",
"fourfold2.Rmd"
)
```
```{r quiz, echo = FALSE, message = FALSE, results = "asis"}
exams2forms(exm, n = 3)
```
The questions.Rmd
file looks similar but contains different sections, each with a single question set up via exams2forms()
.
Both files can be rendered to HTML via rmarkdown::render()
or by clicking the knit button after opening the files in RStudio etc.
When setting up a more elaborate document or even a full webpage or online book with rmarkdown
or quarto
, then the simple webquiz()
HTML document provided by exams2forms
is probably not sufficient. In this case, it is best to take the CSS and Javascript files from the package or download them here:
The files can then be placed in the same folder as the rmarkdown
or quarto
project. They can also be adapted relatively easily by changing the definitions of colors, icons, text, etc. in the first few lines of each file.
To include the CSS and Javascript in an rmarkown
project, the YAML header should include:
output:
html_document:
css: webex.css
includes:
after_body: webex.js
Similarly, in a quarto
project the YAML header or the _quarto.yml
file should include:
format:
html:
css: webex.css
include-after-body: webex.js
While it is not the primary focus of the exams2forms
package, it is also possible to directly include interaction forms in documents like in the webexercises
package, i.e., without setting up full R/exams exercises. A few simple examples are inlcuded below for numeric, text, single-choice, and multiple-choice interactions (both using drop-down interactions here), respectively.
What is the answer to the ultimate question of life, the universe, and everything?
Which superhero is the secret identity of Bruce Wayne?
Which of the following villains is not an adversary of Batman?
Which of the following characters are romantic interests of Spider-Man?
The corresponding code snippets included in the inline code chunks are:
forms_num(42, width = 10)
forms_string("Batman", width = 20, usecase = FALSE)
forms_schoice(c("Bane", "Riddler", "Thanos", "Poison Ivy"), c(FALSE, FALSE, TRUE, FALSE), display = "dropdown")
forms_mchoice(c("Mary Jane Watson", "Pepper Potts", "Selina Kyle", "Gwen Stacy"), c(TRUE, FALSE, FALSE, TRUE), display = "dropdown")
Guest post by Pia Tscholl & Lisa Schlosser (Universität Innsbruck).
Universities in German-speaking countries face above-average dropout rates in STEM (Science, Technology, Engineering, Mathematics) subjects. These high dropout rates are worrying, as the demand for qualified workers in these fields exceeds the available supply. Especially deficient mathematical knowledge plays a major role during the dropout process in STEM fields. For this reason, the majority of the German-speaking universities offer mathematical remedial or bridging courses at the beginning of STEM degree programs. However, researchers criticize that many remedial courses do not offer audience-appropriate assessments to efficiently diagnose mathematical deficits. Echoing this critique, a team at the University of Innsbruck has developed a mathematical self-assessment, which is implemented via R/exams and made available to participants of the mathematical remedial course via the university’s learning management system OpenOlat. The implementation with R/exams offers the advantage of large test sets with (potentially) randomized tasks which can be easily modified and evaluated in an automated manner. Due to the automated evaluation, the self-assessment can be used in courses with a large number of participants. Moreover, the created assessment can be easily exported to other formats (pdf, moodle, …).
The self-assessment covers the following content areas (with abbreviations based on the German titles):
A separate test set with a certain number of R/exams tasks is created for each content area. All tasks are saved in the same folder and named as follows: “ContentArea_AscendingNumber.Rmd”, e.g. the first task of the content area ZF is named ZF_001.Rmd, the second ZF_002.Rmd and so on. Consistent naming makes it possible to quickly create task lists by content area:
testset_ZF <- list.files(path = "wd/task", pattern = "^ZF_[0-9]+\\.Rmd$")
where wd
is your current working directory or the path where your task folder is located and task
is the name of the folder where the tasks are stored. The argument pattern
is a regular expression starting with ZF
(or another content area) followed by _
, some digits and ending with .Rmd
. Subsequently, testset_ZF
contains the ZF task list to be exported to OpenOlat.
As an example, the code for a specific randomized task is presented below. It asks the participants to compare two randomly selected fractions to check which one is greater or whether they are equal. In addition to the code being shown below (in R/Markdown format) it can also be downloaded in both R/Markdown and R/LaTeX format as ZF_071.Rmd or ZF_071.Rnw, respectively.
```{r, include = FALSE}
## Generate two random fractions based on numbers from 1 to 10
f1 <- sample(1:10, 2, replace = FALSE)
f2 <- sample(1:10, 2, replace = FALSE)
a <- f1[1]
b <- f1[2]
c <- f2[1]
d <- f2[2]
## Save random fractions a/b and c/d as character
## (don't forget escaping \\ within the math mode)
fr1 <- paste0("$\\frac{", a, "}{", b, "}$")
fr2 <- paste0("$\\frac{", c, "}{", d, "}$")
## Possible answers
answers <- c(
paste(fr1, "is greater than", fr2),
paste(fr2, "is greater than", fr1),
"Both fractions are equal"
)
## Correct solution
sol <- c(0, 0, 0)
if(a/b > c/d) {
sol[1] <- 1
} else if(c/d > a/b) {
sol[2] <- 1
} else {
sol[3] <- 1
}
## Explanation
k <- answers[as.logical(sol)]
eq <- c(" > ", " < ", " = ")[as.logical(sol)]
explanation <- paste0(k, " since $", a, " \\cdot ", d, eq, b, " \\cdot ", c, "$.")
```
Question
========
Which of these numbers is greater: `r fr1` or `r fr2`?
```{r, echo = FALSE, results = "asis"}
answerlist(answers, markup = "markdown")
```
Solution
========
`r explanation`
Meta-information
================
exname: Comparing fractions
extype: schoice
exsolution: `r paste(sol, collapse = "")`
One random version of the exercise is shown below as rendered in an OpenOlat test after entering an incorrect answer:
Further meta-information could be added to the exercise via the exextra
tags, if needed. For num
questions, the tag extol
defines the tolerance range for numerical solutions. Exercise templates for different task types (num
, mchoice
, schoice
, cloze
, …) are provided on the R/exams web page at https://www.R-exams.org/templates/.
The elegant aspect of randomized tasks is that students can perform the same test multiple times for the purpose of practice or improved self-assessment. If implemented foresighted like in our example, solutions and explanations are automatically adapted - so there is no additional work for teachers despite many task variations.
The next section will explain how to export such randomized tasks from R/exams to OpenOlat.
As explained in the previous section, consistent file naming makes it easy to create a list of test files, hereinafter referred to as testset_ZF
. The following code exemplifies how such a test set, consisting of several randomized tasks, can be exported from R/exams to OpenOlat.
library("exams") # Load R/exams package
seed <- 6020 # Select seed, so randomization can be repeated
set.seed(seed) # Set seed
export_ZF <- exams2openolat( # Function for exporting tasks to OpenOlat
edir = "wd/tasks", # Directory where task files are stored
file = testset_ZF, # File names of the tasks
n = 20, # Number of randomized versions of the test set
name = paste0("ZF_Testset_seed", seed), # Remember seed also in file name
stitle = "ZF", # Section title
ititle = "Aufgaben", # Item title
solutionswitch = TRUE # Display explanation right after saving the answer
)
Further arguments, such as cutvalue
(threshold for passing), navigation
(disabling switching between tasks), and duration
(maximum processing time) can individualize the test implementation regarding the needs of the course (see this blog post for some practical illustration).
The exams2openolat
function creates a zip folder with the chosen name in the current working directory.
Now, you can insert a new Test element in your OpenOlat course. In the editor mode, it is possible to import the created zip folder to the Test-Konfiguration element. At this point, the work is already done - yay! Each time the test is called up, one of the created randomized test sets is selected by OpenOlat. If desired, further settings can be done using the editor mode in OpenOlat, such as displaying the test results after the test has been completed (Testkonfiguration > Report), a winner’s podium for the three best (anonymous) participants (HighScore > Siegertreppchen), or an automated submission confirmation by E-mail (Email Bestätigung).
It is noteworthy - since this question occurs often among our students - that decimal input for num
questions can be entered in the OpenOlat export with dot or comma.
Once a class or group has finished the self-assessment, various tools provide information on the results. For example, an anonymous ranking showing the achieved number of points of the three best performing participants and a histogram over all point results can provide a first overview on the results.
Moreover, each participant is informed about his/her performance after finishing the assessment for one of the content areas. In particular, the total number of achieved points, the time needed to finish and the number of answered questions is listed. Additionally, each question, the given answer, and the expected answer can be accessed by the participant.
A problem that still arises with num
tasks is the incorporation of fractions. While they can be properly in the tasks using mathematical notation, there is still no convenient way to insert fractions as solutions when completing the test set. As of now, fractions can only be transmitted as decimal numbers, which can be tedious and prone to errors. One possibility would be to implement the numerator and denominator as two num
solution-elements of a cloze
task in R/exams. However, this would have the disadvantage that reduced or expanded variants of the respective fractions are not recognized as solutions. In this case, the most reduced fraction would have to be requested as a solution, which in turn is again prone to errors. Furthermore, if only the numerator or only the denominator is inserted correctly (and the solution is therefore wrong overall), 50% of the points are still awarded.
Additionally, it would be desirable to provide students with the solutions, including implemented explanations for the tasks, after completing the test. It is currently only possible to display explanations immediately after saving the answer, but not as final feedback.
However, we received primarily positive feedback from students and colleagues regarding the implementation of our mathematics self-assessment using R/exams in OpenOlat.
The concept for this Bridging Course Mathematics even won the Austrian teaching award Ars Docendi State Prize 2023. The photo below shows the entire team Tobias Hell, Elisabeth Hell, Pia Tscholl, and Lisa Schlosser (left to right) being presented with the award by Secretary of Education Martin Polaschek (Photo: Martin Lusser/BMBWF).
One advantage of this digitalized self-assessment - in addition to those already mentioned - is the ability to collect data in larger quantities and without further effort. For example, a full PhD position has been realized to evaluate the data collected through the presented self-assessment.
]]>Guest post by Andreas Steinmayr (Universität Innsbruck, Faculty of Economics and Statistics).
Large language models such as ChatGPT have received much attention in recent months. There has been a lot of praise, but also criticism. One of the issues raised was that ChatGPT makes up answers to questions, and does so in a way that the answers appear plausible to readers not familiar with the subject. This might be harmful behavior in many instances, especially in academic settings, but there might also be an application: Creating multiple-choice and single-choice questions for exams.
Multiple-choice and single-choice questions are commonly used in assessments, exams, and surveys. They are popular because they can be quickly and efficiently graded by computers or instructors for many test-takers. However, creating high-quality questions in multiple-choice or single-choice format is challenging and time-consuming for various reasons:
I am currently teaching a class in introductory economics, and we have regular short take-home assignments with single-choice questions using some random variation in the selection of questions and answer alternatives. The assignments are prepared with R/exams and administered in our university’s learning management system (OpenOlat).
Recently, I have started to use ChatGPT to create those questions and found it a useful - yet imperfect - tool. Below, I show how I create a question on the income elasticity of demand with ChatGPT and what further steps I take to turn it into an R/exams question with some shuffling and randomization.
Naturally, the illustrative example I use comes from economics but the same strategy can also be applied for creating knowledge quiz questions in other fields.
In general, I don’t ask ChatGPT to come up with the question but provide the question myself. But that’s pretty much it. What I ask for is a list of correct and incorrect answers.
Note that I ask for more than one correct answer, despite creating a single-choice question, because I can use R/exams’ built-in shuffling to select a different correct answer for different test-takers. Similarly, I ask for more incorrect answers than I need for each test-taker in order to apply some subsampling and shuffling. The same strategy can also be leveraged for multiple-choice questions.
Prompt: Create seven correct and seven incorrect answers for an exam in introductory economics based on the following question: Which of the following statements about income elasticity of demand are correct?
Correct answers:
A) If the income elasticity of demand is positive, it is called a normal good.
B) Income elasticity of demand measures the extent to which demand for a good responds to a change in income.
C) Luxury goods typically have a higher income elasticity of demand than necessities.
D) If the income elasticity of demand is negative, it is called an inferior good.
E) Income elasticity of demand can vary between zero and infinity.
F) A Giffen good is a special case of an inferior good in which demand increases as income decreases.
G) An income elasticity of demand of one means that demand for the good increases proportionally to the change in income.Incorrect answers:
H) Income elasticity of demand measures the extent to which demand for a good responds to a change in price.
I) If the income elasticity of demand is positive, it is called an inferior good.
J) Income elasticity of demand is the same for all goods.
K) If the income elasticity of demand is zero, it is called a luxury good.
L) If the income elasticity of demand is greater than one, it is called a normal good.
M) Income elasticity of demand is the same for all consumers.
N) Income elasticity of demand is independent of the price of the good.
Above, I have color-coded the suggested answers. Green means I am okay with the answer being categorized as correct/incorrect. Yellow means it’s not necessarily incorrect but imprecise or ill-crafted. Red means definitely incorrect.
For answer C), ChatGPT seems to conflate the term “luxury good” in everyday language and economics, where luxury goods are defined by having an income elasticity larger than one.
Answer E) is incorrect, as the income elasticity can be negative. But this will make for a nice answer in the incorrect list.
Answer F) is incorrect because a Giffen good is a good for which demand increases if its price increases.
I decided to ask for more incorrect answers, giving ChatGPT a bit more guidance:
Prompt: Add additional incorrect answer options that include the terms “normal goods” and “Giffen goods.”
Additional incorrect answers:
O) If the income elasticity of demand for normal goods is positive, it is called an inferior good.
P) Giffen goods always have a positive income elasticity of demand.
Q) The income elasticity of demand for Giffen goods cannot be greater than one.
R) If the income elasticity of demand for normal goods is negative, it is called a superior good.
S) If the income elasticity of demand for Giffen goods is negative, demand increases with an increase in income.
T) Giffen goods are luxury goods for which demand decreases as income increases.
U) Normal goods are goods preferred by a specific segment of the population.
I decided to use the two answers highlighted in green and added them to the list.
In the next step, I edited the answers I was not fully happy with and added my own answers to the list:
Correct answers:
Incorrect answers:
Finally, let’s turn this into a dynamic R/exams exercise in both R/Markdown (Rmd) and R/LaTeX (Rnw) format. The first version of the exercise, linked in the table below, simply lists all answer alternatives from Step 3, flags them as correct/incorrect in the exsolution
meta-information, specifies extype
as schoice
for single-choice, and sets exshuffle
to 5 so that one correct and 4 incorrect answer alternatives are randomly selected.
The second version in the table below is virtually identical to the first one, but extype
is now set to mchoice
for multiple-choice. In this case five answer alternatives are chosen randomly with at least one correct and at least one incorrect alternative.
# | Exercise templates | Invert? | Type | Description |
---|---|---|---|---|
1 | income_elasticity1.Rmd income_elasticity1.Rnw |
No | schoice |
Select single correct answer from shuffled subset of five answer alternatives. |
2 | income_elasticity2.Rmd income_elasticity2.Rnw |
No | mchoice |
As in #1 but with one or more correct answer alternatives. |
3 | income_elasticity3.Rmd income_elasticity3.Rnw |
Yes | schoice |
As in #1 but randomly vary between selection of correct or incorrect answer. |
4 | income_elasticity4.Rmd income_elasticity4.Rnw |
Yes | mchoice |
As in #3 but with one or more (in)correct answer alternatives. |
In the third and fourth version of the exercise, I add some further dynamic elements to the question in order to create some more random variation and make cheating harder. Specifically, I randomly vary between the original task (Which of the statements is/are correct?) and the inverted task (Which of statements is/are not correct?). Moreover, some of the answer alternatives are randomly modified (increase vs. decrease; different percentages) without changing the nature of the statement (correct vs. incorrect).
The fourth version again differs from the third only by using the mchoice
(instead of schoice
) type and “are” (instead of “is”) in the question.
For more guidance on using the schoice
and mchoice
exercise types in R/exams see the corresponding YouTube tutorial which starts out from a knowledge quiz question about capitals.
Just like it’s not a good idea for students to rely on ChatGPT to answer exam questions, it’s not a good idea to entirely rely on it to create them. Correct answers might be phrased incorrectly, vice versa, or you might find answers to be ill-crafted.
But what ChatGPT is very useful for is to give you a set of potential answers, especially incorrect ones, that sound plausible. You can then use this list, use only selected answer options, or modify others. This is what I find most useful at the moment, as it can take a lot of time and cognitive energy to come up with high-quality answer lists from an empty slate.
]]>lm3
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Interpretation: Consider various diagnostic plots for the fitted linear regression model. Do you think the assumptions of the Gauss-Markov theorem are fulfilled? What are the consequences?
Code: Please upload your code script that reads the data, fits the regression model, extracts the quantities of interest, and generates the diagnostic plots.
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that x
and y
are not significantly correlated (at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-0.55258 -0.15907 -0.02757 0.15782 0.74504
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.007988 0.024256 -0.329 0.743
x -0.031263 0.045420 -0.688 0.493
Residual standard error: 0.2425 on 98 degrees of freedom
Multiple R-squared: 0.004811, Adjusted R-squared: -0.005344
F-statistic: 0.4738 on 1 and 98 DF, p-value: 0.4929
Interpretation: Considering the visualization of the data along with the diagnostic plots suggests that the assumptions of the Gauss-Markov theorem are reasonably well fulfilled.
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
## diagnostic plots
plot(m)
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Interpretation: Consider various diagnostic plots for the fitted linear regression model. Do you think the assumptions of the Gauss-Markov theorem are fulfilled? What are the consequences?
Code: Please upload your code script that reads the data, fits the regression model, extracts the quantities of interest, and generates the diagnostic plots.
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that y
increases significantly with x
(at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-0.71650 -0.17244 -0.00028 0.17813 0.77574
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.14599 0.02877 -5.075 1.84e-06
x 0.85960 0.04804 17.895 < 2e-16
Residual standard error: 0.287 on 98 degrees of freedom
Multiple R-squared: 0.7657, Adjusted R-squared: 0.7633
F-statistic: 320.2 on 1 and 98 DF, p-value: < 2.2e-16
Interpretation: Considering the visualization of the data along with the diagnostic plots suggests that the true relationship between y
and x
is not linear but quadratic (and hence errors do not have zero expectation).
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
## diagnostic plots
plot(m)
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Interpretation: Consider various diagnostic plots for the fitted linear regression model. Do you think the assumptions of the Gauss-Markov theorem are fulfilled? What are the consequences?
Code: Please upload your code script that reads the data, fits the regression model, extracts the quantities of interest, and generates the diagnostic plots.
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that y
increases significantly with x
(at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-1.17761 -0.11645 -0.02677 0.11443 0.90074
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.01586 0.02625 -0.604 0.547
x 0.63255 0.04412 14.337 <2e-16
Residual standard error: 0.262 on 98 degrees of freedom
Multiple R-squared: 0.6772, Adjusted R-squared: 0.6739
F-statistic: 205.5 on 1 and 98 DF, p-value: < 2.2e-16
Interpretation: Considering the visualization of the data along with the diagnostic plots suggests that the errors are heteroscedastic with increasing variance along with the mean.
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
## diagnostic plots
plot(m)
Demo code:
library("exams")
set.seed(403)
exams2html("lm3.Rmd")
set.seed(403)
exams2pdf("lm3.Rmd")
set.seed(403)
exams2html("lm3.Rnw")
set.seed(403)
exams2pdf("lm3.Rnw")
]]>lm2
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that y
increases significantly with x
(at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-0.50503 -0.17149 -0.01047 0.13726 0.69840
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.005094 0.023993 -0.212 0.832
x 0.558063 0.044927 12.421 <2e-16
Residual standard error: 0.2399 on 98 degrees of freedom
Multiple R-squared: 0.6116, Adjusted R-squared: 0.6076
F-statistic: 154.3 on 1 and 98 DF, p-value: < 2.2e-16
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that y
decreases significantly with x
(at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-0.73798 -0.12940 -0.00055 0.17485 0.71948
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.02682 0.02520 1.064 0.2898
x -0.08318 0.04181 -1.990 0.0494
Residual standard error: 0.2515 on 98 degrees of freedom
Multiple R-squared: 0.03883, Adjusted R-squared: 0.02902
F-statistic: 3.959 on 1 and 98 DF, p-value: 0.04941
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
Theory: Consider a linear regression of y
on x
. It is usually estimated with which estimation technique (three-letter abbreviation)?
This estimator yields the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. Which of the following properties are required for the errors of the linear regression model under these assumptions?
Application: Using the data provided in linreg.csv estimate a linear regression of y
on x
. What are the estimated parameters?
Intercept:
Slope:
In terms of significance at 5% level:
Theory: Linear regression models are typically estimated by ordinary least squares (OLS). The Gauss-Markov theorem establishes certain optimality properties: Namely, if the errors have expectation zero, constant variance (homoscedastic), no autocorrelation and the regressors are exogenous and not linearly dependent, the OLS estimator is the best linear unbiased estimator (BLUE).
Application: The estimated coefficients along with their significances are reported in the summary of the fitted regression model, showing that x
and y
are not significantly correlated (at 5% level).
Call:
lm(formula = y ~ x, data = d)
Residuals:
Min 1Q Median 3Q Max
-0.60377 -0.15149 0.00999 0.15205 0.59555
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.02871 0.02338 -1.228 0.222
x 0.03627 0.03927 0.924 0.358
Residual standard error: 0.2329 on 98 degrees of freedom
Multiple R-squared: 0.008632, Adjusted R-squared: -0.001484
F-statistic: 0.8533 on 1 and 98 DF, p-value: 0.3579
Code: The analysis can be replicated in R using the following code.
## data
d <- read.csv("linreg.csv")
## regression
m <- lm(y ~ x, data = d)
summary(m)
## visualization
plot(y ~ x, data = d)
abline(m)
Demo code:
library("exams")
set.seed(403)
exams2html("lm2.Rmd")
set.seed(403)
exams2pdf("lm2.Rmd")
set.seed(403)
exams2html("lm2.Rnw")
set.seed(403)
exams2pdf("lm2.Rnw")
]]>essayreg2
Using the data provided in regression.csv estimate a linear regression of y
on x1
and x2
. Answer the following questions.
Proportion of variance explained (in percent):
F-statistic:
Characterize in your own words how the response y
depends on the regressors x1
and x2
.
Upload the R script you used to analyze the data.
The presented results describe a semi-logarithmic regression.
Call:
lm(formula = log(y) ~ x1 + x2, data = d)
Residuals:
Min 1Q Median 3Q Max
-2.68802 -0.67816 -0.01803 0.68866 2.35064
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.06802 0.13491 -0.504 0.616
x1 1.37863 0.13351 10.326 9.34e-15
x2 -0.21449 0.13995 -1.533 0.131
Residual standard error: 1.052 on 58 degrees of freedom
Multiple R-squared: 0.6511, Adjusted R-squared: 0.6391
F-statistic: 54.12 on 2 and 58 DF, p-value: 5.472e-14
The mean of the response y
increases with increasing x1
. If x1
increases by 1 unit then a change of y
by about 296.94 percent can be expected. Also, the effect of x1
is significant at the 5 percent level.
Variable x2
has no significant influence on the response at 5 percent level.
The R-squared is 0.6511 and thus 65.11 percent of the variance of the response is explained by the regression.
The F-statistic is 54.12.
Using the data provided in regression.csv estimate a linear regression of y
on x1
and x2
. Answer the following questions.
Proportion of variance explained (in percent):
F-statistic:
Characterize in your own words how the response y
depends on the regressors x1
and x2
.
Upload the R script you used to analyze the data.
The presented results describe a log-log regression.
Call:
lm(formula = log(y) ~ log(x1) + log(x2), data = d)
Residuals:
Min 1Q Median 3Q Max
-1.2455 -0.2693 -0.0071 0.2339 0.9976
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.01096 0.04900 -0.224 0.824
log(x1) 0.96949 0.04432 21.875 <2e-16
log(x2) 0.02591 0.05341 0.485 0.629
Residual standard error: 0.447 on 81 degrees of freedom
Multiple R-squared: 0.8555, Adjusted R-squared: 0.8519
F-statistic: 239.8 on 2 and 81 DF, p-value: < 2.2e-16
The mean of the response y
increases with increasing x1
. If x1
increases by 1 percent then a change of y
by about 0.97 percent can be expected. Also, the effect of x1
is significant at the 5 percent level.
Variable x2
has no significant influence on the response at 5 percent level.
The R-squared is 0.8555 and thus 85.55 percent of the variance of the response is explained by the regression.
The F-statistic is 239.78.
Using the data provided in regression.csv estimate a linear regression of y
on x1
and x2
. Answer the following questions.
Proportion of variance explained (in percent):
F-statistic:
Characterize in your own words how the response y
depends on the regressors x1
and x2
.
Upload the R script you used to analyze the data.
The presented results describe a log-log regression.
Call:
lm(formula = log(y) ~ log(x1) + log(x2), data = d)
Residuals:
Min 1Q Median 3Q Max
-0.90603 -0.25642 -0.01465 0.21672 1.25306
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.07904 0.06266 1.261 0.213
log(x1) -0.82335 0.06078 -13.547 <2e-16
log(x2) -0.03764 0.06424 -0.586 0.560
Residual standard error: 0.4479 on 54 degrees of freedom
Multiple R-squared: 0.7784, Adjusted R-squared: 0.7702
F-statistic: 94.85 on 2 and 54 DF, p-value: < 2.2e-16
The mean of the response y
decreases with increasing x1
. If x1
increases by 1 percent then a change of y
by about -0.82 percent can be expected. Also, the effect of x1
is significant at the 5 percent level.
Variable x2
has no significant influence on the response at 5 percent level.
The R-squared is 0.7784 and thus 77.84 percent of the variance of the response is explained by the regression.
The F-statistic is 94.85.
Demo code:
library("exams")
set.seed(403)
exams2html("essayreg2.Rmd")
set.seed(403)
exams2pdf("essayreg2.Rmd")
set.seed(403)
exams2html("essayreg2.Rnw")
set.seed(403)
exams2pdf("essayreg2.Rnw")
]]>The new version 2.4-0 of the R/exams package has been released on the Comprehensive R Archive Network at https://CRAN.R-project.org/package=exams. This is the first CRAN release after 2.5 years of very active development. Due to the challenges from distance learning during the pandemic a lot of new features were added and enhancements made, especially for the support of various e-learning tools and learning management systems.
Many of changes were prompted by the authors’ own needs (especially in the support of OpenOlat) but also by the many questions on StackOverflow, posts in the R-Forge forum, or tweets and e-mails from users all around the world. In the following the most important features from the update are highlighted. For the full list of changes including technical details and smaller bug fixes, see the NEWS of the package release. More blog posts are planned for introducing some of the new features in more detail, e.g., cloze exercises and evaluation strategies.
Initially, the package relied on Rnw (R/LaTeX) exercises only and optionally supported different encodings for non-ASCII characters, e.g., UTF-8 or latin1 (ISO-8859-1) etc. However, it was cumbersome to correctly set the encoding of input exercises and output file formats.
Given that the majority of users by now seem to rely on Rmd (R/Markdown) exercises and that most output file formats involve using the pandoc
document converter (which requires to use UTF-8) for some aspects, the entire exams
package and all its function now exclusively use UTF-8 as well. All templates etc. have been modified to support UTF-8 out-of-the-box. This greatly facilitates working with UTF-8 as there is no need anymore to set any encoding
argument anywhere.
Support for all other encodings like ISO-8859-* (latin1, latin9, etc.), which had previously been available for Rnw exercises in certain exams2xyz()
interfaces, has been disabled. This is a slight reduction of functionality but will be a great benefit for most users of the package.
Finally, to facilitate working with Rmd exercises and embedded graphics or data files, the packages base64enc
, knitr
, and rmarkdown
are now imported in the package (and not just suggested).
The processing of string
exercises for learning management systems like Moodle, OpenOlat, or other QTI-based systems has been extended. By default, string
exercises are intended for closed-format short-text answers that have to be matched exactly by the participants (e.g., asking for the name of an R function or exact value of an output string etc.). Additionally, open-ended text answers can now be enabled by setting the exstringtype
meta-information to essay
and/or file
. The former requests a text editor for entering an answer while the latter requests a file upload dialogue. The essayreg exercise has been modified to leverage this new meta-information. Such exercises then have to be evaluated by the instructors/examiners, i.e., there is no automatic evaluation by the learning management system.
Similarly, the exclozetype
meta-information now also accepts essay
or file
instead of string
for elements of a cloze
exercise. Currently, the combination of these types with num
or schoice
elements etc. is only possible for QTI-based systems (i.e., OpenOlat in particular) but not for Moodle (whose cloze format does not support open-ended text answers). For illustration, see the new "essayreg2"
and "lm3"
exercise templates.
Several enhancements and extensions have been made in exams2openolat()
(or the underlying exams2qti21()
function, respectively). The most important new features are:
cutvalue
(for passing a test/exam/quiz). By default, this is cutvalue = NULL
(or equivalently cutvalue = NA
) which means that no pass/fail status is computed at all, i.e., only the sum of the points is reported. This is particularly useful if individual online tests during a semester contribute to an overall grade but cannot be passed/failed individually.navigation = "nonlinear"
. This can be switched to "linear"
enforcing that questions in the test must be answered sequentially while the default "nonlinear"
means that participants can switch back and forth between questions.selection = "pool"
that controls how exercises and sections are sampled. By default, the function creates one section for each exercise from which one replication will be selected in the exam. If selection = "exam"
each section contains all questions and one section will be selected for the exam. The "exam"
variant has the advantage that nsamp
can be fully used now and that questions that build on each other can be used in the exam.shufflesections = FALSE
can be set to TRUE
in order to randomly shuffle the order of sections/exercises for each participant. For selection = "pool"
this corresponds to shuffling the sections that contain the pools of exercises. For selection = "exam"
it corresponds to shuffling the exercises within each exam section.cloze_schoice_display = "auto"
that controls the display of schoice
elements in cloze
exercises. By default, radio "buttons"
are used if the answer list appears in its own paragraph and a "dropdown"
menu is used if the answer list appears inline (and has no mathematical markup). Both options can also be enforced explicitly, independently from the answer list appearing in a separate paragraph or inline.stitle
(section title) and ititle
(item title) are now NULL
so that items are simply numbered consecutively (1, …, n) and section titles are omitted. Similarly, the default sdescription
is now empty omitting the section description as it is typically not necessary.The are some additional new arguments that allow further fine control of a test’s configuration. The most important and flexible is the argument config = TRUE
. The logical specification config = TRUE
/FALSE
uses the default configuration or switches off the extra configuration entirely, respectively. Moreover, a list of options like config = list(cancel = TRUE, scoreprogress = TRUE)
can be provided for customizing how OpenOlat renders the QTI 2.1 content, see ?openolat_config
for more details.
Finally. a new argument envir
has been added that is passed on to xweave()
. By setting this to a common environment, e.g., envir = .GlobalEnv
(or some new dedicated environment), it is possible to re-use variables generated in one exercise in previous exercises, e.g., for creating a sequence of variables based on the same data set. Note that this needs to be combined with selection = "exam"
(see above).
The popular exams2moodle()
interface has been improved, in particular for more flexible cloze
exercises:
mchoice
elements in cloze
questions are now properly supported. By default they are shown as MULTIRESPONSE
checkboxes and employ Moodle’s default evaluation strategy where each incorrect box eliminates one correct box. A different evaluation strategy can, in principle, be chosen but Moodle might not process all negative points correctly.schoice
elements in cloze
questions are still rendered as MULTICHOICE
drop-down menus by default unless they contain math markup. As this is not rendered by Moodle in drop-down menus, a MULTICHOICE_V
column of radio buttons is used in this case.mchoice
and schoice
elements in cloze
questions, the are now both cloze_mchoice_display
and cloze_schoice_display
arguments. This is not fully backward compatible because in previous versions cloze_mchoice_display
was also used to customize schoice
elements. Now a warning is issued in this case.num
and/or string
sub-items (e.g., when presented in a table), arguments numwidth
and stringwidth
have been added to make_question_moodle()
. Alternatively, they can also be specified through exextra
tags in each exercise. See the fourfold2 exercise for an example and ?make_question_moodle
for more details.Moreover, there is a new experimental function moodle2exams()
that can take a Moodle XML quiz file with numeric, multichoice, shortanswer, and essay exercises and convert them to R/exams exercise files, either in Rmd or Rnw format. If the text formatting is more advanced (e.g., containing mathematical notation or tables etc.) the function might not lead to fully satisfactory results but still provide a useful starting point for subsequent manual editing.
Various new learning management systems or quiz engines can now be interfaced using R/exams. The work on exams2testvision()
and exams2grasple()
was financially supported by the Dutch Ministry of Education, Culture and Science (Project code OL20-06), and the University of Amsterdam.
exams2ilias()
for the open-source ILIAS learning management system. This is essentially a convenience wrapper to exams2qti12()
, tweaking a few defaults and employing a somewhat modified XML template. Not all question types are supported, though, mostly string questions with open-ended answers and multiple-choice and single-choice questions. Numeric and cloze questions are not supported, yet.exams2testvision()
for the Dutch testing platform TestVision. It is essentially a fork of exams2qti21()
that incorporates TestVision’s own strict implementation of QTI 2.1. See the online tutorial on how to upload the zip output from exams2testvision()
into the system by selecting it in the import menu and then moving the imported material to the appropriate directories.testvision2exams()
to convert TestVision’s QTI 2.1 questions to R/exams exercise files, either in Rmd or Rnw format. The supported TestVision question types are ‘invul (numeriek)’, ‘een-uit-meer’, ‘meer-uit-meer’, and ‘open’ which are converted to num
, schoice
, mchoice
, and string
, respectively.exams2grasple()
for Grasple, a Dutch practice platform for mathematics and statistics education. It supports num
and schoice
questions which are exported to a zip file containing Grasple’s JSON format. Note that currently importing cannot be done by users themselves; it requires a request for manual import by a Grasple team member.exams2particify()
that can export exercises to a comma-separated values (CSV) format for import in the audience response system Particify, the successor to ARSnova. In particular, single-choice and multiple-choice exercises are fully supported while num and string question are converted to open-ended text questions.exams2kahoot()
that can export sufficiently simple single-choice and multiple-choice exercises to an Excel sheet via openxlsx::write.xlsx()
that can be imported by the game-based learning platform Kahoot! (suggested by Rushad Faridi). Exercises are converted to plain text and questions must not exceed 120 characters, answers must not exceed 75 characters.The exams2nops() function is the R/exams interface for generating exams of single-choice and multiple-choice questions, that can be scanned and evaluated automatically.
Various improvemens have been made in nops_scan()
, especially for scanning the boxes pertaining to the student registration ID. Rather than reading a very small area around each box and just shaving off its borders, a larger area is read now and then shaved iteratively. Hence, it is also easily possible to further increase the size
of the area which may sometimes lead to improved scanning results.
In the exams2nops()
function itself, the handling of reglength < 7
has been improved. Internally, reglength = 7
is still enforced (and thus necessary in the registration CSV file) but the initial IDs are fixed to “0” in the exam sheet and corresponding boxes ticked already.
Finally, language support has been extended by two new languages: Czech (cz
, by Jindřich Marek) and Galician (gl
, by Marta Sestelo & Nora M. Villanueva).
In the exams2blackboard()
interface the new argument mathjax = NULL
has been added that optionally embeds the MathJax <script>
in each of the exercises. This allows to render mathematical content by the MathJax plugin for all modern browsers. Without MathJax, the browser itself has to do the rendering directly (which is supported by Firefox and Safari but not by Chrome for example). The default is FALSE
unless converter = "pandoc-mathjax"
is used. But also for the default converters (producing MathML output) mathjax = TRUE
can be used. (Suggested and tested by Sean Quallen and Gabriele Cantaluppi.)
Moreover, the following improvements may be useful for Blackboard users: Rendering verbatim code chunks can be fixed from <pre>
to <code>
tags if fix_pre = TRUE
(default) which is necessary in classical Blackboard systems. Points can be specified through expoints
.
knitr::knit()
errors in the R code will stop the evaluation now by default. This was always the default behavior for Rnw exercises (i.e., when processed with engine = "sweave"
) but now is also the default for Rmd exercises and for Rnw exercises via engine = "knitr"
. In exercises processed via knitr::knit()
it is possible to carry on with code evaluation after errors (the default in knitr
) by setting the chunk option error = TRUE
. Similarly, the default handling of warnings has been set to warning = FALSE
so that warnings are reported on the console rather than in the weaved/knitted exercises.texengine = "pdflatex"
to exams2pdf()
which is passed on to tinytex::latexmk(..., engine = texengine)
. Provided that tinytex
support is installed, this option can also be set to texengine = "xelatex"
or "lualatex"
for example.exams2html()
, exams2pdf()
, and exams2pandoc()
gained an argument exshuffle = NULL
which can be modified to overrule the exshuffle
setting within an exercise (if any). For example, setting exams2xyz(..., exshuffle = FALSE)
retains the full answerlist without any permutation.xexams()
a new argument rds = FALSE
was added that indicates whether the list returned by the function should also be saved as an RDS data file. If rds = TRUE
the file name metainfo.rds
is used but alternatively rds
can also be specified as a character file name to be used. The new rds
argument is leveraged in exams2blackboard()
, exams2moodle()
, exams2pdf()
, exams2html()
, exams2qti12()
, and exams2qti21()
, as well as other interfaces built on top of these.The package exams2learnr
mainly provides the function of the same name, exams2learnr()
. It is an interface for embedding exercises written with R/exams in tutorials or quizzes written with the learnr package.
The goal of learnr
is the creation of interactive tutorials in a shiny application written with R/Markdown. The tutorials can contain individual questions (exercises) or quizzes (sets of exercises) along with text, graphics, and other elements you can embed in R/Markdown. Usually, these tutorials can then be used for self-paced learning rather than (summative) assessment.
The exams2learnr()
interface leverages learnr
’s capabilities for questions/quizzes to dynamically insert (random variations) from R/exams exercise templates, either written in Rmd (R/Markdown) or Rnw (R/LaTeX) format. Currently, there is support for the following question types:
Description | R/exams | learnr |
---|---|---|
Single-choice | schoice | learnr_radio |
Multiple-choice | mchoice | learnr_checkbox |
Numeric | num | learnr_numeric |
Text | string | learnr_text |
Thus, there is no support, yet, for cloze
questions that can combine all of the elements above.
For quickly trying out how a certain R/exams exercise is rendered in a learnr
tutorial, there is the convenience function run_quiz()
. This sets up a tutorial embedding R/exams exercises in a temporary directory and directly runs the tutorial in a shiny
app. For illustration, the code below creates a quiz containing one random version of each of these exercises that are shipped with the R/exams package: capitals (multiple-choice), fruit (numeric), function (string/text). A screenshot is included below the code.
library("exams2learnr")
run_quiz(c("capitals.Rmd", "fruit.Rmd", "function.Rmd"))
Thus, the run_quiz()
function is nice for getting a first feeling for what is possible with the exams2learnr
package and for the look & feel of the resulting shiny
apps. For full customization, however, a dedicated Rmd tutorial should be set up. This can then leverage the main exams2learnr()
function to explicitly include questions/quizzes, potentially along with further elements and/or customizations.
The main function exams2learnr()
can take one or more exercise templates, either as a vector or as a list, just as for other exams2xyz()
interfaces. By default, this creates a single random replication from (each of) the exercise(s), converts the text to HTML, and wraps it into a suitable learnr
object. For a single exercise a tutorial_question
is returned and for a vector/list of exercises a tutorial_quiz
. These defaults can be modified and further arguments can be passed on to the underlying learnr
function calls to question()
and quiz()
, respectively. This allows for customizing the appearance or controlling whether multiple attempts are allowed, or whether hints/solutions are shown, etc.
As a first example, we apply exams2learnr()
to the string/text function exercise, readily provided within the R/exams package. The result is represented as a learnr_text
object, inheriting from tutorial_question
:
qn <- exams2learnr("function.Rmd")
class(qn)
## [1] "learnr_text" "tutorial_question"
print(qn)
## Question: "What is the name of the R function for extracting the fitted
## log-likelihood from a fitted (generalized) linear model object?"
## type: "learnr_text"
## allow_retry: FALSE
## random_answer_order: FALSE
## answers:
## ✔: "logLik"
## messages:
## correct: "Correct!"
## incorrect: "Incorrect"
## message: "<code>logLik</code> is the R function for extracting the
## fitted log-likelihood from a fitted (generalized) linear model object.
## See <code>?logLik</code> for the corresponding manual page."
## Options:
## trim: TRUE
Subsequently, we set up a quiz with the single-choice swisscapital exercise and the multiple-choice switzerland exercise, represented as learnr_radio
and learnr_checkbox
objects, respectively, contained in a tutorial_quiz
.
qz <- exams2learnr(c("swisscapital.Rmd", "switzerland.Rmd"))
class(qz)
## [1] "tutorial_quiz"
print(qz)
## Quiz: "Quiz"
##
## Question: "What is the seat of the federal authorities in Switzerland
## (i.e., the de facto capital)?"
## type: "learnr_radio"
## allow_retry: FALSE
## random_answer_order: FALSE
## answers:
## X: "Zurich"
## X: "Basel"
## ✔: "Bern"
## X: "Vaduz"
## X: "Geneva"
## messages:
## correct: "Correct!"
## incorrect: "Incorrect"
## message: "There is no de jure capital but the de facto capital and
## seat of the federal authorities is Bern.</p>
##
## <ul>
## <li>False</li>
## <li>False</li>
## <li>True</li>
## <li>False</li>
## <li>False</li>
## </ul>
## "
##
## Question: "Which of the following statements about Switzerland is correct?"
## type: "learnr_checkbox"
## allow_retry: FALSE
## random_answer_order: FALSE
## answers:
## X: "The currency in Switzerland is the Euro."
## ✔: "Italian is an official language in Switzerland."
## ✔: "The Swiss national holiday is August 1."
## X: "Switzerland is part of the European Union (EU)."
## X: "Zurich is the capital of Switzerland."
## messages:
## correct: "Correct!"
## incorrect: "Incorrect"
## message: "<ul>
## <li>False. The currency is the Swiss Franc (CHF).</li>
## <li>True. The official languages are: German, French, Italian, Romansh.</li>
## <li>True. The establishment of the Swiss Confederation is traditionally dated to August 1, 1291.</li>
## <li>False. Switzerland is part of the Schengen Area but not the EU.</li>
## <li>False. There is no de jure capital but the de facto capital of Switzerland is Bern.</li>
## </ul>
## "
If more than n = 1
random replications are produced then a standard list is returned - because learnr
currently does not provide a compound object that can capture a question bank with multiple replications. The list has one element per random replication, each of which is a list of tutorial_question
objects.
The format of the return value can also be controlled by the output
argument if the desired format is not selected automatically.
For setting up a learnr
tutorial with a quiz based on R/exams exercises the building blocks discussed above can be used. In the YAML header the output
should be learnr::tutorial
and the runtime
should to be shiny_prerendered
. Then the quiz (or “exam”) can be set up as a list of exercise templates, as for other exams2xyz()
interfaces as well. The entire quiz is then prepared in a single exams2learnr()
call. For illustration consider:
--- title: "R/exams quiz" output: learnr::tutorial runtime: shiny_prerendered --- ```{r setup, include = FALSE} ## package and list of various exercises library("exams2learnr") exm <- list( c("swisscapital.Rmd", "capitals.Rmd"), "deriv.Rnw", "deriv2.Rnw", "fruit.Rmd", "boxplots.Rmd", "ttest.Rmd", "function.Rmd" ) ``` ```{r rexams_quiz, echo = FALSE, message = FALSE} exams2learnr(exm, caption = "Please solve the following exercises") ```
The demo file above is actually shipped as a supplementary file learnr_quiz.Rmd within the exams2learnr
package. Hence, it can be run()
as in the following command. The screenshot below shows (one random replication of) the first exercises. The concrete exercises will differ in every new run()
of the tutorial.
rmarkdown::run(system.file("learnr", "learnr_quiz.Rmd", package = "exams2learnr"))
To finetune the appearance of the R/exams exercises within the learnr
tutorial, e.g., for putting different exercises into different sections, it may be necessary to include each question/exercise individually (rather than an entire quiz). For illustration consider the following Rmd document that is provided as supplementary file learnr_questions.Rmd:
--- title: "R/exams questions" output: learnr::tutorial runtime: shiny_prerendered --- ```{r setup, include = FALSE} library("exams2learnr") ``` ## Knowledge quiz (single-choice) ```{r schoice1, echo = FALSE, message = FALSE} exams2learnr("swisscapital.Rmd", allow_retry = TRUE, incorrect = "Incorrect, try again.") ``` ## Knowledge quiz (multiple-choice) ```{r mchoice2, echo = FALSE, message = FALSE} exams2learnr("capitals.Rmd") ``` ## Arithmetic (numeric) ```{r num3, echo = FALSE, message = FALSE} exams2learnr("deriv.Rnw", allow_retry = TRUE) ``` ## Arithmetic (single-choice) ```{r schoice4, echo = FALSE, message = FALSE} exams2learnr("deriv2.Rnw") ``` ## Multiple-choice with graphic ```{r mchoice5, echo = FALSE, message = FALSE} exams2learnr("boxplots.Rmd") ``` ## Multiple-choice with R output ```{r mchoice6, echo = FALSE, message = FALSE} exams2learnr("ttest.Rmd") ``` ## String question ```{r mchoice7, echo = FALSE, message = FALSE} exams2learnr("function.Rmd", allow_retry = TRUE) ```
This tutorial also shows how to use some of learnr
’s customization options such as allow_retry
or incorrect
. To run()
this tutorial the command below can be used. As in the previous section, the concrete exercise variations will differ in every new run()
of the tutorial.
rmarkdown::run(system.file("learnr", "learnr_questions.Rmd", package = "exams2learnr"))
]]>Due to the requirement to move to remote assessment during the global Covid-19 pandemic, Andrew Zammit-Mangion (University of Wollongong, School of Mathematics and Applied Statistics) created introductory materials to help his colleagues to get started with R/exams and turned these into a full online course. The video tutorials in the course show in simple steps how to:
The course is hosted on Thinkific and is completely free, just requires registration or a login via a Google account or similar:
The materials are geared somewhat towards assessments containing mathematical exercises but are also easily accessible for users with other backgrounds. Due to the emphasis on mathematical content, the examples in the course are based on Rnw (Sweave) exercises allowing for seamless LaTeX integration. Converting the exercises to Rmd (R/Markdown) format is also possible as shown in the First steps tutorial but this is not covered in the online course.
]]>LaTeX is a software system for document preparation, see the corresponding Wikipedia page for an overview. The system encompasses various elements that are important to distinguish:
In general LaTeX it is very useful for professional typesetting, especially of scientific documents, and is the most widely-used markup language for mathematical notation. However, the complexity of the LaTeX markup language can also be confusing, especially for newcomers, and hence it can be easier to use other markup languages such as Markdown for many documents. But even when using Markdown as the main markup for a document, LaTeX may still be involved in producing an output document: either only for mathematical notation (e.g., in HTML output) and/or for rendering PDF output. In short, due to its power and flexibility LaTeX is used either explicitly or implicitly in the preparation of a lot of output documents.
Consequently, R/exams leverages LaTeX, mainly for three purposes:
exams2pdf()
for flexible, customizable PDF output and exams2nops()
for standardized PDF output that can be automatically scanned and evaluated.If you use R/exams for rendering exercises without mathematical notation into online exams then you likely do not need LaTeX at all (and can stop reading here).
If you only use mathematical notation but still only generate online HTML-based exams, especially for e-learning materials, then you need a bit of the markup language but do not actually need to install a TeX engine as part of a TeX distribution.
However, when you produce PDF output, be it from Rnw or Rmd exercises, then you need to install a TeX distribution with suitable TeX engines. In case of R/Markdown exercises in Rmd format, R/exams converts the Markdown markup to LaTeX behind the scenes (using the powerful pandoc document converter), before using a TeX engine to render the PDF output. The following comments should help you choose an appropriate way to install and interface a TeX distribution when working with R/exams.
The TeX distribution providing the TeX engine(s) can either be installed “as usual” on the system, e.g., by downloading MikTeX for Windows or installing TeX Live from a standard repository under Linux or OS X. Alternatively, the R package tinytex offers the possibility for installing a lightweight TeX distribution called “TinyTeX”, based on TeX Live. The latter option is particularly attractive for R/exams users that do not actually use LaTeX for other projects. This is due to a couple of reasons: Everything is handled through R; no special super user rights are needed for the installation; and the installation is minimal with additional style files etc. being installed automatically only if needed. To install the R package tinytex
you just need
install.packages("tinytex")
or alternatively use the graphical user interface in RStudio etc. To install the TeX distribution “TinyTeX” you just need
tinytex::install_tinytex()
Finally, even when you have installed a TeX distribution (other than TinyTeX) on your system (e.g., TeX Live or MikTeX), the R package tinytex
may (or may not) still be useful for producing PDF exams. Three cases need to be distinguished.
System LaTeX via tools::texi2dvi()
When: The R package tinytex
is not installed or when setting options(exams_tex = "tools")
.
Why: For R/exams users that also use LaTeX for other projects and do not need any further assistance and hence do not need to introduce another dependency in their code.
System LaTeX via tinytex::latexmk()
When: The R package tinytex
is installed but not the TinyTeX distribution (and the exams_tex
option is not set to "tools"
).
Why: This tries to resolve some dependencies automatically (e.g., additional style files). Also, it not only supports the "pdflatex"
engine but also "xelatex"
or "lualatex"
etc., which may be useful/necessary for handling certain fonts.
TinyTeX via tinytex::latexmk()
When: The R package tinytex
is installed and tinytex::install_tinytex()
was used to install the TinyTeX distribution (and the exams_tex
option is not set to "tools"
).
Why: For users that do not use LaTeX outside of R as it is more lightweight and very easy to install/maintain.