Grading Solutions

Custom algorithms for grading and feedback of submitted R and Python code

Interactive exercises can be dynamically graded by defining a custom grading algorithm for each exercise. Instructors can use specialised testing logic and return specific feedback for common mistakes.

Adding grading to an exercise

Suppose we have the following exercise, asking a learner to calculate the average of the first 10 integers.

R
Python

Calculate the average of all of the integers from 1 to 10.

```{webr}
#| exercise: ex_1_r
______(1:10)
```

Calculate the average of all of the integers from 1 to 10.

```{pyodide}
#| exercise: ex_1_py
n = range(1, 11)
______
```

We hope the learner will replace the ______ with the correct code to calculate the average. To grade an exercise, associate checking code with the exercise using a webr block with the check: true cell option set.

A simple version of exercise grading might check that the student’s result is equal to mean(1:10). If it is, the grading code returns a passing grade, otherwise a failing grade is returned.

R
Python

```{webr}
#| exercise: ex_1_r
#| check: true
if (identical(.result, mean(1:10))) {
  list(correct = TRUE, message = "Nice work!")
} else {
  list(correct = FALSE, message = "That's incorrect, sorry.")
}
```

```{pyodide}
#| exercise: ex_1_py
#| check: true
n = range(1, 11)
mean = sum(n)/len(n)
feedback = None
if (result == mean):
  feedback = { "correct": True, "message": "Nice work!" }
else:
  feedback = { "correct": False, "message": "That's incorrect, sorry." }
feedback
```

Calculate the average of all of the integers from 1 to 10.

Return feedback

Feedback from grading algorithms should be returned in the form of an R list, or Python dictionary, with the following properties:

Property	Type	Description
`correct`	boolean	Is the student’s code correct?
`message`	string	Feedback text to show to the student. Can be a plain string or HTML.
`type`	string	(Optional) Feedback presentation style. Can be “success”, “info”, “warning”, or “error”.

Feedback formatted in this way will be displayed as a styled HTML alert.

Note

Returning and displaying multiple feedback elements with “append”, “prepend”, or “replace” functionality is planned for a future release of quarto-live.

Checking environment

A selection of variables are made available for use by grading algorithms in the checking environment. These variable names have been chosen so as to match closely to the existing {learnr} and {gradethis} grading systems.

Warning

Under Python, there is no leading . character in the variable names. For example, the result of evaluating the learner’s code is given in the variable named result.

Variable	Description
`.check_code`	Code provided for custom checking algorithm.
`.engine`	The execution language, `"r"` or `"python"`.
`.envir_prep`	The environment before the execution of learner code.
`.envir_result`	The environment after the execution of learner code.
`.evaluate_result`	Full evaluation state of evaluating the learner code.
`.label`	The exercise label.
`.last_value`	The last value of the evaluated learner code.
`.result`	The last value of the evaluated learner code.
`.solution_code`	Solution code, if provided by the author.
`.solution`	Result of evaluating the solution code, if provided by the author.
`.stage`	The current checking stage (currently only `check`).
`.user_code`	Exercise code submitted by the user.

More grading examples

Further examples of using the full evaluation state provided by the checking environment with grading algorithms are shown below.

Find any result in learner output

Source

Write R code that returns 2468 somewhere, even invisibly:

```{webr}
#| exercise: example_2
123
invisible(2468)
456
```

```{webr}
#| exercise: example_2
#| check: true
results <- Filter(\(x) inherits(x, "result"), .evaluate_result)
if(is.null(Find(\(x) x$value == 2468, results))) {
  list(correct = FALSE, message = "Incorrect, sorry.")
} else {
  list(correct = TRUE, message = "Correct!")
}
```

Write R code that returns 2468 somewhere, even invisibly:

Feedback for a specific learner error

Source

```{webr}
#| exercise: example_3
123 + "456"
```

```{webr}
#| exercise: example_3
#| check: true
errors <- Filter(\(x) inherits(x, "error"), .evaluate_result)
this_error <- Filter(\(x) x$message == "non-numeric argument to binary operator", errors)
if (length(this_error) > 0) {
  list(
    correct = FALSE,
    type = "info",
    message = "Be careful! In R you cannot add a number and a character string!"
  )
}
```