Service incident - 2nd of March - Incorrect guess score for multiple response questions where 'start with zero points' is disabled

All times in this document are recorded in UTC+1 (CET).

Incident summary

On March 2nd, during internal testing, an issue was identified in the guess score calculation for multiple response questions when the 'start with zero points' option was disabled. In this configuration, the system incorrectly used the maximum possible score as the baseline for unanswered questions when calculating the average guess score. This resulted in an inflated guess score, which could lead to scores that were incorrectly adjusted for guessing and potentially impacted final marks.

A fix was developed, tested, reviewed, and deployed by March 3rd at 13:02. An impact analysis identified 2 affected assignments and 38 impacted results. All affected schools were informed on March 4th. The issue has been fully resolved, and the guess score calculation now correctly excludes the full starting score when no answer is selected.

Lead-up

During internal testing on the morning of March 2nd, an issue was identified in the guess score calculation for multiple response questions when the option “Start with zero points” was disabled.

In this configuration, the calculated guess score was higher than expected.

Fault

When the option “Start with zero points” is disabled, the system incorrectly used the maximum possible score as the baseline in the guess score calculation. Since the guess score is equal to the average achieved points, every permutation of answer alternatives is checked for its point value, after which the average can be determined.

The issue originated from the inclusion of the permutation in which no alternatives are selected. In reality, this permutation results in an unanswered question, which receives a score of 0. Instead, the maximum achievable number of points was used.

As a result, the average guess score was calculated using the maximum score instead of zero for unanswered submissions. This led to an inflated guess score and, consequently, a higher total score calculation than intended.

Example

Configuration:

• Question type: Multiple response

• 3 answer alternatives

• Maximum score: 3 points

• “Start with zero points” option disabled

• Scoring method: Custom

• Each selected alternative awards -1 point

All possible answer combinations and their correct scores:

• No answer: 0

• A: 2

• B: 2

• C: 2

• AB: 1

• AC: 1

• BC: 1

• ABC: 0

Correct average guess score calculation:
(0 + 2 + 2 + 2 + 1 + 1 + 1 + 0) / 7 = 1.29

Current incorrect behaviour:
(3 + 2 + 2 + 2 + 1 + 1 + 1 + 0) / 7 = 1.71

The example shows that the system incorrectly uses the full starting score in the guess score calculation when no answer is selected because the question does not start at zero points.

Impact

The impact of this incident affected the guess score calculation, which affected the obtained points and therefore potentially the final marks. An impact analysis was performed on assignments with the guess correction option enabled containing multiple response questions where the option 'start with zero points' was disabled.

The investigation resulted in 2 affected assignments, with 38 impacted results. 

Impacted schools have been informed.

Detection

The issue was discovered during internal testing on the morning of March 2nd. 

Response

The fix for this issue was directly picked up with high priority. It was completed (written, tested, reviewed, and deployed) on March 3rd at 13:02. 

Recovery

The problem was resolved by identifying the problematic code and rewriting the affected logic. Now, the permutation of answering none of the checkboxes is no longer included in the total points that can be earned, resulting in a more accurate representation of how the platform actually handles this situation.

Timeline

2nd of March

• Morning - Technical team discovers the issue of an incorrect guess score if the option ‘Start with zero points’ is disabled in a multiple response question. 
• 9:45 - Technical team informs support about the issue.

3rd of March

• 13:02 - Fix deploys to production. 

4th of March

• 09:59 - Technical team shares the final impact with support. 
• 13:39 - All affected schools are informed. 

Reflection

This issue was caused by an oversight when guess correction was introduced in June 2022. Though the division of the number of alternatives did not consider the permutation where no answer was selected, the amount of points was still considered.

The issue existed for an extended period of time due to the specific conditions required to cause a change in guess score. The difference in guess score was only present when the start with zero setting was turned off and the average score for all permutations were positive. 

The issue was found during a regression test performed by the Technical team, in which parts of the platform were being validated for correctness. This issue was picked up and investigated after suspicion arose during one of the tests. Further investigation concluded that this issue existed.

Once the issue was discovered, a fix was promptly developed and deployed on March 3rd. The initial incident, impact analysis and communication to affected schools were also handled effectively.