Service Incident - 29th of March 2024 - Unapplied guess correction without at least one multiple choice question defined within the assignment

All times in the document are recorded in UTC+1 (CEST).

Incident Summary

On the 28th of March 2024, at 13:52, the support team received a ticket where the sender stated that the guess correction was not being applied for fill-in questions where the option “show answer options” was enabled. The support team reproduced the issue and forwarded it to the technical team at 14:51. The technical team deployed a hotfix to the production environment on 29th of March, at 17:25. 

Lead-up

The support team received a ticket on the 28th of March 2024, at 13:52. In this ticket, the sender reported that the guess correction was not being applied for fill-in questions where the option “show answer options” was enabled.

Fault

During the investigation, it was discovered that this issue was introduced on September 17th, 2023. On this date, a change was released where improvements were made to the internal codebase. These improvements were aimed to improve the readability and usage of the points calculation. This change contained an oversight where the guess correction would not be applied unless the assignment contained at least one multiple-choice question with a single answer alternative.

Impact

An impact analysis was performed by the technical team in order to find the results that were affected. This analysis was scoped to assignments that were created between the 17th of September 2023 and the 29th of March 2024, as these were the dates from when the change was released until the investigation started. The results from this analysis are as follows:

• A total of 12 assignments were found to be affected among 6 schools. 

The impacted results have their grades calculated without any guess correction, causing the impacted results to have a higher grade than they should have.

Detection

The findings of the user who sent the ticket were reproduced by the support team, after which they were shared with the technical team who verified the issue. 

Response

The support team forwarded the ticket to the technical team on the 28th of March, 2024, at 14:51 alongside another reproduced instance of the same issue. The technical team confirmed the issue at 16:00 and began to analyse the impact and develop a hotfix.

Recovery

A hotfix was deployed to the production environment on the 29th of March, 2024 at 17:25. With this fix a user with access to the assignment settings can retrigger the mark calculation, by updating the grading formula and then changing it back.

Timeline

28th of March 2024

• 13:52 - Support team receives a ticket where the user states that the guess correction was not being applied for fill-in questions where the option “show answer options” was enabled.
• 14:51 - Support team forwards ticket to the technical team with a reproduction of the issue.
• 14:54 - Technical team asks Support team to verify whether other question types are impacted while the issue is being investigated further.
• 15:11 - Support team informs the technical team that other closed question types are impacted as well.
• 15:51 - Support team informs the user that the issue has been forwarded and is being handled with high urgency.
• 16:00 - The technical team discovered the root cause of the issue.
• 16:03 - The technical team started development of the hotfix
• 16:30 - The support team informs the user with the impact within their institution. 

29th of March 2024

• 12:10 - The hotfix is set to be reviewed by the other engineers. 
• 13:31 - Support team asks the technical team to take a look at the question asked by the user.
• 17:25 - The hotfix is deployed to the production environment.
  
  

Reflection

The steps taken in the handling of this incident were followed in procedural order. We have included several internal tests alongside the hotfix which will be run in our continuous integration process to ensure that issues with future changes are immediately discovered. Alongside these tests we have created several tasks to expand the test coverage of our internal tests regarding guess correction to account for more edge cases and prevent future breaking changes.