Service Incident - 17th of April 2024 - Fixed positions were ignored during form generation, but used during grading when shuffle choices was enabled, leading to incorrect automatic grading

All times in the document are recorded in UTC+2 (CEST).

Summary

On the 15th of April, 2024, at 16:41, the support team received a ticket where the sender stated that automatic grading had been done incorrectly for some exercises. The support team verified the issue and forwarded it to the technical team on the 17th of April, 2024, at 10:37. The technical team deployed hotfixes to the production environment on the 17th of April, at 13:30 and at 21:50. 

Lead-up

The support team received a ticket on the 15th of April, 2024, at 16:41. In this ticket, the sender reported that  the automatic grading of an assignment was wrong. Based on the information in this ticket, the ticket got categorised with a lower priority than it turned out to be in the end. On the 17th of April, 2024, at 10:37, the issue was verified and forwarded to the technical team, who then started their investigation. 

Fault

During the investigation, it was discovered that this issue was introduced with the release on 30th of March, 2024. A change was made to ensure that fixed positions were used even when shuffling was not enabled. This had the unintended consequence that during the generation of written assignments, questions with fixed positions had their positions ignored when shuffling was enabled.

Impact

The impacted results have their fixed-position multiple choice questions graded using incorrect positions. In most cases, this led to incorrect marking.

An impact analysis was performed by the technical team in order to find the results that were affected. This analysis was scoped to assignments which had forms generated between the 30th of March, 2024, and the 17th of April, 2024, as these were the dates from when the change was released until the hotfix was deployed. The investigation resulted in six affected assignments. The schools to which these six assignments belong to have been notified.

Detection

The findings of the user who sent the ticket were verified by the support team, after which they were shared with the technical team, who diagnosed the issue. 

Response

The support team forwarded the ticket to the technical team on the 17th of April, 2024, at 10:37. The technical team confirmed the issue at 11:09 and began to analyse the impact and develop a hotfix.

Recovery

A hotfix was deployed to the production environment on the 17th of April, 2024, at 13:30. This fix caused assignments to be generated using fixed positions when appropriate. An additional hotfix was deployed at 21:50 on the same day, so that all assignments generated between the 30th of March, 2024, and the 17th of April, 2024, would either a) be processed correctly if they were uploaded after this fix, or b) be able to be reprocessed correctly if they had already been uploaded before this fix.

Timeline 

15th of April 2024

• 16:41 - Support team receives a ticket where the user states that the automatic grading of the scans seems wrong. The user mentions that students crossed the right answer, but was registered as crossing an incorrect answer.

17th of April 2024

• 10:37 - Support team forwards the issue to the technical team.
• 10:57 - Support team informs the user that the issue has been forwarded and is being handled with high priority.
• 11:09 - Technical team confirms the issue and diagnoses the problem.
• 13:30 - Technical team deploys a hotfix to prevent the issue for newly-generated assignments.
• 21:50 - Technical team deploys a hotfix for assignments generated between the 30th of March, 2024 and the 17th of April, 2024.

Reflection

The steps taken in the handling of this incident were followed in procedural order. We have manually tested the hotfix and already successfully run score reprocessing for affected assignments which were not yet fully graded. We have also created tasks to expand the test coverage of our internal tests regarding fixed positions and grading to account for more edge cases and prevent future breaking changes.

The initial ticket the support team received about the issue was mistakenly categorised with a lower priority and therefore more time had passed before the issue was verified and forwarded to the technical team. We will look at our prioritisation procedures to reduce the chance of mis-categorising in the future.