Service Incident - 18th of Feb 2025 - Incorrect reviewed state for automatically graded questions that reward 0 points for bubble sheet and written assignments

All times in the document are recorded in UTC+1 (CET).

Summary
On February 12th, 2025, a user reported that a bubble sheet assignment was incorrectly marked as fully reviewed despite containing unrecognised choices in several questions. The issue occurred when a question's choices could not be recognised and the question was worth 0 points, causing it to be automatically assigned 0 points and considered "answered" during scan processing. If the point value of the question was then later increased, the grading status was not reset, resulting in an incorrect review status and the student receiving 0 points.

The issue was first reported via a support ticket and escalated to the technical team. On February 13th, the technical team began developing a hotfix and conducting an impact analysis. By February 15th, the support team informed the user about the impact on their institution and confirmed that a hotfix and postmortem were in progress.

A hotfix was deployed on February 18th, 2025, ensuring that unrecognised choices remain unanswered until manually reviewed, preventing incorrect grading statuses in the future.

Lead-up
On February 12th, 2025, a user reported an issue in which a result of a bubble sheet assignment was marked as fully reviewed, while containing questions with unrecognised choices.

Fault
It was discovered that this issue had been present since early January 2022 and would occur under the following conditions:

1. A question contained unrecognised choices during the initial scan processing.
2. The question was assigned 0 points because the system could not recognise its answers and automatically marked it as "answered."
3. Subsequently, the point value for the question was increased.

Since the grading status of the question was not reset when the point value was adjusted, the review status of the assignment remained unchanged, and the student incorrectly received 0 points for the question.

Impact
An impact analysis was performed on written assignments and bubble sheet assignments. The impact analysis was scoped on automatically graded questions where no choice selection was initially recognised, had not been reviewed manually, and scans that were uploaded between September 1st, 2023, and February 17th, 2025.

The investigation resulted in 39 affected assignments, with 393 impacted submissions.

Detection
The issue was first reported by a user via a support ticket. Upon review, the support team escalated the problem to the technical team for further investigation.

Response
On February 12th, 2025, the support team forwarded the ticket to the technical team and informed the user that further investigation was required. The support team assured the user that updates would be provided as more information became available. The technical team confirmed that the issue was caused by questions with unrecognised choices being incorrectly marked as answered.

On February 13th, 2025, the technical team began developing a hotfix while also conducting an impact analysis.

On February 15th, 2025, the support team informed the user about the impact on their institution and confirmed that a hotfix was in development, along with a postmortem report.

Recovery
On February 18th, 2025, a hotfix was released to ensure that unrecognised choices are converted back to ungraded when changing the points of the question from zero to a non-zero value. The reviewer will have to manually review these questions.

Timeline

12th of February, 2025

• 13:39 - Support receives a ticket indicating that a bubble sheet assignment is considered fully reviewed, while containing answers where the choices were not recognised.
• 13:44 - Support forwards the information to the technical team
• 13:56 - Technical team discovers root of the issue 
• 14:02 - Technical team starts development on a hotfix alongside an impact analysis
• 14:15 - Support notifies the sender that the choices on the question were not recognised and will update the sender once further investigation has been performed

13th of February, 2025

• 11:02 - Technical team informs support that a hotfix has been created that needs to be reviewed

14th of February, 2025

• 14:51 - The technical team informs support with an overview of the issue alongside the impact analysis.
• 16:01 - Support informed the user about the impact on their institution and confirmed that a hotfix was in development, along with a postmortem report.

18th of February, 2025

• 10:37 - Hotfix was deployed

Reflection

While the response and resolution of this issue were handled efficiently, our inability to detect it earlier highlights a gap in our testing approach. We are actively working to enhance our testing coverage and continuously improve the system. Our goal is to identify and address issues proactively, ensuring they are resolved before they impact users.