Service incident - 6th of February - Incorrectly calculated guess score with match questions

All times in this document are recorded in UTC+1 (CET).

Incident summary

On the 28th of January, 2026, an issue was identified where the guess score for certain match questions was calculated incorrectly. Specifically, for match questions using a Matrix layout with a single answer per row, the system produced an inflated guess score when Automatic scoring was enabled and Partial scoring was disabled.

After investigating, we discovered that this behaviour had been present since the introduction of guess correction for match questions, in June 2022, due to a missing calculation path for this specific configuration. As a result, the system calculated the guess score as if Partial scoring was enabled, even when it was disabled. This caused more permutations of answering the question to be considered as point awarding, in turn increasing the guess score of the question, which could have impacted student scores.

A fix was deployed on January 29th 2026 at 11:29 to ensure the guess score calculation correctly takes the disabled Partial scoring option into account for this configuration.

Lead-up

On January 27, 2026, at 15:11, a user contacted support after noticing that the guess score for a match question was higher than expected when compared with the example calculation provided in the Help Center. The user’s message was noticed by a member of the technical team, who picked up the possible issue with urgency. 

During the investigation, the team reproduced the behavior using a match question configured with a Matrix layout, a single correct answer alternative, Automatic scoring enabled, and Partial scoring disabled. This confirmed that the guess score calculation did not align with the documented behavior, prompting a deeper review of the scoring logic.

Fault

This issue was caused by an oversight when guess correction was introduced in June 2022. At that time, separate calculations were implemented for the four possible combinations of scoring options:

• Automatic scoring on — Partial scoring on
• Automatic scoring off — Partial scoring on
• Automatic scoring on — Partial scoring off
• Automatic scoring off — Partial scoring off

In the code base, there are separate calculations for each combination. However, the third case (Automatic scoring enabled and Partial scoring disabled) was not implemented for match questions with a Matrix match layout and a single answer per row.

If Partial scoring is enabled, a student can receive points for a partially correct answer. If Partial scoring is disabled, the student only receives points when the answer is completely correct. Because Partial scoring increases the chance of earning points, the guess score is higher when Partial scoring is enabled.

When Automatic scoring was enabled, the system always calculated the guess score as if Partial scoring was enabled as well. As a result, the guess score of these questions was higher than it should have been. For assignments with guess score correction enabled, this may mean that a student was punished more harshly for incorrect answers in the closed questions of their assignment.

Impact

The impact of this incident could have affected the scores of students.

An impact analysis was performed on digital tests created since the introduction of guess correction in June 2022. The investigation resulted in 37 assignments, with 2198 results being impacted. 

The impacted schools have been informed.

Detection

The issue was reported by a user via a support ticket. The technical team investigated the issue described in the ticket and found the cause the next morning, on January 28.

Response

The technical team started investigating the issue in the morning of January 28th and found the cause at 10:05. An initial impact scope was determined later that day. An internal meeting was held with the product and technical teams to confirm whether the behaviour was intended, after which the issue was treated as a defect requiring a fix.

Recovery

On January 29th at 09:24, a fix was created to take the disabled Partial scoring option into account when calculating the guess score for a match question where Automatic scoring is enabled with one correct answer alternative and the Matrix match layout. The fix was deployed on the production environment by January 29th at 11:29.

After deployment, the incorrect calculation no longer occurred for newly calculated guess scores.

Timeline

27th of January, 2026

• 15:11 - The support team receives a ticket where the user reports that the guess score of a match question was higher than expected. 

28th of January, 2026

• 09:00 - The technical team starts investigating.
• 10:05 - The technical team found the cause of the issue after investigating. 
• 14:02 - The initial impact was determined. 
• 15:12 - Internal meeting held with product and technical teams to determine whether the behaviour was intentional.

29th of January, 2026

• 09:24 - The technical team starts with the fix to resolve the issue.
• 11:29 - The fix is deployed on the production environment.
• 13:50 - The impacted assignments are determined. 
• 16:22 - All affected schools are informed of the assignments that contain impacted results. 

30th of January, 2026

• 09:00 - Technical team starts working on the full impact of the issue, including mark and point differences if the results were to be recalculated
• 17:08 - Technical team finds that the implemented fix does not fully cover all scenarios in which the issue can occur

2nd of February, 2026

• 10:21 - A new fix is created to fully cover all scenarios of how the question can be created.
• 11:32 - The new fix is deployed onto production.
• 14:09 - The complete impact is determined including the current and corrected marks and points.
• 17:13 - All affected schools are updated of the complete impact.

Reflection

This issue was caused by an oversight when guess correction was introduced in June 2022. The specific scenario where Partial scoring was disabled but Automatic scoring was enabled, for match questions with one correct alternative and the Matrix match layout, was not implemented in the guess score calculation logic. Because guess score logic depends on configuration combinations, missing a single configuration resulted in the calculation being performed as if it was another.

The issue existed for an extended period because it was limited to a specific configuration and because the incorrect output could look plausible without an accurate comparison reference. In this case, the user comparison against the Help Center example helped make the discrepancy visible. This highlights that scoring and guess correction require strong regression coverage across all option combinations and layouts, as the amount of functionalities in the platform that may impact the guess score are very high.

The ticket was not initially picked up with the correct priority when it was received. An engineer noticed the ticket and picked it up themselves due to its concerning title. Since the ticket mentioned scoring correctness and could affect student outcomes, reports of this nature should be treated as high urgency until triaged and scoped properly. This has been discussed internally, and the technical team will be approached more promptly in similar cases to ensure the priority is set accurately.

Once the issue was escalated, investigation and mitigation were handled effectively. The cause was found quickly on January 28th, and the fix was developed and deployed on January 29th. The initial fix was correct in a large majority of cases, but failed to consider that match question rows may have multiple correct answers, even for questions that only expect the student to give a single answer per row. After finding this, the initial fix was iterated upon to guarantee the guess score calculation would be correct from that point on. The impact analysis and communication to affected schools were also handled effectively.