Service Incident - 27th of March 2025 - Outdated variables

All times in the document are recorded in UTC+1 (CET).

Summary
On March 15th, 2025, a user reported a missing variable during grading of a numerical question. After investigating, the technical team discovered that some variables were not updated or created after changes were made to the script. This could cause variables to either be missing or their value to be outdated when students took their test. This occurred if the teacher previewed the digital test between December 16th, 2024, and March 2nd, 2025, and updated the script after previewing the test. Only students starting a digital test with random variables after March 2nd, 2025 are impacted if the system assigned them the variables with the same seed as the seed given to the teacher during previewing. 

The issue was first reported via a support ticket and escalated to the technical team. On March 17th, the technical team began investigating the report. After identifying the cause on March 18th, the technical team began developing a hotfix and conducting an impact analysis. By March 21st, the support team informed the user about the impact on their assignment and confirmed that the postmortem was in progress.

A hotfix was deployed on March 21st, 2025, ensuring that only up-to-date variables are assigned to students when starting a digital test.

Lead-up
On March 15th, 2025, a user reported an issue that variables were missing in a digital test for a few students.

Fault
It was discovered that this issue had been present since March 2nd, 2025, and would occur under the following conditions:

1. The teacher previewed the digital test between December 16th, 2024, and March 2nd, 2025
2. The teacher changed the script after previewing between December 16th, 2024 and March 2nd, 2025
3. A student started a digital test with random variables after March 2nd, 2025 where the student was assigned the same seed as the teacher during previewing.

This issue was introduced by a refactoring of our backend for storing variables. This is a process that we started in November 2024 and was planned out meticulously to prevent any impact on our users. Previously, variables were each calculated and stored separately when a student began a test. However, this approach was performance-intensive. In order to improve this, we now pre-calculate the variables before the test starts and store them in a set alongside the seed used to generate them. As the student starts the test they are assigned a random set when they begin.  

Impact
The impact of this incident affected variables shown in the content of the question and the variables used in numerical questions, but are depending on the changes made to the script. 

An impact analysis was performed on all digital tests using random variables between March 2nd, 2025, and March 21st, 2025.

The investigation resulted in 7 affected assignments, with 151 impacted results.

Detection
The issue was first reported by a user via a support ticket. Upon review, the support team escalated the problem to the technical team for further investigation.

Response
On March 17th, 2025, the support team forwarded the ticket to the technical team and informed the user that further investigation was required. The support team assured the user that updates would be provided as more information became available. It took the technical team some time to figure out what caused this, but on March 18th the cause was identified and the technical team started working on a hotfix and determining the impact of the incident

On March 21st, 2025, the support team informed the user about the impact on their assignment and confirmed that a hotfix was in development, along with a postmortem report.

Recovery
On March 21st, 2025, a hotfix was released to ensure that the correct variables are assigned to students when starting a digital test. 

Timeline

15th of March, 2025

• 19:51 - Support receives a ticket indicating that a few students are missing variables.

17th of March, 2025

• 09:18 - Support forwards the information to the technical team and the technical team begins their investigation

18th of March, 2025

• 12:51 - The technical team identifies the cause of the issue and starts working on a hotfix and determining the impact.

21st of March, 2025

• 11:27 - The technical team informs support with an overview of the issue alongside the impact analysis.
• 16:55 - Hotfix was deployed
• 17:56 - Support informed the user that reported the issue and the administrators about the impact on their school and confirmed that a postmortem report was being written.

Reflection
Although we took several months to plan and execute the refactoring of our backend for storing variables—carefully considering various scenarios—this specific case was unfortunately missed. We performed several manual checks before deploying the new approach, all of which passed. However, we missed a scenario where previewing a test generated a set of variables with a seed that wouldn’t update after a script change. It was a human oversight, and despite our best efforts, software development inherently carries the risk of bugs.

This incident highlights the importance of expanding test coverage to include more edge cases, particularly around interactions between previewing and variable generation. Moving forward, we will refine our testing strategies to include simulated user workflows that mimic real-world conditions more closely. 

While this issue had a limited impact, affecting 151 results across 7 assignments, we recognize the importance of maintaining user trust. Our response time was reasonable, but improvements can be made in detecting such issues proactively rather than relying on user reports. To that end, we will explore monitoring solutions that can alert us to inconsistencies in variable assignments before they reach students.