Service Incident - 21st of January 2026 - Changes to layout options not being applied due to caching

All times in the document are recorded in UTC+1 (CET).

Incident summary

Between October 12, 2025 and January 15, 2026, changes made to layout options were not always applied correctly when assignments were regenerated or retaken. This resulted in discrepancies between the order of answer alternatives in multiple choice and multiple response questions shown to students, and the order used during grading if the option “Shuffle the answer alternatives for each question” was adjusted. 

The issue was caused by a caching mechanism that kept outdated layout settings for up to one week after results or forms were created. A backend change introduced on October 12, 2025 changed where layout settings were cached, but the corresponding cache invalidation logic was not updated. As a result, layout changes made after deleting results or forms were ignored during that period.

The issue affected both written assignments and digital tests. It was first reported by a user on December 23, 2025, resolved through two hotfixes, the first being deployed on January 7 and was specific for written assignments and the second on January 15, 2026 for digital tests. Impacted users were informed by January 21, 2026.

Lead-up

On December 23, 2025, at 11:07, a user reported an issue where multiple choice questions and multiple response questions in a written assignment were graded incorrectly. The order of the answer alternatives  displayed in the PDF preview did not match the order shown in the grading view, resulting in incorrect grading by the system. The support team escalated the issue to the technical team the same day.

Fault

The root cause of the issue was traced to a backend change introduced on October 12, 2025. This change updated how layout settings were cached in preparation for the future addition of the layout option “Add ‘Leave unanswered’ option to multiple choice and multiple response questions.”

When results are created for digital tests or forms are generated for written assignments, the system saves a temporary copy of the layout settings for one week, called a cache. This cache ensures performance and layout consistency.

If results or generated forms are removed, this cache should be cleared so that subsequent layout changes are applied. However, the October 12 change updated the cache storage location without updating the corresponding cache invalidation logic. As a result, the system attempted to clear the cache from the old location, leaving the active cache intact.

Because of this mismatch, layout changes made within the one-week cache window were ignored.

The issue would occur under the following conditions. 

For digital tests:

1. A student participates in a digital test and a digital result is created
2. The result is removed
3. Changes are made to the layout options
4. The layout changes would not be reflected when the assignment is taken again within a week of the layout change. 

For written assignments:

1. Forms are generated in a written assignment
2. Forms are deleted
3. Changes are made to the layout options
4. Forms are regenerated
5. The layout changes would not be reflected in the generated forms if the forms were generated within a week of the layout change.

In both cases, the assignments continued using the previous layout settings instead of the updated ones.

Impact

The impact of this incident could have affected the automatic grading of the students. 
An impact analysis was performed on digital tests and written assignments that have been taken and forms that have been generated since October 12th, 2025.

The investigation resulted in 3 assignments where the grading was affected. 1 assignment where the layout options were not applied but the assignment was graded correctly. 12 written assignments that have not yet been taken but the forms were generated incorrectly.

Detection

The issue was first reported by a user via a support ticket. Upon review, the support team escalated the problem to the technical team for further investigation.

Response

The support team forwarded the issue on December 23rd at 12:47 to the technical team. 

On January 2nd, 2025, at 10:45, the technical team started investigating the issue. During the investigation it initially seemed like the issue only occurred with the option “Shuffle the answer alternatives for each question” for written assignments. As the cache was not invalidated when forms were deleted.

On January 5th at 12:13, another user reported that the same issue happens in digital tests. On January 8th at 09:07 the support team forwarded the issue to the technical team. Once it was discovered that the issue was also present on digital tests the technical team started investigating further on January 13th. 

Recovery

On January 6th at 15:34 a hotfix was created to invalidate the cache for written assignments once the generated forms were removed, which was deployed on January 7th at 14:41.
On January 14th at 10:12 a hotfix was created which was deployed on January 15th at 16:08 to update the usage of invalidating the cache with the cache key change from October 12th.

The technical team started to investigate the impact. For written assignments this took longer to verify as it was necessary to manually check whether the generated forms contained the changes that were applied in the layout.

Timeline

12th of October, 2025

• 09:44 - Change is introduced to support the addition of the “Add ‘Leave unanswered’ option to multiple choice and multiple response questions” option to the layout.

23rd of December, 2025

• 11:07 - The support team receives a ticket where the user reported that multiple choice questions and multiple response questions were graded incorrectly for a written assignment.
• 12:47 - A task was created for the technical team to investigate and fix the issue.

2nd of January, 2026

• 10:45 - The technical team started investigating the issue.

5th of January, 2026

• 12:13 - The support team received a ticket where the user reported that the same issue occurs in digital tests.

6th of January, 2026

• 15:34 -  A hotfix was created to fix the issue where toggling shuffle answer alternatives in the layout options would not apply after forms had already been generated once.

7th of January, 2026

• 14:41 -  The hotfix to fix the issue where toggling shuffle answer alternatives in the layout options would not apply after forms had already been generated once is deployed.

8th of January, 2026

• 09:07 - The support team notifies the technical team that the issue is also present in digital tests to the technical team.

9th of January, 2026

• 15:34 -  An initial impact analysis was performed for written assignments where the option “Shuffle the answer alternatives for each question” was incorrectly applied. 

14th of January, 2026

• 10:12 -  A hotfix was created to fix the issue where test layout settings could be outdated.

15th of January, 2026

• 14:16 - Support team notifies the technical team that the initial impact analysis needs to be expanded to account for digital tests as well.

• 16:08 -  Hotfix to fix the issue where test layout settings could be outdated is deployed.

20th of January, 2026

• 17:29 - All affected schools have been informed. 

Reflection

This incident highlighted several gaps in how layout-related changes are validated and communicated across different assessment types. Although the issue affected both written assignments and digital tests, the impact initially appeared limited to written assignments. As a result, the investigation and first hotfix focused only on that area.

Two separate engineers worked on addressing the issues for written assignments and digital tests respectively. While both engineers acted appropriately based on the information available at the time, the shared root cause was not immediately identified. This led to duplicated investigation efforts and a delay in recognizing that the same caching behavior affected both assessment types. Earlier coordination could have revealed the shared root cause sooner.

The incident also revealed that changes to cache keys and cache invalidation logic were not sufficiently validated together. Although the October 12th change updated how layout caches were stored, the related cache-clearing logic was not updated at the same time. This mismatch was not detected through testing or monitoring and allowed the issue to remain unnoticed until users were affected.

We have expanded our automated testing to verify that cached values are correctly evaluated and invalidated.