Service Incident - 26th of April 2024 - Use of deprecated guess score correction when copying older assignments

All times in the document are recorded in UTC+2 (CEST).

Summary

On the 25th of April, 2024, at 10:35, the technical team discovered a discrepancy while improving the test coverage of the guess score correction functionality. The technical team discovered that it was possible for newly created assignments to use the deprecated version of the guess score correction.

There were two scenarios in which the deprecated guess score correction would still be applied to assignments created after 27th of August 2023:

1. Using a template of an assignment where the deprecated guess score correction is enabled.
2. Copying a course assignment where the deprecated guess score correction is enabled.

In both cases, the type of guess score correction used would be copied over to the newly created assignment. This means that the new assignment would make use of the old guess score correction of the template or original assignment it was based on.

The technical team deployed a hotfix on 25th of April 2024, at 14:09, to ensure that newly created assignments will always use the most recent version of the guess score correction.

Lead-up

Whilst improving Ans’ internal testing coverage of the guess score correction functionality, the technical team discovered a discrepancy where newly created assignments could have the deprecated version of the guess score correction enabled. 

Fault

During the investigation, it was discovered that this issue was introduced on August 27th 2023, as this was the date Ans enforced the use of a new and expanded version of the guess score correction, which uses a different method to calculate the guess score and takes other closed question types into account as well. Ans would ensure that the deprecated version of the guess score correction was kept for assignments that were taken before August 27th 2023, in order to keep the marks of assignments taken before this date the same as they were when they were first marked.

Impact

An impact analysis was performed by the technical team in order to find the results that were affected. This analysis was scoped to assignments that were created between the 27th of August 2023 and the 25th of April 2024 and where the grade would differ between the two different versions of the guess score correction. The results from this analysis are as follows:

• A total of 123 assignments were found to be affected among 12 institutions. 

The results in the affected assignments were calculated based on the deprecated version of the guess score correction. This means that the guess score correction would only be applied to multiple choice questions in which a student can only select a single alternative.

Detection

The technical team discovered the issue on the 25th of April 2024, at 10:35. The issue was detected whilst improving testing and improving the test coverage of the guess score correction functionality.

Response

After the initial discovery, the technical team verified internally confirming the discrepancy to be an issue. The technical team started on the development of a hotfix to ensure that newly created assignments will always use the newest version of the guess score correction. During the development and deployment of the hotfix, the technical team started working on an impact analysis. 

Recovery

A hotfix was deployed to the production environment on the 25th of April 2024, at 14:09. Which prevents the possibility of copied assignments to continue using the deprecated version of the guess score correction. An impact analysis was performed and shared with the administrators of the institutions that were affected. The impact analysis will show a comparison between the existing mark and what the mark would be with the most recent version of the guess score correction.

Timeline

25th of April 2024

• 10:35 - The technical team discovers a potential issue when copying older assignments which still used the deprecated version of the guess score correction.
• 11:55 - The technical team confirms internally that the use of the deprecated version of the guess score correction is an issue.
• 12:39 - The technical team starts development on the hotfix and investigates the impact.
• 14:09 - The hotfix is deployed and prevents newly created assignments that are created using templates or that are copied will no longer use the deprecated version of the guess score correction.

Reflection

The steps taken in the handling of this incident were followed in procedural order. We have included several internal tests alongside the hotfix which will be run in our continuous integration process to ensure that issues with future changes are immediately discovered. 

Alongside these added tests, we are still continuing to add and improve our existing tests within Ans regarding guess score correction and other critical areas within the platform to ensure we prevent breaking changes in the future.