All times in the document are recorded in UTC+2 (CEST).
Summary
On the 16th of September 2024, at 14:35, the technical team discovered an issue while expanding their internal testing. The issue occurred with open-ended questions that contained the grading type “Slider”. It was possible that participants were awarded more points than the maximum points set on the open-ended question with the grading type “Slider”.
The technical team informed the support team at 15:35 after confirming the issue.
The technical team created a hotfix which was deployed the 16th of September 2024, at 21:39 to ensure that the issue does not persist.
Results that were impacted have been shared with the affected schools.
Lead-up
On the 16th of September, at 14:35, the technical team discovered an issue while expanding their internal testing. During testing it was discovered that it’s possible that participants were awarded more points than the maximum points set on open-ended questions with grading type “Slider”. If the maximum points were reduced to a positive value after the test had been taken, then the participant would keep the amount of points they were originally awarded.
Fault
During the investigation, it was discovered that this issue was created on the 27th of October 2017, the date when background processing was first implemented in Ans. Because the “Slider” grading type allowed for points to be set directly during review, its behavior requires different logic than existing grading types for open-ended questions.
Impact
An impact analysis was performed by the technical team in order to find the results that were affected. This analysis was scoped to assignments that were submitted between the 27th of October 2017 and the 16th of September 2024. The results from this analysis are as follows:
- A total of 175 results were found to be affected among 24 assignments
Detection
The technical team discovered this issue while expanding their continuous test integration.
Response
The technical team immediately started an investigation after discovering the issue. Once the issue was reproduced and verified, a hotfix was created and deployed on the production environment on the 16th of September at 21:39.
Recovery
A hotfix has been deployed in order to prevent the issue from happening again. An impact analysis has been shared with the affected schools. The impact analysis provides a URL to the impacted result and a comparison between the current grade and what the new grade would be.
Timeline
16th of September, 2024
- 14:35 - The technical team discovers the issue while testing
- 15:08 - The technical team starts working on a fix and the impact analysis
- 16:38 - The technical team shares the details of the issue with the support team
- 21:39 - The hotfix is deployed on the production environment
Reflection
The steps taken in the handling of this incident were followed in procedural order. We have included several internal tests alongside the hotfix which will be run in our continuous integration process to ensure that issues with future changes are immediately discovered.
Alongside these added tests, we are still continuing to add and improve our existing tests within Ans to prevent breaking changes in the future and to ensure the reliability of the platform.
Comments
0 comments
Please sign in to leave a comment.