Service Incident - 3rd of May 2022 - Shuffling of answers for written assignments

Incident summary

All times in this document are recorded in UTC+2. On the 3rd of May 2022 at 12:49 and 17:45, users reported an issue where multiple choice submissions were improperly recognized for the first form in written exams, causing students to be incorrectly graded. The event was triggered by adding the possibility of shuffling the answer options in the preview of an assignment on April 10th. This addition contained a change to the randomisation of written assignments, causing the seed used for the randomisation to not be saved for the first form. The event was noted by two support tickets being sent to the Customer Team. The Development team started working on the event by investigating what caused the incident. This incident is identified as severity level 3, which is a minor incident with a low impact. However, for users that were impacted by this incident, this incident was identified as severity level 1, which is a major incident with a significant impact. This event affected 0,003% of our users.

Leadup

A change was introduced in the release of April 10th 2022, adding the possibility to shuffle the answer options in the preview of an assignment. The issue was discovered at 12:49 on the 3rd of May 2022, when the Customer Team received a ticket describing that Ans does not register that the answer options have been shuffled. The change resulted in a bug for written assignments where the option 'shuffle the answer options' was enabled. The first form did not contain the seed used for the randomisation and was not recognised as shuffled. Because of this, multiple choice submissions were improperly recognised, causing students to be graded incorrectly.

Impact

Ans has reviewed all assignments that were generated and uploaded between April 11th and May 4th to determine which results were affected. The scope could be limited to assignments which had shuffling enabled, contained a multiple choice question and where the preview (001.pdf) was used by a student. We identified six results of which two were reported to the Customer team. All impacted schools were informed.

Response

A side conversation was opened in Slack by the Customer Team on the 3rd of May at 15:22 informing the Development Team of the issue. An investigation was opened by the Development Team on the 3rd of May at 17:45. A response to the ticket was made on the 4th of May at 09:16 by the Customer Team, informing the user that the Development team will be investigating the issue.

Recovery

First, the Development Team began investigating the issue by locating the change that introduced this issue. Second, a hotfix was made to ensure this issue does not persist. Third, once the breaking change had been found, the Development Team investigated the impact this change had, by looking at which assignments could have been influenced between the period of April 11th 2022 and May 4th 2022. Fourth, the exams that were impacted during this incident were reprocessed and verified, by confirming that the submissions in the assignments are correctly graded. Finally, the customers were informed that the issue had been resolved.

We regret that this issue has occurred and have reviewed our testing procedures to see where we can improve. Ans has three stages of testing, during code review, with automated tests and a manual test in the interface of our testing environment. This issue should have been found during manual testing, but in this case only the generation was tested, not the impact when uploading the generated form. We have expanded the steps in manual user testing to also better look at consequences of a change in a later stage and we have added automated tests to automatically detect this issue.

Timeline

3rd of May 2022

12:49 - The Customer Team received a ticket detailing that Ans does not recognise answers that have been shuffled.
15:22 - A side conversation was opened by the customer team asking the Development Team for more information.
17:39 - The Customer Team received a second ticket detailing that Ans is showing different answer options for an assignment.
17:45 - An investigation has been opened by the Development Team.
18:38 - The Development Team discovers the change that was made which created this issue.
19:01 - The Development Team issues a hotfix, making sure the issue does not persist and starts investigating the impact.

4th of May 2022

09:16 - The issuer of the first ticket was informed that the Development Team is investigating the issue.
09:16 - The issuer of the second ticket was informed that the Development Team is investigating the issue.
10:45 - The Development Team has discovered which users and results were affected by this incident.
11:30 - The Development Team has reprocessed the assignments that were impacted by this incident.
11:50 - The Development Team has confirmed that the issue has been fixed, verified that the assignments are correctly graded and that the issue has been resolved.

5th of May 2022

11:00 - The customers have been informed that the issues have been resolved.