Service Incident - 19th of July 2023 - Unavailable gaps for fill-in questions

Incident Summary

All times in this document are recorded in UTC+2 (CEST). On the 11th of July, 2023, at 11:21, the support team received a ticket about certain fill-in questions not having any input fields in the preview or in the test. The issue occurred when the name of the gap was changed, but the question was not saved. This caused the name of the gap in the fill text to differ from the actual name of the gap. Because of this, when copying or adding this question to a test, Ans would not be able to find the correct gap and would instead copy the fill text as is. This resulted in the fill-in question to not recognise the gaps as input fields and would instead show them as blue boxes in the preview or in the digital test.

Lead-up

The support team received a ticket on July 11th at 11:21, stating that it was not possible to fill in the gaps during the taking of the test. Instead of the gap showing, a blue box was displayed with the name of the gap. The technical team started an investigation on July 11th, at 11:41.

Fault

According to the impact analysis, the issue has been present since March 19th, 2021. The issue occurred when changing the name of the gap. The change would also display the gap name being updated in the content of the question, however while this is visually shown, the change would not be applied unless the user explicitly saves the question. 

For example in a question bank, if the name of the gap is edited, but the changes to the question are discarded by leaving the page.

For example if the name of the gap is edited in a question bank exercise, but the changes to the question are discarded by leaving the page. The issue would then occur when adding this exercise to an assignment. 

Impact

Users were unable to input values in certain fill-in questions. During the investigation, an impact analysis was performed by the technical team. The results of the impact analysis display tests (some which have already been taken) and question bank exercises that were affected by this issue. The earliest discovered example from the impact analysis dates back to the 19th of March, 2021. 

The support team received several tickets regarding this issue.

Detection

The incident was detected when the support team received several tickets stating the fill-in questions did not contain gaps where students could provide an answer.

Response

The support team informed the technical team about the issue on July 11th at 11:30. Afterwards, the technical team started an investigation but was initially unsuccessful in reproducing the issue. The support team then requested more information from the user and the steps they took on July 11th at 17:16. The user replied on July 12th at 08:56 with a more detailed explanation of the issue. On July 12th at 17:01, the technical team began another investigation with the extra information. Extensive further investigation was required as with the extra information the technical team could still not immediately reproduce the issue, which is why it took longer to create a hotfix and perform an impact analysis than usual. The performance analysis was completed on July 18th. 

Recovery

On July 19th, a hotfix was deployed to prevent this issue from occurring again. Additionally, the results of the analysis will be shared with the impacted customers. The fix was only applied for newly created questions as Ans cannot determine where each gap should be positioned in the fill text. Ans will provide a list of the impacted questions and assignments and offer the possibility to rectify the questions that are affected. In the event that the question is deemed unfixable, then Ans will provide the link to the question or assignment.

Timeline

11th of July 2023

• 11:21 - Initial ticket with the issue about fill-in questions is sent.
• 11:30 - Issue is shared with the technical team.
• 12:13 - Technical team asks for more information as they are unable to reproduce the issue with the provided information.
• 17:16 - Support team asks the user that reported the issue for more information.

12th of July 2023

• 08:56 - User provides more information to reproduce the issue.

14th of July 2023

• 10:42 - Technical team notifies support team that they still require more time to reproduce the issue, however the technical team has started an impact analysis in the meantime.

17th of July 2023

• 16:54 -  Support team informs the user that the technical team is still working on reproducing the issue.

18th of July 2023

• 17:15 - Technical team finished the impact analysis and reproduced the issue.

19th of July 2023

• 15:43 - Deployed a hotfix to prevent the issue from occurring again.
• 17:13 - Support team has informed the customers that sent in the ticket that the issue has been resolved on production.

Reflection

The issue remained unnoticed as there were not many occurrences nor any errors which were directly visible to the user or would show up in Ans’ internal monitoring system. We have included an internal test alongside the hotfix which will be run in our continuous integration process to prevent the issue from occurring again.