Service incident - 29th of April 2026 - Timeslots shifted by timezone mismatch

All times in the document are recorded in UTC+2 (CEST).

Incident summary

With last weekend's release, the server timezone changed from CEST to UTC due to a configuration mismatch in our servers. As a result, scheduling-related times were shifted by 2 hours.

The issue was first reported by a customer via support ticket. A fix was deployed at 17:26 on 26 April 2026 to restore correct runtime timezone behaviour.

After the fix:

• 52 timeslots and 25 publication timeslots that were changed during the impacted window were corrected.

Lead-up

On the night of April 25th to April 26th, a new Ans version was released. This release introduced a server timezone issue, where runtime timezone shifted from CEST to UTC.

The same release included migration work from the old file storage to the new file storage for exercise attachments. More information can be found here.

To ensure this migration would not affect Schoolyear exams, a maintenance task was run after the release to synchronise whitelisted URLs between Ans and Schoolyear.

Because the timezone issue was active at that time, the synchronisation updated Schoolyear exams with incorrect start and end dates (2-hour difference).

Fault

The root cause was an incomplete configuration change in the Docker image flow (server configuration):

• The time settings were updated in the part of the process used to prepare the application.
• The same update was not applied to the version of the application that actually runs in production.

Impact

• 6 students lost 2 hours of test-taking time when the fix was deployed as time moved forward by 2 hours.
• 114 students of 1 assignment were impacted by the the wrong timeslots in Schoolyear
• 52 timeslots and 25 publication timeslots were corrected after the fix.
• No data loss or data corruption occurred.
• Additional effects:
  • For records that were changed during the impacted window, some timestamps shown in the interface appear incorrect, for example the modified column in the question bank.
  • Logs were not impacted.

Detection

The incident was detected through a customer support ticket.

At the time of the incident, we had monitoring for clock drift (NTP) but no explicit monitoring or verification for runtime timezone configuration.

Response

The engineer on call handled the incident and investigation.

After identifying the runtime timezone mismatch, a hotfix was deployed at 17:26 on 26 April to restore CEST behaviour.

Impact analysis and correction were then executed for scheduling data changed during the impacted window.

Recovery

After the fix:

• Timeslots and publication timeslots changed between release and fix were corrected.

After a customer report on April 27 about incorrect Schoolyear times:

• The maintenance task was re-run to correct Schoolyear exam timeslots.

No manual action is required from customers.
 

Timeline

26th of April, 2026

• 00:01 - A new version of Ans is released which changes the timezone from CEST to UTC
• 00:09 - A maintenance task is run to synchronise all scheduled Schoolyear exams.
• 13:18 - A ticket is received reporting incorrect timeslot times.
• 14:13 - The engineer on call sees the ticket and starts investigating the issue
• 16:03 - The engineer on call pings another engineer to discuss the issue and verify the proposed fix
• 17:26 - A fix is released to correct the runtime timezone behaviour back from UTC to CEST.
• 17:30 - Impact assessment and correction start for affected timeslots/publication timeslots.

27th of April, 2026

• 12:04 - A ticket is received reporting that Schoolyear start and end times are incorrect.
• 13:22 - Engineer on call handles the ticket and reruns the maintenance task to sync the timeslots with Schoolyear.

Reflection

No proactive weekend communication was sent at the time, as initial impact was estimated to be low.

In retrospect, we recognise the importance of clear communication even under low initial impact estimates. We apologise for any inconvenience this may have caused.

To reduce recurrence risk, we have updated our release and incident procedures:

1. Post-release verification update: Added a mandatory manual release-manager check to verify that the correct runtime timezone is active after deployment.
2. Playbook update: On-call and release checklists now explicitly include cross-system synchronisation validation after releases affecting scheduling behaviour.
3. Testing focus: Post-release validation now emphasises verification in the final runtime environment (not only build/preparation stages).