All times in the document are recorded in UTC+1 (CET).
Incident summary
On December 3rd, 2025, the platform experienced downtime from 10:28 to 10:31 after the database exhausted its available memory, causing it to shut down and automatically reboot.
This issue was the result of backend changes made in the October 12th release, where the existing OpenSearch-based search system was replaced with a PostgreSQL-powered search implementation. The new search implementation created larger search queries, which caused the database to have insufficient memory, forcing it to reboot itself.
The technical team was immediately notified by our monitoring tools, and a hotfix was developed and deployed the same day to resolve the issue.
Lead-up
On December 3rd, 2025, at 10:27, the PostgreSQL database experienced a sharp spike in memory usage caused by unusually large search queries being executed.
Fault
It was discovered that this issue had been present since October 12th, 2025 when a change was introduced in the backend where the existing OpenSearch-based search system was replaced with a PostgreSQL-powered search implementation. The new system generated larger and more complex search queries than anticipated, which led to unexpected memory spikes and caused the database to exhaust its memory, forcing it to reboot itself.
The issue only occurred when the search query contained long sentences when searching for question bank exercises.
Impact
Beginning at 10:25, the platform experienced degraded performance due to insufficient database memory. This issue escalated, and between 10:28 and 10:31, the platform became fully unavailable to all users.
Detection
On December 3rd, 2025, at 10:27, the technical team received alerts from our application monitoring system about slow or timed-out browser requests. On December 3rd, 2025, at 10:28 our monitoring system, Pingdom, triggered an alert indicating application downtime.
Response
On December 3rd, 2025, at 10:45, the technical team started investigating the issue. At 11:20 the technical team confirmed the root cause of the issue and informed the support team to update the status. On December 3rd, 2025 at 11:42, the support team posted a status message to inform users of degraded performance.
Recovery
On December 3rd, 2025, at 14:59, a hotfix was deployed to prevent the issue from recurring. This hotfix moved PostgreSQL-powered search implementation to the secondary read only instance of the database. This way the primary writer instance would not be affected by the long queries. At 21:32 a second hotfix was deployed, which limits search queries on question bank exercises to a maximum of 50 characters, reducing the risk of memory exhaustion on the PostgreSQL database.
Timeline
3rd of December, 2025
- 10:28 - Our monitoring system, Pingdom, triggered an alert indicating application downtime.
- 10:31 - Database rebooted itself and the platform is available again.
- 10:45 - The technical team is investigating the issue
- 11:20 - The technical team informs the support team of the issue.
- 11:42 - The support team posts a status message to inform users of degraded performance
- 12:04 - The technical team confirms the root cause of the issue and created a hotfix
- 12:27 - The support team updates the status that the issue is identified
- 14:59 - First hotfix was deployed to the production environment
- 21:32 - Second hotfix was deployed to the production environment
4th of December, 2025
- 08:56 - The support team posts a status message that the issue has been resolved
Reflection
The incident occurred when users submitted large search queries, such as entire sentences, when searching for question bank exercises. This caused the PostgreSQL database to exhaust its available memory, leading to temporary downtime. Our team will continue to monitor the application closely to ensure its stability and performance.
During the incident, the technical team did not immediately notice the downtime, which delayed response and communication. Improved alerting systems have now been implemented to ensure incidents are detected quicker.
The technical team prioritised investigating the issue and did not inform the support team directly, which caused delays in updating the status page. Moving forward, support will be notified as soon as an issue is identified.
This incident highlighted that we can improve the performance of our PostgreSQL-powered search implementation. We will focus on optimising it to make the system more resilient going forward, ensuring the database performance is not affected under high-stress conditions.
Comments
0 comments
Article is closed for comments.