All times in this document are recorded in UTC+1 (CET).
Incident summary
On February 24th, 2026 an issue was identified that affected the calculation of the Rir-value (Response-Item Residual correlation). The Rir-value is a statistical measure that indicates how well a specific question aligns with the overall test performance. Its value ranges from -1 to 1:
- Close to 1: Strong positive correlation
- Close to 0: Little or no correlation
- Close to -1: Negative correlation
Due to an error in the calculation process, Rir-values were incorrect if a question was answered incorrectly.
Lead-up
A user reported on February 11th, 2026, at 13:45 via a support ticket that there was a difference in the Ans Rir-value and the Rir-value they calculated themselves. The ticket was forwarded to the technical team, who started their investigation.
It was not directly clear how the Rir-value was calculated by the user. Their steps were shared on February 18th, and the technical team confirmed on February 23rd that they found the difference between the calculations. The technical team confirmed on February 24th that there was a mistake in the Ans calculation of the Rir-value.
Fault
This issue has existed since the introduction of the Rir-value in Ans on July 15th, 2022. Before calculating the Rir-value, we applied a step called normalisation.
What is normalisation?
Normalisation is a way of adjusting numbers so they can be compared fairly. In our case, normalisation was originally introduced because assignments could use flow. When flow is used, students’ results may have a different amount of maximum points. To make results comparable, we normalised scores before calculating correlations.
The Issue
The problem occurred in the normalisation step. When normalising values, the number we divided by (the “normalising value”) was incorrectly determined based on the correctness of the students’ answers.
This means:
- Incorrectly answered questions would receive a different value from what they should have.
- As a result, the data used for calculating the correlation was unintentionally distorted.
- This led to incorrect Rir-values if a question was ever answered incorrectly in the assignment.
The impact of this issue could result in any deviation within the full Rir range of -1 to 1, depending on the dataset.
Average Impact
A sample of the most recent 1000 calculations based on 50 or more results, showed an average absolute difference of 0.056.
Detection
This issue was reported by a user via a support ticket on February 11th, 2026, at 13:45. The ticket was forwarded to the technical team the next day on February 12th, 2026, at 14:33.
Response
We have taken the following actions:
1. Removed normalisation from the calculation
We have removed the normalisation step entirely from the Rir-value calculation.
The original reason for normalisation was to account for differences in maximum possible scores caused by assignments’ flow. We have now removed that dependency from the calculation logic.
2. Aligned Rit calculation
Although the Rit-values were not incorrect, we have applied the same adjustment to their calculation method. This ensures:
- Consistency between Rir and Rit
- Alignment with supported and validated statistical methods
- Clearer and more transparent calculations
3. Updated all future calculations
All new and future Rir and Rit calculations are now performed without normalisation. This ensures that all future calculations are correct.
4. Recalculation available upon request
Schools who would like previously calculated Rir-values to be recalculated can submit a request to our support team. We will recalculate the values using the corrected calculation method.
Timeline
11th of February, 2026
- 13:45 - A user reported the issue to the support team via a support ticket.
12th of February, 2026
- 13:35 - The support team investigated the report.
- 14:33 - The ticket was forwarded to the technical team for further investigation.
13th of February, 2026
- 09:49 - The technical team asked the reporter for more information on which questions had a different Rir-value.
16th of February, 2026
- 11:59 - The reporter shared images of the questions that had a different Rir-value in Ans and in their own calculation.
- 15:37 - The technical team could not reproduce the reporter’s calculations and asks for the calculation steps.
18th of February, 2026
- 13:35 - The reporter shares the calculation steps.
23rd of February, 2026
- 09:27 - The technical team finds the difference between the calculations and will investigate which one is correct.
24th of February, 2026
- 09:25 - The technical team confirms that the Ans calculation is incorrect.
28th of February, 2026
- 20:31 - A fix was deployed to remove the normalisation step from the calculation of the Rit- and Rir-values.
Reflection
The incident with the Rir-value calculation in Ans showed that a flaw in the normalisation step of the correlation logic led to incorrect values when questions were answered incorrectly. Because normalisation was originally introduced to handle differences in maximum scoring (for assignments using flow), the calculation inadvertently adjusted data in a way that distorted results.
The issue persisted for some time since it only surfaced in specific datasets and the outputs could appear plausible without a direct reference for comparison. A user report ultimately exposed the discrepancy, demonstrating how external feedback and validation are important safeguards.
The response addressed the problem by removing normalisation from the calculation and aligning related metrics for consistency. This simplifies the methodology and reduces the risk of similar errors. The incident highlights the need for continuous validation of statistical methods and strong regression testing across all scenarios to maintain accuracy and trust in calculated results.
Comments
0 comments
Article is closed for comments.