Alignment insights

Analysing the alignment of reviewers can provide different insights in regards to an assignment after it has been graded. Using this feature achieves fair and transparent grading.

To view insights into the alignment of reviewers, follow the steps below.

What the reviewer alignment shows

As you can see in the screenshot below, the following data is provided per question in the reviewer alignment insights:

• The alignment of the grading by different reviewers
  The top left hand corner, indicates if the alignment for this question is Good (shown in green) or if there is deviation detected (shown in yellow) or if the question Needs Attention (shown in red). The calculation of the deviation is explained at the bottom of this article.
  
  
• The number of results
  The overall number of results for the question is displayed in the top bar. In the reviewer breakdown you can see how many results each reviewer has graded per question.
  
  
• The number of minutes spent reviewing the question
  The total amount of minutes it took all reviewers to grade a question is displayed in the top bar. In the reviewer breakdown you can see how many minutes each reviewer took to grade this question.
  
  
• The average number of points given
  The average number of points given per question is displayed in the top bar. In the reviewer breakdown you can see how many points each reviewer has awarded for that question.
  
  
• Deviation detected
  The icon warningwill display next to the reviewers name if that reviewer has deviated more than 10% from the normalised average score for that question.

How we calculate the reviewer alignment

Ans shows whether the reviewers are aligned for reviewers that reviewed at least 5 results. There are two possible data fields:

• Good alignment
• Deviation detected

The determination of the quality of the alignment depends on the average score, on the average points given by a grader and by a deviation threshold. The following formula is used:

((average_question_score - grader_average) / points) > 0.1

For example:

average_question_score = 1.0
grader_average for a reviewer is 1.3
points = 5
(1.0 - 1.3) / 5 = 0.06 results in a value that is smaller than 0.1, indicating that no significant deviation is detected. Therefore, the field displays 'Good alignment'.