Analysing the alignment of reviewers can provide different insights in regards to an assignment after it has been graded. Using this feature achieves fair and transparent grading.
To view insights into the alignment of reviewers, follow the steps below.
- Click the domain School name in the menu on the left.
- Click label_importantCourses in the menu at the top.
- Select your course or use the search bar.
- Select your assignment or use the search bar.
- Click analytics Insights in the menu at the top.
- Click Alignment in the menu on the left.
What the reviewer alignment shows
As you can see in the screenshot below, the following data is provided per question in the reviewer alignment insights:
The alignment of the grading by different reviewers
The top left hand corner, indicates if the alignment for this question is Good (shown in green) or if there is deviation detected (shown in yellow). The calculation of the deviation is explained at the bottom of this article.
The number of results
The overall number of results for the question is displayed in the top bar. In the reviewer breakdown you can see how many results each reviewer has graded per question.
The number of minutes spent reviewing the question
The total amount of minutes it took all reviewers to grade a question is displayed in the top bar. In the reviewer breakdown you can see how many minutes each reviewer took to grade this question.
The average number of points given
The average number of points given per question is displayed in the top bar. In the reviewer breakdown you can see how many points each reviewer has awarded for that question.
The icon warningwill display next to the reviewers name if that reviewer has deviated more than 10% from the normalised average score for that question.
How we calculate the reviewer alignment
Ans shows whether the reviewers are aligned. There are two possible data fields:
- Good alignment
- Deviation detected
The determination of the quality of the alignment depends on the average score, on the average points given by a grader and by a deviation threshold. The following formula is used:
((average_question_score - grader_average) / points) > DEVIATION_THRESHOLD
We use a DEVIATION_THRESHOLD of 0.2
average_question_score = 1.0
grader_average for a reviewer is 1.5
points = 5
(1.0 - 1.5) / 5 = 0.1 which is smaller than 0.2 so there is no deviation detected. In this case, the field shows 'Good alignment'.