Analysing the alignment of reviewers can provide different insights in regards to an assignment after it has been graded. Using this feature achieves fair and transparent grading.
To view insights into the alignment of reviewers, follow the steps below.
- Click the domain School name in the menu on the left.
- Click label_importantCourses in the menu at the top.
- Select your course or use the search bar.
- Select your assignment or use the search bar.
- Click analytics Insights in the menu at the top.
- Click Alignment in the menu on the left.
What the reviewer alignment shows
As you can see in the screenshot below, the following data is provided per question in the reviewer alignment insights:
- The alignment of the grading by different reviewers (the determination of this field is shown below)
- The number of results
- The total number of minutes spent reviewing
- The average number of points given
- A breakdown of the results, duration and average points given per reviewer
How we calculate the reviewer alignment
Ans shows whether the reviewers are aligned good or not. There are two possible data fields possible:
- Good alignment
- Deviation detected
The determination of the quality of the alignment depends on the average score, on the average points given by a grader and by a deviation threshold. The following formula is used:
((average_question_score - grader_average) / points) > DEVIATION_THRESHOLD
We use a DEVIATION_THRESHOLD of 0.2
average_question_score = 1.0
grader_average for a reviewer is 1.5
points = 5
(1.0 - 1.5) / 5 = 0.1 which is smaller than 0.2 so there is no deviation detected. In this case, the field shows 'Good alignment'.