Skip to main content

Alignment insights

Brooke Henshall
  • Updated

Analysing the alignment of reviewers can provide different insights in regards to an assignment after it has been graded. Using this feature achieves fair and transparent grading.

To view insights into the alignment of reviewers, follow the steps below.

  1. Click the domain  School name in the menu on the left.
  2. Click label_importantCourses in the menu at the top.
  3. Select your course or use the search bar. 
  4. Select your assignment or use the search bar. 
  5. Click analytics Insights in the menu at the top.
  6. Click Alignment in the menu on the left.


What the reviewer alignment shows

As you can see in the screenshot below, the following data is provided per question in the reviewer alignment insights:

  • The alignment of the grading by different reviewers
    The top left hand corner, indicates if the alignment for this question is Good (shown in green) or if there is deviation detected (shown in yellow). The calculation of the deviation is explained at the bottom of this article.
  • The number of results
    The overall number of results for the question is displayed in the top bar. In the reviewer breakdown you can see how many results each reviewer has graded per question.

  • The number of minutes spent reviewing the question
    The total amount of minutes it took all reviewers to grade a question is displayed in the top bar. In the reviewer breakdown you can see how many minutes each reviewer took to grade this question.
  • The average number of points given
    The average number of points given per question is displayed in the top bar. In the reviewer breakdown you can see how many points each reviewer has awarded for that question.

  • Deviation detected
    The icon warningwill display next to the reviewers name if that reviewer has deviated more than 10% from the normalised average score for that question.

Screen_Shot_2022-07-14_at_13.22.20.png

How we calculate the reviewer alignment

Ans shows whether the reviewers are aligned. There are two possible data fields:

  • Good alignment
  • Deviation detected

The determination of the quality of the alignment depends on the average score, on the average points given by a grader and by a deviation threshold. The following formula is used:

((average_question_score - grader_average) / points) > DEVIATION_THRESHOLD

We use a DEVIATION_THRESHOLD of 0.2

For example:

average_question_score = 1.0
grader_average for a reviewer is 1.5
points = 5
(1.0 - 1.5) / 5 = 0.1 which is smaller than 0.2 so there is no deviation detected. In this case, the field shows 'Good alignment'. 

Related articles

Comments

0 comments

Article is closed for comments.