Start reviewing

Review overview

On the left side of the review overview, the amount of results (number of participants) and the amount of exercises included in the assignment is displayed. If you are an instructor of the course, you will see the button how_to_reg.If you only have a 'reviewer' role in the course, this button will not be visible. If you click on this button, you can assign reviewers to help distribute the workload. Alternatively assigning reviewers can be done from the settings Settings menu at the top.

At the top of the page, you can make use of the three filters. The 'Grading progress' filter allows you to filter result on 'All questions', 'Graded questions' or 'Ungraded questions'. If you prefer to review based on group(s) or class(es), you can use of these two filters. If you begin reviewing when a filter is active, Ans will only show submissions of the participants included in the filtered grading progress and group(s) and/or class(es). 

On the right side of the review overview, you will see the Review button and a dropdown menu. The dropdown menu allows you to choose how you would like to review the submissions of the participants.

Review per result

To view a submission of a participant as a whole, you can choose to review per result by clicking the Review button or Continue reviewing button. 

To view the grading progress of all participants, you can hover over the purple grading progress bar.

Review per exercise

It is also possible to review a submission per exercise. By doing so, you will review one exercise for all participants before navigating to the next exercise. The same options three filters apply for Review per exercise as they are for Review per result. In addition, you have the possibility to view either all subquestions or a selection of subquestions. 

To view the grading progress of an exercise, you can hover over the purple grading progress bar.

It is important to note, that multiple reviewers cannot grade the same submission. Once a reviewer is grading a submission, it is automatically locked for other reviewers. If multiple reviewers are grading at the same time, Ans automatically distributes them over different submissions. However, there is an exception when another user accesses a student's answer through the student's result overview or a shared link. This exception is necessary for situations where teachers need to discuss a result together.

Reviewing menu

When you begin reviewing submissions, you will be directed to the following screen. Please note that the screen differs based on the question type, assignment type and the chosen grading scheme.

When reviewing a question, the scroll will focus on the given answer. This could mean, for longer questions, you may need to scroll up to view the entire question. The reasoning behind scrolling to the answer is that as a teacher/grader you probably know the question, and want to focus on reading the students reply. The question you are reviewing is shown on the rights, so if you don't know the question, you can check on the right side of the screen.

When reviewing a question, the view will automatically scroll to the given answer. For longer questions, this might require you to scroll up to see the entire question. The reasoning behind this approach is that, as a teacher or grader, you're likely already familiar with the question and want to focus on the student’s response. The exercise name and question are also displayed on the right-hand side (above the grading scheme) of the screen for easy reference.

Top menu

• The arrow_back-icon: Clicking on this will direct you back to the Review overview.
• The assignment name: The assignment name is displayed.
• beenhere #945261: This indicates the result number. This is not the same as the student number.
• The keyboard_arrow_left x/y keyboard_arrow_right-icon: switch between questions or participants (depending if you chose to Review per result or Review per exercise). When viewing the answers of an individual result, you can easily move between all the attempts using the arrows. For peer-reviewed digital tests, you can navigate between all the peer reviews from a specific result.
• Review:
  • Grading scheme: Here, you can grade the participant's answer or make adjustments if needed.
  • Flags: Add, edit or delete flags. Flags are often used in cases where you need a second opinion from another reviewer on how to grade the participant's answer. A comment can be added to the flag.
  • Discussion: If this option is enabled, participants can start a discussion. The messages of the participant of this question will be displayed. 
• Annotations: You can add annotations to the answer of a participant.
• The search -icon: By using the search function, you can  find previously viewed results. Results can be filtered according to grading progress, such as fully graded, partially graded, or not graded. In instances where grading is not conducted anonymously, a search for a specific student is also possible. 
• The settings-icon: This option allows you to edit the grading settings. The grading settings are personal settings and do not apply for other graders. The following grading settings can be enabled:
  • Display exercise
  • Display question
  • Display model answer
  • Display objectives

lightbulb_outline The results are sorted first by the date they were created and then randomised within each day. Suppose there are 3 results: 1 and 2 started yesterday and 3 started today, 1 and 2 are randomly ordered and 3 will always be the third.

lightbulb_outline For open questions in digital tests, it is possible to navigate through the saved answers by using the navigation buttons of the timestamp just below the question.

error_outline The steps in this dropdown refer to the previous grading experience. The new grading experience is outlined in the next dropdown. There is currently no set timeline for when the new grading experience will become the default. Feedback from users will help determine when the transition will occur.

Reviewing questions using the previous grading experience

The reviewing menu differs per grading scheme. For open-ended questions, there are three possible options:

• Points per criterion
• Rubrics with levels
• Slider

Adjustments can be made to to the grading types points per criterion and rubrics with levels. This is explained below.

Points per criterion

Points per criterion is the default grading type. All criteria are uncoloured (grey) when reviewing begins, indicating an unsaved state. If a criterion has been met, click the applicable criterion, and it will turn green. Your changes will be automatically saved. If no criteria is met, click None of the above.

When grading with points per criterion, you can make use of the following shortcuts:

• 1, 2, 3, ... : (un)toggle criteria
• Spacebar: (un)toggle all criteria (not available in the new grading experience)
• 0, § (Apple), ~ (Windows): (un)toggle "None of the above"
  
  

Rubrics with levels

When using the grading type 'rubrics with levels,' you can define levels (e.g., insufficient, sufficient, good), include descriptions, and assign points to each level. You can select one level per criterion when grading. This is the key difference between points per criterion and rubrics with levels, as with points per criterion, you can select multiple (or all) criteria.

When grading with rubrics with levels, you can make use of the following shortcuts:

• 1, 2, 3, ... : to select a level
  
  

Slider

With the grading type 'slider,' you can either manually enter the number of points or use the slider to assign points. To include decimals, use a period to separate the whole number from the fractional part.

When grading with slider, you can make use of the following shortcuts:

• 1, 2, 3, ... : to fill in the points
  
  

Adjustments to the grading scheme

If you need to deviate from the grading scheme, you can use the option to adjust question points to either award or deduct points for a question. Adjustments to question points are possible for all question types, provided the 'Adjustments to question points' option is enabled in the Review Options tab within your assignment settings.

When making adjustments, you must first select the number of points before applying the adjustment. The adjustment is then applied afterward. This calculation method is designed to provide maximum flexibility, allowing students to be awarded extra points that exceed the question's original point value.

This flexibility enables you to give students a point for their effort, even if they scored only on negative criteria, or to reward exceptionally good answers with points above the maximum.

To make adjustments to question points, follow the steps below.

1. Navigate to the question you want to review/make adjustments.
2. You will first need to select the amount of points before you can add an adjustment.
3. Click the tab +Adjustment.
4. Fill in the adjustment (either positive or negative), add a comment and click Save.
5. To make another adjustment to the points for that question, click the tab +Adjustment.
6. Fill in the amount of points, add a comment and click Save.

Reviewing questions using the new grading experience

The new grading experience is available for questions that are graded using the grading types 'points per criterion' or 'rubric with levels'. Therefore, this dropdown does not explain how to review questions using the grading type 'slider'. Please refer to the previous dropdown for this information.

You can choose when to switch to the new version by clicking on ✨ Try the new grading scheme experience at the bottom of the grading scheme in the review screen. The new grading scheme can be tried out at any time. In case you want to return to the previous version, click on Switch back to the previous grading scheme experience at the bottom of the grading scheme in the review screen. 

In the new grading scheme experience, the total awarded points are now always visible, as the total is sticky when scrolling down. The colour of the score also changes based on whether the question is ungraded, fully graded, or partially graded. This colour matches the question tiles on the left of the review screen, providing consistent visual feedback.

Points per criterion

The criteria in the 'points per criterion' grading type are showcased on the right of the screen. All criteria are uncoloured when you start with reviewing, indicating it is ungraded. If a criterion has been met, click the applicable criterion, and it will turn green. Your changes are automatically saved. If no criteria is met, click None of the above. 

In the new grading experience, it is possible to toggle the description of criteria by clicking on the unfold_less-icon. The title is a truncated version of the description. To customise this, a title can be added or edited directly in the criterion settings. Collapsing criteria reduces the space used by the grading scheme and allows more room for flags or discussions. To undo this, click on the unfold_more-icon. 

When grading with points per criterion, and you have enabled shortcuts in your accessibility settings, you can make use of the following keyboard shortcuts:

• 1 to 9: (un)toggle criteria
• Q, W, E, R, T, Y: (un)toggle criteria, starting from the 10th criterion
• 0, § (Apple), ~ (Windows): (un)toggle "None of the above"
• Shift + C: (un)toggle criteria descriptions

If criteria groups are used, they will also appear during grading. You can move criteria between groups or remove them from a group using the dropdown menu. Grading is considered complete when at least one criterion is selected, regardless of whether groups exist.

Rubric with levels

When using the grading type 'rubric with levels', you can define levels (e.g. insufficient, sufficient, good), include descriptions, and assign points to each level. You can select one level/criterion per question when grading. This is the key difference between points per criterion and rubrics with levels, as with points per criterion, you can select multiple (or all) criteria.

In the new grading experience, it is possible to toggle the description of criteria by clicking on the unfold_less-icon. When collapsed, only the level title is shown. Collapsing criteria reduces the space used by the grading scheme and allows more room for flags or discussions. To undo this, click on the unfold_more-icon. 

When grading with rubrics with levels, and you have enabled shortcuts in your accessibility settings, you can make use of the following shortcuts:

• 1 to 9: (un)toggle criteria
• Q, W, E, R, T, Y: (un)toggle criteria, starting from the 10th criterion
• Shift + C: (un)toggle criteria descriptions

If criteria groups are used, you can move criteria between groups or remove them from a group using the dropdown menu. Grading is complete when one criterion is selected for each set. Criteria outside of groups are treated as their own set.

Edit the grading scheme while grading

In case the grading scheme needs to be adjusted during the review process, such as changing how points are awarded of deducted, it is possible to do edit the grading scheme directly while grading with the new grading experience. Descriptions can be changed, and criteria can be added, copied, or deleted without leaving the review screen.

To adjust the grading scheme, follow the steps below.

1. Navigate to the question you want to review
2. Click on the more_vert-icon next to the criterion name.
3. Click on Edit. 
4. Adjust the fields that you want to edit. 

To add criteria to the grading scheme, follow the steps below.

1. Click on + Criterion.
2. Add the Points the criterion is worth.
3. Add the Name of the criterion (optional for the grading type 'points per criterion'). 
4. Add the description of the Evaluation criterion. 
5. Click on Save. 

To copy a criterion, follow the steps below.

1. Navigate to the question you want to review
2. Click on the more_vert-icon next to the criterion name.
3. Click on Copy. 
4. The criterion is copied and added at the bottom of the grading scheme. Adjust the necessary fields.
5. Click on Save.

To delete a criterion, follow the steps below.

1. Navigate to the question you want to review
2. Click on the more_vert-icon next to the criterion name.
3. Click on Delete. 
4. Click on Confirm. 

Deleting a criterion from the review screen will also remove the criterion from the grading scheme. Already graded results will also be impacted by this.

All changes are reflected in the grading scheme of the assignment. If the exercise is linked to a question bank exercise, you can synchronise the question bank exercise with the adjusted version in your course assignment. Refer to this article for more information.

Adjustments to the grading scheme

If you need to deviate from the grading scheme, you can use the option to adjust question points to either award or deduct points for a question. Adjustments to question points are possible for all question types, provided the 'Adjustments to question points' option is enabled in the Review Options tab within your assignment settings.

When making adjustments, you must first select the number of points before applying the adjustment. The adjustment is then applied afterward. This calculation method is designed to provide maximum flexibility, allowing students to be awarded extra points that exceed the question's original point value.

This flexibility enables you to give students a point for their effort, even if they scored only on negative criteria, or to reward exceptionally good answers with points above the maximum.

To make adjustments to question points, follow the steps below.

1. Navigate to the question you want to review/make adjustments.
2. Click on + Adjustment.
3. Fill in the number of Points (this can be positive or negative points). 
4. Add a Comment. 
5. Click on Save. 

After creating an adjustment, it is saved and can be used when you are reviewing other results in the assignment. Recent adjustments are shown when adding a new one, making it easier to grade consistently and efficiently. When reusing adjustments, both the comment and point value are applied.

Provide feedback using annotations

A useful way to leave feedback for open questions is by using annotations. When adding annotations, you can highlight text in a student's answer, and a tooltip will appear. From there, you can add a comment, change the highlight colour, or delete the highlight. The name of the reviewer that created the annotation is never visible to a student. 

To make use of annotations, follow the steps below.

1. Navigate to the open question you want to annotate.
2. Select text in the student's answer that you want to annotate.
3. The highlighted text will turn yellow, and the annotations panel will open automatically.
4. To add a comment, click the highlighted text so the tooltip appears. Click on the rate_review-icon, and the comment box will open in the annotations panel. Click Save.
5. To delete or edit your comment, click the more_vert-icon in the annotations panel, then click Edit or Delete.
6. To change the colour of the highlight, click the highlighted text so the tooltip appears. Click on the coloured circle, then select the desired colour.
7.  To delete the highlight, click the highlighted text so the tooltip appears, then click the delete-icon. 

Add a comment (using the previous grading experience)

A helpful way to provide feedback for participants while grading is to leave a comment under the awarded points. Adding comments is possible for all open-ended question types.

To add a comment, follow the steps below.

1. Navigate to the question you want to grade and leave a comment on.
2. Grade the question accordingly and then click +Comments. The exact position of this button depends on the chosen grading type.
3. A text editor will open where you can add your comment.
4. Fill in the comment or select a Quick comment.
5. Click Save.

Add a comment (using the new grading experience)

To add a comment in the new grading experience, follow the steps below.

1. Hover over the criterion you want to comment on.
2. Click on Add comment. 
3. A text editor will open where you can add your comment.
4. Fill in the comment or open the rich text editor by clicking the text_format-icon to use a quick comment.
5. Click on Save.

Use comment suggestions

To save time while reviewing, you have the option to reuse the same feedback that you gave to a previous participant. When you have added a comment to a criterion and you are giving a comment to a different participant for the same criterion, Ans will show you the three most used comments you have given for that specific criterion. Your suggested comments will be displayed under the text editor and can be used by clicking on them.

For the previous grading experience:

For the new grading experience:

Use quick comments (using the previous grading experience)

If you often use the same comments, you can save them as quick comments. This way you just need to click on quick comments and select the comment you want to use. In every place where you can add comments, it is possible to save comments for later use or reuse previously given comments.

In order to do so, click on the + Comment button. In the screen that pops up, you can click on Quick comments. You can show and choose an existing quick comment by clicking Show quick comments. It's also possible to store text as quick comment by clicking Save as quick comment. A comment always has a name and the content of the comment:

Use quick comments (using the new grading experience)

To create quick comments in the new grading experience, follow the steps below.

1. Hover over the criterion you want to use the quick comment on.
2. Click on Add comment.
3. Write the comment you want to save as quick comment.
4. Click on the text_format-icon to open the text editor.
5. Click on Quick comments.
6. Click on Save as quick comment.
7. Enter a name for the quick comment
8. Click on Save

To view your quick comments, click on Quick comments in the text editor, and then Show quick comments.

lightbulb_outline You can also type a hashtag (#) with the name of a saved quick comment in the comment field. Ans will directly show the comment after pressing enter.

Discussions

After participants have submitted an assignment and the results have been published, they have the option to start a discussion (if enabled with the publication of the assignment). More information about using discussions can be found in this article.

Flags

Reviewers can flag answers if they are uncertain about the number of points to award the participant for a specific question. Instructors can then take a look at the flagged answer to give a decisive answer. More information about using flags can be found in this article.

Grading filter

With the grading filter, it is possible to filter results directly from the grading screen. An icon in the grading navigation bar opens a side panel called Filter results, allowing reviewers to refine which answers they want to see.

Applying filters
In the side panel, all questions with criteria are listed vertically along with their available options. Each criterion has two toggle buttons:

• Checkmark - include results where this criterion is selected
• Block - include results where this criterion is not selected

Navigating to matching results
After filters are applied, the number of matching results is displayed. Clicking Filter X Results navigates to the first match, and the remaining matches can be browsed using the standard navigation buttons.

Clearing filters
The view can easily be reset by clicking Clear filters in the side panel.