Start reviewing

On the left side of the review overview, the amount of results (number of participants) and the amount of exercises included in the assignment is displayed. If you are an instructor of the course, you will see the button how_to_reg. If you only have a 'reviewer' role in the course, this button will not be visible. If you click on this button, you can assign reviewers to help distribute the workload. Alternatively assigning reviewers can be done from the settings Settings menu at the top.

At the top of the page, you can make use of the three filters. The 'Grading progress' filter allows you to filter result on 'All questions', 'Graded questions' or 'Ungraded questions'. If you prefer to review based on group(s) or class(es), you can use of these two filters. If you begin reviewing when a filter is active, Ans will only show submissions of the participants included in the filtered grading progress and group(s) and/or class(es). 

On the right side of the review overview, you will see the Review button and a dropdown menu. The dropdown menu allows you to choose how you would like to review the submissions of the participants.

Review per result

To view a submission of a participant as a whole, you can choose to review per result by clicking the Review button or Continue reviewing button. 

To view the grading progress of all participants, you can hover over the purple grading progress bar.

play_circle_outline  Check out how to review per result on our  YouTube channel.

Review per exercise

It is also possible to review a submission per exercise. By doing so, you will review one exercise for all participants before navigating to the next exercise. The same options three filters apply for Review per exercise as they are for Review per result. In addition, you have the possibility to view either all subquestions or a selection of subquestions. 

To view the grading progress of an exercise, you can hover over the purple grading progress bar.

It is important to note, that multiple reviewers cannot grade the same submission. Once a reviewer is grading a submission, it is automatically locked for other reviewers. If multiple reviewers are grading at the same time, Ans automatically distributes them over different submissions. However, there is an exception when another user accesses a student's answer through the student's result overview or a shared link. This exception is necessary for situations where teachers need to discuss a result together.

play_circle_outline  Check out how to review per exercise on our  YouTube channel.

When you begin reviewing submissions, you will be directed to the following screen. Please note that the screen differs based on the question type, assignment type and the chosen grading scheme.

Top menu

• The arrow_back-icon: Clicking on this will direct you back to the Review overview.
• The assignment name: The assignment name is displayed.
• beenhere # 942690: This indicates the result number. This is not the same as the student number.
• The keyboard_arrow_left x/y keyboard_arrow_right-icon: switch between questions or participants (depending if you chose to Review per result or Review per exercise). When viewing the answers of an individual result, you can easily move between all the attempts using the arrows. For peer-reviewed digital tests, you can navigate between all the peer reviews from a specific result.
• Review: Here you can grade the participant's answer.
• Annotations: You can add annotations to the answer of a participant.
• Discussion: If this option is enabled, participants can start a discussion. The messages of the participant of this question will be displayed. 
• Flags: Add, edit or delete flags. Flags are often used in cases where you need a second opinion from another reviewer on how to grade the participant's answer. A comment can be added to the flag.
• The search -icon: By using the search function, you can  find previously viewed results. Results can be filtered according to grading progress, such as fully graded, partially graded, or not graded. In instances where grading is not conducted anonymously, a search for a specific student is also possible. 
• The settings-icon: This option allows you to edit the grading settings. The grading settings are personal settings and do not apply for other graders. The following grading settings can be enabled:
  • Display exercise
  • Display question
  • Display model answer
  • Display objectives

The reviewing menu differs per grading scheme. For open-ended questions, there are three possible options: points per criterion, slider or rubrics with levels.

Points per criteria

Points per criteria is the default grading type. All criteria is uncoloured (grey) when reviewing begins. This is the unsaved state. If a criterion has been met, click the criterion that applies and it will turn green. The system will automatically save your changes. If no criteria is met, click None of the above.

When grading with a slider or rubrics with levels, you have the option to click Undo grading. This will set your grading for this question back to zero points. It is also possible to select the criteria with the help of your keyboard. Every criteria has a number. Press '1' to enabled the first criteria, '2' to enable the second criteria and so on. It is possible to use your keyboard to review a question with criteria:

• 1, 2, 3, ... : toggle criteria.
• Spacebar: toggle all criteria.
• 0, § (Apple), ~ (Windows): toggle "None of the above".
• f: view flags.

Information on comments and adjustments are explained below, as they also apply to the other grading scheme types.

Rubrics with levels

The second grading type is the rubrics with levels. If you use a rubric, you have the possibility to define levels (for example insufficient - sufficient - good), include a description and add points to each level. You can select one level when grading. This is the main difference between points per criteria and rubrics with levels, as for criteria you can select multiple (or all) criteria.

Slider

The last grading type option is the slider option. With the slider option, you have the possibility to either fill in the amount of points or use the slider to award the number of points. If you want to use decimals, use the point to separate the whole number from the fractional part of the number.

Adjustments to the grading scheme

If you need to deviate from the grading scheme, you can use the option to make adjustments to question points to either award or deduct points of a question. Adjustments to question points are possible for all question types. This option is possible if 'Adjustments to question points' is enabled in the review options tab in the settings of your assignment.

To make adjustments to question points, follow the steps below.

1. Navigate to the question you want to review/make adjustments.
2. You will first need to select the amount of points before you can add an adjustment.
3. Click the tab +Adjustment.
4. Fill in the adjustment (either positive or negative), add a comment and click Save.
5. To make another adjustment to the points for that question, click the tab +Adjustment.
6. Fill in the amount of points, add a comment and click Save.

A useful way to leave feedback for open questions, is to make use of annotations.

To add an annotation, follow the steps below.

1. Navigate to the open question that you want to annotate.
2. Select text in the answer of the open question to annotate it.
3. The highlighted text will turn yellow and the Annotations-tab will open automatically.
4. Click on the yellow highlighted text and add your comment. 
5. Click Save to add the comment. The annotation will then turn purple.
6. If you do not wish to save or add a comment, click Back to overview or Cancel to continue reviewing without adding a comment to the annotation.  
7. Highlighted text can be deleted by clicking on it in the answer box. The highlight will be selected and you can click on the close-icon that appears at the top-right of the highlight.

Use comment suggestions

To save time while reviewing, you have the option to reuse the same feedback that you gave to a previous participant. When you have added a comment to a criterion and you are giving a comment to a different participant for the same criterion, Ans will show you the three most used comments you have given for that specific criterion. Your suggested comments will be displayed under the text editor and can be used by clicking on them.

Use quick comments

If you often use the same comments, you can save them as quick comments. This way you just need to click on quick comments and select the comment you want to use. In every place where you can add comments, it is possible to save comments for later use or reuse previously given comments.

In order to do so, click on the + Comment button. In the screen that pops up, you can click on Quick comments. You can show and choose an existing quick comment by pressing Show quick comments. It's also possible to store text as quick comment by pressing Save as quick comment. A comment always has a name and the content of the comment:

lightbulb_outline You can also type a hashtag (#) with the name of a saved quick comment in the comment field. Ans will directly show the comment after pressing enter. 

After participants have submitted an assignment and the results have been published, they have the option to start a discussion (if enabled with the publication of the assignment).

To review discussions from participants, follow the steps below.

1. Navigate to thecontent_paste_search Review overview.
2. If a discussion has been started by a participant, a message will be displayed with the number of messages. Under the Review per result, the total number of discussions is shown. For each question, the number of discussions for that question is shown.
   
3. Click on # open discussions and an open discussion will be opened. 
4. Fill in your reaction and then: 
   • arrow_drop_down-icone and Save as Draft: This function will allow you to save the response as draft (like a concept e-mail) so the response can be altered after reading and responding to all questions so all participants will be treated equally. All draft messages can be sent later on in one click all together. Only the author of a draft message, whether it's a reviewer or an instructor, can send it. Administrators are also unable to send draft messages created by instructors or reviewers. This is to prevent unintended delivery of draft messages to students.
   • Send: This function will send the response directly to the participant. The message cannot be altered later on.
5. After sending a reply to a message, the discussion will be closed automatically. A discussion can be closed without responding to the message by clicking Close. Closing a discussion means that both the participant and instructor cannot send messages anymore. If necessary, a closed discussion can be opened again by an instructor. 

In your personal settings, you can enable receiving an email regarding discussions between you and the student. Click on your user icon at the top right of your screen, click on 'Settings' and go to the submenu Communication to enable it. 

Reviewers can flag answers if they are uncertain about the number of points to award the participant for a specific question. Instructors can then take a look at the flagged answer to give a decisive answer. It is also possible to solve all flags for a specific question or all flags at once. 

Flag an answer

If you are reviewing a test and would like to flag an answer, follow the steps below.

1. Click on the Flags tab within the exercise you are reviewing.
2. Click the +Flag button
3. Add your comment and click Save.
4. You can also click the more_vert-icon to edit or delete your flag.  

View and resolve unresolved flags

If you would like to review unresolved flags, follow the steps below.

1. From the content_paste_search Review overview of your assignment, click # unresolved flags. This option only appears in the dropdown when the exercise has been flagged.
2. Check the flag, and re-grade the answer if needed.  
3. Click Flags in the menu at the top and Reply or click Resolve.
4. When more flags need to be reviewed, click Next Flag.

It is possible for multiple reviewers to resolve a flag simultaneously. We do not lock a submission when a reviewer is addressing an unresolved flag. A submission is only locked when it is still ungraded.