When the taking session of a digital test is complete or when a written assignment has been uploaded into Ans, you can start reviewing. Keep in mind, that in order to review an assignment or digital test, there needs to be a grading scheme. The grading scheme is dependent on the question type. For open-ended questions, you can make use of grading methods: points per criterion, slider or rubrics with levels.
Reviewing can be can divided between instructors and reviewers. Instructors are assigned to all exercises and all groups by default. Reviewers are only assigned to all groups by default. You can learn how to assign reviewers to certain groups an/or exercises here.
To navigate to the review menu, follow the steps below.
- Click the domain School name in the menu on the left.
- Click on label_importantCourses in the menu at the top.
- Click on your course or use the search bar.
- Click on your assignment or use the search bar.
- Click content_paste_search Review in the menu at the top.
You will be directed to the review overview. From here, you can start reviewing.
On the left side of the review overview, the amount of results (number of participants) and the amount of exercises included in the assignment is displayed. If you are an instructor of the course, you will see the button how_to_reg. If you only have a 'reviewer' role in the course, this button will not be visible. If you click on this button, you can assign reviewers to help distribute the workload. Alternatively assigning reviewers can be done from the settings Settings menu at the top.
If you prefer to review based on group(s) or class(es), you can make use of the filters at the top of the page. If you begin reviewing when a filter is active, Ans will only show submissions of the participants included in the filtered group(s) and/or class(es). The blue progress bar shows how far the review progress is.
On the right side of the review overview, you will see the Review button and a dropdown menu. The dropdown menu allows you to choose how you would like to review the submissions of the participants.
Review per participant
To view a submission of a participant as a whole, you can choose to review per participant by clicking the Review button. Keep in mind, If all submissions have been reviewed, the Review button will be greyed out.
If you click on the dropdown menu right next to the Review button, you may see the following options:
-
- Review sequentially: To view and adjust all existing submissions in the order of when the participants started the digital test. For written assignments, the order is based on when the result ID was scanned.
- Review randomly: To view and adjust all existing submissions in a random order. This option is only clickable in case (a part of) the assignment has not been reviewed at all.
- Review discussions: Show all discussions that need a review.
- View closed discussions: Show all closed discussions of participants.
- View resolved flags: Show all flags that have been resolved.
- Review flags: Show all open flags that need a review.
- Resolve all flags: Automatically resolve all flags.
Review per exercise
It is also possible to review a submission per question. By doing so, you will review one exercise for all participants before navigating to the next exercise. The same options are available for Review per exercise as they are for Review per participant (see above). In addition, you have the possibility to view either all subquestions or a selection of subquestions. As an example, in the screenshot below, Ans will only show subquestion b.
It is important to note, that multiple reviewers cannot grade the same submission. Once a reviewer is grading a submission, it is automatically locked for other reviewers. If multiple reviewers are grading at the same time, Ans automatically distributes them over different submissions.
When you begin reviewing submissions, you will be directed to the following screen. Please note that the screen differs based on the question type, assignment type and the chosen grading scheme.
Top menu
- The arrow_back-icon: Clicking on this will be direct you back to the Review overview.
- The assignment name: Clicking on this will direct you back to the Results overview.
- beenhere # 942690: This indicates the result number. This is different from the student number.
- The keyboard_arrow_left x/y keyboard_arrow_right-icon: switch between questions or participants (depending if you chose to Review per exercise or Review per participant).
- Review: Here you can review the assignment and the participants answer.
- Annotations: You can add annotations to the answer of a participant.
- Discussion: If this option is enabled, participants can start a discussion. The messages of the participant of this question will be displayed.
- Flags: Add, edit or delete flags. Flags are often used in cases where you need a second opinion from another reviewer on how to grade the participant's answer. A comment can be added to the flag.
- The settings-icon: This option allows you to edit the grading settings. The grading settings are personal settings and do not apply for other graders. The following grading settings are available:
- Display exercises
- Display questions
- Display model answer
- Display objectives
Bottom menu
- The arrow_back-icon: Navigate back to the previous question or participant, depending if you chose to Review per participant or Review per exercise.
- Question buttons: Click the button to go directly to that question.
- The arrow_forward-icon: Navigate to the next question or participant.
lightbulb_outline Did you know you can use keyboard shortcuts to grade faster? Use the following keys:
- Arrow pointed left: Previous question.
- Arrow pointed right: Next question.
- Enter: Next question.
- B: Previous participant.
- N: Next participant.
The reviewing menu differs per grading scheme. For open-ended questions, there are three possible options: points per criterion, slider or rubrics with levels.
Points per criteria
Points per criteria is the default grading type. All criteria is uncoloured when reviewing begins. This is the unsaved state. If a criterion has been met, click the criterion that applies and it will turn green. The system will automatically save your changes. If no criteria is met, click None of the above.
When grading with a slider or rubrics with levels, you have the option to click Undo grading. This will set your grading for this question back to zero points. It is also possible to select the criteria with the help of your keyboard. Every criteria has a number. Press '1' to enabled the first criteria, '2' to enable the second criteria and so on. It is possible to use your keyboard to review a question with criteria:
- 1, 2, 3, ... : toggle criteria.
- Spacebar: toggle all criteria.
- 0, ยง (Apple), ~ (Windows): toggle "None of the above".
- f: view flags.
Information on comments, adjustments and the grading history are explained below, as they also apply to the other grading scheme types.
Rubrics with levels
The second grading type is the rubrics with levels. If you use a rubric, you have the possibility to define levels (for example insufficient - sufficient - good), include a description and add points to each level. You can select one level when grading. This is the main difference between points per criteria and rubrics with levels, as for criteria you can select multiple (or all) criteria.
Slider
The last grading type option is the slider option. With the slider option, you have the possibility to either fill in the amount of points or use the slider to award the number of points. If you want to use decimals, use the point to separate the whole number from the fractional part of the number.
If you need to deviate from the grading scheme, you can use the option to make adjustments to question points to either award or deduct points of a question. Adjustments to question points are possible for all question types. This option is possible if 'Adjustments to question points' is enabled in the review options tab in the settings of your assignment.
To make adjustments to question points, follow the steps below.
- Navigate to the question you want to review/make adjustments.
- You will first need to select the amount of points before you can add an adjustment.
- Click the tab +Adjustments.
- Fill in the adjustment (either positive or negative), add a comment and click Save.
- To make another adjustment to the points for that question, click the tab +Adjustments.
- Fill in the amount of points, add a comment and click Save.
A useful way to leave feedback for open questions, is to make use of annotations.
To add an annotation, follow the steps below.
- Navigate to the open question that you want to annotate.
- Select text in the answer of the open question to annotate it.
- The highlighted text will turn yellow and the Annotations-tab will open automatically.
- Click on the yellow highlighted text and add your comment.
- Click Save to add the comment. The annotation will then turn purple.
- If you do not wish to save or add a comment, click Back to overview to continue reviewing without adding a comment to the annotation.
- Highlighted text can be deleted by clicking on it in the answer box. The highlight will be selected and you can click on the close-icon that appears at the top-right of the highlight.
Add a comment
A helpful way to provide feedback for participants while grading is to leave a comment under the awarded points. Adding comments is possible for all open-ended question types.
To add a comment, follow the steps below.
- Navigate to the question you want to grade and leave a comment on.
- Grade the question accordingly and then click +Comments. The exact position of this button depends on the chosen grading type.
- A text editor will open where you can add your comment.
- Fill in the comment or select a Quick comment
- Click Save
Use comment suggestions
To save time while reviewing, you have the option to reuse the same feedback that you gave to a previous participant. When you have added a comment to a criterion and you are giving a comment to a different participant for the same criterion, Ans will show you the three most used comments you have given for that specific criterion. Your suggested comments will be displayed under the text editor and can be used by clicking on them.
Use quick comments
If you often use the same comments, you can save them as quick comments. This way you just need to click on quick comments and select the comment you want to use. In every place where you can add comments, it is possible to save comments for later use or reuse previously given comments.
In order to do so, click on the + Comment button. In the screen that pops up, you can click on Quick comments. You can show and choose an existing quick comment by pressing Show quick comments. It's also possible to store text as quick comment by pressing Save as quick comment. A comment always has a name and the content of the comment:
lightbulb_outline You can also type a hashtag (#) with the name of a saved quick comment in the comment field. Ans will directly show the comment after pressing enter.
After participants have submitted an assignment and the results have been published, they have the option to start a discussion (if enabled).
To review discussions from participants, follow the steps below.
- Navigate to the content_paste_search Review overview.
- If a discussion has been started by a participant, a message will be displayed with the number of messages. Under the Review per participant, the total number of discussions is shown. For each question, the number of discussions for that question is shown.
- Click the arrow_drop_down-icon and select Review discussions.
- Click Discussions in the menu at the top.
- Fill in your reaction and then:
- Save as draft: This function will allow you to save the response as draft (like a concept e-mail) so the response can be altered after reading and responding to all questions so all participants will be treated equally. All draft messages can be sent later on in one click all together. Only the author of a draft message, whether it's a reviewer or an instructor, can send it. Administrators are also unable to send draft messages created by instructors or reviewers. This is to prevent unintended delivery of draft messages to students.
- Send: This function will send the response directly to the participant. The message cannot be altered later on.
- After sending a reply to a message, the discussion will be closed automatically. A discussion can be closed without responding to the message by clicking Close. Closing a discussion means that both the participant and instructor cannot send messages anymore. If necessary, a closed discussion can be opened again by an instructor.
In your personal settings, you can enable receiving an email regarding discussions between you and the student. Click on your user icon at the top right of your screen, click on 'Settings' and go to the submenu 'Emails' to enable it.
Reviewers can flag answers if they are uncertain about the number of points to award the participant for a specific question. Instructors can then take a look at the flagged answer to give a decisive answer. It is also possible to solve all flags for a specific question or all flags at once.
Flag an answer
If you are reviewing a test and would like to flag an answer, follow the steps below.
- Click on the Flag tab within the exercise you are reviewing.
- Click the +Flag button
- Add your comment and click Save.
- You can also click the more_vert-icon to edit or delete your flag.
View and resolve flags
If you would like to review unresolved flags, follow the steps below.
- From the content_paste_search Review overview of your assignment, click the dropdown menu arrow_drop_down-icon and select Review flags. This option only appears in the dropdown when the exercise has been flagged.
- Check the flag, and re-grade the answer if needed.
- Click Flag in the menu at the top and Reply or click Resolve.
- When more flags need to be reviewed, click Next unresolved flag.
Comments
0 comments
Please sign in to leave a comment.