When the taking session of a digital test is complete or when a written assignment has been uploaded into Ans, you can start reviewing. Keep in mind, that in order to review an assignment or digital test, there needs to be a grading scheme. The grading scheme is dependent on the question type. For open-ended questions, you can make use of grading methods: points per criterion, slider or rubrics with levels.
Reviewing can be divided between instructors and reviewers. Instructors are assigned to all exercises and all groups by default. Reviewers are only assigned to all groups by default. You can learn how to assign reviewers to certain groups an/or exercises here.
To navigate to the review menu, follow the steps below.
- Navigate to your assignment.
- Click content_paste_search Review in the menu at the top.
You will be directed to the review overview. From here, you can start reviewing.
Review overview
On the left side of the review overview, the amount of results (number of participants) and the amount of exercises included in the assignment is displayed. If you are an instructor of the course, you will see the button how_to_reg. If you only have a 'reviewer' role in the course, this button will not be visible. If you click on this button, you can assign reviewers to help distribute the workload. Alternatively assigning reviewers can be done from the settings Settings menu at the top.
At the top of the page, you can make use of the three filters. The 'Grading progress' filter allows you to filter result on 'All questions', 'Graded questions' or 'Ungraded questions'. If you prefer to review based on group(s) or class(es), you can use of these two filters. If you begin reviewing when a filter is active, Ans will only show submissions of the participants included in the filtered grading progress and group(s) and/or class(es).
On the right side of the review overview, you will see the Review button and a dropdown menu. The dropdown menu allows you to choose how you would like to review the submissions of the participants.
Review per result
To view a submission of a participant as a whole, you can choose to review per result by clicking the Review button or Continue reviewing button.
To view the grading progress of all participants, you can hover over the purple grading progress bar.
Review per exercise
It is also possible to review a submission per exercise. By doing so, you will review one exercise for all participants before navigating to the next exercise. The same options three filters apply for Review per exercise as they are for Review per result. In addition, you have the possibility to view either all subquestions or a selection of subquestions.
To view the grading progress of an exercise, you can hover over the purple grading progress bar.
It is important to note, that multiple reviewers cannot grade the same submission. Once a reviewer is grading a submission, it is automatically locked for other reviewers. If multiple reviewers are grading at the same time, Ans automatically distributes them over different submissions. However, there is an exception when another user accesses a student's answer through the student's result overview or a shared link. This exception is necessary for situations where teachers need to discuss a result together.
Reviewing menu
When you begin reviewing submissions, you will be directed to the following screen. Please note that the screen differs based on the question type, assignment type and the chosen grading scheme.
When reviewing a question, the scroll will focus on the given answer. This could mean, for longer questions, you may need to scroll up to view the entire question. The reasoning behind scrolling to the answer is that as a teacher/grader you probably know the question, and want to focus on reading the students reply. The question you are reviewing is shown on the rights, so if you don't know the question, you can check on the right side of the screen.
When reviewing a question, the view will automatically scroll to the given answer. For longer questions, this might require you to scroll up to see the entire question. The reasoning behind this approach is that, as a teacher or grader, you're likely already familiar with the question and want to focus on the student’s response. The exercise name and question are also displayed on the right-hand side (above the grading scheme) of the screen for easy reference.
Top menu
- The arrow_back-icon: Clicking on this will direct you back to the Review overview.
- The assignment name: The assignment name is displayed.
- beenhere # 942690: This indicates the result number. This is not the same as the student number.
- The keyboard_arrow_left x/y keyboard_arrow_right-icon: switch between questions or participants (depending if you chose to Review per result or Review per exercise). When viewing the answers of an individual result, you can easily move between all the attempts using the arrows. For peer-reviewed digital tests, you can navigate between all the peer reviews from a specific result.
-
Review:
- Grading scheme: Here, you can grade the participant's answer or make adjustments if needed.
- Flags: Add, edit or delete flags. Flags are often used in cases where you need a second opinion from another reviewer on how to grade the participant's answer. A comment can be added to the flag.
- Discussion: If this option is enabled, participants can start a discussion. The messages of the participant of this question will be displayed.
- Annotations: You can add annotations to the answer of a participant.
- The search -icon: By using the search function, you can find previously viewed results. Results can be filtered according to grading progress, such as fully graded, partially graded, or not graded. In instances where grading is not conducted anonymously, a search for a specific student is also possible.
- The settings-icon: This option allows you to edit the grading settings. The grading settings are personal settings and do not apply for other graders. The following grading settings can be enabled:
- Display exercise
- Display question
- Display model answer
- Display objectives
lightbulb_outline The results are sorted first by the date they were created and then randomised within each day. Suppose there are 3 results: 1 and 2 started yesterday and 3 started today, 1 and 2 are randomly ordered and 3 will always be the third.
Side menu
- Question buttons: Click the button to go directly to that question. The auto_fix_high icon indicates that the question has been graded automatically.
You can use the following keyboard shortcuts to grade faster:
- Arrow pointed left / up: Previous question.
- Arrow pointed right / down: Next question.
- Enter: Next question.
- B: Previous result.
- N: Next result.
- S: Search.
- D: Opens discussions panel (only if a discussion has been created).
- F: Opens flags panel.
Bottom menu
- Previous Ungraded: Navigate to the previous ungraded question.
- Next Ungraded: Navigate to the next ungraded question.
- Previous Question: Navigate to the previous question (when all results have been graded).
- Next Question: Navigate to the next question (when all results have been graded).
- Previous Flag: Navigate to the previous flag.
- Next Flag: Navigate to the next flag.
- Previous Discussion: Navigate to the previous discussion.
- Next Discussion: Navigate to the next discussion.
lightbulb_outline For open questions in digital tests, it is possible to navigate through the saved answers by using the navigation buttons of the timestamp just below the question.
Reviewing questions
The reviewing menu differs per grading scheme. For open-ended questions, there are three possible options:
- Points per criterion
- Rubrics with levels
- Slider
Adjustments can be made to to the grading types points per criterion and rubrics with levels. This is explained below.
Points per criterion
Points per criterion is the default grading type. All criteria are uncoloured (grey) when reviewing begins, indicating an unsaved state. If a criterion has been met, click the applicable criterion, and it will turn green. Your changes will be automatically saved. If no criteria is met, click None of the above.
When grading with points per criterion, you can make use of the following shortcuts:
- 1, 2, 3, ... : (un)toggle criteria
- Spacebar: (un)toggle all criteria
-
0, § (Apple), ~ (Windows): (un)toggle "None of the above"
Rubrics with levels
When using the grading type 'rubrics with levels,' you can define levels (e.g., insufficient, sufficient, good), include descriptions, and assign points to each level. You can select one level per criterion when grading. This is the key difference between points per criterion and rubrics with levels, as with points per criterion, you can select multiple (or all) criteria.
When grading with rubrics with levels, you can make use of the following shortcuts:
-
1, 2, 3, ... : to select a level
Slider
With the grading type 'slider,' you can either manually enter the number of points or use the slider to assign points. To include decimals, use a period to separate the whole number from the fractional part.
When grading with slider, you can make use of the following shortcuts:
-
1, 2, 3, ... : to fill in the points
Adjustments to the grading scheme
If you need to deviate from the grading scheme, you can use the option to adjust question points to either award or deduct points for a question. Adjustments to question points are possible for all question types, provided the 'Adjustments to question points' option is enabled in the Review Options tab within your assignment settings.
When making adjustments, you must first select the number of points before applying the adjustment. The adjustment is then applied afterward. This calculation method is designed to provide maximum flexibility, allowing students to be awarded extra points that exceed the question's original point value.
This flexibility enables you to give students a point for their effort, even if they scored only on negative criteria, or to reward exceptionally good answers with points above the maximum.
To make adjustments to question points, follow the steps below.
- Navigate to the question you want to review/make adjustments.
- You will first need to select the amount of points before you can add an adjustment.
- Click the tab +Adjustment.
- Fill in the adjustment (either positive or negative), add a comment and click Save.
- To make another adjustment to the points for that question, click the tab +Adjustment.
- Fill in the amount of points, add a comment and click Save.
Provide feedback using annotations
A useful way to leave feedback for open questions is by using annotations. When adding annotations, you can highlight text in a student's answer, and a tooltip will appear. From there, you can add a comment, change the highlight colour, or delete the highlight.
To make use of annotations, follow the steps below.
- Navigate to the open question you want to annotate.
- Select text in the student's answer that you want to annotate.
- The highlighted text will turn yellow, and the annotations panel will open automatically.
- To add a comment, click the highlighted text so the tooltip appears. Click on the rate_review-icon, and the comment box will open in the annotations panel. Click Save.
- To delete or edit your comment, click the more_vert-icon in the annotations panel, then click Edit or Delete.
- To change the colour of the highlight, click the highlighted text so the tooltip appears. Click on the coloured circle, then select the desired colour.
- To delete the highlight, click the highlighted text so the tooltip appears, then click the delete-icon.
Provide feedback using comments
Add a comment
A helpful way to provide feedback for participants while grading is to leave a comment under the awarded points. Adding comments is possible for all open-ended question types.
To add a comment, follow the steps below.
- Navigate to the question you want to grade and leave a comment on.
- Grade the question accordingly and then click +Comments. The exact position of this button depends on the chosen grading type.
- A text editor will open where you can add your comment.
- Fill in the comment or select a Quick comment.
- Click Save.
Use comment suggestions
To save time while reviewing, you have the option to reuse the same feedback that you gave to a previous participant. When you have added a comment to a criterion and you are giving a comment to a different participant for the same criterion, Ans will show you the three most used comments you have given for that specific criterion. Your suggested comments will be displayed under the text editor and can be used by clicking on them.
Use quick comments
If you often use the same comments, you can save them as quick comments. This way you just need to click on quick comments and select the comment you want to use. In every place where you can add comments, it is possible to save comments for later use or reuse previously given comments.
In order to do so, click on the + Comment button. In the screen that pops up, you can click on Quick comments. You can show and choose an existing quick comment by clicking Show quick comments. It's also possible to store text as quick comment by clicking Save as quick comment. A comment always has a name and the content of the comment:
lightbulb_outline You can also type a hashtag (#) with the name of a saved quick comment in the comment field. Ans will directly show the comment after pressing enter.
Discussions
After participants have submitted an assignment and the results have been published, they have the option to start a discussion (if enabled with the publication of the assignment). More information about using flags can be found in this article.
Flags
Reviewers can flag answers if they are uncertain about the number of points to award the participant for a specific question. Instructors can then take a look at the flagged answer to give a decisive answer. More information about using flags can be found in this article.
Comments
0 comments
Please sign in to leave a comment.