Automatic scoring is an option which can be enabled for a question where students can select multiple answers. The option allows you to select which answers are correct, instead of providing scores per answer alternative. The score for the student is calculated based on the fraction of correct options selected and the fraction of incorrect options selected. The automatic scoring option can only be applied to the following question types:
- Match question
- Multiple choice question
- Order question
- Fill-in question
- Hotspot match question
For all question types mentioned above, an example of automatic scoring is provided at the bottom of the article.
To enable automatic scoring, follow the steps below.
- Click the domain School name in the menu on the left.
- Click label_importantCourses in the menu at the top.
- Select your course or use the search bar.
- Select your assignment or use the search bar.
- Navigate to the question that you want to change the question type of.
- Click the more_horiz-icon and click Edit.
- Click the more_vert-icon and click Use automatic scoring.
- Click Save.
lightbulb_outline It is possible that the automatic scoring option is enabled as default by your administrator for all new questions that are created. In that case, you don't have to enable the option yourself.
The automatic scoring option prevents a student from receiving points for a question when the student selects all possible answers. In case automatic scoring is enabled, Ans also takes the fraction of incorrect answers selected into account to calculate the score for a student. In case you don't want to provide full points when a student selects all possible answers, you can use the automatic scoring option to prevent this from happening. If automatic scoring is enabled, the score for a student that selects all answers for a question where multiple answers can be selected should be (approximately) equal to the guess score of a question.
For questions without the automatic scoring option enabled, it is possible to determine the number of points per answer alternative. For questions with automatic scoring enabled, you can only select which answers are correct and set the total amount of points for the question. In the figures below, you can see the difference for a multiple choice question answer alternative.
Automatic scoring disabled: enter the number of points to be earned for the answer alternative.
Automatic scoring enabled: select whether an answer alternative is correct or not.
To calculate the score Ans uses a formula. The formula uses the total amount of correct and incorrect options. For the multiple-choice-, match- and order question types, it's possible to determine the amount of correct and incorrect options. For these question types, the formula is used.
For the fill-in and the hotspot match question type, the amount of incorrect options is not known. Therefore, Ans applies the automatic scoring option in a different way for the fill-in and the hotspot match question type.
As mentioned above, the formula is used for the multiple-choice-, match- and order question types. Ans uses a formula defined by E. Terwisscha van Scheltinga (2009)*.
The formula is as follows:
- Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for the question
Where:
- proportion correct = number of correct options selected / number of correct options
- proportion incorrect = number of incorrect options selected / number of incorrect options
- deduction factor = 1/a + ((N-C)/N)*(1-1/a)
- N = total number of options
- C = number of correct options
- a = weight factor (set standard to 2.0)
For the fill-in and the hotspot match question types, automatic scoring is applied in a different way. The total amount of incorrect options are not known for these question types. Because this is not known, the result of the formula cannot be calculated.
Fill-in question
For the fill-in question type, Ans evenly distributes the number of points over the different gaps. In case the question is worth 2 points and there are 2 gaps in the question, both gaps are worth 1 point.
Hotspot match question
For the hotspot match question type, Ans awards the points evenly for every marker that is placed correctly. In case the question is worth 4 points and has 4 markers, a student will receive 1 point per marker which is placed correctly.
For both question types, an extended example can be found below.
*E. Terwisscha van Scheltinga (2009). Van quotiëntregel naar productregel. EXAMENS, 1, 26-27.
Example: Multiple choice question
Automatic scoring can be used for multiple choice questions when multiple answers are correct. In case you use automatic scoring when only 1 answer is correct, the student will automatically get full points for the selected answer. In this example, a multiple choice question is used with four answer alternatives in total. Two answer alternatives are correct and two alternatives are incorrect. The total amount of points to be earned for the question is 2. In the example shown below, the student needs to select which cities are the capital cities of their country.London and Amsterdam are correct, Munich and Milan are incorrect.
In case automatic scoring is disabled, you insert the number of points a student earns when selecting an answer. In this example, London and Amsterdam are worth 1 point and Munich and Milan are worth 0 points. In case a student selects all four answer possibilities, Ans only takes the number of correct answers into consideration for the score calculation.
Example situation A:
When a student selects all 4 answers, the student did also select the 2 correct answers. Therefore, the student receives the maximum score of 2 points, despite having selected 2 incorrect answers as well.
Example situation B:
When a student selects 2 correct answers and 1 incorrect answer, the same applies. A student will get 2 points, as both correct answers are selected and the incorrect answer is not taken into consideration for calculating the score.
Example situation A:
When all 4 answers are selected, the following applies:
- proportion correct = 1 (2 correct options selected / 2 correct options)
- proportion incorrect = 1 (2 incorrect options selected / 2 incorrect options)
- deduction factor = 0,75 because:
- 1/a+((N-C)/N)*(1-1/a) where:
- N = 4 (total number of options)
- C = 2 (number of correct options)
- a = 2 (standard weight factor)
- 1/a+((N-C)/N)*(1-1/a) where:
- points for question = 2
- This results in:
- Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
- Score = (1 * (1-1*0,75))*2
- Score = 0,5
Example situation B:
When all 2 correct answers and 1 incorrect answer is selected, the following applies:
- proportion correct = 1 (2 correct options selected / 2 correct options)
- proportion incorrect = 0,5 (1 incorrect options selected / 2 incorrect options)
- deduction factor = 0,75 because:
- 1/a+((N-C)/N)*(1-1/a) where:
- N = 4 (total number of options)
- C = 2 (number of correct options)
- a = 2 (standard weight factor)
- 1/a+((N-C)/N)*(1-1/a) where:
- Points for question = 2
- This results in:
- Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
- Score = (1 * (1-0,5*0,75))*2
- Score = 1,25
Example: match question
The automatic scoring option can be enabled for the match question type in case multiple answers are selected under correct choices. The match question has 2 layout options: matrix & drag and drop. The automatic scoring can be used for both layouts. However, the determination of incorrect & correct answers works slightly different for drag and drop. In the example for the match question, we are asking the student to match capital cities with their countries. In this case, two countries need to be matched with two cities. The correct matches for this question are Belgium - Brussels and France - Paris. The student can earn a maximum of 2 points. The same question can be used for both layout types, as you can see in the screenshots below.
Example A: matrix layout
Example B: drag and drop layout
Example A: Matrix layout
If the matrix layout is chosen, the calculation works the same as for the multiple-choice question when multiple answers are correct. In case a student selects all four options, the two correct options are selected, therefore the student will receive the full 2 points.
Example B: Drag and drop layout
For the drag-and-drop layout, the calculation works slightly different. As the student has to drag the answer alternatives to the rows, there are fewer options that a student has compared to the matrix layout. The student will receive a point for each correctly placed answer alternative. Therefore, if a student places both Brussels and Paris under Belgium, the student will receive 1 point (as Ans only takes the correctly placed answer alternatives into consideration for the grade calculation).
Example A: Matrix layout
When automatic scoring is enabled for the matrix layout, the calculation goes the same as for the multiple-choice question. When all four answers are selected, the calculation is as follows:
- proportion correct = 1 (2 correct options selected / 2 correct options)
- proportion incorrect = 1 (2 incorrect options selected / 2 incorrect options)
- deduction factor = 0,75 because:
- 1/a+((N-C)/N)*(1-1/a) where:
- N = 4 (total number of options)
- C = 2 (number of correct options)
- a = 2 (standard weight factor)
- 1/a+((N-C)/N)*(1-1/a) where:
- points for question = 2
- This results in:
- Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
- Score = (1 * (1-1*0,75))*2
- Score = 0,5
Example B: Drag and drop layout
For each question that has automatic scoring enabled, both the amount of correct and incorrect options chosen are taken into account. The determination of these amounts is different for drag-and-drop layouts. When a student drags and drop an answer alternative, Ans determines both a correct and an incorrect option. In the example, the student can drag Brussels to Belgium. By doing this the student not only selects a correct match (Brussels-Belgium), but it also does not select an incorrect match (Brussels - France). The amount of correct and incorrect options necessary for the formula is still the same (in this example 2 correct and 2 incorrect options). The difference with the Matrix alternative is that the student will not have the possibility to select all four options, as for each match the student makes, one other possible match is not available anymore. In our example, it's possible that a student matches both Paris and Brussels to Belgium. In this case, the following formula is applied:
- proportion correct = 0,5 (1 correct option selected / 2 correct options)
- proportion incorrect = 0,5 (1 incorrect option selected / 2 incorrect options)
- deduction factor = 0,75 because:
- 1/a+((N-C)/N)*(1-1/a) where:
- N = 4 (total number of options)
- C = 2 (number of correct options)
- a = 2 (standard weight factor)
- 1/a+((N-C)/N)*(1-1/a) where:
- points for question = 2
- This results in:
- Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
- Score = (0,5 * (1-0,5*0,75))*2
- Score = 0,625
Example: Order question
For an order question type, the automatic scoring option can be enabled as well. To calculate the score for an order question, Ans checks the amount of correct 'pairs' of options. A correct pair is considered as an answer alternative that have been placed correctly compared to the other alternative of the pair. We have provided an example with four answer alternatives. The student needs to put the Dutch cities in the correct order, from North to South. The correct order is: Groningen - Utrecht - Eindhoven - Maastricht.
In total, there are 12 pairs that can be made. 6 out of 12 are correct and 6 out of 12 are incorrect. The overview has been placed in the table below.
| Correct pairs | Incorrect pairs |
| Groningen - Utrecht | Maastricht - Eindhoven |
| Groningen - Eindhoven | Maastricht - Utrecht |
| Groningen - Maastricht | Maastricht - Groningen |
| Utrecht - Eindhoven | Eindhoven - Utrecht |
| Utrecht - Maastricht | Eindhoven - Groningen |
| Eindhoven - Maastricht | Utrecht - Groningen |
If the automatic scoring is disabled, Ans will only take the correct pairs into consideration to calculate the score. To make the example easier, the amount of points that can be earned has been set to 3. As there are 6 correct alternatives, this means a student can get 0,5 points per correct pair.
Example
For this example, we assume that the answer of the student is:
- Eindhoven
- Maastricht
- Groningen
- Utrecht
In this case, the student has got 2 correct pairs out of the 6 possible correct pairs. Eindhoven - Maastricht and Groningen - Utrecht have been placed correctly. Therefore, the student receives 1 point in total (0,5 points per correct pair).
For the calculation of the score with automatic scoring, Ans takes the amount of correct and incorrect alternatives into account. For the order question, an alternative equal a pair. For the formula, some details are different compared to others. If we use the same example as above, the answer of the student is:
- Eindhoven
- Maastricht
- Groningen
- Utrecht
Using the automatic scoring formula, this answer results in:
- proportion correct = 0,33 (2 correct option selected / 6 correct options)
- Eindhoven-Maastricht and Groningen-Utrecht are selected
- proportion incorrect = 0,66 (4 incorrect option selected / 6 incorrect options)
- deduction factor = 0,75 because:
- 1/a+((N-C)/N)*(1-1/a) where:
- N = 12 (total number of options)
- C = 6 (number of correct options)
- a = 2 (standard weight factor)
- 1/a+((N-C)/N)*(1-1/a) where:
- points for question = 3
- This results in:
- Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
- Score = (0,33 * (1-0,66*0,75))*3
- Score = 0,50
Example: Fill-in & hotspot match question
For the fill-in and the hotspot match question, the automatic scoring rule can be applied. However, for these question types the number of incorrect answers cannot be calculated. Therefore, the formula as used in the other question types cannot be applied. Instead, the automatic scoring option will make sure that points will be awarded evenly for every possible correct answer.
For the fill-in question, the automatic scoring option will take a look at the amount of gaps that has been inserted in the question. In the example below, the student needs to fill-in three capital cities of Dutch provinces. The correct answers from top to bottom are: Groningen, Maastricht and Haarlem.
When the automatic scoring option is enabled, Ans will check how many gaps have been answered correctly. In case 2 out of 3 answers are answered correctly, the student receives 2/3 of the number of points (in this case 2 points). For the fill-in question type, it is possible to enter multiple answers that are correct. If all answers that are correct for a gap have an equal amount of points, automatic scoring can be enabled. However, if you for example would like to give half of the points for one of the answer alternatives, the automatic scoring option cannot be used. In this case, you need to disable the automatic scoring option and insert the number of points per answer alternative per gap.
For the hotspot match question, the number of incorrect answers cannot be determined as well. Therefore, the calculation in case automatic scoring is enabled is similar to the fill-in question. Ans checks all correctly placed markers and distributes the scores evenly. As an example, we use a question where the student needs to place three markers on the cities of Groningen, Maastricht and Haarlem on a map of the Netherlands.
If a student places 2 out of 3 markers correctly, the student will receive 2/3 of the number of points for the questions (in this case 2).
Comments
0 comments
Article is closed for comments.