Use automatic scoring for closed-ended questions

The automatic scoring option prevents a student from receiving points for a question when the student selects all possible answers. In case automatic scoring is enabled, Ans also takes the fraction of incorrect answers selected into account to calculate the score for a student. In case you don't want to provide full points when a student selects all possible answers, you can use the automatic scoring option to prevent this from happening. If automatic scoring is enabled, the score for a student that selects all answers for a question where multiple answers can be selected should be (approximately) equal to the guess score of a question. 

For questions without the automatic scoring option enabled, it is possible to determine the number of points per answer alternative. For questions with automatic scoring enabled, you can only select which answers are correct and set the total amount of points for the question. In the figures below, you can see the difference for a multiple choice question answer alternative.

Automatic scoring disabled: enter the number of points to be earned for the answer alternative.

Automatic scoring enabled: select whether an answer alternative is correct or not.

To calculate the score Ans uses a formula. The formula uses the total amount of correct and incorrect options. For the multiplechoice, match and order question types, it's possible to determine the amount of correct and incorrect options. For these question types, the formula is used.

For the fill-in and the hotspot match question type, the amount of incorrect options is not known. Therefore, Ans applies the automatic scoring option in a different way for the fill-in and the hotspot match question type.

As mentioned above, the formula is used for the multiple choice, match and order question types. Ans uses a formula defined by E. Terwisscha van Scheltinga (2009)*.

The formula is as follows:

• Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for the question

Where:

For the fill-in and the hotspot match question types, automatic scoring is applied in a different way. The total amount of incorrect options are not known for these question types. Because this is not known, the result of the formula cannot be calculated.

Fill-in question

For the fill-in question type, Ans evenly distributes the number of points over the different gaps. In case the question is worth 2 points and there are 2 gaps in the question, both gaps are worth 1 point.

Hotspot match question

For the hotspot match question type, Ans awards the points evenly for every marker that is placed correctly. In case the question is worth 4 points and has 4 markers, a student will receive 1 point per marker which is placed correctly.

For both question types, an extended example can be found below.

*E. Terwisscha van Scheltinga (2009). Van quotiëntregel naar productregel. EXAMENS, 1, 26-27.

In case automatic scoring is disabled, you enter the number of points a student earns when selecting an answer. In this example, London and Amsterdam are worth 1 point and Munich and Milan are worth 0 points. In case a student selects all four answer possibilities, Ans only takes the number of correct answers into consideration for the score calculation.

Example situation A:

When a student selects all 4 answers, the student did also select the 2 correct answers. Therefore, the student receives the maximum score of 2 points, despite having selected 2 incorrect answers as well.

Example situation B:

When a student selects 2 correct answers and 1 incorrect answer, the same applies. A student will receive 2 points, as both correct answers are selected and the incorrect answer is not taken into consideration for calculating the score.

Example situation A:

When all 4 answers are selected, the following applies:

• proportion correct = 1 (2 correct options selected / 2 correct options)
• proportion incorrect = 1 (2 incorrect options selected / 2 incorrect options)
• deduction factor = 0,75 because:
  • 1/a+((N-C)/N)*(1-1/a) where:
    • N = 4 (total number of options)
    • C = 2 (number of correct options)
    • a = 2 (standard weight factor)
• points for question = 2
• This results in:
  • Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
  • Score = (1 * (1-1*0,75))*2
  • Score = 0,5

If automatic scoring is disabled, Ans will only consider the correct pairs when calculating the score. For simplicity, the total points available have been set to 3. With 6 correct pairs, this means a student can earn 0.5 points for each correct pair.

Example

For this example, we assume that the answer of the student is:

• Eindhoven
• Maastricht
• Groningen
• Utrecht

In this scenario, the student has correctly matched 2 out of the 6 possible pairs. The pairs Eindhoven - Maastricht and Groningen - Utrecht have been placed correctly. As a result, the student receives 1 point in total (0.5 points per correct pair).

For the calculation of the score with automatic scoring, Ans takes the amount of correct and incorrect alternatives into account. For the order question, an alternative equal a pair. For the formula, some details are different compared to others. If we use the same example as above, the answer of the student is:

• Eindhoven
• Maastricht
• Groningen
• Utrecht

Using the automatic scoring formula, this answer results in:

• proportion correct = 0,33 (2 correct option selected / 6 correct options)
  • Eindhoven-Maastricht and Groningen-Utrecht are selected
• proportion incorrect = 0,66 (4 incorrect option selected / 6 incorrect options)
• deduction factor = 0,75 because:
  • 1/a+((N-C)/N)*(1-1/a) where:
    • N = 12 (total number of options)
    • C = 6 (number of correct options)
    • a = 2 (standard weight factor)
• points for question = 3
• This results in:
  • Score = (proportion correct * (1 - proportion incorrect * deduction factor))*points for question
  • Score = (0,33 * (1-0,66*0,75))*3
  • Score = 0,50

For the fill-in question, the automatic scoring option will take a look at the amount of gaps that has been inserted in the question. In the example below, the student needs to fill-in three capital cities of Dutch provinces. The correct answers from top to bottom are: Groningen, Maastricht and Haarlem.

When the automatic scoring option is enabled, Ans will check how many gaps have been answered correctly. In case 2 out of 3 answers are answered correctly, the student receives 2/3 of the number of points (in this case 2 points). For the fill-in question type, it is possible to enter multiple answers that are correct. If all answers that are correct for a gap have an equal amount of points, automatic scoring can be enabled. However, if you for example would like to give half of the points for one of the answer alternatives, the automatic scoring option cannot be used. In this case, you need to disable the automatic scoring option and insert the number of points per answer alternative per gap.

For the hotspot match question, the number of incorrect answers cannot be determined as well. Therefore, the calculation in case automatic scoring is enabled is similar to the fill-in question. Ans checks all correctly placed markers and distributes the scores evenly. As an example, we use a question where the student needs to place three markers on the cities of Groningen, Maastricht and Haarlem on a map of the Netherlands. 

If a student places 2 out of 3 markers correctly, the student will receive 2/3 of the number of points for the questions (in this case 2).