Questions insights

You can use different filters to reduce the overview of questions. You can use multiple filters at the same time, for example a combination of contribution filters and quality filters. After you have selected a filter, you have the option to clear all filters by making use of the option 'Clear filters'. If you have used filters before on the question insights of an assignment, you can also restore the last used filters by clicking 'Restore last used filters'. The following options are available to filter questions:

• Contribution: Show questions with a contribution status: contributes, bonus, full points or disabled. You can select one contribution status.
• Quality: Show questions with a quality status: easy, difficult, needs attention or good. You can select one quality status.
• P-value, Rit-value and Rir-value: Show questions with a value between two numbers. For each value you can select a lower and upper boundary. All questions with a value in between the boundaries will be visible.

The option to change the question contribution is on the top of each question insight. In the dropdown menu, four options are shown:

1. Contributes: This is the default option; the question contributes to the grades of the participants.
2. Bonus: The question will not be taken into account in the maximum score that can be gained, but it can be a bonus for the participants who answer correctly.
3. Full points: Every participants is awarded the maximum amount of points possible for this question.
4. Disabled: The question and the points awarded for the question will be taken out of the assignment and will not be taken into account for the grading.

If you select another contribution type in the dropdown menu, the predicted consequences of this edit will be displayed. You can check the consequences and click Confirm to edit the contribution.

In the questions insights, each question is analysed individually. The following values are calculated by Ans: Advice on question quality, P-value, Rit-value, Rir-value, p', Rk , Guess score and average duration.

The visibility is applicable depending on the question type:

For multiple choice questions, Ans also calculates the a-value, Rat-value and Rar-value. As these values tell something about the answer alternatives, these values are explained in grading scheme breakdown part of this article.  

The meaning of the other values is as follows:

• Advice on question quality
  

  For every question, based on the P- and Rit-values, one of the following conclusions is made:

  • Good: All values are within the appropriate ranges.
  • Easy: The P-value is above 0.9.
  • Difficult: The P-value is below 0.1.
  • Needs attention: The Rit-value is below 0.2.
  
  
• P-value: Rate of difficulty
  The P-value represents the difficulty of the question. It shows how many participants correctly answered the question on a scale from 0 to 1. The ideal value depends on the goal of the assignment. For example, if the assignment is the only assignment in the course, the aim could be to have a P-value of 0.8. A lower P-value is acceptable as well, for example in case it's a formative assignment which is part of a series of assignments. The P-value is calculated as follows:
  P = ([average score of the participants] / [total points for the question]) / [total number of participants]

Ans chose this calculation as individual answer alternatives can contain a negative score. Therefore, the score of the question needs to be taken into account.

• Rit-value: Relation of the question to all questions in the test
  The Rit-value indicates how well the question correlates with the test; the higher, the better. Rit stands for Relation Item Test. For the Rit-value, Ans is using Pearsons correlation coefficient. This is a statistical method to determine the correlation between two values. The Rit-value plots the question score against the total score of the test. As with all values, the interpretation depends on your assessment goal. An indication of the value interpretation is given below:
  • A value greater than 0,40 is very good.
  • A value between 0,30-0,39 is good, but the question still has room for some improvement.
  • A value between 0,20-0,29 is mediocre.
  • Any value lower than 0,19 indicates that reviewing the question should be considered. 

• Rir-value: Relation of the question to all other questions (so all questions except this one)
  This value measures the same as the Rit-value, but the Rir-value is more exact, as it is not compared to itself. As with the Rit-value, the Pearsons correlation coefficient is used to calculate the Rir-value. This time, the question score is plotted against the total score of the test, minus the score of this question. An indication of the value interpretation is given below:
  • A value higher than 0,40 is very good.
  • A value between 0,30-0,39 is good, but the question still has room for some improvement.
  • A value between 0,20-0,29 is mediocre.
  • Any value lower than 0,19 indicates that reviewing the question should be considered.

• p': Rate of difficulty corrected for the guess correction
  The corrected P-value (p'-value) represents the difficulty of the question as well, however, it corrects the value for the statistical possibility of guessing the right answer. The value is scaled from -1 to 1. If the p'-value is close to 1, the question was too easy and it didn't separate participants based on their performance. For extremely low p'-values, the question was likely to be difficult. The p'-value is calculated as follows: 

p' = P - [ (1-P) / (number of options for this multiple choice question - 1) ]

• Rk: Corrected guess correction
  Rk is the corrected guess effect, where the choices from others are taken into consideration in order to judge such information more accurately. In a question with, for example, four alternatives and 1 correct answer, there's a one in four, or 25% chance, the answer can be guessed correctly. This is translated into a guess score (see below). The Rk value also takes the actual results of the participants into account. In the example of four possible alternatives, there are three incorrect alternatives (distractors). Ideally, the distribution of incorrect answers of all participants is evenly divided over the distractors (roughly). The Rk value takes this distribution into account. The Rk value is calculated as follows:
  
  • In the example to calculate the Rk value, we use a multiple choice question with 1 correct answer out of 4 possible answers. The following distribution is applicable:
    • Answer A (correct answer): 40 answers
    • Answer B (distractor): 22 answers
    • Answer C (distractor): 3 answers
    • Answer D (distractor): 14 answers
    • The total amount of answers: 79
  • Calculate the number of incorrect answers: 22+3+14 = 39.
  • Determine the ideal (evenly) distribution of the incorrect answers per distractor: 39 divided over three distractors is 13 answers per distractor.
  • Determine per distractor the amount of participants that need to be 'moved' to get the ideal distribution: (22-13) + (14-13) = 9 + 1 = 10 answers need to be 'moved'.
  • The ration 10 / 39 on three distractors is (10/39 * 3) = 0,77 alternatives.
  • The amount of correct alternatives is 4 - 0,77 = 3,23 alternatives.
  • The Rk value is 1 / 3,23 = 0,31 (or 31%). The original guess correction was 25%. The closer the Rk value and the guess score are, the better the ideal distribution of the distractors.
    
    
• Guess score: The score that can be statistically scored by guessing the answer
  For all closed-ended questions where the answer can be guessed, a guess score is calculated. The overview of applicable question types and the calculation of the guess score for different situations can be found in the guess correction article.
  
  
• Average duration: The average duration of all participants for this question
  This value represents the time taken to answer the question.

error_outline In a 'randomised per participant' assignment, not all participants will receive the same questions. The Rit- and Rir-values indicate a correlation between the questions and the final result of an assignment. Because not every participant is given the same questions, this correlation cannot be calculated.

Ans provides the possibility to show differences in the number of participants, P-value, Rit-value and Rir-value for different groups. Groups are mostly used for the taking of a test, but it is also possible to create groups after the taking of an assignment. The group comparison can be used to see how different groups perform compared with each other.

By viewing the rank correlation insights of an assignment, you can see if the question correlates well compared to the whole assignment. For the rank correlation, we sort participants of the assignment by their final mark and divide them into five groups. Ans shows the average score of each group for that specific question.

If the line in the graph is roughly horizontal, it means that the question was answered just as well by the overall higher-scoring participants as by the overall lower-scoring participants.

If the line is more ascending, it means that the overall higher scoring participants answered this question better than the overall lower scoring participants, as you would expect for most questions.

• Y-axis: Normalised average score
  The normalised average score contains the score of the question on a scale from 0 to 1.

• X-axis: Quintile group
  For the quintile groups, the participants are sorted by their final mark for the assignment and divided into five groups. Quintile 1 contains the 20% of participants with the lowest marks and quintile 5 contains the 20% of participants with the highest marks.

For each question, the breakdown of the answers of the participants or from the review of the teacher is given. The representation depends on the question type and chosen grading method. All possible combinations are shown below.

Multiple choice
For each answer alternative and for the 'no alternative chosen', Ans shows the number of participants that selected the alternative. Also the a-value, Rat-value and Rar-value are visible.

• a-value: The number of participants that selected the answer alternative, on a scale of 0 to 1.
• Rat-value: Correlation of the answer alternative versus the test score. Ans uses Pearsons correlation coefficient and plots the candidates that chose this answer alternative against the test score.
• Rar-value: Correlation of the answer alternative versus the test score, except this answer alternative. Ans uses Pearsons correlation coefficient and plots the candidates that chose this answer alternative against the test score minus the score of this alternative.

Open, file upload, code editor, drawing, worksheet and mathematical equation questions (if manual grading is used)
Points per criterion: Per criterion, the number of participants that received the points is shown.
Rubric with levels: Per rubric level, the number of participants that received the points of the level is shown. 
Slider: No breakdown is shown.

Points per criterion:

Rubric with levels:

Fill-in question
Ans shows for each unique answer given the number of participants that gave that answer. Additionally, it is possible to approve additional answer alternatives via the grading scheme option.

Match & statement questions
For each possible combination of rows and columns, Ans shows the number of participants that gave that answer combination.

Hotspot, Hotspot match, Order, Numerical and mathematical equation questions (if automatic grading is used)
No breakdown is shown.