With questions insights, you have the possibility to check the quality of your questions based on statistical metrics. By making use of the insights per question you can determine followup actions if necessary. The insights can substantiate a decision to change the contribution of a question or to change the grading scheme.
To view insights into the questions of your assignment, follow the steps below.
 Click the domain School name in the menu on the left.
 Click label_importantCourses in the menu at the top.
 Select your course or use the search bar.
 Select your assignment or use the search bar.
 Click analyticsInsights in the menu at the top.
 Click Questions in the menu on the left.
Firstly, at the top of the page, you have the option to filter the question insights. Subsequently, the following elements are included in the analysis per question:
 Changing the question contribution
 Question analytics
 Group comparison
 Rank correlation
 Multiple choice or grading scheme breakdown
Filtering options
You can use different filters to reduce the overview of questions. You can use multiple filters at the same time, for example a combination of contribution filters and quality filters. After you have selected a filter, you have the option to clear all filters by making use of the option 'Clear filters'. If you have used filters before on the question insights of an assignment, you can also restore the last used filters by clicking 'Restore last used filters'. The following options are available to filter questions:
 Contribution: Show questions with a contribution status: contributes, bonus, full points or disabled. You can select one contribution status.
 Quality: Show questions with a quality status: easy, difficult, needs attention or good. You can select one quality status.
 Pvalue, Ritvalue and Rirvalue: Show questions with a value between two numbers. For each value you can select a lower and upper boundary. All questions with a value in between the boundaries will be visible.
Changing the question contribution
The option to change the question contribution is on the top of each question insight. In the dropdown menu, four options are shown:
 Contributes: This is the default option; the question contributes to the grades of the participants.
 Bonus: The question will not be taken into account in the maximum score that can be gained, but it can be a bonus for the participants who answer correctly. The guess score of these questions is still taken into account for the guess score correction.
 Full points: Every participants is awarded the maximum amount of points possible for this question. The guess score of these questions is not taken into account for the guess score correction.
 Disabled: The question and the points awarded for the question will be taken out of the assignment and will not be taken into account for the grading. The guess score of these questions is not taken into account for the guess score correction.
If you select another contribution type in the dropdown menu, the predicted consequences of this edit will be displayed. You can check the consequences and click Confirm to edit the contribution.
Question analytics
In the questions insights, each question is analysed individually. The following values are calculated by Ans: Advice on question quality, Pvalue, Ritvalue, Rirvalue, p', Rk , Guess score and average duration.
The visibility is applicable depending on the question type:
Guess correction possible for question  No guess correction possible for question  
Advice on question quality  Yes  Yes 
Pvalue  Yes  Yes 
Ritvalue  Yes  Yes 
Rirvalue  Yes  Yes 
p'  Yes  No 
Rk  Yes  No 
Guess score  Yes  No 
Average duration  Yes  Yes 
For multiple choice questions, Ans also calculates the avalue, Ratvalue and Rarvalue. As these values tell something about the answer alternatives, these values are explained in grading scheme breakdown part of this article.
The meaning of the other values is as follows:

Advice on question quality
For every question, based on the P and Ritvalues, one of the following conclusions is made: Good: All values are within the appropriate ranges (Pvalue is between 0.10.9 and Ritvalue is above 0.2).
 Easy: The Pvalue is above 0.9.
 Difficult: The Pvalue is below 0.1.
 Needs attention: The Ritvalue is below 0.2.

Pvalue: Rate of difficulty
The Pvalue represents the difficulty of the question. Without the presence of negative points it shows how many participants correctly answered the question on a scale from 0 to 1. The ideal value depends on the goal of the assignment. For example, if the assignment is the only assignment in the course, the aim could be to have a Pvalue of 0.8. A lower Pvalue is acceptable as well, for example in case it's a formative assignment which is part of a series of assignments. The Pvalue is calculated as follows:
P = ([total score of the participants] / [total number of participants]) / [total points for the question]
With the presence of negative points it is possible for the p value to be negative, if most students scored negatively. In this case the scale will range from 1 to 1.
Ans chose this calculation as individual answer alternatives can contain a negative score. Therefore, the score of the question needs to be taken into account.

Ritvalue: Relation of the question to all questions in the test
The Ritvalue indicates how well the question correlates with the test; the higher, the better. Rit stands for Relation Item Test. For the Ritvalue, Ans is using Pearsons correlation coefficient. This is a statistical method to determine the correlation between two values. The Ritvalue plots the question score against the total score of the test. As with all values, the interpretation depends on your assessment goal. An indication of the value interpretation is given below: A value greater than 0,40 is very good.
 A value between 0,300,39 is good, but the question still has room for some improvement.
 A value between 0,200,29 is mediocre.
 Any value lower than 0,19 indicates that reviewing the question should be considered.

Rirvalue: Relation of the question to all other questions (so all questions except this one)
This value measures the same as the Ritvalue, but the Rirvalue is more exact, as it is not compared to itself. As with the Ritvalue, the Pearsons correlation coefficient is used to calculate the Rirvalue. This time, the question score is plotted against the total score of the test, minus the score of this question. An indication of the value interpretation is given below: A value higher than 0,40 is very good.
 A value between 0,300,39 is good, but the question still has room for some improvement.
 A value between 0,200,29 is mediocre.
 Any value lower than 0,19 indicates that reviewing the question should be considered.

p': Rate of difficulty corrected for the guess correction
The corrected Pvalue (p'value) represents the difficulty of the question as well, however, it corrects the value for the statistical possibility of guessing the right answer. The value is scaled from 1 to 1. If the p'value is close to 1, the question was too easy and it didn't separate participants based on their performance. For extremely low p'values, the question was likely to be difficult. The p'value is calculated as follows:
p' = P  [ (1P) / (number of options for this multiple choice question  1) ]

Rk: Corrected guess correction
Rk is the corrected guess effect, where the choices from others are taken into consideration in order to judge such information more accurately. In a question with, for example, four alternatives and 1 correct answer, there's a one in four, or 25% chance, the answer can be guessed correctly. This is translated into a guess score (see below). The Rk value also takes the actual results of the participants into account. In the example of four possible alternatives, there are three incorrect alternatives (distractors). Ideally, the distribution of incorrect answers of all participants is evenly divided over the distractors (roughly). The Rk value takes this distribution into account. The Rk value is calculated as follows:
 In the example to calculate the Rk value, we use a multiple choice question with 1 correct answer out of 4 possible answers. The following distribution is applicable:
 Answer A (correct answer): 40 answers
 Answer B (distractor): 22 answers
 Answer C (distractor): 3 answers
 Answer D (distractor): 14 answers
 The total amount of answers: 79
 Calculate the number of incorrect answers: 22+3+14 = 39.
 Determine the ideal (evenly) distribution of the incorrect answers per distractor: 39 divided over three distractors is 13 answers per distractor.
 Determine per distractor the amount of participants that need to be 'moved' to get the ideal distribution: (2213) + (1413) = 9 + 1 = 10 answers need to be 'moved'.
 The ration 10 / 39 on three distractors is (10/39 * 3) = 0,77 alternatives.
 The amount of correct alternatives is 4  0,77 = 3,23 alternatives.
 The Rk value is 1 / 3,23 = 0,31 (or 31%). The original guess correction was 25%. The closer the Rk value and the guess score are, the better the ideal distribution of the distractors.
 In the example to calculate the Rk value, we use a multiple choice question with 1 correct answer out of 4 possible answers. The following distribution is applicable:

Guess score: The score that can be statistically scored by guessing the answer
For all closedended questions where the answer can be guessed, a guess score is calculated. The overview of applicable question types and the calculation of the guess score for different situations can be found in the guess correction article.

Average duration: The average duration of all participants for this question
This value represents the time taken to answer the question.
error_outline In a 'randomised per participant' assignment, not all participants will receive the same questions. The Rit and Rirvalues indicate a correlation between the questions and the final result of an assignment. Even though not every participant is given the same questions, this correlation will be calculated.
Group comparison
Ans provides the possibility to show differences in the number of participants, Pvalue, Ritvalue and Rirvalue for different groups. Groups are mostly used for the taking of a test, but it is also possible to create groups after the taking of an assignment. The group comparison can be used to see how different groups perform compared with each other.
Rank correlation
By viewing the rank correlation insights of an assignment, you can see if the question correlates well compared to the whole assignment. For the rank correlation, we sort participants of the assignment by their final mark and divide them into five groups. Ans shows the average score of each group for that specific question.
If the line in the graph is roughly horizontal, it means that the question was answered just as well by the overall higherscoring participants as by the overall lowerscoring participants.
If the line is more ascending, it means that the overall higher scoring participants answered this question better than the overall lower scoring participants, as you would expect for most questions.

Yaxis: Normalised average score
The normalised average score contains the score of the question on a scale from 0 to 1.

Xaxis: Quintile group
For the quintile groups, the participants are sorted by their final mark for the assignment and divided into five groups. Quintile 1 contains the 20% of participants with the lowest marks and quintile 5 contains the 20% of participants with the highest marks.
Grading scheme breakdown
For each question, the breakdown of the answers of the participants or from the review of the teacher is given. The representation depends on the question type and chosen grading method. All possible combinations are shown below.
Multiple choice
For each answer alternative and for the 'no alternative chosen', Ans shows the number of participants that selected the alternative. Also the avalue, Ratvalue and Rarvalue are visible.
 avalue: The number of participants that selected the answer alternative, on a scale of 0 to 1.
 Ratvalue: Correlation of the answer alternative versus the test score. Ans uses Pearsons correlation coefficient and plots the candidates that chose this answer alternative against the test score.
 Rarvalue: Correlation of the answer alternative versus the test score, except this answer alternative. Ans uses Pearsons correlation coefficient and plots the candidates that chose this answer alternative against the test score minus the score of this alternative.
Open, file upload, code editor, drawing, worksheet and mathematical equation questions (if manual grading is used)
Points per criterion: Per criterion, the number of participants that received the points is shown.
Rubric with levels: Per rubric level, the number of participants that received the points of the level is shown.
Slider: No breakdown is shown.
Points per criterion:
Rubric with levels:
Fillin question
Ans shows for each unique answer given the number of participants that gave that answer. Additionally, it is possible to approve additional answer alternatives via the grading scheme option.
Match & statement questions
For each possible combination of rows and columns, Ans shows the number of participants that gave that answer combination.
Hotspot, Hotspot match, Order, Numerical and mathematical equation questions (if automatic grading is used)
No breakdown is shown.
error_outline Updating statistics is a resourceintensive task, so it might take some time for the latest data to become visible.
Comments
0 comments
Article is closed for comments.