Posted by Austin Fossey
The Questionmark OData API, which offers flexible access to data for the creation of custom reports, can help you ensure the quality of your tests.
For instance, you can use OData to create a frequency table of item keys in a multiple choice assessment. This report shows the number of items that have the first choice as the correct answer, the number of items that have the second choice as the correct answer, et cetera.
Why do we care about how often each choice number is the item key? If there is a pattern in how correct choices are assigned, it may affect how participants perform on the test, and this can lead to construct-irrelevant variance; i.e., the scores are being affected by factors other than the participant’s knowledge and abilities.
Let’s say that our assessment has 50 items, and on 30 (60%) of those items the second choice is the item key. Now let’s put ourselves in the shoes of a qualified participant. Halfway through the assessment, we might start thinking, “Gosh, I just picked the second choice four times in a row. Maybe I should go back and check some of those answers.” Because of poor test design, we are second-guessing our answers. Even if we do not change our responses, time is being wasted and test anxiety is rising, which might negatively affect our responses later in the assessment.
The opposite problem may arise too. If an unqualified participant figures out that the second choice is most often the key, he or she may pick the second choice even when he or she does not know the answer, resulting in an inflated score.
When looking at the distribution of keys across a selected response assessment, we expect to see an even distribution of the keys across the choices. For example, if we have a multiple choice assessment with four choices in each item labeled A, B, C, and D, we would like to see the 25% of the keys assigned to each of these choices.
You do not have to limit your assessment research to this example. The beauty of OData is that you can access your data whenever you have a new question you would like to investigate. For example, instead of reviewing the frequencies of your keys, you may want to determine the ratio of the length of the key to the length of the other options (a common item writing mistake is to write keys that are noticeably longer than the distractors). You may also want to look for patterns in the keys’ text (e.g., 10 items all have “OData” as the correct choice).
Click here for more information about the OData API. (If you are a Questionmark software support plan customer, you can get step-by-step instructions for using OData to create the item key frequency table in the Premium Videos section of the Questionmark Learning Café.)