Posted by John Kleeman
What makes a good diagnostic question?
First, it should be almost impossible for someone to get the right answer for the wrong reason: A participant will only get the question right if he/she has the right idea about whatever the instructor wants them to be able to know, understand or do.
Second, wrong answers should be interpretable: If a participant chooses a particular wrong response, the instructor should be able to guess why the person has done so and what misconception he/she has.
So suggests Dr. Dylan William in his excellent new book, Embedded Formative Assessment (published by Solution Tree Press, and recommended). A common use for diagnostic questions is to find out whether participants have understood your instruction – telling you whether you can go onto another topic or need to spend more time on this topic. And if participants get it wrong, you want to understand why they have done so in order to correct the misconception. Good diagnostic questions involve deep domain knowledge, as you have to understand why learners are likely to answer in a particular way.
One tactic for creating diagnostic questions is to look at answers that students give in open-ended questions and choose common ones as distractors in multiple choice questions.
Here is an example of a multiple response diagnostic question quoted in the book:

Example question

There are 64 possible answers to the question; the right answer is B. and D. It’s pretty unlikely that someone who does not understand Pythagoras’ rule will get the question right, and if they get it wrong, there will be good clues as to why.
Questions like this can be hinge-points in instruction – they give you evidence as to what you do next. Do you need to continue instruction and practice on this topic, or can you move on to the next topic?