Joan Phaup
Posted by Joan Phaup
Item and test analyses bring the most value when understood interpreted in an organizational context. Questionmark Analytics and Psychometrics Manager Greg Pope’s upcoming workshop on this subject will help participants make the most effective use of the valuable information they get from test and item analysis. The workshop will combine classical test theory with hands-on learning using Questionmark reporting tools to analyze exemplar assessments and test questions. Attendees are welcome to bring in their own item and test analysis reports to discuss during the session.
I spent a few minutes with Greg the other day, asking him for more details about this workshop, which will take place the morning of Tuesday, March 15th —  one of two workshops preceding the Questionmark 2011 Users Conference.

Greg Pope

Q: What value do organizations get from item analysis and test analysis reports?
A: Item and test analysis report provide invaluable psychometric information regarding the performance of assessments and the building blocks of assessments, items. Creating assessments that are composed of questions that all perform well benefits the organization funding the assessment program as well as the participant taking the assessment. The organization benefits by providing assessments that are valid and reliable (and therefore legally defensible) and potentially the organization is able to use fewer questions on assessments to get the same measurement power. Organizations and participants can have confidence that the scores that they obtain from the assessments reflect to a high degree what participants know and can do. Item and test analyses allow organizations to know which questions are performing well, which questions are not performing well, and most importantly, WHY.
Q: What are the challenges in using these reports effectively?
I think the main challenges center around a psychological barrier to entry. Many people feel anxiety at the thought of having to read and interpret something they have likely had little to no exposure to in their life. Psychometrics is a specialized area, to be sure, but to apply the basic foundations of it does not need to be akin to summiting Everest. I feel strongly that it is possible to give people the basic knowledge around item and test analysis in only a few hours to break down the psychological firewalls that often hinder using these reports effectively.
Q: How can individuals and organizations surmount these challenges?
A: I feel a gentle introduction to the subject area with lots of practical examples in plain English does the trick nicely. Sometimes psychometricians are accused of being pedantic, whether it is intentional or unintentional, making this information inaccessible for more people to understand and apply. I want to break down these barriers because I feel that the more people  who understand and can use psychometrics to improve assessment, the better off we all will be. I have tried to increase people’s understanding through my blog posts and I am really looking forward to personalizing this approach further in the workshop at the users conference.
Q: How have you structured the workshop?
A: I have structured the workshop to provide some of the basic theory behind item and test analysis and then get hands on to look at practical examples in different contexts. When I have done these workshops in the past, I have found that at first people can be sceptical of their own capacity to learn and apply knowledge in this area. However, by the end of the workshops I see people excited and energized by their newfound knowledge base and getting really involved in picking apart questions based on the item analysis report information. It is really inspiring for me to see people walk away with new found confidence and motivation to apply what they have learned when they get back to their jobs.
Q: What do you want people to take away with them from this session?
A: I want people to take away a newfound comfort level with the basics of psychometrics so that they can go back to their desks, run their item and test analysis reports, have confidence that they know how to identify good and bad items, and do something with that knowledge to improve the quality of their assessments.
You can sign up for this workshop at the same time you register for the conference (remembering that this Friday, January 21st, is the last day for earlybird savings). If you’re already registered for the conference, email to arrange for participation in the workshop. Click here to see the conference schedule.