Posted by John Kleeman, Founder
I was fortunate to attend the ITCC Spring Meeting and ATP Innovations in Testing conference in Florida last week. The ATP conference is the largest testing conference and is a great place to learn what is new in assessment. It’s been virtual the last two years, and it was wonderful to see people in person this year. The ITCC meeting is a smaller affair, but very effective in allowing those involved in IT certification to communicate.
Here are some of the things I learned:
ATP opening keynote
There was a powerful opening keynote speech by Jennifer Randall and Pamela Paek focused at improving how the testing world looks at bias and equity. They argued that testing has been for too long guided by an outdated majority view and that we need to be more culturally open in how we write items. For example, some item writing guidelines suggest avoiding pictures of pregnant women as context for items, but they suggest that most women are pregnant for some of the time and it’s wrong to exclude them.
Questionmark wins an award for an innovation that aids equity
Another area where equity in testing is important is ensuring that migrants and non-native language speakers are supported in taking a test when not being fluent in the language of the test. For example, if you are taking a test in a technical area, you shouldn’t be penalized if you don’t speak English well- unless being fluent in English is part of what is being tested.
Questionmark, in partnership with our customer SAP, has recently developed an innovation to help. In this innovation, non-native speakers taking a test can highlight parts of a question they want to understand better and get an instant, computerized translation, as shown below.
This “Instant Translate” feature will be coming to Questionmark OnDemand shortly and will help provide more of a level playing field for non-native speakers. This new development won SAP and Questionmark the prestigious 2022 ITCC Innovation Award, which the ITCC presents each year. You can see more HERE.
Standardization vs inclusiveness
Equity and inclusiveness were a big theme at both ITCC and ATP. One of the ATP attendees was leading psychometrician, Steve Sireci, who I’ve recently interviewed for the Questionmark podcast.
I recommend a paper by him as being an excellent explanation for the need for more inclusivity in testing: “Standardization and UNDERSTANDization in Educational Assessment”, published in 2020 in Educational Measurement Issues and Practice.
In this paper, he argues that standardization or uniformity was critical in the early years of testing to help develop concepts of measurement and fairness. A key tenet of psychometrics became that the test should be delivered in a standardized way, similarly for every test taker. But in this paper, he suggests we need to understand the diversity of our test takers better and set testing conditions that are best for each individual. We should be flexible to accommodate and account for test taker personal characteristics to measure more effectively. For example, he suggests we need to consider whether time limits are still needed in most tests.
Remote proctoring was widely discussed. Everyone has been doing it over the pandemic and almost all are going to continue to do it as the pandemic wanes. Three important themes with slightly converging pressures are:
- Inclusivity. We need to accommodate those whose equipment may vary and whose home environment is not spacious. One admissions organization at the conference shared that they give a credit to candidates to allow them to rent quiet space elsewhere if their home environment doesn’t suit. Not all organizations can afford this, but everyone is thinking how to be more inclusive.
- Candidate convenience. It’s important to reduce the number of times a candidate tries to take the test and fails, these are called “spoiled sessions”. Providers are working to improve the test taker experience around software installation, device support, how authentication happens to make the candidate journey smoother.
- Integrity. Although inclusivity and candidate convenience are important, it’s critical to the validity of assessments that people take them honestly and that we reduce test fraud.
Lastly some tips to reduce test fraud
There were many sessions on security and test fraud including some excellent ones by Questionmark partner Caveon. A common way of thinking about test fraud is that we should seek to prevent it, seek to deter it but then detect and respond if it happens. Some deterrence tips:
- Ensure that you have a prominent test taker agreement in which people commit to take the test honestly and understand the consequences of not doing so.
- Test takers chat amongst themselves, and it’s worth broadcasting when sanctions are taken against someone found cheating. This reminds that there are consequences of test fraud and helps deter it.
- It can be worth suggesting that if people use external services to help them pass a test illegitimately, they not only risk disqualification, but also that they are vulnerable to intimidation. These external services are sometimes criminals and there is a risk of being told “Hey we passed the test for you, give us more money or we’ll let people know”.
Attending an in-person conference is invigorating and I hope to see many Questionmark blog readers at future conferences. Please reach out to me with any comments.