Russian Psychometrics at an International Level

© Высшая школа экономики

Dmitry Abbakumov is the first researcher from a Russian university to present their work at the Psychometric Society conference, founded in 1935. He told the HSE new service about his study of online learning and why Russia has not played a major role on the international psychometrics scene – until now.

What is psychometrics all about?

The major players in this field are the Netherlands, Belgium, the United Kingdom and the United States. For more than a hundred years, these countries have been developing new approaches and mathematical models for behavioural science.

The classical psychometrics of the first half of the twentieth century measured human knowledge using groups of tasks. The logic was simple: the more tasks the person solved correctly, the smarter the person was and the better prepared they were. Then, in the 1960s, there was a crisis. Why? Here’s an example.

Let’s take two students. One solves 20 simple tasks, the other only 12, but more complicated. How should we compare them? Georg Rasch from Copenhagen University came up with the answer. His model enables researchers to compare students, even if the tasks are different. We can consider the level of a set task as an indicator of a latent variable (e.g. knowledge, preparedness, ability). That is, we examine the latent variable not from the point of view of the test overall, but from the point of view of the individual tasks, each of which is treated as an indicator.

Frederic Lord from ETS – America's largest testing company, which develops tests such as SAT and TOEFL, as well as tests for MBA – also came up with the same model.

How to Measure the Effectiveness of Online Learning

The models of Rasch and Lord were applied throughout the world, but not in the field of online learning. Both models considered the level of preparedness to be a static parameter, that is, they assumed that the level of a person’s knowledge does not change during the test. However, in the case of online education, if student solves a new task, they gain knowledge, whereas if they miss a week’s lectures, their level of preparedness decreases.

To take this factor into account, my colleagues and I decided to expand Rasch’s model by adding a few parameters. This enables us to measure the preparedness of students in a dynamic online learning environment. It takes into account not only whether or not the answer is correct, but also how the student dealt with the task: how many attempts were made and the effectiveness of these attempts. The model takes into account not only how the student copes with the tasks, but also how they use the course material (do they watch the video lectures, download materials for reading, etc.). The model is stable regardless of whether the tasks are fully or partially replaced in the exam – the results of the two students will still be comparable.

Such a tool can increase the credibility of exam results in online courses, as well as the credibility of online courses in general. Traditionally, tests are treated with distrust, likened to ‘guess work’ or the well-known game show where the participant can ‘call a friend’.

What’s Happening with Psychometrics in Russia

It may seem strange that our presentation was the first from Russia in the eighty-year history of the Psychometric Society. However, there is an explanation: there haven’t been any actual studies carried out in Russia for a long time. Actually, psychometrics as a science began to develop only a few years ago.

Under the Soviet Union, the Russian version of psychometrics was banned, along with cybernetics and genetics. All this was considered pseudoscience. By the 1970s, the situation had improved slightly, but by this time the West had already made significant advancements. We were only just starting to get into it. The whole approach was built around applied psychometrics using Western mathematical models and not offering anything new. Only now do we have results that influence the development of psychometric models and psychometric theories, and this is what we can present to the international psychometric community.

HSE’s Centre for Psychometrics in eLearning conducts a lot of applied research, however our main goal is to develop science in the field of online learning. We consider preparedness to be a multidimensional construct. A student's knowledge is not a single whole, but a system of many components, each of which contributes to a student’s results in an online exam. We are trying to create models closer that are closer to reality. This is perhaps something that no one else in the world is doing.