Challenging the reliability and validity of cognitive measures: the case of the numerical distance effect

Challenging the reliability and validity of cognitive measures: the case of the numerical distance effect

Click the buttons to download the infographic and transcript!

Title:

• Challenging the reliability and validity of cognitive measures: The case of the numerical distance effect

Authors:

• Erin A. Maloney, Evan F. Risko, Frank Preston, Daniel Ansari, Jonathan Fugelsang

Journal and DOI:

• Acta Psychologica, 134, 2010, 154-161, doi: 10.1016/j.actpsy.2010.01.006

Previous Research

Researchers have long argued that the ability to decide which of two numbers represents a larger quantity (e.g., 7 vs 2) is a fundamental cognitive skill. Researchers also argue that difference between how quickly people can discriminate between far numbers (e.g., 7 and 2) and close numbers (e.g., 1 and 2) is an indication of how precisely they represent numbers in their mind. This “numerical distance effect” has been assessed using two types of tasks: (1) simultaneous comparison (comparing two quantities presented at once) (2) comparison to a standard (comparing one number to a set standard (e.g., always deciding if something is more than or less than 5). These tasks have also been done using Arabic digits and sets of objects (e.g., rectangles).

What did we ask?

How strongly does an individuals’ performance on the first half of trials within a given task relate to their performance on the second half of trials on the same task (split-half reliability)?

How strongly does an individuals’ performance on each variant of the task relate to their performance on the other variants of the task (construct validity)?

How did we do it?

Adults completed four versions of the numerical comparison task:

(1) comparison to a standard with Arabic digits,

(2) comparison to a standard with sets of rectangles,

(3) simultaneous comparison with Arabic digits, and

(4) simultaneous comparison with sets of squares.

What did we find?

The results indicate that the comparison tasks using Arabic digits are not particularly reliable (with the simultaneous comparison task being slightly more reliable than the comparison to a standard). Also, measures that use non-symbolic stimuli (i.e., the sets of rectangles) are more reliable than measures that use Arabic digits. We also found evidence that the distance effect that arises from comparison tasks using Arabic digits is uncorrelated with the distance effect that is elicited using non-symbolic stimuli.

Why is this important?

These findings highlight the importance of moving beyond solely looking for replicable measures and towards an assessment of replication, reliability, and test validity of effects used to gain insight into cognitive processes.

Brought to you by Dr. Erin Maloney’s Cognition and Emotion Lab at the University of Ottawa.