Role of test motivation in intelligence testing

PNAS – Role of test motivation in intelligence testing

The results indicate that being highly motivated in tests improves IQ tests scores and being highly motivated in life improves life outcomes. “IQ scores are absolutely predictive of long-term outcomes. But what our study questions is whether that’s entirely because smarter people do better in life than other people or whether part of the predictive power (is) coming from test motivation,” Duckworth said. “This means that for people who get high IQ scores, they probably try hard and are intelligent,” she said. “But for people who get low scores, it can be an absence of either or both of those traits.”

Intelligence tests are widely assumed to measure maximal intellectual performance, and predictive associations between intelligence quotient (IQ) scores and later-life outcomes are typically interpreted as unbiased estimates of the effect of intellectual ability on academic, professional, and social life outcomes. The current investigation critically examines these assumptions and finds evidence against both. First, we examined whether motivation is less than maximal on intelligence tests administered in the context of low-stakes research situations. Specifically, we completed a meta-analysis of random-assignment experiments testing the effects of material incentives on intelligence-test performance on a collective 2,008 participants. Incentives increased IQ scores by an average of 0.64 SD, with larger effects for individuals with lower baseline IQ scores. Second, we tested whether individual differences in motivation during IQ testing can spuriously inflate the predictive validity of intelligence for life outcomes. Trained observers rated test motivation among 251 adolescent boys completing intelligence tests using a 15-min “thin-slice” video sample. IQ score predicted life outcomes, including academic performance in adolescence and criminal convictions, employment, and years of education in early adulthood. After adjusting for the influence of test motivation, however, the predictive validity of intelligence for life outcomes was significantly diminished, particularly for nonacademic outcomes. Collectively, our findings suggest that, under low-stakes research conditions, some individuals try harder than others, and, in this context, test motivation can act as a third-variable confound that inflates estimates of the predictive validity of intelligence for life outcomes.

University of Pennsylvania – The researchers discovered individuals with above-average scores at baseline were marginally influenced by motivation as their scores increased by only about a quarter of a standard deviation, or about four points.

However, for those who had below-average scores, motivation made up almost a whole standard deviation, or around 16 points.

The second study involved an experiment in which researchers observed video footage of adolescent boys taking a standard IQ test to rate their motivation and then measured how well they fared in terms of criminal record, job status and educational attainment more than a decade later.

Coders, who were not aware of subjects’ IQ scores or the hypothesis of the study, rated each subject’s motivation based on a standard rubric of behaviors, such as refusing to answer questions or obviously rushing through the test to make it end as quickly as possible.

Ratings of test motivation and IQ scores were about equally predictive of the adult outcomes of years of education, employment status and criminal record.

“What we were really interested in finding out was when you statistically control for motivation, what happens to the predictive power of the IQ tests? What we found is that the predictive power goes down significantly,” Duckworth said.

5 pages of supplemental information

What Intelligence Tests Test: Individual Differences in Test Motivation and IQ (52 pages)

Angela Lee Duckworth University of Pennsylvania

Patrick D. Quinn The University of Texas at Austin

Donald Lynam Purdue University

Rolf Loeber, Magda Stouthamer-Loeber University of Pittsburgh

Terrie E. Moffitt and Avshalom Caspi Duke University and Institute of Psychiatry, King’s College London

If you liked this article, please give it a quick review on ycombinator or StumbleUpon. Thanks