Machine learning psychometrics

Improved cognitive ability validity from supervised training on item level data

Andrew Cutler, Boston University (USA)

Shane McLoughlin, University of Chester (UK)

Curtis Dunkel, Western Illinois University (USA)

Emil O. W. Kirkegaard, Ulster Institute for Social Research (UK/DK)

​

Predict or Understand?

• Psychometricians like to understand constructs
• Use strong assumptions in scoring tests
• Equality of item validity
• Linear
• Non-interactive
• How much more validity do those tests have if they are scored to maximize predictive power (of eg. income, education)?

Let’s try! First, the data

Example: online vocabulary test

• 45 multiple choice items with response level data
• https://openpsychometrics.org/_rawdata/ ← Great resource!
• 12,173 people took online vocabulary test, 9,278 English natives
• Criterion variables:
• Sex
• Age
• Education
• Various others not used here (e.g. country, OCEAN personality)

Binary vs. categorical item coding

• Most items in available datasets use multiple choice format
• But usually data only saved in binary correct/incorrect format
• But you should never throw away data!

​

Methods

• Sum scores: we just sum tests
• Simplest possible
• Item response theory: compute item properties, person scores
• Better than sum scores, usually
• Ridge regression (l2 penalization): estimate weight of each item
• Optimal weights for prediction, assumes no sparsity of effects
• Random forest regression: build very complex decision trees
• Catches interaction effects
• Honorable mentions: deep learning, support vector machines

​

Meta-analysis approach

• Multi-method/coding/dataset/outcome
• 80/20 train/test splits. Loop 50 times
• Quantify sources of validity gain by variation

All datasets: boxplot results

All datasets: average results

• On average across datasets and outcomes:
• IRT: +5.2%
• RF: +2.6%
• Ridge: +22.2%

Shorter tests

Conclusion: implications

• Conclusion: ML can beat traditional scoring for prediction purposes
• Regression > Random Forest: items quite independent
• Predictive validity of cognitive data has been underestimated
• Cognitive tests for hiring purposes
• SAT/ACT/GRE for enrollment
• Cognitive tests for predicting dementia

​

Thank you

• Slides available
• Data available
• Analysis code available

Random Forest

Predict weight from: age, gender, height

Male?

>5’8?

>5’2?

>28 yo?

>40 yo?

Two cultures: understanding and prediction

• Traditional psychometrics/psychology is firmly in the statistics/understanding camp
• But their work often is to predict
• So they should gain from using prediction-optimized methods

Scoring of cognitive data

• Scoring of cognitive data seems quite archaic
• Based on strong statistical assumptions:
• Equality of item validity
• Linear
• Non-interactive
• Assumptions necessary to compute scores when you don’t have strong computers

Cognitive tests are used for prediction in practice

• So why not optimize for predictive validity?
• Supervised vs. unsupervised learning
• Classical methods are all unsupervised learning methods
• Jensen: g is the active ingredient in tests
• Special case of the assumption underlying principal components regression
• But usually only roughly true