Machine Learning for Social Scientists

James Hall

Associate Professor of Psychology of Education:

Southampton Education School, University of Southampton

​

Deputy Director:

ESRC National Centre for Research Methods

​

11/7/2025

J.E.Hall@Soton.ac.uk

Aims

To develop attendees’ understanding of:

• What machine learning (ML) is (and is not)
• When and why ML might be useful in social science research
• How ML compares to traditional statistical methods in social science research in terms of goals, assumptions, and outputs
• Gain awareness of the major ML methods relevant to social science
• How to carry out two basic ML methods in the R software package
• Where to go next for further reading on machine learning

Content

• Part 1: Lecture
• Defining Machine Learning (ML)
• A brief history of ML– and how it relates to other disciplines
• Why use ML in the Social Sciences
• How ML differs from traditional inductive statistical methods
• Common Types of ML: Supervised vs. Unsupervised
• ML in practice: Examples from the Social Sciences
• Decision Trees (supervised, computational)
• Gradient Boosting (supervised, computational)
• Software for ML and follow-on reading
• Part 2: Workshop in R
• Setup & Overview (load packages, explain dataset)
• ML method 1: Decision Trees
• ML method 2: Gradient Boosting
• Wrap-Up & Q&A

Part 1: Machine Learning for Social Scientists

Defining Machine Learning (ML)�

• Machine Learning (ML) refers to a range of computer-powered, algorithm-based, inductive pattern-finding processes
• Used to predict outcomes, classify cases, or discover hidden structures (factors, components, clusters)

​

• These inductive pattern-finding processes can be carried out on both numeric and non-numeric data
• (though non-numeric data still needs to be converted in numeric format)

​

• Some of these processes are mathematical while some are statistical.

A brief history of ML– and how it relates to other disciplines

• 1940s–1970s: ML emerges as a subfield of Artificial Intelligence (AI); early focus on rule learning, games, neural networks
• 1980s: ML and AI diverge; ML retains statistical and computational focus (AI turns toward logic-based systems)
• 1990s: ML formalised as its own field; rise of statistical data mining as an off-shoot
• 2010s–present: ‘Deep learning’ transforms ML via GPU-powered neural networks
• Today: ML spans disciplines -- overlapping with statistics, computer science, cognitive science, and even psychometrics

Why use ML in the Social Sciences?

• Handles large, complex, or messy datasets
• Especially datasets that are clearly unsuitable for traditional statistical analyses such as those that contain thousands of machine or software-generated variables
• E.g. log files from learning platforms (Blackboard, Moodle, Canvas), key stroke data, wearable sensor data, social media data app interaction data
• Finds patterns without needing a pre-specified model
• Useful when theory is weak, or exploratory analysis is needed
• Works well with non-numeric data (text, images, sound etc)
• There are packages that automatically generate numeric data (and passes this to the ML process) from the supply of such non-numeric data
• Numeric data and non-numeric data can thus be entered into a single analysis
• E.g. Responses from closed-ended and open-ended questionnaire items
• Supports prediction and classification tasks, not just explanation
• E.g. who is likely to drop out of a study or from school
• Can complement -- not only replace -- traditional statistical analysis

Why use ML in the Social Sciences?

• You may not have experience with datasets better suited to ML
• But why would you have gathered them in the past knowing that traditional statistical analysis techniques would not have been useful
• After this session: More open to find such data, gather such data, & analyse it?

Data better suited to traditional statistical analysis techniques

Data better suited to ML techniques

Data suited to traditional statistical analysis or ML techniques

Data Conditions (in combination)

• Hundreds/Thousands of variables;
• Variables generated by machines or software
• Irregular formats of data
• Plentiful missing data
• Non-numeric data;
• Numeric with non-numeric data

Data conditions

• Tens/Hundreds of variables
• Variables generated by human processes
• Less complex numeric data formats
• Numeric data without non-numeric data

​

How ML differs from traditional inductive statistical methods

• “I am familiar with inductive statistical techniques – so do I need ML?”
• (see previous slide)
• “How do inductive statistical techniques compare in form and purpose to ML?”
• They can overlap in purpose. Many inductive statistical techniques exist within ML techniques (e.g. EFA, PCA, cluster analysis)
• Comparison:

Common Types of ML: Supervised vs. Unsupervised

• The notion of how much a ML process is ‘supervised’ is a crucial way to group and distinguish ML processes – this concept also helps to distinguish the different purposes to which ML can be put
• There are many types of ‘supervision’ (an area under continual development). Perhaps the two most relevant to statistics in the social science are Supervised ML and Unsupervised ML
• An example of a supervisory ML process less relevant today is the ‘self-supervision’ the Large Language Models (LLMs) use.
• Note: the supervisory status of an ML process is often left unmentioned in speech and/or writing.

Prevalence of ML in social science research

• Supervised ML is much more prevalent in social science research than unsupervised ML at the moment.
• E.g.:
• “About 96% of the studies used supervised methods, two employed unsupervised methods (Abarca-Alvarez, Reinoso-Bellido, & Campos-Sánchez, 2019; Crossley et al., 2020), and one study used a semi-supervised method (Nguyen et al., 2017a). Among supervised methods, classification and regression trees (CART) were the most common (n = 20), followed by random forests (n = 17).”
• Kino, S., Hsu, Y. T., Shiba, K., Chien, Y. S., Mita, C., Kawachi, I., & Daoud, A. (2021). A scoping review on the use of machine learning in research on social determinants of health: Trends and research prospects. SSM-population Health, 15, 100836.

​

• Why the lack of unsupervised ML in social science research?
• Speculation: Many of the algorithms of unsupervised ML are already used within standard inductive statistical methods: EFA, PCA, LCA (etc).

Supervised ML models

• These work in two phases:
• A training phase where the model is given both inputs and outputs in order to learn the patterns that link them
• A prediction (or testing) phase where the model is given new input data only and it tries to predict the output based on what it learned. This tests whether the model can generalise to unseen data.

​

• How these two phases are often approaches in practice:
• The dataset is typically split into a training set (to build the model) and a test set (to evaluate it). Often 70(training)/30 or 80(training)/20
• Alternatively, cross-validation is used. The data are repeatedly split into multiple training/test combinations to get more reliable performance estimates. (5 or 10)
• Performance metrics are used to assess how well the model did on the test data.

Two ML Methods We’ll Explore Today

1. Decision Trees, a supervised computational ML method.
• It predicts an outcome (e.g. whether a student needs support) based on input features, and it gives you a decision-flow you can read

​

• Gradient Boosting, a supervised computational ML method that combined multiple Decision Trees.

​

• Aside Note: A machine learning method is statistical (rather than computational) when:
• It is based on a statistical model
• It assumes distributions, error terms, and parameters
• It can, in principle, support inference (even if used for prediction)
• Example: Regression-based ML

Decision Trees

• A supervised computational machine learning method that learns to make predictions by asking a series of yes/no questions (or threshold-based decisions) about the input data.
• These predictions can either be towards a categorical outcome (‘classification’) or towards a continuous outcome (‘regression’)

​

• It splits your data into branches, based on the best variable at each point, to predict an outcome.
• If you’ve ever worked through a diagnostic flowchart or troubleshooting guide, you’ve followed a decision-tree logic.

​

• Output is a flowchart-like tree structure.

​

• What makes it machine learning is that the tree structure (the questions, cut-offs, and outcomes) is learned from the data, not written by hand.

Example ‘real-world’ Decision Tree

• Koon, S., Petscher, Y., & Foorman, B. R. (2014). Using evidence-based decision trees instead of formulas to identify at-risk readers. Regional Educational Laboratory Southeast REL, 36.
• “Educators need to understand how students are identified as at risk for reading problems.
• …Data from a sample of Florida public school students in grades 1 and 2 in 2012/13…
• This study found that the classification and regression tree (CART) model -- a type of predictive modeling that presents results in an easy-to-interpret “tree” format -- predicted poor performance on the reading comprehension subtest of the Stanford Achievement Test as accurately as the logistic regression model, which is more difficult to interpret.
• The CART model’s ease of communication enables parents, teachers, principals, and school district leaders to better understand how a student is predicted to be at risk.”
• (Note: CART was the first type of developed Decision Tree)

​

Gradient Boosting

• A supervised machine learning method that builds a strong predictive model by combining many small decision trees, each trained to fix the errors of the previous ones.�
• Each tree focuses on correcting mistakes made by the model so far -- gradually improving accuracy.
• This sequential learning process is called boosting.�
• Gradient boosting is used to predict outcomes when relationships between variables are complex, messy, or nonlinear.�
• It is widely used for classification and regression tasks in real-world problems.�
• Though less interpretable than simple models, it often outperforms them in accuracy and robustness.

Example ‘real-world’ Gradient Boosting

• Fernandes, J. P. C., Braga, C. G., & Yokota, R. T. C. (2024). A model for predicting academic performance on standardised tests for lagging regions based on machine learning and Shapley additive explanations. Scientific Reports, 14, Article 76596. https://doi.org/10.1038/s41598-024-76596-3
• Context: National dataset from Colombian public schools used to predict student performance on standardised tests.
• Used Gradient Boosting ML to uncover which factors drive student underperformance, and framed that insight for policy.
• The most important predictors for attainment were found to be:
• socioeconomic index
• rural/urban setting
• student age
• school region

​

Software for ML

​

• R: Widely used in social science; many ML packages (e.g., caret, xgboost, glmnet, rpart)
• it’s open-source,a continuation of a package widely used for statistical analsyes, and integrates ML libraries easily

​

• Python: Most popular language for ML overall; libraries include scikit-learn, xgboost, tensorflow, pandas
• dominates tech-driven ML

Suggested follow-on reading�

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021).�An Introduction to Statistical Learning (2nd ed.). Springer.

​

Widely regarded as the best intro to ML for people with a stats background

Uses R; clear explanations; focuses on supervised learning

Kuhn, M., & Johnson, K. (2013).�Applied Predictive Modeling. Springer.

​

Practical, code-heavy, social-science friendly

Focuses on model tuning, performance, and caret package in R

Great bridge between traditional stats and ML

Efron, B., & Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press

​

Broader perspective on how statistical ideas evolved into ML�Mixes theory and algorithms; good for context and reflection�For readers wanting depth, not just recipes

Questions?

​

15-minute break

Part 2: ML in the R Software Package

Aims for this ML (via RStudio) workshop

1. Load required packages
2. ML method 1: Decision Trees
3. ML method 2: Gradient Boosting
4. Discuss what next

Important note:

R needs straight single quote marks ' and not curly quote marks ‘ .

Load required SEM packages

• Open Rstudio
• Open a new R Script (File > New File > R Script)
• Save it as ml_workshop.R
• This is your working file

​

• Loading R ml packages. Text to enter into Rstudio (into the “script window”):

install.packages("rpart")

install.packages("rpart.plot")

install.packages("tidyverse")

​

library(rpart)

library(rpart.plot)

library(tidyverse)

• Highlight each of those lines one at a time (from the top) and then, for each, hit the “run” button on the top right-hand side of the script window

​

Screenshot

• Script Window (top-left)

​

• Button to press to either run:
• All script in script window
• Just highlighted script

​

• Console Window (bottom-left)

1. ML method 1: Decision Trees

• Data: Cortez, P., & Silva, A. (2008). Using data mining to predict secondary school student performance. In Proceedings of 5th Future Business Technology Conference (FUBUTEC 2008) (pp. 5–12). EUROSIS.
• 2005-2006; Two Portuguese secondary schools (one urban, one rural). 395 students.
• Variables on: math attainment (across 3 periods), age, sex, address type, parental education and job, study time, absences, alcohol use (and others)
• Load and look at data for this ML method:

# Download dataset directly from UC Irvine

temp <- tempfile()

download.file("https://archive.ics.uci.edu/ml/machine-learning-databases/00320/student.zip", temp)

​

# 2. Unzip and read the math course data

unzip(temp, files = "student-mat.csv")

student <- read.csv("student-mat.csv", sep = ";")

​

# 3. Clean up temporary file

unlink(temp)

​

# Create a binary outcome: pass = G3 >= 10 so that our Decision Tree is also a Classification Tree

# Lets us consider a real practical outcome -– risk of failing the grade

student$pass <- ifelse(student$G3 >= 10, "pass", "fail")

student$pass <- as.factor(student$pass)

Screenshot after running: head(student)

Screenshot after running: summary(student)

Run, view, and plot the decision tree ML

# Build a classification tree

tree_model <- rpart(

pass ~ studytime + failures + absences + G1 + G2 + sex + age,

data = student,

method = "class"

)

​

summary(tree_model)

​

rpart.plot(tree_model, type = 2, extra = 104, fallen.leaves = TRUE)

Screenshot after running code

Looking more closely at the decision tree output

​

33% failed, 67% passed the grade

​

G2<10? 🡪 the first and most important split.

​

37% of the sample had “G2<10”. Of these 84% failed G3.

​

Similarly, of those for whom “G2>=10”: 97% of the passed G3.

​

That’s the predictive power of the “G2<10 split and an intuitive showcase of the utility of this method and ML.

Looking more closely at the output

Info about tree complexity control and ‘pruning’ – too much for this intro session

​

​

Matches the tree – variables G2 and then G1 were the strongest predictors of G3 pass/fail

​

Tells us about the nodes on the tree we just saw (same info)

Get quick feedback on the quality of this model in it’s ‘classification’

• # Predict on training data

pred_tree <- predict(tree_model, type = "class")

​

• # Confusion matrix

confusion <- table(Predicted = pred_tree, Actual = student$pass)

print(confusion)

​

• # Accuracy

accuracy <- sum(diag(confusion)) / sum(confusion)

paste("Accuracy:", round(accuracy * 100, 2), "%")

​

Notes:

A first look at how well the model fits the data (same idea as in SEM – “model fit”)

In real ML, models are (as earlier) not evaluated on the same data they are fitted on

Looking more closely at the output

Clear to see how accurate the predictions were.

​

This evaluation of the predictive power of the decision tree model is overestimating predictive power because it is applied to the data it was training on.

2. ML method 2: Gradient Boosting

• We will use the same dataset as before – and the same outcome. But we need some new R packages now:

install.packages("xgboost")

install.packages("caret")

install.packages("Matrix")

library(xgboost)

library(caret)

library(Matrix)

Data preparation

• We need to make a slightly different version of the G3 pass/fail variable – because of the demands on data format within the new package we are using

# Convert factor outcome to numeric: 1 for pass, 0 for fail

student$pass_numeric <- ifelse(student$pass == "pass", 1, 0)

​

# Define the predictors and outcome

predictors <- student[, c("studytime", "failures", "absences", "G1", "G2", "age")]

outcome <- student$pass_numeric

​

# Convert predictors to a matrix for xgboost

x <- as.matrix(predictors)

y <- as.numeric(outcome)

Data fit, prediction, and evalaution

• Fit

# Basic gradient boosting model

gb_model <- xgboost(data = x,

label = y,

objective = "binary:logistic",

nrounds = 100,

verbose = 0)

• Prediction

# Predict probabilities

gb_pred_probs <- predict(gb_model, x)

# Convert to class predictions: 1 = pass, 0 = fail

gb_pred <- ifelse(gb_pred_probs > 0.5, 1, 0)

​

• Evaluation

# Confusion matrix

conf_matrix <- table(Predicted = gb_pred, Actual = y)

print(conf_matrix)

# Accuracy

gb_accuracy <- sum(diag(conf_matrix)) / sum(conf_matrix)

paste("Accuracy:", round(gb_accuracy * 100, 2), "%")

Looking more closely at the output

Clear to see how accurate the predictions were.

Top is from the decision tree ML.

Bottom is from the Gradient Boosting ML.

​

Logically, we would expect the Gradient Boosting ML to perform at-least as well as the Decision Tree ML. Here it performs better.

However, this evaluation of the predictive power of the MLs is overestimating predictive power because it is applied to the data it was training on.

Strong risk of “overfitting our data” – lack of ability to predict future new data

​

​

What next?

• Apply what you’ve learned to your own data
• Start with decision trees -- low barrier, high insight
• Try gradient boosting for better performance when needed

​

• Experiment with other ML methods
• Logistic regression with glmnet (Lasso/Ridge)
• Random Forests, Support Vector Machines (SVMs), k-NN
• Unsupervised methods like k-means, topic modelling, or PCA

• Use cross-validation and proper evaluation
• Learn how to split, test, and tune ML models like a pro

​

• Go deeper with structured reading or courses
• Introduction to Statistical Learning (James et al., 2021)
• Online courses (e.g., Coursera, edX, DataCamp)
• R package vignettes & GitHub examples

​

• Talk to others in your field
• Look for colleagues experimenting with ML
• Share ideas on when and how ML complements theory-driven work

​