1 of 17

Enhancing Causal Inference in Educational Data Mining:�Integrating Observational and Experimental Data for�Greater Precision� �EDM 2025

1

Yanping Pei Adam C. Sales

Johann A. Gagnon-Bartsch

2 of 17

RCTs in Education Research are Great!

2

3 of 17

RCTs in Education Research Face Challenges!

  • Randomized A/B tests in digital experiment platforms often face
    • low participation
    • underpowered statistical inference
  • Meanwhile, CBLPs generate rich student log data

​

​

​

​

​

​

​

3

Ed Tech Log Data:

How Students and Educators Engage with Intelligent Tutoring Systems? – log data

Learning behaviors

Student historical performance

…

  • Integrating this observational data with experimental data can enhance causal effect estimates??

4 of 17

The Problem in RCT: �Small Sample Size → Large Variance

4

→ → Augmented with Data from Observational Studies

But why??

​

RCT

Observational Studies

Bias

Unbiased

Biased

Variance

Small sample size → Large variance

Large sample size → Small variance

5 of 17

How to Combine Observational and RCT data?

  • Remnant Leave-One-Out Potential Outcomes (ReLOOP)1
    • Imputes control potential outcomes for experimental units using flexible predictive models (e.g., machine learning, neural networks) trained on observational data
    • Emphasizes unbiasedness through Leave-One-Out

5

2 M. Oberst, A. D’Amour, M. Chen, Y. Wang, D. Sontag, and S. Yadlowsky. Understanding the risks and rewards of combining unbiased and possibly biased estimators, with applications to causal inference, 2023.

​

​

A minor tweak to reduce MSE

  • Balancing variance and bias2
    • Weight observational data and the experiment data

​

1 Gagnon-Bartsch, J.A., Sales, A.C., Wu, E., Botelho, A.F., Erickson, J.A., Miratrix, L.W. and Heffernan, N.T., 2023. Precise unbiased estimation in

randomized experiments using auxiliary observational data. Journal of Causal Inference, 11(1), p.20220011.

6 of 17

Goal: Utilize data you often throw out to make better estimates

6

  • CHALLENGE: How to Define Supplemented Observational Data?

Observational Data

RCT Data

Summarized Information

​

  • CHALLENGE: How to Extract Insights from Observational Data?

​

  • CHALLENGE: How to Incorporate Observational Data in Estimating Treatment Effects from RCTs?

7 of 17

Goal: Utilize data you often throw out to make better estimates

7

  • CHALLENGE: How to Define Supplemented Observational Data?

Observational Data

 

8 of 17

Goal: Utilize data you often throw out to make better estimates

8

Observational Data

Summarized Information

​

  • CHALLENGE: How to Extract Insights from Observational Data?
  • Approach: Predictive Modeling with Observational Data
    • Use observational data as the training dataset
    • Train a prediction model where student performance is the target outcome
      • Machine Learning & Deep Neural Networks & Other predictive algorithms
      • Model choice is flexible1 → As long as it achieves high predictive accuracy

J. A. Gagnon-Bartsch, A. C. Sales, E. Wu, A. F. Botelho, J. A. Erickson, L. W. Miratrix, and N. T. Heffernan. Precise unbiased estimation in randomized experiments using auxiliary observational data. Journal of Causal Inference, 11(1):20220011, 2023

​

​

9 of 17

Goal: Utilize data you often throw out to make better estimates

9

Observational Data

Summarized Information

​

  • CHALLENGE: How to Extract Insights from Observational Data?

10 of 17

Goal: Utilize data you often throw out to make better estimates

10

Observational Data

RCT Data

Summarized Information

​

  • CHALLENGE: How to Incorporate Observational Data in Estimating Treatment Effects from RCTs?

Conduct Combined Counterfactual Potential Outcomes (CCPO) estimator to reduce Mean Squared Error (MSE) in estimating the Average Treatment Effect on the Treated (ATT)

​

11 of 17

How to Incorporate Observational Data in Estimating Treatment Effects from RCTs?

11

Observational Data

RCT Data

Summarized Information

 

12 of 17

CCPO Estimation Process Overview

12

Predictive Modeling with Observational Data

Predicted Counterfactual Potential Outcomes

 

Combined Counterfactual Potential Outcomes (CCPO) estimator

 

13 of 17

Preliminary Simulation

13

 

14 of 17

Application

14

RCT: Digital Education Experiment

​

  • Treatment: Reflection Prompt
  • Outcome: % Correctness
  • Observational Data: Student logs

​

15 of 17

Discussion

  •  

15

16 of 17

Q & A

Email: ypei@wpi.edu

16

Questions?

THANK YOU!

17 of 17

Goal: Utilize data you often throw out to make better estimates

17

Observational Data

RCT Data

Summarized Information

​

  • CHALLENGE: How to Incorporate Observational Data in Estimating Treatment Effects from RCTs?