Stay the Course: �Causal Insights from SRL Patterns in Online Professional Learning

Sophia Soomin Lee, Jacob Dirghalli, Dr. Walter Leite

University of Florida

1

Research Motivation

• Understanding how learners engage in Online Professional learning (OPL) is crucial for improving retention and learning outcomes.
• Digital footprints offer an opportunity to model learner behavior in action and using data-driven clustering techniques like Latent Class Analysis (LCA) allows us to summarize these patterns into meaningful learner profiles.
• Many studies describe associations between engagement and outcomes, but these do not imply causality.

​

2

The purpose of the study is to identify meaningful patterns of engagement in an OPL course using behavioral data and estimate their causal effects on course completion.

Quasi-experimental designs with latent classes

• Students self-select to engage in complex behavioral patterns that cannot be randomly assigned to treatment or control conditions.
• Latent class analysis can group students based on similarity of their behavioral patterns.
• Propensity score analysis can be used to balance pre-existing differences between student groups who engaged in different behavioral patterns and allow causal comparisons.
• Key assumptions:
• Behavioral patterns can be manipulated, and there are potential outcomes for each pattern.
• Weak ignorability of latent class membership: Potential outcomes of behavioral patterns represented in latent classes are independent of potential outcomes of other behavioral patterns given observed covariates.
• The potential outcomes of one student are independent of the behavioral patterns of another student.

3

Research Questions

RQ1 What are the latent classes that summarize learner’ engagement of the OPL course?​

RQ2 What are the average treatment effects (ATEs) of engaging with OPL course in ways consistent with the characteristics of latent classes on course completion?

4

Sample

• The sample consisted of 1,322 learners from one online professional development course for literacy educators in a Southeastern State.
• System log data between January 2023 and December 2024.

5

Measures

6

Theoretical Framework: Self-Regulated Learning

7

(Zimmerman, 2000)​

Model

8

Behavior latent classes

Completion

Pre-assessment score

Previous knowledge

handout

Active days

logins

Regularity

Video

Interactive

Reflect

Transcript

Study guide

Analysis Steps

9

Data Preparation

Latent Class Analysis

Propensity Score Analysis

Variable selection

Outlier removal

MICE

Model selection

Save posterior probabilities

Compute GPS

Compute IPTW

Balance check

Estimate ATE

BIC

Entropy

Interpretation

Sensitivity

Logistic regression

Neural networks

Random forest

PS model

Latent Class Model Selection

10

Latent Class Estimates

Learners were categorized into three distinct profiles based on their engagement patterns.

• Class 1, referred to as Minimalist Browsers, showed consistently low levels of interaction across all learning resources, indicating a minimal engagement approach.
• Class 2, named Strategic Skimmers, demonstrated moderate and selective engagement, suggesting a tendency to access resources purposefully rather than broadly.
• Class 3, identified as Engaged Explorers, exhibited the highest levels of interaction and time investment across various materials, reflecting a highly active and exploratory learning style.

11

12

13

14

Propensity Score Analysis Details

• Generalized propensity score (GPS):
• IPTW: 1/𝝅i
• Covariate balance check: � absolute standardized mean differences < 0.05
• ATE estimation:
• Sensitivity analysis: Amount of bias needed to change inference (Konfound package in R)

15

ATE estimates

16

Sensitivity Analysis Results

• 87.8% of the ATE would need to be explained by unobserved bias to invalidate the result
• Would require replacing 698 learners with null-effect cases
• Enhances the credibility of causal inference under non-random exposure

17

Limitation

1. Conceptual misalignment
1. Our behavioral indicators only partially reflect SRL processes.
2. Partially due to the data format (granularity).
2. Limited consideration of temporality
3. Limited Generalizability
3. The analysis was applied to one specific context.

18

Discussion

1. Manipulability of Behavioral Classes

• Behavioral patterns (like being an Engaged Explorer) are not fixed traits.
• Through scaffolding, prompts, and system design (e.g., reminders, structured reflection), learners can be nudged toward more effective strategies.

2. Causal Inference in Non-Random Environments

• This study demonstrates how quasi-experimental design can be applied to log data, where random assignment is not possible.
• The use of LCA + GPS-based weighting allows for nuanced comparisons between naturally occurring behavior patterns.

19

Thank you!

20