1 of 40

Leveraging External Data Sources to Improve Probability of Success

Associate Director, Oncology Statistics

Takeda Pharmaceuticals

Nov. 1, 2024

Veronica Bunn, Ph.D.

2 of 40

Outline

2

01

03

04

An Application in Non-Small Cell Lung Cancer

04

02

Hierarchical Models for External Data Borrowing

03

Defining Probability of Success (PoS)

02

Background and Motivation

01

3 of 40

Challenges and Innovations in Phase III Clinical Trials

Confirmatory phase III clinical trials are lengthy and expensive

Average cost: 12-53 million USD
Average duration: 4 years

To mitigate the risk of conducting a futile trial, researchers have traditionally relied on statistical power

Trials are typically powered ≥ 80%

Failure rate of Phase III trials is still approx. 50%

Statistical power fails to incorporate uncertainty

The probability of achieving statistical significance at a hypothesized effect size
Based on previous trials, external data, expert opinion, or budgetary considerations

Need: A more robust method to quantify the true likelihood of success for a future phase III trial

4 of 40

Common Probability of Success (PoS) Metrics

Assurance / Average Success Probability

Hybrid Bayesian -Frequentist Approach

Probability of Study Success

Marginalizes power function over a prior distribution for the unknown effect size
Analogous to expected power

Posterior predictive probability of obtaining a significant result in phase III given phase II data

Two step process:

Use Bayesian modeling to derive posterior distribution
Estimate probability of a successful trial analytically or via simulation

5 of 40

Probability of Success in Go/No-Go Decision Making

PoS has become a popular tool in the pharmaceutical industry to inform go/no-go decisions after early phase trials
Additional methods have been developed that expand the definition of success:

Comprehensive PoS Approach

(Hampson et al.)

Composite definition of success

(Saint-Hilary et al.)

Successful outcome in pivotal trial

Requirements for market access

Regulatory approval

Achieving statistical significance

Clinical relevance

Favorable risk-benefit profile

6 of 40

Limited Ph II Data Availability Increases Uncertainty of the True Treatment Effect

Distribution of the unknown treatment effect is generally estimated using observed phase II data

Often sparse due to budgetary, logistical, and/or ethical considerations, resulting in:

Increased variability of efficacy endpoints
Increased uncertainty associated with the true treatment effect

Sparse phase II data can also increase the tendency of PoS to overestimate the true probability of success!

Outcome: Phase III pivotal trial doesn’t meet primary endpoint, despite positive results in Ph 2

7 of 40

Improve PoS by Enhancing Reliability of the Estimated Distribution of the Unknown Treatment Effect

Observe data from Ph 2 trial

Leverage external data source(s) for the control arm

Estimate the posterior distribution of the treatment effect using the augmented Ph II control arm

Compute Probability of Success

8 of 40

Defining Probability of Success

9 of 40

Augmenting Control Arm with External Data from Similar Populations

Advantages	Disadvantages
Increase power Reduce type I error rate Improve accuracy of treatment effect estimate Reduce patient burden Accelerate drug development	Studies often differ with respect to patient characteristics, study design, clinical site, etc. If not properly accounted for, confounding factors result in Bias Inflated type I error rate

10 of 40

Two Popular Bayesian Methods for Borrowing External Data

10

Both methods:

Can account for the presence of prior data conflict
Quantify external data into a prior distribution for the current control arm

Raises external data likelihood to some power bounded between 0 and 1
Ignores between-study heterogeneity
Must pre-specify amount of borrowing

Power Prior

Bayesian hierarchical model accounts for between-study heterogeneity
Adaptively determines how much to borrow
Generally outperforms power prior when heterogeneity is present

Meta-Analytic Predictive (MAP) Prior

11 of 40

Meta-Analytic Predictive (MAP) Prior

Total number of successes in j^thcontrol group

Models the log odds of response

Assumes the log-odds of response are exchangeable

12 of 40

An Important Extension: Using Patient-Level Covariates

px1 vector of baseline covariates

Indicator for belonging to the current study

13 of 40

Propensity-Score-Based Meta-Analytic Predictive (PS-MAP) Prior

01

02

03

14 of 40

Propensity-Score-Based Meta-Analytic Predictive (PS-MAP) Prior

04

Derive stratum-specific MAP priors

15 of 40

Propensity-Score-Based Meta-Analytic Predictive (PS-MAP) Prior

Propensity Score

Current study

External Data

…

Trimmed

Strata 1

Strata 2

Strata S

16 of 40

Propensity-Score-Based Meta-Analytic Predictive (PS-MAP) Prior

05

Derive the PS-MAP prior using a weighted sum of the stratum-specific MAP Priors

03

06

Tune the value of t using an iterative procedure in order to achieve a prespecified target effective sample size (ESS)
Approximate PS-MAP using a mixture of beta distributions

17 of 40

Outcome Analysis

18 of 40

Application in Non-Small Cell Lung Cancer (NSCLC)

18

Consider: Two-arm Ph II RCT comparing a new treatment to chemotherapy in patients with NSCLC

Ph II Study Design

ORR in the treatment arm: 27/80 (33.8%)
ORR in the control arm: 8/40 (20%)
Difference in response rate: 13.8%

Ph II Study Results

19 of 40

Planned Phase III Trial

Primary endpoint: progression-free survival (PFS)

Hypothesis: new treatment will improve median PFS by 3.5 months

Hypothesized hazard ratio (HR): 0.65
227 events required for 90% power based on a 2-sided log-rank test

Investigators would like to augment the control arm of the phase II trial prior to computing PoS:

One source with 200 subjects
5 baseline covariates: age, sex, race, smoking status, presence of brain metastases
PS-MAP with S=3 and PESS = 40

20 of 40

Computing PoS

21 of 40

Utilizing External Data Increases Probability of Success in Ph III

	No Borrowing	With Borrowing
Probability of Success	48.3%	64.2%
Posterior Hazard Ratio	0.81	0.72
Posterior 95% Credible Interval	0.54, 1.14	0.51, 0.98

22 of 40

Leveraging External Data Sources Can Improve Probability of Success

Augmenting the control arm

Improves the Performance of PoS

Facilitates robust decision-making:

Straightforward Implementation

Increases power
Increases precision
Reduces RMSE

Larger values, on average, when treatment is effective
Smaller values, on average, when treatment is ineffective

Less likely to abandon a promising treatment
Less likely to waste resources on unpromising treatment

MAP: population summary data is available
PS-MAP: patient level covariates are available

23 of 40

Thank You

Further details and a full simulation study are available:

Proper, J. L., Bunn, V., Hupf, B., & Lin, J. (2024). Predicting Probability of Success for Phase III Trials via Propensity-Score-Based External Data Borrowing. Statistics in Biopharmaceutical Research, 16(3), 348–360. https://doi.org/10.1080/19466315.2023.2292815

24 of 40

Backups

25 of 40

Simulation Study Design

26 of 40

Context

Phase II Trial

80% power

Study Success Criterion: > 0.95
One sided alpha level = 0.05

External Controls

3 external controls of size 100
Augment the current control using:

MAP
PS-MAP; PESS = 30
PS-MAP; PESS = 60

Phase III Trial

Frequentist inference
One-sided alpha level = 0.025

90% power

Fisher’s exact test

Goal: Evaluate the impact of external data borrowing on PoS compared to no borrowing

27 of 40

Scenarios

We assumed that each of these covariates was associated with the outcome of interest and therefore generated binary outcomes using a logistic regression model

We considered 4 different treatment effect configurations in order to determine whether the impact of external data borrowing on PoS differs according to the actual power of the design.

We considered 3 different amounts of between study heterogeneity by modifying the expected response rates in the 3 external controls.

Panel 2 is the mild heterogeneity scenario as the expected response rates in the external controls are still similar, however the average response rate across the external controls is lower than the current trial.

Overall, we considered a total of 12 different scenarios since for each treatment effect, we considered 3 different amounts of heterogeneity.

28 of 40

Evaluating PoS

29 of 40

Overview

Simulate phase II and external data

Approximate PoS

Was the treatment declared efficacious in phase II or were results clinically meaningful?

If NO

If YES

Stop

Was the estimated treatment effect from phase II ≤ truth?

Optimistic ph2 Results

Pessimistic ph2 Results

If YES

If NO

Estimate p(Δ|data) using 4 IBB models

Repeat until 2000 in groups 1 & 2

30 of 40

Simulation Study Results

31 of 40

Data generation: homogeneous

32 of 40

Data generation: mild heterogeneity

33 of 40

Data generation: extreme heterogeneity

34 of 40

Estimated Treatment Effect ≤ Truth and Treatment is Effective

Takeaways:

External data borrowing produced larger PoS values compared to no borrowing
PS-MAP (PESS = 60) exhibited best performance

35 of 40

Treatment is Effective and Observed Treatment Effect ≤ Truth

Takeaways:

Pr(PoS > given threshold) generally largest for PS-MAP
Larger increases in PoS were observed as the observed power of the phase II trial decreased

36 of 40

Treatment is Ineffective

Takeaways:

Borrowing has a large impact on PoS in scenarios where you observe a positive result by chance
Borrowing greatly reduces our chances of incorrectly proceeding to a phase III trial when committing a type I error

37 of 40

External Data Sources

Real-World Data (RWD): “Data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources”
Real-World Evidence (RWE): “The clinical evidence regarding the usage, and potential benefits or risks, of a medical product derived from analysis of RWD”
Sources of RWD:

Historical data from previous clinical trials
Procedure or disease registry
Electronic health records (EHRs)
Medical claims and billing data
Patient-reported outcomes

38 of 40

Probability of Success: Estimation

39 of 40

Overlapping Coefficient

40 of 40

Finding the Target PESS for PS-MAP