AI/ML in Drug Development:�Industry trends in decision-making and other applications

​

E. Andrés “Andy” Houseman

​

Senior Project Leader

Statistical Innovation Hub

Biostatistics, Sanofi

The opinions expressed in this presentation are those of the individual author and should not be interpreted as official Sanofi position.

Personal Introduction

E. Andrés Houseman, Sc.D.

18-Apr-24

2

Outline

AI in Drug Development: Increasing interest in AI/ML in Pharmaceutical Development

​

Statistician vs. Data Scientist: what are the roles?

​

Sanofi’s Statistical Innovation Hub: How we are using AI/ML

• Trial Design
• Decision-making

E. Andrés Houseman, Sc.D.

18-Apr-24

3

What is Artificial Intelligence (and Machine Learning)?

What is your definition?

E. Andrés Houseman, Sc.D.

18-Apr-24

4

What is Artificial Intelligence (and Machine Learning)?

Some different definitions

​

• AI is the endeavor of simulating human intelligence through machines, especially computer systems.
• Encapsulates processes like learning, reasoning, and self-correction.

​

• AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems, as opposed to the natural intelligence of living beings.

​

• ML is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus perform tasks without explicit instructions.

​

• AI is the theory of computer systems that can perform complex tasks. Machine learning (ML) is the process by which computers, including AI, learn new things

​

• FDA: “Artificial Intelligence (AI) and Machine Learning (ML) can be described as a branch of computer science, statistics, and engineering that uses algorithms or models to perform tasks and exhibit behaviors such as learning, making decisions, and making predictions. ML is considered a subset of AI that allows models to be developed by training algorithms through analysis of data, without models being explicitly programmed.”

​

​

​

​

​

E. Andrés Houseman, Sc.D.

18-Apr-24

5

Regulatory Stance on AI/ML

• Major Health Authorities are interested in the topic of AI/ML in drug development and drug manufacturing

​

• Both the FDA and EMA have recently issued position papers, guidance documents, etc.

E. Andrés Houseman, Sc.D.

18-Apr-24

6

FDA (CDER, CBER, CDRH)

Role AI/ML is playing in drug development:

• Increased use
• Significant increase in the number of drug and biologic application submissions using AI/ML components over the past few years, with more than 100 submissions reported in 2021
• Drug discovery, clinical R&D, post-market safety surveillance, CMC

​

FDA’s perspective on the use of AI/ML in drug development:

• Committed to ensuring that drugs are safe and effective while facilitating innovations in their development.
• Plans to develop and adopt a flexible risk-based regulatory framework that promotes innovation and protects patient safety
• Discussion paper: AI/ML for Drug Development Discussion Paper

E. Andrés Houseman, Sc.D.

18-Apr-24

7

Why? Drug Development is Expensive!

E. Andrés Houseman, Sc.D.

18-Apr-24

8

“Data Science” (AI/ML) in the Pharmaceutical Industry

• Expand your definition of "data"

​

• Be bold (innovative)

​

• Balance promise & impact:
• Need for systematic process for showing value generation.

​

E. Andrés Houseman, Sc.D.

18-Apr-24

9

Examples from PharmaDS 2024

Regulatory & R&D Operations

• Moderna: Regbot (NLP-based solutions integrated into the business process)
• AstraZeneca: Automatic TLF generation
• AstraZeneca: Document creation for payers
• Amgen: Q&A system
• Amgen: Documentation summarization & comparison

​

Strategy/Competitive intelliegence

• Cornell: RWE/RWD to investigate drug repurposing (Cornell)
• BI: “Academic Assist” (real-time synchronization with medical literature)
• AbbVie: Marketing (modified multi-touch attribution methodology)
• Amgen: biomarker information extraction

​

CMC

• Moderna: Optimized scheduling of manufacturing patient-specific neoantigen therapies

E. Andrés Houseman, Sc.D.

18-Apr-24

10

Other Takeaways from PharmaDS 2024

Successful implementation: for AI to be successful, you need

• Good starting point (good target)
• Good feedback, and
• Timely iteration.

​

Importance of domain knowledge:

• Knowledge graphs / ontologies

​

Successful adoption:

• Data, technical infrastructure, skillset, change management, explainability, & trust

​

Challenges:

• Reproducibility of outputs (especially when regulators are involved)
• Traceability
• Data privacy, patient consent, ethical considerations, biases
• Intellectual Property
• Cross-pharma standards

E. Andrés Houseman, Sc.D.

18-Apr-24

11

Career Development: Statistician vs. Data Scientist

How do you distinguish the two roles?

E. Andrés Houseman, Sc.D.

18-Apr-24

12

PharmaDS 2024: Career Development

Role of Statistician vs. Data Scientist

​

Statistician

• Hypothesis-testing
• Focused prospectively on eliciting the true question from stakeholders

​

Data Scientist

• Learning & Decision-making
• Exploration/creativity/problem-solving
• “Art of the possible”
• Comfort with different data sources

Note: FDA has a new job category recognizing DS, distinct from statisticians

​

• DS is different to different people
• Need for multidisciplinary approach: spectrum of various technical roles & business functions
• Tradeoffs between compliance, creativity: need for rigor/formality to increase impact of DS in pharma
• “Embrace excitement in the DS field”
• “Recognize diversity of talent”
• Focus on making an impact (not just the appearance of innovation), at scale
• Communication skills as important as technical skills

E. Andrés Houseman, Sc.D.

18-Apr-24

13

• Study design
• Validation of results
• Assessment of bias & generalizability
• Causality vs. Association

​

Friedrich, S., Antes, G., Behr, S., Binder, H., Brannath, W., Dumpert, F., Ickstadt, K., Kestler, H.A., Lederer, J., Leitgöb, H. and Pauly, M., 2022. Is there a role for statistics in artificial intelligence?. Advances in Data Analysis and Classification, 16(4), pp.823-846.

• Model robustness & reproducibility
• Uncertainty & interpretability
• Data quality

​

Contributions of the Statistics discipline to AI/ML methods (alongside CS)

​

Rigorous evaluation of:

Role of Statisticians in AI

E. Andrés Houseman, Sc.D.

18-Apr-24

14

AI/ML for increased efficiency in trial design

McConnell, K.J. and Lindner, S., 2019. Estimating treatment effects with machine learning. Health services research, 54(6), pp.1273-1282.

E. Andrés Houseman, Sc.D.

18-Apr-24

15

AI/ML for increased efficiency in trial design - PROCOVA

RCT

prognostic scores

ANCOVA

For continuous (normal) y, no bias or Type I error inflation.

E. Andrés Houseman, Sc.D.

18-Apr-24

16

AI/ML for increased efficiency in trial design – simulations (1)

• Goal: Use simulation to investigate operating characteristics for different AI/ML methods
• Challenge: nonlinearities & high-order interactions will be very indication/population specific

​

• Requirement: simulations should replicate, as closely as possible, the actual associations present in the targeted population

​

• Partial Solution: Use RWD to learn the associations

​

E. Andrés Houseman, Sc.D.

18-Apr-24

17

AI/ML for increased efficiency in trial design – simulations (2)

E. Andrés Houseman, Sc.D.

18-Apr-24

18

AI/ML for increased efficiency in trial design – simulations (3)

Generalized Adversarial Networks

E. Andrés Houseman, Sc.D.

18-Apr-24

19

AI/ML for increased efficiency in trial design – simulations (4)

Preliminary Results (continuous endpoint):

PROCOVA, GRF, and (maybe) DML seem promising

Notes:

1. DML Type I error control appears to depend on sample size.
2. PROCOVA always requires RWD:
• If used only prospectively in a proper RCT context, the method controls Type I error and offers modest power improvements.
• Unclear how to apply the methodology with EC augmentation, a-priori reasonable approaches appear to inflate Type I error.

E. Andrés Houseman, Sc.D.

18-Apr-24

20

Decision Problem (sig testing to 3ODM)

Sargent, D.J., Chan, V. and Goldberg, R.M., 2001. A three-outcome design for phase II clinical trials. Controlled Clinical Trials, 22(2), pp.117-125.

• Three-outcome-decision-making (3ODM) was designed to address ambiguous results with an additional outcome, “consider”
• Allows additional data sources to be incorporated in the decision.
• Typically requires two “reference values”, an upper and lower target.

E. Andrés Houseman, Sc.D.

18-Apr-24

21

Decision Problem

E. Andrés Houseman, Sc.D.

18-Apr-24

22

Decision Problem (Conditional Assurance)

Temple, J.R. and Robertson, J.R., 2021. Conditional assurance: the answer to the questions that should be asked within drug development. Pharmaceutical Statistics, 20(6), pp.1102-1111.

E. Andrés Houseman, Sc.D.

18-Apr-24

23

Decision Problem (Conditional Assurance: an example)

• Conditional Assurance provides a quantitative understanding of the information (de-risking of overall project success) gained by a given trial.

​

• Our new system operationalizes this concept at scale:
• Internal R package: use now required for presentations to governance committees.
• Software design based on S4 classes that represent interoperable components that can be combined in novel ways to represent a large variety of different project strategies.

​

​

​

​

Temple, J.R. and Robertson, J.R., 2021. Conditional assurance: the answer to the questions that should be asked within drug development. Pharmaceutical Statistics, 20(6), pp.1102-1111.

E. Andrés Houseman, Sc.D.

18-Apr-24

24

Focus on the label!

Project teams forced to consider the definition of project success.

• Ultimately depends on the aspirational label:
• Title 21 of the Code of Federal Regulation:
• Established name of the drug and the name and quantity of each components
• Adequate direction for use, including conditions and purposes, drug dosage, timing and route of administration
• Conditions and purposes are often presented quantitatively in terms of specific indications & endpoints
• Requires a statistically valid multiple testing strategy
• State-of-the-art is graphical hierarchical techniques.

​

Aspirations: support for optimizing the label, including possibly multiple testing strategy

Bretz, F., Maurer, W. and Maca, J., 2014. Graphical approaches to multiple testing. In Clinical trial biostatistics and biopharmaceutical applications (pp. 349-394). Taylor & Francis, Boca Raton, FL.

Maurer, W. and Bretz, F., 2013. Multiple testing in group sequential trials using graphical approaches. Statistics in Biopharmaceutical Research, 5(4), pp.311-320.

E. Andrés Houseman, Sc.D.

18-Apr-24

25

Decision Problem: Incorporating Additional Data

Need to understand the link between phase 2 and phase 3 endpoints (if different):

• Meta-analysis from literature review
• RWD

​

Need a better understanding of how dosing impacts efficacy & safety:

Quantitative Systems Pharmacology (QSP), Pharmacokinetics & Pharmacodynamics (PK/PD) [subdisciplines of translational modeling]

Proper, J.L., Bunn, V., Hupf, B. and Lin, J., 2024. Predicting Probability of Success for Phase III Trials via Propensity-Score-Based External Data Borrowing. Statistics in Biopharmaceutical Research, pp.1-13.

QSP Example: Thiel, C., Smit, I., Baier, V., Cordes, H., Fabry, B., Blank, L.M. and Kuepfer, L., 2018. Using quantitative systems pharmacology to evaluate the drug efficacy of COX-2 and 5-LOX inhibitors in therapeutic situations. NPJ systems biology and applications, 4(1), p.28.

Use of PK data: Gerard, E., Zohar, S., Thai, H.T., Lorenzato, C., Riviere, M.K. and Ursino, M., 2022. Bayesian dose regimen assessment in early phase oncology incorporating pharmacokinetics and pharmacodynamics. Biometrics, 78(1), pp.300-312.

E. Andrés Houseman, Sc.D.

18-Apr-24

26

Conclusions

AI in Drug Development: There is considerable AI/ML activity in the development sector of the pharmaceutical business

• Regulatory interest
• Numerous examples of applications in endpoint development, trial design, safety, regulatory, strategy, and CMC

​

Statistician vs. Data Scientist: Both data scientists and statisticians are still defining their roles relative to each other, especially with respect to AI/ML.

​

Sanofi’s Statistical Innovation Hub: active explorations in AI/ML:

• To design more efficient trials
• To improve decision-making

E. Andrés Houseman, Sc.D.

18-Apr-24

27

Questions?

28