1 of 24

Credibility Assessment of Patient-Specific Computational Models��Pras Pathmanathan, PhD�Division of Biomedical Physics�Office of Science and Engineering Laboratories (OSEL)�Center for Devices and Radiological Health (CDRH)�FDA

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

1

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

2 of 24

Disclaimer

The mention of commercial products, their sources, or their use in connection with material reported herein is not to be construed as either an actual or implied endorsement of such products by the Department of Health and Human Services.

The following slides represent personal views and opinions, and not those of FDA

2

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

3 of 24

Avenues for M&S in device regulatory submissions

In Silico Device Testing

Simulate a physical device to generate evidence for safety and/or effectiveness

Software in/as medical device

Algorithm within the device takes in patient data and simulates the patient

In Silico Clinical Trials

Proposed method where device performance is evaluated using a ‘virtual cohort’ of simulated patients.

Medical Device Development Tools (MDDTs)

MDDTs are tools, qualified by FDA, that device manufacturers can use to evaluate their device

OSEL VICTRE project

IT’IS

Heartflow

CardioInsight

Patient-specific models – Digital Twins

3

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

4 of 24

Cardiac electrophysiological models

4

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

5 of 24

Clinical tools vs virtual cohorts

g

Virtual cohort of ‘synthetic patients’

Model is/in the device

In scope

Out of scope as not patient-specific models

PSM workflow used in a clinical tool

Virtual cohort of patient-specific models

Used to evaluate a device (incl. in silico clinical trials)

Virtual cohort of ‘synthetic patients’

New model generated for every new patient

Fixed set of pre-generated PSMs, each corresponds to a real patient

Large number of models generated by sampling space – models do not correspond to any real patient

OSEL VICTRE project

5

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

6 of 24

Little formal credibility assessment

Growing awareness in devices community (incl. CDRH) of need for better credibility assessment

ASME V&V40 subcommittee formed

CDRH M&S Reporting Guidance

ASME V&V40 Standard

OSEL Credibility of Models Regulatory Science Program

2000s

~2010

2013

2016

2018

2020

Draft Guidance on Credibility Assessment

2021

Model Credibility Assessment

6

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

7 of 24

Verification, validation and uncertainty quantification

Computational Model

p1

p2

p3

p4

p5

y

‘quantity of interest’ (QOI)

Parameters

Verification

Validation

7

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

8 of 24

ASME V&V40 Standard

8

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

9 of 24

Project goals and approach

Goals

Approach

Identify unique considerations when assessing credibility of PSMs

Demonstrate VVUQ process for PSMs

Consider how ASME V&V40 can be applied to PSMs

Use cardiac electrophysiology as exemplar field

Review how cardiac PSMs are developed and consider how verification, validation and uncertainty quantification apply to cardiac PSMs

Perform two simulation studies using a cohort of 24 ventricular PSMs

Use results from above to generate PSM-specific V&V40 credibility factors and gradations

9

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

10 of 24

Project goals and approach

Goals

Approach

Identify unique considerations when assessing credibility of PSMs

Demonstrate VVUQ process for PSMs

Consider how ASME V&V40 can be applied to PSMs

Use cardiac electrophysiology as exemplar field

Review how cardiac PSMs are developed and consider how verification, validation and uncertainty quantification apply to cardiac PSMs

Perform two simulation studies using a cohort of 24 ventricular PSMs

Use results from above to generate PSM-specific V&V40 credibility factors and gradations

10

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

11 of 24

Verification of patient-specific models

	PSM as clinical tools	Virtual cohorts of PSMs
Code verification	Code verification theory and methods equally applicable to PSMs as generic models
Calculation verification	How much does discretization error vary across patients? Theoretically, software could do patient-specific automated mesh resolution analyses (or be adaptive)	How much does discretization error vary across patients?
Use error	For PSMs with semi-manual stages, possible user variability in subjectively-chosen inputs

11

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

12 of 24

Verification of patient-specific models

	PSM as clinical tools	Virtual cohorts of PSMs
Code verification	Code verification theory and methods equally applicable to PSMs as generic models
Calculation verification	How much does discretization error vary across patients? Theoretically, software could do patient-specific automated mesh resolution analyses (or be adaptive)	How much does discretization error vary across patients?
Use error	For PSMs with semi-manual stages, possible user variability in subjectively-chosen inputs

12

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

13 of 24

Verification of patient-specific models

	PSM as clinical tools	Virtual cohorts of PSMs
Code verification	Code verification theory and methods equally applicable to PSMs as generic models
Calculation verification	How much does discretization error vary across patients? Theoretically, software could do patient-specific automated mesh resolution analyses (or be adaptive)	How much does discretization error vary across patients?
Use error	For PSMs with semi-manual stages, possible user variability in subjectively-chosen inputs

13

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

14 of 24

Multi-patient mesh resolution study

We investigated if discretization error varies across patients

Used 24 ventricular models – generated four meshes for each at different resolutions

Computed apex-to-base activation time (normalized by distance) and ECG (lead V1)

	Average edge length (microns)	Number of elements (millions of elements)
Very low	1029 ± 1.6	1.15 ± 0.28
Low	630 ± 0.1	5.57 ±1.35
Medium	426 ± 0.5	16.7 ± 4.1
High	275 ± 0.02	69.1 ± 16.9

14

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

15 of 24

Multi-patient mesh resolution study

For lower resolution meshes, little variability in discretization error across patients

Multi-patient mesh resolution studies may not be critical (for these QOIs)

Greater variability for medium resolution meshes

Single patient convergence analysis could be misleading

15

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

16 of 24

Validation of patient-specific models

Review identified myriad approaches to validation of PSMs

Three example approaches validating a virtual cohort of PSMs:

Validate the COU quantity of interest for a subset of patients in the virtual cohort
Validate a different QOI for all (or some) patients in the virtual cohort
Perform population-level validation

For example, three approaches for validating a PSM clinical tool:

A clinical study directly validating the final model-derived tool output

A clinical study validating predictions of ‘intermediate’ model outputs (not the final tool output)

Patient-specific validation - every time the tool is used on a new patient, some patient data is used for personalization (as normal) and some further data is used in validation.

16

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

17 of 24

Validation of patient-specific models

Review identified myriad approaches to validation of PSMs

Three example approaches validating a virtual cohort of PSMs:

Validate the COU quantity of interest for a subset of patients in the virtual cohort
Validate a different QOI for all (or some) patients in the virtual cohort
Perform population-level validation

For example, three approaches for validating a PSM clinical tool:

A clinical study directly validating the final model-derived tool output

A clinical study validating predictions of ‘intermediate’ model outputs (not the final tool output)

Patient-specific validation - every time the tool is used on a new patient, some patient data is used for personalization (as normal) and some further data is used in validation.

17

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

18 of 24

Uncertainty quantification for patient-specific models

Similarly, there are many possibilities when considering UQ of PSMs

Critical to distinguish between personalized and non-personalized inputs:

Personalized inputs:

Uncertainty due to measurement error

Non-personalized:

Uncertainty due to

population variability

18

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

19 of 24

UQ for a virtual cohort study

Goal: show that the study conclusion is insensitive to uncertainty in key inputs such as border zone region

For border zone (personalized input), we generated perturbed meshes for each patient
Level of expansion/contraction based on assessment of error in image acquisition/segmentation

19

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

20 of 24

UQ for a virtual cohort study

VT risk was computed at both pacing sites using all new meshes

Considered three cases:

Systematic error, BZ always under-estimated:

Using ‘Expanded BZ’ results (left), differences still statistically significant

Systematic error, BZ always under-estimated:

Using ‘Contracted BZ’ results (left), differences still statistically significant

Unbiased error

T-tests with different meshes for each patient
Randomly sampled N=10⁶sets of results from the 3²⁴possibilities
For all cases, differences still statistically significant

20

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

21 of 24

PSM considerations using ASME V&V40 2018

Category	V&V40 Credibility factor	PSM-CT	PSM-VC
Verification	Software quality assurance	No unique PSM considerations identified.
	Numerical code verification
	Discretization error	Gradations could account for number of patients these activities are performed on (see Table 5 & Supplement).
	Numerical solver Error
	Use error	For PSMs, with manual stages, there is a possibility of user variability related to subjectively chosen inputs. See supplement for example gradation accounting for this.	No unique PSM considerations identified.
Validation – model	Model form	No unique PSM considerations identified.
	Model inputs – quantification of sensitivities	For PSMs, sensitivity analysis and uncertainty quantification are intimately linked, so an alternative approach is have a single sensitivity analysis and uncertainty quantification factor with possible subfactors: Inputs analyzed Rigor of input uncertainty characterization Number of patients Output quantities analyzed
	Model inputs – quantification of uncertainties
Validation -comparator	Quantity of Test Samples	There may be unique PSM considerations, but these will be dependent on the specific validation activities performed. For some cases where a PSM is validated against clinical data, these factors could be interpreted as: Number of validation subjects Range of characteristics of validation subjects Patient data Patient measurements See supplement for example gradations.
	Range of Characteristics of Test Samples
	Characteristics of Test Samples
	Measurements of Test Samples
	Quantity of Test Conditions	There may be unique PSM considerations, but these will be dependent on the specific validation activities performed.
	Range of Test Conditions
	Measurements of Test Conditions
	Uncertainty of Test Condition Measurements
Validation - comparison	Equivalency of Input Parameters	No unique PSM considerations identified.
	Output Comparison – quantity
	Equivalency of Output Parameters
	Agreement of Output Comparison
	Rigor of Output Comparison
Applicability	Relevance of the QOIs
	Relevance of the Validation Activities to the COU	For PSMs, assessing the relevance of the validation subjects to the full patient population / full virtual cohort is a component of applicability assessment

Goal: show how important characteristics of PSMs can be considered when using ASME V&V40

Approach:

Considered each ASME V&V40 credibility factor in turn
Identified any unique considerations related to PSMs for each factor
Proposed gradations relevant to PSMs

See publication for full details

21

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

22 of 24

Example – “Discretization error” factor

Example gradation in ASME V&V40	Possible gradation for PSMs as clinical tools	Possible gradation for virtual cohorts of PSMs
(a) No grid or time-step convergence analysis was performed to estimate the discretization error. (b) Applicable grid or time-step convergence analyses were performed and their respective convergence behaviors were observed to be stable, but the discretization error was not estimated. (c) Applicable grid or time-step convergence analyses were performed and discretization error was estimated.	(a) No grid or time-step convergence analysis was performed to estimate the discretization error. (b) Applicable grid or time-step convergence analyses were performed, using one or a small number of patients, and their respective convergence behaviors were observed to be stable, but the discretization error was not estimated. (c) Applicable grid or time-step convergence analyses were performed and discretization error was estimated, using one or a small number of patients. (d) Applicable grid or time-step convergence analyses were performed and discretization error was estimated using a range of representative patients (e) Applicable grid or time-step convergence analyses are automatically performed on each new patient, and discretization error is estimated, and used in processing the results.	(a) No grid or time-step convergence analysis was performed to estimate the discretization error. (b) Applicable grid or time-step convergence analyses were performed and their respective convergence behaviors were observed to be stable, but the discretization error was not estimated, using one or a small number of subjects. (c) Applicable grid or time-step convergence analyses were performed and discretization error was estimated, using one or a small number of subjects. (d) Applicable grid or time-step convergence analyses were performed and discretization error was estimated, using a range of patients covering the virtual cohort.

22

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

23 of 24

Summary and take-home messages

Patient-specific modeling has the potential to help achieve personalized medicine
Patient variability leads to many challenges in both development and assessment
Paper is the first general publication on credibility assessment of PSMs

Highlighted two distinct use cases of PSMs: clinical tools and virtual cohorts
Verification: relatively straightforward
Validation: many approaches to validating PSMs which makes unified framework challenging
UQ: important to distinguish between personalized vs non-personalized, we demonstrated UQ process with a virtual cohort study

Future frameworks for PSM credibility assessment will help accelerate translation of PSMs from research to clinical

23

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023

24 of 24

Thank you for your attention

Acknowledgements (paper co-authors)

Suran Galappaththige (FDA)
Richard Gray (FDA)
Steven Niederer (King’s College London)
Caroline Mendonca Costa (King’s College London)

24

OSEL Accelerating patient access to innovative, safe, and effective medical devices through best-in-the-world regulatory science IMAG Feb 2023