1 of 23

Risk Calculators

Sam Weisenthal

MEP Lecture

Preceptors: Dr. Carroll, Dr. Mooney

2 of 23

Learning Objectives

​

​

  • Know enough about risk calculator development to be informed users
  • Be able to incorporate risk estimates with patient costs to make optimal medical decisions

​

3 of 23

Definition

​

  • Risk = (Probability Event) x (Cost Event)
  • Risk = Probability Event

4 of 23

Broad Categories

​

  • Diagnosis
  • Prognosis

​

5 of 23

How Risk Calculators Can Help You

​

  • High stakes decisions
    • Procedure → complications
    • Medication → toxicity
  • Single-number summary
  • Training

​

6 of 23

Terminology

​

  • P(Stroke given patient’s Age, Cholesterol, etc.)

  • Outcome
  • Response
  • Endpoint
  • Target
  • Dependent Variable

​

  • Covariates
  • Risk Factors
  • Predictors
  • Features
  • Explanatory Variables
  • Independent Variables

​

7 of 23

Example Calculators

​

  • Atherosclerotic Cardiovascular Disease
  • Breast Cancer
    • Gail Risk Model
      • Test result as risk factor

​

8 of 23

Model Development: Variables

  • Outcome
  • Outcome of interest?
  • Covariates
  • All necessary covariates?

​

​

9 of 23

Model Development: Form

  • Assumptions
    • Distributions
    • Equation
  • Assumptions

​

​

10 of 23

Model Development: Study Design

  • Given model, collect data
    • Prospective (cohort) - prognostic
    • Cross sectional - diagnostic
  • Collect data, make model
    • Retrospective (case-control)
  • Randomized Controlled Trial

​

​

11 of 23

Model Development: Sample

  • Underrepresentation → poor predictions
    • Rare groups
  • Selection bias

​

​

12 of 23

Model Development: Validation

  • Clinical Benefit
    • Decision curve analysis
  • Accuracy of risk estimates (calibration)
  • Ability to rank (discrimination)
  • Validated in setting of use?
  • Validated recently?
  • Validated for different groups?

​

​

13 of 23

Things to Watch for

  • Outcome of interest?
  • Accounts for necessary variables?
  • Assumptions
  • Data Collection
  • Underrepresentation?
  • Selection bias
  • Validated in setting of use?
  • Validated recently?
  • Validated for different groups?

​

​

​

14 of 23

Other Data Sources

  • Wearables
  • Images
  • Genetic

​

​

15 of 23

Using Risk, Cost to Make Decision

​

  • Patient: 95 y.o. male, tumor
  • Decision: observe, radiotherapy, or surgery?
  • Costs/utilities:
    • No treatment: 6 months
    • Radiotherapy: 15-1 = 14 months
    • Surgery: if successful, 20-1 = 19 months
      • Risk of death 40%
      • 0.40 x 0 + 0.60 x 19 = 11.4 months

​

16 of 23

Using Risk, Cost to Make Decision

​

  • Patient: 95 y.o. male, tumor
  • Decision: observe, radiotherapy, or surgery?
  • Costs/utilities:
    • No treatment: 6 months
    • Radiotherapy: 15-1 = 14 months
    • Surgery: if successful, 20-1 = 19 months
      • Risk of death 40%
      • 0.40 x 0 + 0.60 x 19 = 11.4 months

​

17 of 23

Using Risk, Cost to Make Decision

​

  • Patient: 95 y.o. male, tumor
  • Decision: observe, radiotherapy, or surgery?
  • Costs/utilities:
    • No treatment: 6 months
    • Radiotherapy: 15-1 = 14 months
    • Surgery: if successful, 20-1 = 19 months
      • Risk of death 20%
      • 15.2 months

​

18 of 23

Using Risk, Cost to Make Decision

​

  • Patient: 95 y.o. male, tumor
  • Decision: observe, radiotherapy, or surgery?
  • Costs/utilities:
    • No treatment: 6 months
    • Radiotherapy: 15-1 = 14 months
    • Surgery: if successful, 20-1 = 19 months
      • Risk of death 20%
      • 15.2 months

​

19 of 23

Using Risk, Cost to Make Decision

​

  • Patient: 95 y.o. male, tumor
  • Decision: observe, radiotherapy, or surgery?
  • Costs/utilities:
    • No treatment: 6 months
    • Radiotherapy: 15-1 = 14 months
    • Surgery: if successful, 20-1 = 19 months
      • Risk of death interval: (15%, 50%)
      • (9.5 months, 16.15 months)

​

20 of 23

Using Risk, Cost to Make Decision

​

  • Patient: 95 y.o. male, tumor
  • Decision: observe, radiotherapy, or surgery?
  • Costs/utilities:
    • No treatment: 6 months
    • Radiotherapy: 15-1 (???) = 14 months
    • Surgery: if successful, 20-1 (???) = 19 months

​

​

21 of 23

References

​

​

​

​

​

​

  • Van Smeden, Maarten. Introduction to Prediction Modeling. Berlin, 2018. https://www.slideshare.net/MaartenvanSmeden/introduction-to-prediction-modelling-berlin-2018-part-i
  • Yadlowsky S, Hayward RA, Sussman JB, McClelland RL, Min YI, Basu S. Clinical implications of revised pooled cohort equations for estimating atherosclerotic cardiovascular disease risk. Ann Intern Med. 2018 Jul 3;169(1):20.
  • Harrell Jr FE. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Springer; 2015 Aug 14.
  • Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian data analysis. Chapman and Hall/CRC; 2013 Nov 27.

​

22 of 23

Other Resources

​

  • Clinical prediction models:
    • http://hbiostat.org/doc/bbr.pdf, https://www.fharrell.com
    • http://www.clinicalpredictionmodels.org
    • Vickers, Andrew J., and Elena B. Elkin. "Decision curve analysis: a novel method for evaluating prediction models." Medical Decision Making 26.6 (2006): 565-574.
    • Collins, Gary S., et al. "Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement." BMC medicine 13.1 (2015): 1.
  • Decision theory:
  • Risk communication:

​

23 of 23

Thank You!

​

  • samuel_weisenthal@urmc.rochester.edu

​