Elicitation Training for Survival Extrapolation

��

January 2025

Instructions

• Suggested training materials including content specific to survival extrapolation
• Edit and incorporate within any other training content as you wish!
• But please could you give an acknowledgement, referring to the slides provided at https://shelf.sites.sheffield.ac.uk/survival-extrapolation

Contents

1. General probability elicitation

​

• Survival extrapolation

General probability elicitation

My uncertainty about the distance

My uncertainty about the distance

• I think distances around 200 miles are more likely than distances around 300 miles

My uncertainty about the distance

• For the same reward, I’d have no preference on betting whether the distance is inside or outside the red section (190-240 miles)

Poor choices of distribution

• Obtaining a suitable distribution is hard, but easy to identify inappropriate representations of uncertainty

How we obtain probability distributions

• We do not ask you to specify distributions directly
• We will ask you to make a small number of probability judgements
• We will then choose a distribution consistent with these, and present to you for feedback

Biases

• A large body of research on how people make probability judgements, and what can go wrong.
• Some biases to be aware of:
• Overconfidence
• Anchoring
• Availability

Overconfidence

• Out of all occasions we are ‘90% sure’ we are right, we should be right 90% of the time (so wrong 10% of the time)
• but experiments typically show when we claim to be 90% sure we are right, we are right far less often, e.g. 60% of the time

​

Overconfidence

• Unlikely events/small probabilities difficult to think about
• tails of a distribution should not look too plausible

Anchoring

Undue influence of a single value. Example:

• Q1a Is the population of South Africa greater/less than 8 million?
• Q2a Estimate the population of South Africa. Answers in millions:

25, 17, 11, 14, 14, 10, 15, 17, 11, 15, 100, 20, 12, 30, 11, 21, 20, 10, 16, 20, 18

Anchoring

• Q1b Is the population of South Africa greater/less than 85 million?
• Q2b Estimate the population of South Africa. Answers in millions:

60, 75, 70, 55, 60, 60, 50, 70, 50, 85, 100, 45, 35, 15, 90, 90

Availability

• Probability linked to how easily some event can be recalled
• Some events more memorable than others
• Need to be cautious with anecdotal evidence; individual patients

Elicitation Protocol

• We will use the Sheffield Elicitation Framework (SHELF) to elicit probability judgements in a way that reduces these biases, whilst enabling sharing of expertise
• You will be asked to make probability judgements individually first, without conferring
• Judgements will be presented to the group (next meeting)
• Typically suggests more uncertainty than in any one individual; provides multiple anchors

​

Elicitation Protocol

• Facilitated group discussion will follow
• Includes discussion of qualitative opinions
• Group will propose single set of probability judgements from perspective of a "Rational Impartial Observer" as a fair way of reporting any residual disagreement

​

​

​

Survival extrapolation

• Survival data from trial available: presented as survivor functions
• Aim is to quantify uncertainty about extrapolated survival probabilities
• Will review survivor/hazard functions
• Will consider qualitative judgements about hazard functions

Survivor function

Hazard functions

Hazard functions

• e.g. consider cohort-level ‘risk of death’ over a 6 month period. For those surviving this period, does the cohort-level risk of death over the next 6 months
• stay the same;
• decrease;
• increase?
• Factors influencing hazard include characteristics of patient group, disease natural history, and mechanism of treatment action

UK female population hazard function

Survivor and hazard functions

Model-based versus expert extrapolation

From NICE TA621 Osimertinib for untreated EGFR mutation-positive non-small-cell lung cancer

• Questions?

​

• Training exercise

References

• Office for National Statistics (ONS), released 11 January 2024, ONS website, statistical bulletin, National life tables – life expectancy in the UK: 2020 to 2022
• Loprinzi CL. Laurie JA. Wieand HS. Krook JE. Novotny PJ. Kugler JW. Bartel J. Law M. Bateman M. Klatt NE. et al. Prospective evaluation of prognostic variables from patient-completed questionnaires. North Central Cancer Treatment Group. Journal of Clinical Oncology. 12(3):601-7, 1994.
• Therneau T (2024). A Package for Survival Analysis in R. R package version 3.7-0, https://CRAN.R-project.org/package=survival.
• NICE (2021). Single Technology Appraisal: Osimertinib for untreated EGFR-positive non-small-cell lung cancer [ID1302]; Committee Papers. https://www.nice.org.uk/guidance/ta654/documents/committee-papers-5

​