1 of 93

Study design & analysis,� part II

Carl

MMED 2024

Earlier versions by Jim Scott, Travis Porco, & Reshma Kassanjee

000

2 of 93

In some population, people are dying from an illness
A researcher claims to have a treatment that may reduce illness-related mortality
He had volunteers from the population try the treatment
Mortality among those treated was lower than what was observed in the general population�

Should people be convinced to buy the treatment? � Why or why not?�

2

Study 1

3 of 93

Study 1

Volunteers may be different in some way to the rest�of the population

E.g. Volunteers were younger on average compared �to people selected at random�

Was the lower mortality observed because

the participants were volunteers (who were younger) �-OR-
the treatment actually worked?

3

4 of 93

Study 1

Now, what if the results had gone the other way? That is, higher mortality in that population - do we know the treatment is harmful?

Again: volunteers may be different in some way to the rest of the population

E.g. Volunteers might know they are at high risk, and seek any possible way to mitigate that risk�

4

5 of 93

The researcher developed a treatment that he felt could alleviate symptoms
He selected a group of participants, and provided the treatment over 6 months
He found that symptoms measured at 6 months were reduced compared to symptoms measured at the start

… yet people were not convinced to buy the treatment. � Why?�

5

Study 2

6 of 93

Study 2

If the participants were people having particularly bad weeks, then we would see them return to their average over time

If the participants were people who had just acquired the illness, it could be the natural course the illness

Maybe people were just relieved to be trying a treatment, which led them to experience their symptoms differently�

Was the reduction in symptoms due to the treatment?

6

7 of 93

Study design and analysis,�part II:�Randomised controlled trials

000

8 of 93

Goals

Confounding
Describe randomised controlled trials (RCTs) � and the central design elements
Limitations, bias and variability

Tutorial

8

9 of 93

Goals

Confounding
Describe randomised controlled trials (RCTs) � and the central design elements
Limitations, bias and variability

Tutorial

9

10 of 93

Confounding

Study 1:

Was the lower mortality observed because

the participants were volunteers (who were younger) -OR-
the treatment actually worked?

11 of 93

Confounding

A concept relevant to causal inference�I.e. when we are trying to estimate the direct impact of an exposure on an outcome
In a setting and analysis:

the relationship or association between the exposure and outcome ≠ the causal effect

There is some common cause of the exposure and the outcome

12 of 93

Confounding

Even if there is no treatment effect, treatment will be associated with lower mortality

Exposure:�Treatment

Age

Outcome:

Mortality

13 of 93

Confounding

Even if there is no treatment effect, treatment will be associated with lower mortality

Exposure:�Treatment

Age

Outcome:

Mortality

14 of 93

Confounding

Backdoor non-casual paths between the exposure and outcome

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

15 of 93

Confounding

Exposure:�Treatment

Age

Outcome:

Mortality

Questions?

16 of 93

Confounding

There are methods to account for confounding in the analysis stage: e.g. stratification, and adjusting
But we cannot always adjust for confounders in an analysis. Why?

Exposure:�Treatment

Confounder:�Age

Outcome:

Mortality

17 of 93

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

18 of 93

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

Unknown

confounder

19 of 93

We know how confounding occurs, �but we have not or cannot measure all the confounders

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

20 of 93

Do not know what other confounders there are

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

Unknown

confounder

21 of 93

Goals

Confounding
Describe randomised controlled trials (RCTs) � and the central design elements
Limitations, bias and variability

Tutorial

21

22 of 93

Randomised controlled trials

23 of 93

Randomisation

Randomly assign participants to one of the treatment groups. I.e. what treatment a person receives depends only on chance.

Ensures that the treatment assignment is independent of all possible confounders
Groups are essentially the ‘same’ with respect to all variables – they differ only by chance
Only difference between the groups is in what treatment they receive

23

24 of 93

Randomisation

Exposure:�Treatment

Age

Outcome:

Mortality

25 of 93

Randomisation

Exposure:�Treatment

Age

Outcome:

Mortality

26 of 93

Randomisation

Exposure:�Treatment

Age

Outcome:

Mortality

27 of 93

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

Unknown

confounder

28 of 93

Exposure:�Treatment

Age

Outcome:

Mortality

Attitude -�wellbeing

Exercise

Wealth

Free time

Access to healthcare

Unknown

confounder

29 of 93

Randomisation

Exposure:�Treatment

Age

Outcome:

Mortality

Questions?

30 of 93

Study designs

Correlational/ecological
Cross-sectional
Case-control
Cohort

RCTs (clinical trials) Experimental

30

Observational

Ethics

31 of 93

Randomised controlled trials

Randomisation

32 of 93

Randomised controlled trials

?

33 of 93

Study 2:

Was the reduction in symptoms due to the treatment?

What can we include in our study design to help us answer this?

34 of 93

Randomised controlled trials

Suitable ‘control’ group (or arm) for comparisons

Ethical considerations

35 of 93

Masking and placebos

Knowing the treatment assignments may lead to a biased (often inflated) estimate of the treatment effect
Participants or evaluators may be biased in reporting or measuring of outcomes (e.g. symptoms)

Masking/blinding: Participants (and evaluators) do not know to which treatment group each participant has been assigned�

Single-blind (participants), double-blind (and evaluators), triple-blind (and data analysts)

35

Ask why first

E.g. flu shot to those who do not.

Those who have it, more inclined to dismiss symptoms as tiredness or a cold,

Those who did not, may really be looking out for every symptom and reporting each

Inflated effect.

Those who did not – may be more cautious about their exposure, knowing they are still at higher risk and quickly protect themselves when ppl in their networks have flu.

Behaviourial change.

Dampening of effect.

Smoking cessation – tablets.

Evaluators who are excited about the tablet – may reach out to participants taking the drug more to ask them whether they’ve quit, encourage them more to do so.

This makes them quit more rathe than the drugs.

Inflated effect.

Introduced Bias.

Placebo – treatment that has no therapeutic value. But in this case, would be made to resemble the treatment being tested.

So in the smoking example, sugar tablet that the arm not getting the drug takes – so the evaluators and participants do not know.

A somewhat separate form of bias can be introduced by the ‘Placebo’ effects.

Suppose a treatment should have no real value – you do not know this, so testing it.

For participants, the belief that they are treating something causing them distress and they have been in some way empowered can be powerful.

It is argued that that belief can actually translate into physiological changes in the body, especially for conditions such as depression.

The act of feeling there has been an intervention itself could improve outcomes – even if the treatment is not actually not really useful otherwise.

And so you want to try have this affect of in arms to reduce bias that would be caused if it were only in some.

e.g not always possible – surgery.

36 of 93

Masking and placebos

Placebo: a ‘treatment’ that has no therapeutic effect, that resembles and is administered in the same way as the treatment being tested

Assists with masking
Accounts for the placebo-effect: change in a �person’s outcomes because of the perception of the intervention

Not always possible or ethical to mask treatment

36

Ask why first

E.g. flu shot to those who do not.

Those who have it, more inclined to dismiss symptoms as tiredness or a cold,

Those who did not, may really be looking out for every symptom and reporting each

Inflated effect.

Those who did not – may be more cautious about their exposure, knowing they are still at higher risk and quickly protect themselves when ppl in their networks have flu.

Behaviourial change.

Dampening of effect.

Smoking cessation – tablets.

Evaluators who are excited about the tablet – may reach out to participants taking the drug more to ask them whether they’ve quit, encourage them more to do so.

This makes them quit more rathe than the drugs.

Inflated effect.

Introduced Bias.

Placebo – treatment that has no therapeutic value. But in this case, would be made to resemble the treatment being tested.

So in the smoking example, sugar tablet that the arm not getting the drug takes – so the evaluators and participants do not know.

A somewhat separate form of bias can be introduced by the ‘Placebo’ effects.

Suppose a treatment should have no real value – you do not know this, so testing it.

For participants, the belief that they are treating something causing them distress and they have been in some way empowered can be powerful.

It is argued that that belief can actually translate into physiological changes in the body, especially for conditions such as depression.

The act of feeling there has been an intervention itself could improve outcomes – even if the treatment is not actually not really useful otherwise.

And so you want to try have this affect of in both arms to reduce bias that would be caused if it were only in some.

e.g not always possible – surgery.

37 of 93

Let’s talk through a simple randomized controlled trial….

37