Comparing More Than Two Means

CHANDLER, AZ

The 6 steps of statistical investigation

• Ask a research question
• Design a study and collect data
• Explore the data
• Draw inferences beyond the data (Logic of inference: Significance and estimation)
• Formulate conclusions (Scope of inference: Generalization and causation)
• Look back and ahead

Exercise �and �Brain Volume

Step 1: State the Research Question �

• Brain size usually shrinks as we age, and such shrinkage may be linked to dementia.
• Can we do something to protect against this shrinkage?
• More specifically, can different types of exercise/activity help prevent brain shrinkage in older adults?

​

​

​

J. Mortimer, D. Ding, A. R. Borenstein, C. DeCarli, Q. Guo, Y. Wu, Q. Zhao, & S. Chu (2012). “Changes in brain volume and cognition in a randomized trial of exercise and social interaction in a community-based sample of non-demented Chinese elders,” Journal of Alzheimer's Disease, 30(4): 757-66

Step 2: Design a Study and Collect Data �

• A study done in China randomly assigned older (non-dementia) subjects to one four groups: tai chi, walking, social interaction, none.
• The study started with a random sample of 250 residents (125 men and 125 women) aged 60-79 from a government list in a specific neighborhood in Shanghai.
• The potential subjects were visited to determine their willingness to participate and if they fit the inclusion criteria which include, no history of stroke, Parkinson’s, or other neurologic diseases, ability to walk 2 km, maintain balance, have a cognitive score high enough to rule out dementia, cardiovascular and muscular health strong enough for any intervention, or not currently participating in regular vigorous exercise or not currently participating in tai chi.

Step 2: Design a Study and Collect Data �

• Of the 250 potential participants, 51 men and 97 women (148 total) agreed to participate and met the inclusion criteria
• 11 dropped out before the study began
• 17 more were randomly dropped to obtain a sample size of 120
• 13 dropped out during the study
• Except for the group with no intervention, each group met for about an hour three times a week for 40 weeks to participate in their assigned activity.
• The tai chi group was led by a tai chi master
• The walking group walked around a track at their own speed
• The social interaction group met at a community center and discussed topics of interest to them (they also continued to meet after the study)
• The control group just received four phone calls during the 40 weeks

Step 2: Design a Study and Collect Data �

• Each participant had an MRI to determine brain volume before the study began and again at its end.
• Computed percentage increase or decrease in brain volume over the 40-week study period
• What is the explanatory variable? Type?
• What is the response variable? Type?

​

Step 2: Design a Study and Collect Data �

Did random assignment work?

Hypotheses

• Null hypothesis: The long-run average change in brain volume is the same for each group (μ_None = μ_Walking = μ_TaiChi = μ_Social)
• Alternative hypothesis: The long-run average change in brain volume is different for at least one group (At least one of μ_None, μ_Walking, μ_TaiChi, μ_Social is not the same as the others)

• Null hypothesis: Exercise intervention does not have an effect on change in brain volume in the long run
• Alternative hypothesis: Exercise intervention does have an effect on change in brain volume in the long run

Step 3: Explore the Data

First, I like to look at all the data combined

Step 3: Explore the Data

• Now let’s add in the explanatory variable to separate the data into the four groups.

​

• What does it mean for the mean to be negative? Positive?
• Does exercise intervention explain some of the variability in the change in brain volume?

Step 3: Explore the Data

• R² is the proportion of variation in the response variable explained by the explanatory variable
• Here, we can see that 8.3% of the variability in all the brain change percentages can be explained by exercise intervention.
• Does this seem like a lot?
• What is the smallest possible value of R²? Largest possible value?

Step 4: Make inferences beyond �the data

1. Statistic: Compute a statistic from the observed data that measures how different the groups are in terms of the response.
2. Simulate: Identify a “by-chance-alone” explanation for the data. Repeatedly simulate values of the statistic that could have happened when the chance model is true.
3. Strength of evidence: Consider whether the value of the observed statistic from the research study is unlikely to occur when the chance model is true

​

Step 4: Make inferences beyond �the data

1. Statistic: Compute a statistic from the observed data that measures how different the groups are in terms of the response.

​

Mean Group Diff Statistic

​

​

-0.240 % 0.406 % -0.150 % 0.471 %

​

​

​

​

0.556

0.065

0.646

0.090 + 0.065 + 0.646 + 0.556 + 0.621 + 0.711

6

= 0.448

Average distance between the four group means 0.448 %

0.621

0.090

0.711

Step 4: Make inferences beyond �the data

1. Statistic: Compute a statistic from the observed data that measures how different the groups are in terms of the response.
2. Simulate: Identify a “by-chance-alone” explanation for the data. Repeatedly simulate values of the statistic that could have happened when the chance model is true.

​

• Assume the Null hypothesis is true: Exercise intervention does not have an effect on change in brain volume in the long run

​

• 🡺 Change in brain volumes would have happened regardless of the intervention – could have come from any of the interventions

None

Social

Walking

Tai Chi

Each subject is randomly assigned to one of four groups

None

Social

Walking

Tai Chi

-1.35

1.67

-1.67

1.05

-1.96

-0.56

0.61

-1.54

1.27

0.13

1.00

-0.13

-1.10

0.71

0.67

0.49

0.82

-0.90

0.78

1.80

-1.12

0.99

0.84

-0.43

-0.58

0.27

1.02

-0.75

-1.40

0.38

0.99

1.96

0.30

0.01

-0.75

1.23

0.18

0.54

-0.39

0.61

R² = 0.083, Group Mean Diff = 0.448

Original Data:

None

Social

Walking

Tai Chi

Shuffle Data:

None

Social

Walking

Tai Chi

Shuffle Data:

None

Social

Walking

Tai Chi

0.45

Shuffle Data:

More Simulations and 3: Strength of Evidence

0.21

0.53

0.77

0.18

0.60

0.11

0.49

0.42

0.07

0.32

0.39

0.74

0.53

0.63

0.21

0.05

0.18

0.32

0.39

0.07

0.07

0.32

0.11

0.46

0.45

With 30 repetitions of creating simulated Mean Group Diff statistics, we found 5 were as large or larger than 0.45 giving us an estimated p-value of 5/30 ≈ 0.17

Shuffled Mean Group Diff Statistics

0.18

Multiple Means Applet

• Let’s see this with the entire data set in the Multiple Means applet
• In the applet main page, go to Quantitative > Multiple Means
• In Select data box, choose the Brain and click Use Data

​

​

​

• https://www.isi-stats.com/isi2nd/ISIapplets2021.html

Step 5: Formulate Conclusions

• Because we have a small p-value we have strong evidence that at least one of the long-run mean brain change percentages is different.
• Can we tell which one or ones?
• Go back to dotplots and take a look.
• We can do pairwise confidence intervals to find which means are significantly different than the other means and will do that when we look at theory-based methods.

Step 5: Formulate Conclusions

• Are you comfortable with concluding from this study that the activity type causes a difference in brain volume change?
• To what population are you comfortable generalizing the results of this study?

Step 5: Formulate Conclusions

• The authors stated that a strength of the study was to “assess the effectiveness of interventions in a population-based sample, a necessary step to understand how interventions are likely to affect the general population. ”
• They added, “It is also unclear how our findings would generalize to populations in other communities and countries. Although there is no reason to conclude that Chinese elders would respond differently from other elders, studies of similar interventions in Western populations are needed.

Step 6: Look back and ahead

• The researchers thought that physical activity would help increase brain volume, so going into the study they thought that both walking and tai chi show significant increases.
• What they found was that tai chi and social interaction showed significant increases, but walking did not.
• They did, however, find that the walkers who walked faster had improved scores on some cognitive tests compared to slower walkers.
• They thought that the subjects routinely walked quite a bit anyway just as a function of living in Shanghai and perhaps walking to maintain a higher heart rate might be beneficial.

Step 6: Look back and ahead

• The authors thought that the level of mental involvement for the novice tai chi subjects was thought to have made the difference and perhaps this might not be the same with more experience in tai chi.
• “The magnitude of brain growth demonstrated was relatively small. A longer duration study with increased sample size will be necessary to more clearly evaluate the potential of interventions to reverse brain atrophic processes and maintain cognitive abilities.”

Transitioning to theory-based methods

Validity Conditions for the ANOVA F-Test�

The F-distribution is a good approximation to the null distribution of the F-statistic as long as:

• Either the sample size is at least 20 for all the groups without strong skewness or outliers in the response variable, or if the sample sizes are less than 20, then the distribution of the response variable is approximately symmetric in all the samples (examine the dotplots for skewness or outliers).

• The standard deviations of the samples are approximately equal to each other (Largest standard deviation is not more than twice the value of the smallest standard deviation.)

​

​

We will also look at how our simulation-based results compares to the theory-based results in the applet

​

• Sample sizes?
• Skewedness?
• SDs

Strength of Evidence

​

• As sample size increases, strength of evidence increases.
• As the means move farther apart, strength of evidence increases. (This is the variability between groups.)
• As the standard deviations increase, strength of evidence decreases. (This is the variability within groups.)
• These are all exactly the same as when we compared two means.

Transitioning to theory-based methods

• I’m going back to the applet to show how the F-statistic can be used in simulation-based inference.
• We will see how an F-distribution will fit our simulated null distribution nicely and how the theory-based p-value lines up with the simulation-based p-value.
• We’ll look at follow-up tests
• And finally take a look at an ANOVA table.

Sources of Variation Diagram