Assessing Variation due to Random Assignment

KAREN MCGAUGHEY CAL POLY STATE UNIVERSITY

DAY 2 JULY 17, 2024 HANDOUT 1

Big Ideas

• Assessing variation due to random assignment
• 2 group comparison of quantitative response
• 3 group comparison of quantitative response

Recap

• Multivariable thinking…….
• What variables will be studied, measured?
• What variables will likely contribute to variation in the response?
• Inclusion criteria for both observational and designed experiments 🡺 limitations on scope of inference
• In a designed experiment, what variables can be controlled? Randomized? Blocked?

​

• To isolate the relationship between 2 variables requires accounting for many other variables as much as possible

Example 1: Tai Chi Study 2 groups

• The New England Journal of Medicine (Feb 2012) study investigating the effects of tai chi on postural stability in people with Parkinson’s.
• The disease has five stages, with stage 1 being the most mild and stage 5 the most severe.
•  
• Two different exercise programs were compared: (1) tai chi and (2) stretching.
• n = 130 people with Parkinson’s (stages 1-4), ages 40 to 85, medical clearance to participate

​

Example 1: Tai Chi Study 2 groups

• 65 participants randomly assigned to each exercise program for a period of 12-weeks.

​

​

​

​

• Weekly frequency and duration of exercise sessions were kept the same for all participants.
• Measured change in functional reach (centimeters, cm). Functional reach is the distance one can reach out away from the body to retrieve an item without losing balance and falling over.

​

Background Questions

• Response Variable?
• Explanatory Variable?
• Type of Study?
• Type of Conclusions?

• Change in functional reach (cm)
• Exercise Program: Tai Chi, Stretching
• Experiment
• If we find a significant association between the RV and EV, we tend towards a cause-and-effect conclusion
• well-designed, controlled experiment with replication using random assignment of treatments to participants
• Randomize other things, such as order of testing

Demographic Characteristics

Sources of Variation

• What are possible sources of variation in the change in functional reach?

​

• (Type into the chat)

Sources of Variation

• What would the histogram of change in functional reach look like if there was no variation in the change in functional reach values?

Sources of Variation Diagram

Change in FR (cm)

Stages 1-4, ages 40-85; stable med use; clearance

Frequency, duration, measurement technique

Exercise Program (Tai Chi vs. Stretching)

Age

Initial fitness

Stress level

Stage of Parkinson’s

:

:

Unknown

Change in FR (cm)

Stages 1-4, ages 40-85; stable med use; clearance

Frequency, duration, measurement technique

Exercise Program (Tai Chi vs. Stretching)

Age

Initial fitness

Stress level

Stage of Parkinson’s

:

:

Unknown

• Word Model

Change in FR (cm)

Exercise Program

Unexplained Variation

+

Fitted Model

Total Variation in ΔFR

Explained

Unexplained

4.89 – 0.86 = 4.03 cm

Statistical Inference: Theory-based test

Unpooled:

Pooled:

Simulation-based Inference

• What statistic can we use to measure how different the 2 group means are?

4.89 – 0.86 = 4.03 cm

Simulation-based Inference

4.89 – 0.86 = 4.03 cm

• Assess statistical significance: for these data what kinds of differences in means could we expect just from the random assignment process alone?

• i.e., what’s the goal of random assignment?

🡺 Shuffle the observed change in FR values and re-randomize them to Tai Chi and stretching 🡺

Simulation-based Inference

• Multiple Means Applet:
• https://www.isi-stats.com/isi2nd/ISIapplets2021.html

​

• Data: TaiChi2.txt

Example 2: Tai Chi Study 3 groups

Actual study: 3 different exercise programs were compared: (1) tai chi, (2) stretching and (3) resistance training. A group of 195 people with Parkinson’s (stages 1-4) were enrolled in the study. One-third (n = 65) were randomly assigned to each exercise program for a period of 12-weeks.

Change in FR (cm)

Stages 1-4, ages 40-85, stable med use; clearance

Frequency, duration, measurement technique

Exercise Program (Tai Chi, Stretching, Resistance)

Age

Initial fitness

Stress level

Stage of Parkinson’s

:

:

Unknown

• Word Model

Change in FR (cm)

Exercise Program

Unexplained Variation

+

Fitted Model

Total Variation in ΔFR

Explained

Unexplained

Is exercise program explaining variation in change in functional reach?

Simulation-based Inference

• What statistic can we use to measure how different the 3 group means are?

Mean of Pairwise Group Differences:

Simulation-based Inference

• Multiple Means Applet:
• https://www.isi-stats.com/isi2nd/ISIapplets2021.html

​

• Data: TaiChi3.txt

Theory-based Test

Designing Out a Source of Variability

HANDOUT 2

Big Idea

• Previous examples used random assignment of treatments to experimental units to ‘protect against’ confounding variables

​

• Use blocking to ‘design out’ an inherent source of variation of the experimental unit

Example 3: Strawberry Storage

• Study to investigate the effect of storage method on the firmness of berries.
• 3 storage methods: Air, Modified Air, Control
• Experimental unit was a clamshell of strawberries
• Storage time for air and modified air was 42 hours; control clamshells were measured immediately after packing
• 5 different varieties of strawberries (Allstar, Bounty, Kent, Selva, Vesper)
• Large differences in inherent firmness of these varieties
• RV: clamshell firmness in Newtons (each berry was measured, average)

Smith & Skogg 1992

Sources of Variation Diagram

Clamshell firmness (N)

5 varieties; California

Size, color of berries, time, measurement technique

Storage Method (air, modified air, control)

Variety (5)

Slight variations in color, size

Slight variations in timing of harvest, weather conditions

:

:

Unknown

Fitted Model

Why not statistically significant?

Higher mean is better

Block Design

• How could the researchers design their study to take a source of variation, e.g., variety into account?

Analysis needs to match the design

Students have already seen a paired design

Method 1

• Quantify the variation explained by Variety
• Remove it from Error

Clamshell firmness (N)

5 varieties; California

Size, color of berries, time, measurement technique

Storage Method (air, modified air, control)

Variety (5)

Slight variations in color, size

Slight variations in timing of harvest, weather conditions

:

:

Unknown

Method 1

• Quantify the variation explained by Variety
• Remove it from Error

Clamshell firmness (N)

5 varieties; California

Size, color of berries, time, measurement technique

Storage Method (air, modified air, control)

​

Variety (5)

​

Slight variations in color, size

Slight variations in timing of harvest, weather conditions

:

:

Unknown

SSError

Method 1

Effect of each variety = variety mean – overall mean

Method 1

63.51

67.44 – 63.51 = 3.93

15.88

0.49

11.4

Method 1

• One Blocking Variable Applet

​

• https://www.rossmanchance.com/applets/2021/anovashuffle/rcbd.html

Method 2

• Account for blocks in the simulation
• Re-randomize using the randomization process used in the original study

Null and Alternative Hypotheses

• H_o: µ_Air = µ_MA = µ_Control, after accounting for strawberry variety
•  
• H_a: At least one of the µi differs, after accounting for strawberry variety
•  
• OR
• H_o: There is not a true or real effect of storage method on strawberry firmness after accounting for strawberry variety
•  
• H_a: There is a true or real effect of storage method on strawberry firmness after accounting for strawberry variety
•  

​

Summary

• In a designed experiment:
• If possible, design out a source or sources using blocking
• Blocking variables are inherent characteristics of the experimental unit
• E.g., strawberry variety, plot location, participant weight, etc.
• Analysis will need to account for blocks
• Randomize everything else