CMSC 320 • PART 03

ASSUMES DISTRIBUTION

Parametric Tests

• Assume the data follow a particular distribution (often normal).
• Usually make assumptions about population parameters.
• Often more powerful when assumptions are satisfied.

Examples: t-test, ANOVA

DISTRIBUTION-FREE

Non-Parametric Tests

• Make fewer assumptions about the underlying distribution.
• Often use ranks rather than original values.
• Useful for skewed, ordinal, or non-normal data.

Examples: Mann–Whitney U, Wilcoxon, Kruskal–Wallis

Key Idea: Choose the test based on your data, research question, and assumptions.

Overview & Key Assumptions

Assumes data follow a continuous, normally distributed scale with equal variances.

• Relies on mean & variance parameters
• Requires interval or ratio scale data
• Provides higher statistical power when assumptions are met

Core Parametric Tests (Comparing Means)

• 1-Sample t / Z-Test: Compares sample mean to a target value
• 2-Sample t-Test: Compares means of 2 independent groups
• Paired t-Test: Compares 2 related/dependent samples
• 1-Way ANOVA: Compares means across 3+ independent groups

Decision Rule (Sample Size & Groups)

Use Z-Test for n > 30, t-Test for n < 30 or unknown variance, and ANOVA when evaluating 3 or more sample groups.

Z-Test

Use when population standard deviation (σ) is known.

• Tests a hypothesis about a population mean
• Population σ is known
• Data are approximately normal or sample size is large enough for the CLT
• Uses the standard normal (Z) distribution

Z = (x̄ - μ) / (σ / √n)

Think: σ known → Z-test

T-Test

Use when population standard deviation (σ) is unknown.

• Tests hypotheses about population mean(s)
• Population σ is unknown
• Uses the sample standard deviation (s) instead
• For small samples, approximate normality is important
• Can also be used with large samples
• Uses the t-distribution

t = (x̄ - μ) / (s / √n)

Think: σ unknown → t-test

​

x̄ = sample mean | μ = population mean | σ = population std. deviation | s = sample std. deviation | n = sample size

​

KEY TAKEAWAY: Main distinction is σ known vs. unknown, not sample size alone.

We have noticed most humans fall into one of two distinct categories—male or female.

We would like to know if our sample of males is taller than our sample of females.

Key Research Question

Can we just take the average of the two samples?

​

One-Sample T-Test

Compare one sample mean with a known/reference value.

Example: Is average sleep time different from 7 hours?

Two-Sample T-Test (Independent)

Compare the means of two independent groups.

Example: Is average height different between Group A and Group B?

Paired T-Test

Compare two related measurements from the same/matched subjects.

Example: Blood pressure before vs. after treatment.

Often measured from the same subjects at different times or under different conditions.

KEY DECISION GUIDE

• One group vs. a value? → One-Sample T-Test   |   • Two independent groups? → Two-Sample T-Test

• Same subjects twice? → Paired T-Test

If the two groups have very different spreads, use Welch’s t-test.

Simply taking the average of two samples is not sufficient to determine if groups differ (doesn't account for variability). Conduct a statistical test to determine significant differences.

How to run a one-sample t-test:

import numpy as np

from scipy import stats

stats.ttest_1samp(your_data, popmean=0.5)

>>> TtestResult(statistic=2.456, pvalue=0.0176, df=49)

Why Simple Comparison Is Not Enough

Two groups can have different sample means, but that alone does not tell us whether the difference is statistically significant.

We also need to consider:

• Variability — how spread out the values are
• Sample size — how much data we have

The Statistical Solution

A Two-Sample T-Test helps determine whether the difference between the means of two independent groups is statistically significant.

Example: Is the average height different between Group A and Group B?

Key Takeaway: Difference in means + Variability + Sample size → Statistical evidence

Continuous Outcome

Outcome is numerical / continuous.

Independence of Observations

Observations are appropriately independent (except paired t-test).

Normality Assumption

Data are approximately normal, especially important for small samples.

Non-Parametric Alternative

If normality is not reasonable → consider a non-parametric alternative.

The Experiment Scenario

The aliens monitor a bunch of humans, test them for intelligence, and then run one half of them through a machine to make them smarter. Afterwards, they want to know if their machine worked.

Experimental Method: This study design is analyzed using a paired t-test.

Null Hypothesis (H₀)

The machine did nothing

Assumes no significant difference in intelligence before and after the intervention.

Alternative Hypothesis (H₁)

The machine came from a different distribution

Assumes the machine produced a real, statistically measurable effect on intelligence.

Question for Discussion

The aliens get a p-value of .05. What can they conclude?

Consider the standard significance threshold (α = 0.05) and whether this provides sufficient evidence to reject the null hypothesis.

ANOVA: Comparing Multiple Groups

Statistical scenario, hypothesis formulation, and group variance logic

The Scenario

The aliens want to know whether human growth differs across diets. They observe three groups: Vegan Diet, Meat Diet, & Vegetarian Diet.

Question: Do the groups have different average growth over the next few years?

ANOVA

When comparing the means of 3 or more groups, we use ANOVA (Analysis of Variance). Tests if at least one group mean differs.

H₀: All group means are equal.

Hₐ: At least one group mean is different.

Remember: ANOVA considers both within-group and between-group variability.

Key Idea & Rule

2 groups → T-Test  |  3+ groups → ANOVA

Important: A significant ANOVA tells us that a difference exists, but not which groups differ.�→ That's where post-hoc tests come next.

Overview & Key Assumptions

Used when data do not meet assumptions of parametric tests (e.g., normality, equal variances).

• Does not rely on population parameters (mean/variance).
• Based on ranking data instead of raw values.
• Suitable for nominal/ordinal data, skewed distributions, small samples.

Rank-Based Tests (work on ranked data)

• Mann–Whitney U: 2 independent groups
• Wilcoxon Signed-Rank: 2 paired groups
• Kruskal–Wallis: 3+ independent groups
• Spearman’s Rank: Correlation

Frequency-Based Tests (categorical counts)

• Chi-Square Test: Goodness of Fit & Independence

Kruskal–Wallis Test Alt to One-Way ANOVA

• Compares medians of 3 or more independent groups
• Use when data are non-normal or variances are unequal

Mann–Whitney U Test Alt to Independent t-Test

• Compares two independent groups on median ranks
• Useful for ordinal data or non-normal distributions

Wilcoxon Signed-Rank Test Alt to Paired-Samples t-Test

• Compares two related/paired groups (e.g., before vs. after)
• Assesses differences in median ranks when normality is violated

Spearman’s Rank Correlation

Measures correlation between two variables based on ranks. Useful for ordinal/non-normal data.

Key Takeaway

Nonparametric tests are mostly based on ranked data instead of raw values, making them significantly more robust when the underlying assumptions of parametric tests are not met.

How Ranking Works

Instead of using actual numerical values (raw scores), nonparametric tests convert data into ranks (positions).

Example: Exam Scores

Raw: 45, 80, 60, 90, 75

45 → Rank 1

60 → Rank 2

75 → Rank 3

80 → Rank 4

90 → Rank 5

Why use ranks?

Tests look at rank distributions rather than exact values. Ranks are less sensitive to outliers and don't require normal distributions.

Chi-Square Test: Categorical Data

What does it test?

The Chi-Square (χ²) test checks whether there is an association between categorical variables.

Works with:

• Categorical data
• Counts / frequencies, not means
• li { color: #334155; } Two or more categories

Key Takeaway:

Numerical means → t-test / ANOVA

Categorical counts → Chi-Square

Example:

Is bird species associated with location?

H₀ (Null Hypothesis):

Bird species and location are independent.

Hₐ (Alternative Hypothesis):

Bird species and location are associated.

We observe bird species at Location A and Location B.

Question: Is the distribution of bird species the same across the two locations?

Analysis Process

Observed counts → Chi-Square Test

→ Compare observed vs. expected counts

Hypotheses

H₀ (Null Hypothesis):

Bird species and location are independent.

Hₐ (Alternative Hypothesis):

Bird species and location are associated.

01

Research Goal

What are you curious about?

Mean? Standard deviation? Frequency?

02

Data Type

Is your data categorical or continuous?

03

Sample Count

Do you have one or two samples?

04

Distribution

Is your data normally distributed?

If normal, use a parametric test. If very non-normal, use a non-parametric test.

05

Dependency

Is your data paired?

Is there a before and after?

WHEN TO USE

ANOVA tells us that at least one group differs, but not which groups are different.

Post-hoc analysis identifies specific group differences after ANOVA is significant when more than two groups are compared.

PURPOSE

Core Objectives

• Compare pairs of groups
• Control error rate from making multiple statistical comparisons
• Pinpoint differences to identify exactly where significant variation occurs

COMMON POST-HOC TESTS

• Tukey’s Honest Significant Difference (HSD)
• Bonferroni Correction
• Duncan’s Multiple Range Test

Key Idea: Post-hoc tests show which group means differ significantly from each other.

WHY POST-HOC TESTS?

Why Do We Need Post-Hoc Tests?

A significant ANOVA tells us that at least one group mean is different, but it does not tell us which groups differ.

COMMON METHODS

Common Post-Hoc Methods

SCENARIO

We conducted a study to compare the test scores of students from three different schools: School A, School B, and School C.

Initial Assessment: More than 2 groups (A, B, C) → Apply ANOVA

ANOVA Result: p-value < 0.05 (indicating significant overall differences among schools).

Next Step: Post-hoc testing is required to pinpoint which specific schools differ.

Apply Tukey’s HSD: Balanced Approach

HSD = Honestly Significant Difference

Tukey compares all pairs of group means and controls the overall chance of making one or more Type I errors across that family of comparisons.

Goal: Find which pairs differ — Ideal when comparing every group with every other group.

Interpretation: School A and School B, as well as School A and School C, have significantly different test scores (p < 0.05).

Key Result: There is no significant difference in test scores between School B and School C.

Next Step: Compare all three pairs while using a stricter significance level to control Type I error.

Adjusted α = 0.05 / 3 = 0.0167(3 comparisons) — Goal: Strong protection against false positives by using a stricter threshold.

Interpretation: Using the Bonferroni-adjusted threshold (α = 0.0167), none of the pairwise comparisons are statistically significant.

Key Idea: Bonferroni is more conservative because it uses a stricter threshold to reduce false positives.

Summary: Key Statistical Tests

Overview of core testing methods and their application criteria

Statistical tests let us reason about one or more samples and how they relate to each other and the population.

LARGE SAMPLE

z-test

A test we can use if your data is normally distributed and we have a large number of samples.

SMALL SAMPLE

t-test

A test we can use if your data is normally distributed and we have a small number of samples.

LONGITUDINAL

Paired tests

Used when we follow specific population members through time.

DIRECTIONALITY

One tailed vs two tailed tests

Two tailed: Look for any difference in population.

One tailed: Require us to pick a specific direction.

1

1. State the Null Hypothesis: Assumption what you're trying to test

2

2. State the Alternative Hypothesis: what you believe might be true

3

3. Pick a Level of Significance α: the probability of rejecting the null hypothesis when it's actually true. Common values for α are 0.05 or 0.01.

4

4. Choose a Test: Select the right tool to check your guesses based on your data.

5

5. Collect Data: Get the information/data you need through observation or experimentation.

6

6. Calculate a test statistic: Using the collected data, calculate the appropriate test statistic.

7

7. Calculate P-Value and compare with α: Based on the comparison, decide if you have enough evidence to believe your guess is right or if you need to keep looking.

8

8. Draw a Conclusion: Based on your decision in the previous step, draw a conclusion regarding the null hypothesis. If you reject it, you accept the alternative hypothesis. If you fail to reject it, you do not have enough evidence to support the alternative hypothesis.