Unit 3 Sampling and Selection �

1

2

Lesson 1 Does Experimental Design = Good Research?

3

Welcome Back! To start off this unit let’s test your knowledge on sampling and selection in experiments:

​

True or False?

​

1. Experiments are always better than correlational research.
2. If an experiment is a “randomized control” experiment then it has to be good.
3. We should always trust experimental research
4. Experimental research can sometimes have flaws, and it’s important that I can recognize them.

Lesson 1: Experimental Design = Good Research?

4

How was it? If you were unsure about some of the questions, don’t fret! Understanding what makes a good experiment can be tricky.

​

Lesson 1: Experimental Design = Good Research?

This unit is about understanding issues related to sampling or selection in experiments - that is what or who is being sampled in an experiment, and where did they come from?

​

5

Lesson 1: Experimental Design = Good Research?

Issues of sampling and selection are particularly important for research related to humans. However, these issues are relevant in any kind of experiment regardless

of what or who is being sampled.

​

6

Lesson 1: Experimental Design = Good Research?

7

Let’s consider the following scenario:

Marissa noticed that it is hotter than normal outside, so she decides to study whether climate change is affecting the weather in her hometown of Miami, Florida. To test this question, she records the temperature outside her house everyday for a 1-week period.

Example

8

She then finds a chart showing the average U.S. temperatures for the same dates in the year 2000.

Example

Example

9

She compares the average temperature from her notes to the average temperature from the chart that she found in order to see whether the average temperature in Miami, Fl has increased over the last 21 years.

Average Temperature = 79.7 °

Average Temperature = 54.9 °

Example

10

Marissa is shocked to find that the average temperature has increased from 54.9 to 79.7 degrees! She concludes that the average temperature in the U.S. has increased dramatically.

Example

Do you see an issue with Marissa’s conclusion that average temperatures in the U.S. have increased based on her data?��Please explain why or why not.

11

12

One major issue with Marissa’s experiment is that she is comparing weather from Florida to weather across the United States.

​

By saying that temperatures are increasing across the United States, she is generalizing the temperatures she collected in Florida to the entire country.

Example

13

This highlights a key issue in sampling and selection – representativeness.

​

The weather in Florida is not representative of the entire country, so she should not compare Florida weather to U.S. weather.

​

Her claim overgeneralizes her findings.

Example

Marissa falsely claims that “the average temperature in the U.S. has increased dramatically”.��What claim can she make from her data if any?

14

15

The only claim Marissa can make from the data is that Florida’s March 2021 temperature is hotter than the U.S. temperature in March 2000.

Example

How could Marissa design a study where she could test her hypothesis that temperatures have increased in the U.S. over the last 21 years?

16

17

She could find out the average temperature across the U.S. for each of the days in 2021. She could then compare this average to the average U.S. temperature in 2000.

Example

Even the media sometimes makes the mistake of overgeneralizing findings!

18

Lesson 1: Experimental Design = Good Research?

In this unit, you will learn some of the tips and tricks for spotting the good experiments from the bad.

You will also learn how to determine whether or not a finding is generalizable.

​

19

Lesson 1: Experimental Design = Good Research?

By recognizing good research from bad, this will give you the power to make better choices on how you incorporate scientific research into your life!

​

20

Lesson 1: Experimental Design = Good Research?

Lesson 2 Representativeness

​

21

22

In this lesson you will learn about one common error researchers make: choosing a non-representative sample. Or another way of putting it... choosing a sample that doesn’t accurately reflect the population being hypothesized about.

Lesson 2: Representativeness

23

By non-representative sample, what do we really mean?

Lesson 2: Representativeness

24

When a researcher presents their question, they should consider the group that they want their results to generalize to.

Lesson 2: Representativeness

By generalize, we mean “hold true for everyone in the target population”.

​

25

Lesson 2: Representativeness

This target population can vary by the researcher’s question, for example...

​

26

Lesson 2: Representativeness

Are first-generation college students more likely to drop out of college?

​

Do vaccines lead to longer lifespans in women?

​

Do people prefer chocolate or vanilla ice cream?

​

27

Lesson 2: Representativeness

Once the researcher has defined their research question, it is important that they recruit a sample of participants that is representative of their population.

​

That is, they should include participants that represent all subsets of that population (to the best of their ability).

​

Let’s walk through an example!

​

28

Lesson 2: Representativeness

Let’s consider how light and water play a role in plant growth.

​

29

Example

Marco has just purchased a plant for his bedroom. To understand how to take care of it, he reads about an experiment online:

30

Science News

Watering Plants Improves Growth

​

A group of students examined the importance of water for plant growth. The students took a collection of 10 vining houseplants of the same species and gave them different amounts of water.

​

They found that the plants grew best when watered once per week. This suggests that watering weekly is best for houseplants.

Example

Based on the article’s advice, Marco waters his new plant once per week.

​

31

Example

However, after a few weeks, Marco’s plant starts to rot, and it eventually dies.�

32

Example

Marco followed the article’s instructions, so what do you think went wrong?

33

Science News

Watering Plants Improves Growth

​

A group of students examined the importance of water for plant growth. The students took a collection of 10 vining houseplants of the same species and gave them different amounts of water.

​

They found that the plants grew best when watered once per week. This suggests that watering weekly is best for houseplants.

Marco’s plant died because the author of the science text overgeneralized the study’s findings!

The author said that watering once per week is best for houseplants, however the researchers only tested one kind of plant. Based on the study’s results, we can’t say that watering once per week is best for all types of houseplants.

34

Example

Example

Since Marco purchased a cactus which is native to the desert. His cactus needs less watering than the vining houseplants the researchers used, so his cactus died when he followed the article’s advice.

​

35

Example

We have the same issue when we are studying people!

Imagine that a researcher is interested in whether people prefer chocolate or vanilla ice cream. Which group of participants do you think the researcher should recruit?

36

You should have answered Group B.

Selecting a random sample of people from the city he lives in increases the likelihood that his results will generalize to his target population (all people). This is because he will get a more diverse sample when compared to recruiting from the college he works at or a group of preschoolers.

37

Example

38

Example

But wait a minute, if the researcher wants his findings to generalize to all people, shouldn’t he have collected participants from across the state, or the country, or the whole world?

​

39

Example

In theory, YES! The researcher should get as representative of a sample as possible, however, collecting from a truly representative sample may be:

Impractical

Impossible or

Too Costly

40

Example

Researchers have to do their best to collect a representative sample, and you must be the judge of whether or not you think that their results generalize.

​

41

Lesson 2: Representativeness

42

Now it’s your turn!

​

Choose a research question and the group you want your findings to generalize to.

​

Then, provide an example of a “good” and “bad” sample of participants to use in your study.

43

Example:

My research question is: Does using flashcards improve vocabulary learning?

I want my findings to generalize to:

Elementary school students

Good Sample Bad Sample

Students from multiple grades 3rd grade boys

Example

44

Lesson 3 Thinking about representative samples

45

In the last lesson, you read about the importance of representative samples. Let’s try again with another example.

Lesson 3: Thinking About Representativeness

46

Remember to use your critical thinking skills and think about the following:

​

1. What is the evidence?�2. Do the findings support the conclusions?�3. Does the sample allow the researcher to answer their question?

Lesson 3: Thinking About Representativeness

47

Who is going to be homecoming king?

​

The chess club at Valley High School was interested in determining who classmates would choose to be homecoming king. They set up a poll on the chess club website, asking people who visited their website to report who they planned to vote for.

​

They found that 75% of the people were planning

to vote for the president of the chess club and 25%

were planning to vote for the football captain.

This study showed that the president of the chess

club would probably win homecoming king.

​

Science News

Example

48

Does the chess club’s study convince you that the chess club president is going to win homecoming king?

​

Why or why not?

49

Do you think that the sample recruited for the study allowed the chess club to determine who would win?

​

Why or why not?

50

Do you think that the chess club used a representative sample?

​

Why or why not?

51

The chess club did NOT use a representative sample.

​

It is more likely that people in the chess club will

visit their website, and it is more likely that those people will vote for the chess club president for homecoming king.

Example

52

In other words, they thought that they had a sample that represented the opinion of all the students in their high school, but their sample actually only represented people who were in the chess club.

Example

53

Population

Chess Club’s Sample

Representative Sample

Example

What could the chess club have done differently to ensure that their sample was representative of the population?

​

54

55

What could the chess club have done differently to ensure that their sample was representative of the population?

​

Answer: There are many possible answers here, but one possibility would have been for the chess club to randomly poll people walking around the school.

Let’s Recap!

​

In this unit you learned about the importance of sampling and selection in studies.

​

It is important that a researcher selects a sample that is representative of the population that they wish to study.

​

When reading science in the news, check to see whether the findings of a study are over-generalized!

​

56