Biologist’s Interview with Google Notebook

Deborah Frazier

Vinay K. Chaudhri

Outline

• Evaluating an AI System
• Why Expert Interviews?
• An Expert Interview in Biology
• Rating scale
• Example Interview
• Results
• Lessons Learned

​

Evaluating an AI System

• Computer passing an exam is not the same as a human passing the same test
• “Proxy Failure”
• Humans have general-purpose knowledge that they can use for multiple purposes
• What should be a test of “knowledge” for computers?

Standing an Expert Interview

• Styled as a Turing Test
• No need to determine if the conversation is with a human
• Need to assess that the program has genuine expertise
• It is not BS-ing
• It is not pretending

Standing an Expert Interview

• Styled as a Turing Test
• Conversation that lasts over a period of time (months)
• Interviewer could be a panel of experts
• Probes for the basic knowledge and ability to use it for multiple purposes

Standing a Biologist Interview

• Interested in better teaching
• Probes for the basic knowledge and ability to use it for multiple purposes
• Foundational knowledge of biology
• Application to specific examples
• Expanding on implicit knowledge
• Making connections across the book
• Drawing appropriate analogies

Interview Playing Ground

Chapter 39

Physiology, Homeostasis and

Temperature Regulation

Rubric

Case #1: Cell specialization, multicellularity, tissue classification, evolution

Interview Questions

1) Why are internal tissues of the sponge specialized in structure if they can do nutrient or gas exchange like all other sponge cells?

Had to restate/break down question (4 total)

​

2) If we were to classify the two layers of cells in a sponge, what kind of tissues would they be?

Had to restate/break down question (4 total)

​

3) So, considering where sponges are in the evolutionary lineage of animals, might this argue for one of these tissue types to have been the first to evolve? Explain why or why not.

​

Interview Questions – Support of Student Learning

1) Why are internal tissues of the sponge specialized in structure if they can do nutrient or gas exchange like all other sponge cells?

Harmful (3), Helpful (1)

2) If we were to classify the two layers of cells in a sponge, what kind of tissues would they be?

Harmful (2), Helpful (2)

3) So, considering where sponges are in the evolutionary lineage of animals, might this argue for one of these tissue types to have been the first to evolve? Explain why or why not.

Helpful

Interview Questions – Answer Quality

1) Why are internal tissues of the sponge specialized in structure if they can do nutrient or gas exchange like all other sponge cells?

Correct part, book dump, error, ambiguity

Correct part, connect to big picture, misleading, error

Cannot answer

Cannot answer, book dump

2) If we were to classify the two layers of cells in a sponge, what kind of tissues would they be?

Oversimplification, chooses not to answer

Correct, nice parallel

Error, overgeneralization

Correct, connect to big picture, nice parallel

3) So, considering where sponges are in the evolutionary lineage of animals, might this argue for one of these tissue types to have been the first to evolve? Explain why or why not. Correct, connect to big picture

​

Case #2: Physiology, Homeostasis

• Builds on very early (elementary school) knowledge of “warm- and cold-blooded” creatures. Students tend to accept the dichotomy as “how things are” and don’t question (but I will).
• Granularity opportunities here, from reaction to organism to ecosystem

Interview Questions

1) Why can’t a cold-blooded animal get warm like a warm-blooded animal is just by doing exothermic reactions?

​

2) You said endotherms have leaky cells to sodium and potassium ions. Does this make their nervous  responses poorer compared to ectotherms?

​

3) Leakiness is generally seen as a bad thing. Would you say the leakiness of endothermal membranes is a beneficial evolutionary adaptation for animals?

Had to drive explanation (3 total)

​

Interview Questions – Support of Student Learning

1) Why can’t a cold-blooded animal get warm like a warm-blooded animal is just by doing exothermic reactions? Harmful

​

2) You said endotherms have leaky cells to sodium and potassium ions. Does this make their nervous  responses poorer compared to ectotherms? Harmful

​

3) Leakiness is generally seen as a bad thing. Would you say the leakiness of endothermal membranes is a beneficial evolutionary adaptation for animals?

May be harmful / little helpful

May be harmful / little helpful

Helpful

Interview Questions – Answer Quality

1) Why can’t a cold-blooded animal get warm like a warm-blooded animal is just by doing exothermic reactions?

Oversimplified, misleading, cannot interpret what info to compare

​

2) You said endotherms have leaky cells to sodium and potassium ions. Does this make their nervous  responses poorer compared to ectotherms? Book dump, cannot answer

​

3) Leakiness is generally seen as a bad thing. Would you say the leakiness of endothermal membranes is a beneficial evolutionary adaptation for animals?

Correct part, book dump

Good parts, overgeneralization, ambiguous, book dump

Good part, verbose

Case #3: Work with math

1) Based on a body mass of 110 pounds, how much water is in your body? What’s your intracellular fluid volume, your extracellular fluid volume, and your blood plasma volume?

2) If Q10 is 1.5 and the rate of reaction is 2 at 10 degrees, what is the rate of reaction at 20 degrees?

3) Considering your answer and the curvilinear nature of biological data displays, At 15 degrees, would you expect a rate of reaction above or below 2.5?

4) You know what Q10 is. How would Q30 differ? Why is Q30 not used, considering homeostasis?

5) Why is Q10 almost always 1 or higher? Can you explain a situation where Q10 might be less than one?

​

​

​

Case #3: Work with Math - Support of Student Learning

1) Based on a body mass of 110 pounds, how much water is in your body? What’s your intracellular fluid volume, your extracellular fluid volume, and your blood plasma volume? Helpful, Insightful

2) If Q10 is 1.5 and the rate of reaction is 2 at 10 degrees, what is the rate of reaction at 20 degrees? Helpful, Insightful

3) Considering your answer and the curvilinear nature of biological data displays, At 15 degrees, would you expect a rate of reaction above or below 2.5? Helpful, Insightful

4) You know what Q10 is. How would Q30 differ? Why is Q30 not used, considering homeostasis? Helpful

5) Why is Q10 almost always 1 or higher? Can you explain a situation where Q10 might be less than one? Helpful

​

​

​

Case #3: Work with math – Answer Quality

1) Based on a body mass of 110 pounds, how much water is in your body? What’s your intracellular fluid volume, your extracellular fluid volume, and your blood plasma volume? Correct, Step by step answer

2) If Q10 is 1.5 and the rate of reaction is 2 at 10 degrees, what is the rate of reaction at 20 degrees? Correct, Step by step answer

3) Considering your answer and the curvilinear nature of biological data displays, At 15 degrees, would you expect a rate of reaction above or below 2.5? Correct, Step by step answer w more detail than expected

4) You know what Q10 is. How would Q30 differ? Why is Q30 not used, considering homeostasis? Correct, Connect to big picture

5) Why is Q10 almost always 1 or higher? Can you explain a situation where Q10 might be less than one? Correct, 2 examples

​

​

​

Case #4: Exercise

• Text uses this as a context throughout.
• Will centering questions on this topic improve responses?

​

​

Interview Questions

1) My hiking buddy noted that her heart rate felt higher when hiking while wearing long sleeves, compared to when her sleeves are short. She was drinking water, so dehydration was not an issue. Why do you think her heart rate was higher? What effectors were involved? What could she do to drop her heart rate, without changing her sleeve length?

2) I really like hiking. Skeletal muscle propels me down a trail. What are the roles played by the other three tissue types in this process?

3) The text described a study exploring heat exchange at the palms and the amount of work (weight lifting) several young people could do. Would wearing wicking fabrics be more like the palm-cooling group OR the control group, if this study were repeated with wearing /not wearing wicking fabric replacing the experimental treatment.

Interview Questions – Support of Student Learning

1) My hiking buddy noted that her heart rate felt higher when hiking while wearing long sleeves, compared to when her sleeves are short. She was drinking water, so dehydration was not an issue. Why do you think her heart rate was higher? What effectors were involved? What could she do to drop her heart rate, without changing her sleeve length? Helpful, Insightful, Inspirational

2) I really like hiking. Skeletal muscle propels me down a trail. What are the roles played by the other three tissue types in this process? Helpful, Insightful, Inspirational

3) The text described a study exploring heat exchange at the palms and the amount of work (weight lifting) several young people could do. Would wearing wicking fabrics be more like the palm-cooling group OR the control group, if this study were repeated with wearing /not wearing wicking fabric replacing the experimental treatment. Helpful, Insightful, Inspirational

Interview Questions – Answer Quality

1) My hiking buddy noted that her heart rate felt higher when hiking while wearing long sleeves, compared to when her sleeves are short. She was drinking water, so dehydration was not an issue. Why do you think her heart rate was higher? What effectors were involved? What could she do to drop her heart rate, without changing her sleeve length? Correct, Connect to big picture, rich parallels

2) I really like hiking. Skeletal muscle propels me down a trail. What are the roles played by the other three tissue types in this process? Correct, Connect to big picture, rich parallels & details

3) The text described a study exploring heat exchange at the palms and the amount of work (weight lifting) several young people could do. Would wearing wicking fabrics be more like the palm-cooling group OR the control group, if this study were repeated with wearing /not wearing wicking fabric replacing the experimental treatment. Correct, rich parallels & details

Case #5: Spatial organizations of fluids & cell (using a figure)

• Students may be confused about where the cell & these fluids are.
• I add a common process as context so it’s not a definitional question.

1) A cell is undergoing cellular respiration. Using the terms blood plasma, interstitial fluid, and intracellular fluid, describe the movement of key reactants TOWARDS the cell from these regions, as well as products moving AWAY from the cell into one or more of these regions.

Support of Student Learning: Helpful

Answer Quality: Correct spatial answer, connect to big picture, ambiguous reactants

​

Summary of Evaluation�Uses overall rating for each question

Lessons Learned

Strengths:

• Using emphasized content repeated across a chapter
• Responding to leading or repeated questioning
• Use of figures
• Working with math
• Attempt to connect to some overarching principles

​

Opportunity for growth:

• Ambiguity/Overgeneralization 🡪 Misleading
• Unwilling to answer
• Verbose
• Would more text confuse it or help orient it?
• Would a foundational database/standards help it draw out foundational concepts?

​

​