Human-Computer Interaction

saadh.info/hci

​

Week 2 (Tuesday): Perception Cont., Cognition

​

1

Attendance and Agenda

1. Hearing

​

• Rest of senses

​

• Cognition
• Three levels & Seven Stages
• Memory
• Attention

2

1. Perception and Cognition

3

Humans and Computers

​

​

4

Human Operator in a Work Environment

Hearing

5

Hearing

• Provides information about environment: distances, directions, objects.

• Physical apparatus:
• Pinnea: collects sound (part of the ear you can see)
• Outer ear: protects inner and amplifies sound
• Middle ear: transmits sound waves as vibrations to inner ear
• Inner ear: chemical transmitters are released and

cause impulses in auditory nerve

​

6

Sound

• Sound
• Pitch: Frequency
• Loudness: Amplitude
• Timbre: Type or quality (“Color of sound”)
• Envelope: Note or harmonics build up transition over time
• Location: Where is the sound coming from?

7

Hearing Capabilities

• Capabilities (best-case scenario)
• Pitch: frequency (20 - 20,000 Hz)
• Loudness: amplitude (30 - 130dB)
• Location: 5° source & stream separation

​

• Often take for granted what it indicates about computer state, e.g., disk whirring

​

• Auditory system filters sounds and can attend to sounds over background noise, e.g., the cocktail party phenomenon

8

Designing for Hearing: Skeuomorphism vs Metaphors

9

Skeuomorphism: Sound enhances user action on Mac. You hear crumpling of paper when you put a file in recycle bin. Example of sound that mimic real world sound.

Designing for Hearing: Skeuomorphism vs Metaphors

10

Metaphors: Sound enhances user interaction in Discord: a bright, uplifting tone plays when someone joins, evoking joy, while a mellow, downward tone signals someone leaving.

Designing for Hearing: Duration

11

Keep sounds simple: Transition or micro-interaction sound should never last more than 0.3 seconds longer than its associated animation

Designing for Hearing: Personification

12

Sound can contribute a lot to the perception of user experience. These two sounds above accompany the same visual but provide two different aural experiences.

Designing for Hearing: Repetition

13

There’s a limit to the number of times a sound is repeated before we wish it never existed. What’s more, the threshold falls significantly lower the more complex a sound becomes.

Designing for Hearing: Frequencies

14

Does the sound feel heavy or muffled? Remove some of the lower frequencies and accentuate the higher ones.

​

Does the sound feel squeaky or brittle? Carve out some of the higher frequencies and fatten up the medium to low range.

​

Designing for Deaf and Hard of Hearing Users

15

Use Plain English

Provide Summaries

Simple Layout

Closed Captions

Visuals

Hearing: Spatial Audio

16

Hearing: Spatial Audio

17

Touch

• Provides important feedback about environment

​

• Some areas more sensitive, e.g. fingers

​

• Tactile feedback in devices like mobile phones, mice, keyboards, vibrating beepers, and force-feedback tools enhances user interaction by providing physical confirmation of actions.

​

• Maybe key sense for someone who is visually impaired

​

​

18

Limbs

• Movement in limbs is coupled to somatosensory system, particularly proprioception
• Coordination of limb movement and position through perception of stimuli within muscles and tendons often without visual cues, e.g., walking without looking at the floor or touching elbow in dark

​

​

19

Touch

20

Touch

21

(Shultz and Harrison, 2023)

Other Responders

22

Responders

23

Lips

24

Mouth

25

Smell and Taste

• Smell comes from receptors are in the olfactory lining of the nasal passages
• Sensory nerve cells, or neurons, with hairlike fibers called cilia on one end; brain organizes information from these receptors into patterns interpreted as different odors.
• Older adults often lose their sense of smell with time
• Taste is the direct chemical reception of sweet, salty, bitter, and sour sensations through taste buds in the tongue and oral cavity
• Flavor is a perceptual process in the brain that occurs through a partnering of the smell and taste senses
• Aid and recall… but poorer performance

26

Smell and Taste Design

27

Virtual Lemonade

Image Labels using Scent

Moral Sense?

28

Moral Sense?

29

Reputational Concerns, Human Cooperative Behavior

Human Operator in a Work Environment

30

​

​

Human Operator in a Work Environment

Humans and Computers

​

​

31

Newell’s Time Scale of Human Action

Cognitive Operations

32

Three Levels of Processing

33

Visceral

Behavioral

Reflective

Three Levels of Processing and Seven Stages of Action

34

1. Visceral Level

• Basic level of processing
• “Lizard Brain”
• Responsible for fast and completely subconscious visceral responses

​

• Basic biology of visceral system minimizes its ability to learn

​

• Tightly coupled with the body’s musculature or the motor system

​

• Determined by analyzing the tension in the body
• Tense state (negative)
• Relaxed state (positive)

35

1. Visceral System: Relevance for Design

• Immediate perception
• Pleasantness of a mellow, harmonious sound
• Jarring irritating scratch of fingernail

​

• Appearance, whether sight, sound, touch, taste, smell, drive the visceral response
• Little or nothing to do with the usability, effectiveness, or understandability

36

2. Behavioral Level

• Learned Skills
• Triggered by situations that match appropriate patterns
• Aware of our actions, unaware of the details
• Think of the goal and the behavioral level handles all the details

​

• Every action associates with expectations and behavioral states are learned
• Feeling of control when things go right
• Feeling of frustration when things go wrong

​

• Emotional system reacts to changes
• Lack of feedback creates lack of control which is unsettling

​

37

3. Reflective Level

• Conscious cognition
• Deep and slow understanding

​

• Highest level of emotions come from this level

​

• Reflection or looking back over them, evaluating the circumstances, actions, and outcomes, often assessing blame or responsibility

​

• Reflective memories can be more important than reality

​

38

Three Levels of Processing and Design

• Strongly positive visceral response but disappointing usability problems at the behavioral level?

​

• Too much frustration towards the end stage of use but good initial visceral responses?

​

• Vacation experiences

​

• Thoughts and Emotions: High-level reflective cognition can trigger lower-level emotions. Lower-level emotions can trigger higher-level reflective cognition.

​

​

39

Memory

40

Memory

• Most basic of cognitive process

​

• Required for
• Perception and pattern recognition
• Thinking, reasoning, problem solving
• Attention
• Learning
• Language

41

Multi-Store Model of Memory

42

Sensory Store

• Sensory Store
• Visual memory = 150 to 200 ms
• Echoic memory = 3 to 4 seconds (length of a sentence)

​

43

Experiment Time

• 12 letters will appear, after they disappear, write as many as you can

44

SDFG

PWHJ

XCVN

Experiment Time

• I will say TOP, MIDDLE, or BOTTOM after the letters disappear. Write the corresponding row

45

CTJR

ULQW

NSFX

Experiment Time

Bottom

46

Experiment Time

• 12 letters will appear, after they disappear, write as many as you can

47

KCTB

NKIP

LRSW

Experiment Time

Middle

48

Sperling’s Experiments: How did you do?

49

SDFG

PWHJ

XCVN

CTJR

ULQW

NSFX

KCTB

NKIP

LRSW

All

Bottom

Middle

Iconic memory holds a vast amount of visual information for a very short time (less than a second). This information decays rapidly unless attention is focused on specific details, which is what the cue helps to do.

Experiment Time

3021302230233024

​

​

Memorize the number

You will have 3 seconds

​

50

Experiment Time

6021 6022 6023 6024

​

​

Memorize the number

You will have 3 seconds

​

51

Experiment Time

• How did you do?

​

• 3021302230233024

​

• 6021 6022 6023 6024

​

​

52

Short-Term Memory as Working Memory

• Working Memory
• Lasts 10-15 seconds (Goldstein, 2010)
• Fragile, displacement

​

• Capacity
• 7±2 (AT&T, Miller)
• Maximum number of “units” able to be held in short term memory at any one time

​

• When presented with more than 7 ± 2 units, reorganize and chunk the data to create 7 ± 2 units
• For example: N B C M T V C I A F B I C N
• NBC MTV CIA FBI CNN
• Went from 15 unites to 5 units

53

A Decent Starting Point for Menus

• Avoid menus with more than 7±2 items

​

​

54

Three Parts of Working Memory

55

Central Executive System

Auditory Loop

Visuo-spatial Sketchpad

Three Parts of Working Memory

56

Central Executive System

Short-Term Memory as Working Memory

• Task 1:
• Visual: pressing keys on keyboard in clockwise fashion
• Phonological: repeating words “see-saw ”

​

​

• Task 2:
• Visual: Write your name with one hand
• Visual: Make circles with the other hand

57

Attention

Verbal Rehearsal

Visual/Spatial

Information

Short-Term Memory as Attention

• Whatever we are focused on is our short-term memory
• Mental focus on stimuli
• Filter Theory

​

• Selective Attention
• Cocktail party effect
• Daydreaming while walking

58

Short-Term Memory as Attention

59

Vigilance and Attention

• Continuous attention to monotonous, long-lasting tasks
• Inability to perceive stimulus right in front of you
• Problems detecting events

​

• Csikszentmihalyi’s (chik-sent-mee-hai-ee) “flow”: Most engaging tasks are difficult enough to provide a challenge and require continued attention, but not so difficult that it invokes frustration and anxiety
• Breaking down tasks

​

​

​

​

60

Some more concepts…

• Primacy effect
• Tendency to remember first items in a list
• May be due to more rehearsal early

​

• Recency effect
• Tendency to remember last items
• Could be from sensory memory
• Items in the middle of the list not remembered as well

​

• Memory Trace: The encoded memory imprint

​

• Interference or Displacement: New memories displace older ones in STS

61

Experiment Time

Jungle

Bird

Forest

​

​

Memorize the following words

You will have 3 seconds

​

62

Experiment Time

• What did you do in your head to memorize the words?

​

• Did you:

– Memorize each word?

– Picture each word in your head?

– Repeat the sound each word makes?

– Picture a bird in a jungle with feathers?

​

• What you did in the previous experiment was encode the information.
• How you represent the physical stimuli in memory

​

63

Long Term Memory

• Coding
• While acoustic/visual similarity affects short-term memory, semantic similarity effects long-term memory
• Words that sound alike are acoustically similar
• Words that mean the same thing are semantically similar

​

• Capacity
• Unknown
• Could be in billions of bits of data
• Not always retrievable
• Depends on the amount of rehearsal

​

64

Long Term Memory

• Can be like flashbulb especially with negative/traumatic events

​

• Transfer from short-term memory to long-term memory also results from coding
• Adding extra information from long term stored to the presented information
• Not perfect or high resolution
• Subject to interference

65

Ebbinghaus’ Forgetting Curve

66

Announcements

• Fill out the project preference form
• Quiz 2 due on Friday
• CITI Training (Optional but recommended)

​

67

Attendance & Next Time

1. Cognition Cont.

​

• Mistakes and Slips

​

68