Welcome!
LINGUISTICS
UG Advanced Experimental & Quantitative Workshops
10-11 June 2019
Before we start...
Link to slides on: https://www.adamjchong.com/ug-quantexp-workshops-2019.html
(go to www.adamjchong.com and navigate from there - Teaching)
For today, make sure you have downloaded and installed: �OpenSesame - https://osdoc.cogsci.nl/3.2/
You’ll need R and R Studio for tomorrow - see the links online and in the overview
2
What?
3
Who?
4
Dr. Adam Chong
Dr. Kathleen McCarthy
Dr. Adib Mehrabi
What we’re assuming you know
These topics would have been covered in LIN5202 Research Methods last semester
5
How to best make use of these workshops
6
After attending these workshops, I will:
7
After attending these workshops, I will:
8
More broadly: Transferable skills
•Dealing with complex data - how manipulate, query, visualise and understand
•Experiments and data analysis: logical problem solving
•Technical skills – e.g. R, some basic coding
•Being able to learn some basics, and consult further resources independently to learn more, as needed (independence)
9
Overview of workshops:
Today (Monday):
Tomorrow (Tuesday):
10
Overview of today:
First half (morning)
Second half (afternoon)
11
Recap: What is an experiment?
Way to scientifically test what affects a particular outcome, while controlling for extraneous factors (cf. reliability, validity)
12
(Experimental) Variables
Variable: something that can take on multiple values (general definition)
Dependent variables: the specific thing you are measuring in an experiment (e.g. reaction time, VOT, % correct etc.)
Independent variables: what you manipulate to see if it has an effect on what you measure (i.e. your dependent variable)
Note: we’ll be using the term ‘variable’ to mean a couple of different things, but they all derive from the general definition above.
13
Types of measures (dependent variables)
14
Types of measures (dependent variables)
NOTE: You will almost certainly be working with a behaviourial measure.
15
What type of variable are you dealing with?
16
A particular variable can be classified into two main types:
i. Continuous:
ii. Categorical:
Discrete groups/types
intervals between the values are equally spaced, counting starts at absence/0 (ratio) or relative position (interval)
e.g., body weight. vowel formant frequencies, reaction times, etc.
A. Nominal:�discrete categories; no relative/intrinsic ordering (i.e., no agreed way of ordering these values from highest to lowest); no values in intervening space
B. Ordinal:
discrete but ordered e.g., high to low; even though they are ordered the spacing between the values may not be the same
e.g., sex, ethnicity, presence/absence of a word, phonological environment, etc.
e.g., education level, socio-economic status, age (if binned), etc.
Why does the type of variable matter?
Note: that you can transform data from one kind to another - e.g. you can ‘bin’ age (continuous) to just two categories (young vs. old).
You generally don’t do the reverse though...
17
Types of experimental designs
Between-subjects designs: comparing behaviour between two (or more) groups of participants�
18
Types of experimental designs
Within-subjects: interested in comparing effect of some variable on the same participant
E.g., does a given listener show a similar perceptual judgement of stigmatised vs. non-stigmatised variants in speech? (same listener but hearing both types of variants)
19
Types of experimental designs
Mixed-Designs: some between- and some within-subject variables
E.g., discrimination task: Japanese vs. English listeners’ discrimination /r/ and /l/ (contrast in only Eng.) vs. /t/ and /s/ (contrast in both)
Within-subject IV: Type of contrast (contrastive in both or just in English) - each listeners hears both!
Between-subject IV: Language background (i.e. Japanese vs. English)
20
Group Activity
Read ‘methods’ sections of published papers
21
What was the point of the ‘methods’ activity?
22
Dealing with confounds/controlling factors
Sampling - this is the one most of you are probably most aware of
‘Carryover effects’ - ordering effects, item specific effects, etc.
23
Once you’ve designed your experiment, i.e. You’ve thought about:
You’re now ready to think even more specifically about HOW your experiment will run - i.e. what happens when?
24
Some key terms:
Phase: a particular stage of an experiment (e.g. practice phase, test phase)
Trial: a series of procedures in an experiment, where a participant’s behaviour is measured
Block: a set of trials �(a given phase can have one or more blocks of trials)
25
Experiments - as a series of ordered events
Next step, implementation.
26
Thinking about experiments schematically
(this will significantly help implementing/building it using software, for later)
We’ll look through some simple examples then think about how this applies to the examples you just examined.
27
There are a number of levels to experimental structure
28
Instructions
Test Phase
Feedback/�End
I. Global sequence
There are a number of levels to experimental structure
29
Instructions
Test Phase
Feedback/�End
Block �(of trials)
I. Global sequence
II. Block
sequence
There are a number of levels to experimental structure
30
Instructions
Test Phase
Feedback/�End
Block �(of trials)
I. Global sequence
Trial
III. Trial sequence
II. Block
structure
There are a number of levels to experimental structure
31
Instructions
Test Phase
Feedback/�End
Block �(of trials)
Trial
Loops as many times as there are trials
I. Global sequence
II. Block
sequence
III. Trial sequence
You can have more complex Global sequences
32
Instructions
Practice Phase
Feedback/�End
Block �(of trials)
Trial
Loops as many times as there are trials
I. Global sequence
II. Block
sequence
III. Trial sequence
Block �(of trials)
Trial
Test Phase
Break
33
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
Break
Fixation Cross
Sound 1
Make response
Sound 2
500 ms
Sound 3
500 ms
500 ms
2 seconds
Group Activity
Can you draw out the sequence of events that occur in the experiments you looked at?
34
Local Summary
Next up, actually implementing an experiment - this is more straightforward if you understand how to think through the procedure of an experiment schematically as we’ve just done!
35
LUNCH BREAK
See you back here at 1.30pm
36
Simple semantic priming example.
(similar to what you did in Prac Ex 4 in Research Methods)
Inspired by Meyer & Schvaneveldt (1971), Milberg & Blumstein (1981)
The authors were interested if the relationship between words (i.e. the semantic relation) can facilitate lexical retrieval. The notion of ‘semantic priming’ refers to the idea that a response to a word (e.g. ‘doctor’) is faster when it is primed (i.e. preceded by) a word that is semantically related (e.g. ‘nurse’). The task that people have to do in a lexical decision task is to decide if the word they see (i.e. the target) is a word or not.
37
Simple semantic priming example.
Stimuli:
Stimuli pairs consisted of pairs of real or nonwords. Each pair consisted of a prime (the first member) and a target (the second member). There were three main experimental types of word-pairs. In the first three pairs, a real word target was preceded either by a related word (e.g., dog-cat), an unrelated word (e.g., land-cat), or a nonword prime (e.g., lak-cat). In the final two, a non-word target was preceded by either a real-word prime (e.g. dog-lak) or a nonword prime (e.g. mion-lak).
[AC: what information is missing here? What else would you want to know?]
38
Simple semantic priming example.
Stimuli:
In total there were 30 stimuli pairs, 5 real word targets preceded by a related prime, 5 preceded by an unrelated prime, 5 preceded by a nonword prime. The remaining 15 pairs consisted of nonword targets, with 10 preceded by real word primes, and 5 preceded by nonword primes. To control for priming effects across the experiment, targets and primes only appeared once. Three experimental lists were created to counterbalance any word specific effects.
5 additional stimuli pairs, 1 of each stimulus type, were created for practice trials. None of these were used in the test trials.
39
Simple semantic priming example.
Quick check-in:
How many trials does this experiment have?
How many conditions are there?
40
Simple semantic priming example.
Apparatus & Procedure: (sometimes you see these separated out!)
Test trials were all presented in one test block. The experiment was presented through OpenSesame (Mathôt, Schreij & Theeuwes, 2012). At the start of each experiment, participants saw a set of instructions printed on the computer screen in front of them. They were instructed that they were going to see two letter strings, and to decide, as quickly as possible, if the second string was a word or not. They were instructed to press the Z key if it was a word and M key if it now a nonword. Participants first completed the 5 practice trials, before moving on to the test phase.
On a given trial, a participant first saw a fixation cross in the centre of the screen that lasted for 500 ms. Then the printed prime and target were shown consecutively. The interstimulus interval was 500ms, and the inter-trial interval was 2 seconds. Participants were instructed to respond as quickly as possible, and responses above 2000ms were not counted. The entire task lasted approximately 15 minutes.
41
Let’s schematise this!
We won’t worry about the multiple experimental lists for today.
42
Existing experimental software
OpenSesame
Gorilla (online)
PsyToolkit (online)
E-Prime
Praat
Etc.
43
What is OpenSesame?
44
Now let’s go to OpenSesame
Open up OpenSesame on your own computers
If you open OpenSesame, it should by default start with a blank experiment.
Let’s first get oriented with the general layout.
In more detail: https://osdoc.cogsci.nl/3.2/tutorials/beginner/
Consult the documentation manual and forums - these are really useful!
45
46
Variable Inspector
File pool
Hide this bit for now - we won’t really be using this.
47
Let’s start today with the ‘default template’
Before moving on
Save the file in a new folder that you can access straightforwardly.
I save EVERYTHING onto Dropbox - you should do the same.
This way when you test run your experiment the data will all go to the same folder.
48
49
General script = the under-the-hood code that actually runs your experiment.
You can get by without worrying about this - but there are some nice tutorials that show you how to modify small pieces of this to do more complex manipulations
Various elements that OpenSesame provides for putting together an experiment.
50
Two important structural components
Holds other set of objects together, and ensures that they occur in a particular order - i.e. a sequence of events
As its name suggests, this is a general object the runs another item multiple times.
This is also where you will be defining your independent variables - what you will want to manipulate
51
Loop-Sequence structure
To repeat a sequence of events, you’ll need a loop item that contains a sequence item.
A sequence item will only execute once.
52
53
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
Fixation Cross
Prime
Make response
Target
500 ms
500 ms
ITI
2 seconds
Some other bits we’ll be using
54
Ways to display stimuli
Sketchpad: allows you to display text, images etc.
What do you think this does?
Ways to record responses
What do you think this does?
Keyboard response: registers if, when, and what key was used
How to start:
You can select an icon from the left hand bar and drag them to the Overview panel, directly under where you want the item to occur (note you can also change this after).
55
Closer look at the Sketchpad
56
Tools: like inserting images ,etc.
Canvas: this is what you’ll see on the screen in the experiment.
Notice the grids!
Name of item - you can rename this here
Duration: default = until a key is pressed
Name this ‘instructions’
Closer look at the Sketchpad
57
Type some text here:
Let’s create the global structure
Refer back to the global structure of the experiment you came up with before.
Task:
58
A quick comment about naming conventions
You will notice that I did two things:
59
60
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
Fixation Cross
Prime
Make response
Target
500 ms
500 ms
ITI
2 seconds
Practice and Test Phases
Both of these are quite similar! So once we’ve got one of them, we’ve got the other.
Let’s start with the Practice Phase:
What we need first is a Loop Object - let’s call it practice_Loop
This loop object will loop over the practice Block Sequence (remember: a block = a collection of trials)
61
62
I have created a variable called ‘practice’ which has the value “yes”
If you have more than one block of trials, you would create another variable to deal with that - see some of the examples in the online documentation.
Loop-Sequence
A loop on its own does nothing. It needs something to run!
This is where the Sequence comes in.
Note that if you drag it and place it under prac_loop, it will prompt you to ‘insert into’ or ‘insert after’.
‘Insert into’ = the sequence is within the loop itself (this is what we want here)
63
64
65
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
Fixation Cross
Prime
Make response
Target
500 ms
500 ms
ITI
2 seconds
We need a Trial Loop within the block sequence
If you drag another Loop Object under prac_sequence, you’ll get the same prompt of where in the experiment you want this to occur.
Should the Trial Loop go into or after the prac_sequence?
66
67
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
So far.
We now have the general structure of the entire experiment (more or less).
And we have most of a practice phase done.
But we haven’t actual set up what happens on each trial yet! (i.e. the trial_sequence)
68
How the trial enfolds
69
III. Trial sequence
Fixation Cross
Prime
Make response
ITI:
2 seconds
Target
ISI:
500 ms
For each red box, you’ll need to use an OpenSesame item (i.e. drag it from the left bar) - can you think of what type of item you’ll need for each?
500 ms
Blank screen
How the trial enfolds
70
III. Trial sequence
Fixation Cross
Prime
Make response
ITI:
2 seconds
Target
ISI:
500 ms
500 ms
These are SKETCHPAD items
This is a KEYBOARD RESPONSE item
Add these to your overview.
Name them appropriately, as we’ve been doing.
E.g. prac_fixation_cross
Blank screen
prac_fixation_cross
Draw a fixation cross in the middle of the canvas
We should also change the duration to 500 ms - why?
71
prac_prime
prac_target
For now, type ‘prime’ and ‘target’ in the centre of the canvas, respectively.
Make sure these are 50 px in size.
What are the duration for prac_prime and prac_target? Are they the same?
72
prac_target
One weird thing: the duration for prac_target is 0.
The idea is that the minute the display is shown, the next action, prac_keyboard_response is also initiated.
73
prac_target
One weird thing: the duration for prac_target is 0.
The idea is that the minute the display is shown, the prac_keyboard_response is also initiated.
(this was one of those things which took me some time to figure out...it’s a little counter-intuitive…)
74
prac_keyboard_response
We need to set up the response keys
75
We’ll leave this empty since we are going to use “correct_response” but more on this in a few slides
Possible response keys
Upper limit for response - i.e. if a response is not made in 2000ms (2s), then it’s logged as NULL
prac_blank
What should be on this?
What would its duration be?
76
One important thing: logging data!
OpenSesame does not actually log any responses etc, without being told to do so!
So we need to do this!
Here we will use the ‘logger’ item!
77
Drag and drop this after the prac_blank item
Logger
By default, it logs EVERYTHING (all the variables in the experiment.
This will get written into a results file.
Leave it that way - better to log everything and select what you need after, than potentially lose data that cannot be retrieved...
78
We’ve basically got the trial sequence set up
79
500 ms
III. Trial sequence
Fixation Cross
Prime
Make response
ITI:
2 seconds
Target
ISI:
500 ms
500 ms
Blank screen
Taking stock
80
Data source
This is the data source with all your variables that you set (column headers)
You can create this is Excel and copy-paste it here. This is easier when you have a larger datasource than this.
Let’s inspect what I have here for the set of practice trials:
81
Repeat: each cycle = all the listed trials (number of rows).
Here the entire set repeats once.
You can set this lower if you only want to run a subset (e.g. 0.5 runs half)). This is useful if you’re just testing the experiment, and don’t want to run the entirety of it.
One problem...
Let’s go back to:
prac_prime
prac_target
Is this displaying what we want?
82
One problem...
Let’s go back to:
prac_prime
prac_target
Right now - it’s just displaying the text ‘prime’. And so on every trial, it’ll just show ‘prime’! That’s not what we want.
83
84
How do we get OpenSesame to print each cell in this column on each trial?
We use variable notation!
Remember that what we’re doing is essentially ‘coding’ up an experiment.
In coding (as in mathematics), we often resort to the use of variables.
Variables - something that can take multiple values
This variable can just be called when it’s needed, each time with a different value!
In the data source, we’re defining variables, and the values that each variable can take!
85
We use variable notation!
To call a variable in OpenSesame, we use the [variable] notation.
E.g. If I want to call the ‘prime’ variable, I type in [prime].
This can be done ANYWHERE!
E.g., if you want to play a sound file, you can type in [prime].wav
What’s also neat is that all the other info on the same row is recorded with this.
86
87
On each trial, OpenSesame will pick a row, and print the ‘prime’ value for that row, and the ‘target’ value - and all the other information on the same row is recorded with it!
Using variables notation to print the relevant items
88
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
ITI
2 seconds
Fixation Cross
Prime
Make response
Target
500 ms
We’ve basically set up the practice phase!
Good time to test to see if everything is working fine.
89
Select ‘Quick Run’ - the blue play buttons.
You can also select the others, but this is the easiest!
Once you’re done
Let’s inspect the output file briefly...we’ll come back to this:
Check - ‘response’ is registered, and that the ‘response_time’ seems reasonable
90
91
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
Fixation Cross
Prime
Make response
Target
500 ms
500 ms
2 seconds
Some good news
Since the basic structure is the same, we can reuse a lot of what we’ve done already!
If you select ‘prac_loop’, you can copy it and paste it after ‘end_prac’
You’ll notice that there are two copy options - linked and unlinked. Select ‘unlinked’.
92
One small thing.
Under prac_loop, copy logger (linked version) and then replace the new logger item under the new loop you just created with the link-copied one.
What we’re doing is to ensure there’s only ONE logging action across the entire experiment (across both phases).
93
Great! Now your turn
Task:
94
95
Instructions
Practice Phase
Feedback/�End
I. Global sequence
II. Block
sequence
III. Trial sequence
Test Phase
Break
Block(s):
Trial loop
Fixation Cross
Prime
Make response
Target
500 ms
500 ms
2 seconds
DONE!!
Great job everyone!
96
What you’ve accomplished today
You’ve learnt to think much precisely about experimental design and procedure.
You’ve also learnt to implement an simple experiment (with a little bit a ‘coding’ involved)
Tomorrow: we’ll look at the analysis side of things. There there’ll bit quite a bit more coding involved, and the fundamental notion of a variable will play a big role.
97
For tomorrow
If you have already, please make sure to download R and R studio - see the workshop website and overview for links.
98
Some final comments
OpenSesame comes with some fully-implemented examples
The ‘Extended-Template’ gets you a long way towards building a typical experiment (but good to know you can do it from scratch so you actually know what you’re doing…)
To run the experiment, you click the big green arrow - this will then prompt you to type in a subject number. This is where you would be keeping track of your participant IDs (and will be the name of the results file).
The online tutorials are also incredibly helpful!
99