The future is now - research experiments
via online platforms - a brief update
PsyBSc 10: EXPRA - Group 3
Winter term 2021
Peer Herholz (he/him)
Habilitation candidate - Neurocognitive Psychology at Goethe-University Frankfurt
Research affiliate - NeuroDataScience-ORIGAMI lab at MNI, McGill & BRAMS
Member - BIDS, ReproNim, Brainhack, UNIQUE, CNeuroMod
@peerherholz
13/01/2022
A few things before we start...
https://giphy.com/embed/4Z9jjKGJVM9Ck8E4cN
I hope y’all had a nice and relaxing break!
It’s great to see y’all again and I’m looking forward
to the last 4 weeks of this course!
A few things before we start...
https://giphy.com/embed/JRmenOACUaRFCCgSqp
As it’s been a while, we covered a lot of content already and transition
to a new topic, we will spend the first few minutes on bringing things back
to our memory as well as re-situate certain aspects.
EXPRA - 101 - the life cycle of a scientific project from ideas to results
Please make sure to check https://undraw.co/
analyses
results
data acquisition
project creation
not sufficient
not sufficient
not sufficient
not sufficient
sufficient if done right
communication
not sufficient
A holistic perspective: everything is important
EXPRA - 101 - the life cycle of a scientific project from ideas to results
Every steps needs to have/generate FAIR outcomes
Findable
(machine readable metadata essential for automatic discovery)
Accessible
through well-defined and ideally standardized protocols
Interoperable
other (meta)data
Re-Usable
https://ogsl.ca/wp-content/uploads/Fair-rectangle-en.png
EXPRA - 101 - the life cycle of a scientific project from ideas to results
Every steps needs to follow open-science principles
assessment
outreach
publication
writing
analysis
search
Bianca Kramer & Jeroen Bosman https://101innovations.wordpress.com
EXPRA - 101 - the life cycle of a scientific project from ideas to results
life-cycle of a scientific project
modules
content �&
aspects
FAIR
outcome
execution
finalization
introduction & prerequisites
data acquisition & processing
results communication
scientific method
theory building
literature/data surveys
validity & reliability
hypothesis generation
power analysis
preregistration
preregistration
DMP
bibliography
OSF project
(open lab notebook)
data structures
data management
(version control)
understanding data
preprocessing
inferential measures
results diagnostic
preregistration (updated)
DMP (updated)
reproducible analyses
walkthroughs
(open lab notebook)
presenting results
design posters/talks
write articles
publication models
peer review
beyond publication
outreach
poster
talk
report
reviews
(open lab notebook)
idea & conceptualization
We will divide the life-cycle of our scientific projects into 3 modules/blocks and incorporate the respective FAIR/open-science resources/tools .
EXPRA - 101 - the implementation
A few more details
You will be conducting two scientific research experiments:
demo experiment
guided under close supervision
work in groups
same experiment for everyone
lots of feedback from instructor & classmates
your own experiment
(more) independent work
work in groups
hypotheses vary between groups
less direct feedback
Along the way you’ll: - receive input and background information from the instructor for all steps
of a given scientific research experiment
- implement the respective sub-tasks & work together with your classmates on
certain problems & provide feedback
EXPRA - 101 - how to get those credits
A few more details
Successful participation and your grade will be based on different aspects:
demo experiment
guided under close supervision
work in groups
same experiment for everyone
lots of feedback from instructor & classmates
your own experiment
(more) independent work
work in groups
hypotheses vary between groups
less direct feedback
report, one per group,
~15 pages, no grading,
deadline: 15/12/2021 12 PM CET
report, individually,
~15 pages, 100% of your grade,
deadline: ~-09/03/2022 12 PM CET
course work
participation/readings
reviews/feedback
journal club presentation
poster presentation
in groups/individually, no grading,
deadline: throughout semester
Objectives for this session
https://media.makeameme.org/created/look-at-the-25f4b725ed.jpg
Outline for this session
https://twitter.com/OfficeMemes_/status/1298982572848869380/photo/1
Outline for this session
https://twitter.com/OfficeMemes_/status/1298982572848869380/photo/1
Online experiments via meadows.com - monitoring experiments
documentation
Please also consult these pages in addition to this session.
researcher login
Will be provided!
Online experiments via meadows.com - monitoring experiments
Main/landing/overview page
The page you’ll see after the log in.
your experiments
Experiments you created in the past/that are currently running.
Account information
Info related to a researcher’s account.
Researcher account
Everyone will be using the same account for billing reasons.
Online experiments via meadows.com - monitoring experiments
Experiment overview page
The page you’ll see after the clicking on your experiment, e.g. EXPRA_2021_MusicListening.
name & description
experiment settings
structure & tasks of experiment
duration & settings for each task
experiment duration est. & preview
Online experiments via meadows.com - monitoring experiments
How to monitor your experiment
You can monitor your experiment via two options: Activity or Versions.
evaluate/check different versions of your experiment and respective participants’ data in detail
get a general overview how much “traffic” (online minutes spents) wrt your experiment
Online experiments via meadows.com - monitoring experiments
How to monitor your experiment
You can monitor your experiment via two options: Activity or Versions.
Online experiments via meadows.com - monitoring experiments
How to monitor your experiment
You can monitor your experiment via two options: Activity or Versions.
all versions of your experiment
total number of versions differs between experiments based on problems with SONA
Online experiments via meadows.com - monitoring experiments
How to monitor your experiment
You can monitor your experiment via two options: Activity or Versions.
tasks and structure of specific version of experiment
participants who started the experiment
Outline for this session
https://twitter.com/OfficeMemes_/status/1298982572848869380/photo/1
Online experiments via meadows.com - getting data
How to get data from your experiment
You can get from your experiment via two options: browse all -> for each participant or download all to get everyone who started.
current “batch” of participants, i.e. round of data acquisition
check each participant in detail
download all participant’s data
currently running until 31/01/2022 or 40 participants, we’ll discuss strategy next week
Online experiments via meadows.com - getting data
all participants who started
the experiment
random participant ID
participant status
download all data
started experiment but didn’t finish/timed out yet
started experiment but didn’t finish within 24 hours
How to get data from your experiment
You can get from your experiment via two options: browse all -> for each participant or download all to get everyone who started.
Online experiments via meadows.com - getting data
How to get data from your experiment
You can get from your experiment via two options: browse all -> for each participant or download all to get everyone who started.
Y’all should download this version of the data as it will be identical/comparable to the data we analyzed within the demo experiments and thus feasible for the R analyzes script.
Online experiments via meadows.com - getting data
How to get data from your experiment
You can get from your experiment via two options: browse all -> for each participant or download all to get everyone who started.
information on all tasks, including duration
Online experiments via meadows.com - getting data
How to get data from your experiment
You can get from your experiment via two options: browse all -> for each participant or download all to get everyone who started.
Spoiler: most participants timed out and stopped at the sound test!
Online experiments via meadows.com - getting data
How to get data from your experiment
A note on participants in general and specifically online settings.
A Comparison of Crowd Types: Idea Selection Performance of Students and Amazon Mechanical Turks (Banken, 2021)
Online experiments via meadows.com - getting data
How to get data from your experiment
A note on participants in general and specifically online settings.
https://giphy.com/embed/l0MYC0LajbaPoEADu
… to “real-life” research! Seriously, that’s a classic and very expectable. Quite often participants don’t pay attention, just stop, etc. . That’s not you, that’s just how it is and there’s only so much we can do about it at this point.
Any ideas how to get more participants to finish the experiment?
https://giphy.com/gifs/Friends-episode-12-friends-tv-the-one-with-embryos-WOIGpnJ3ye445BUQl4
https://giphy.com/gifs/season-7-the-simpsons-7x20-3o6MbnqpZBwWdWg3gk
Suggested task for y’all: check your experiment once per day and download data as soon as possible. The same holds true for exploring the data following the outline of the demo experiment and respective R script, as well as your data management plans and pre-registration.
(Regarding all these points: try to remember the different project management options we discussed.)
Research experiment groups - monitoring experiments & getting data
Outline for this session
https://twitter.com/OfficeMemes_/status/1298982572848869380/photo/1
https://giphy.com/gifs/Friends-episode-12-friends-tv-the-one-with-embryos-WOIGpnJ3ye445BUQl4
https://giphy.com/gifs/season-7-the-simpsons-7x20-3o6MbnqpZBwWdWg3gk
Task for y’all: present your study/experiment in a short presentation (up to 20 min.), outlining background & hypothesis (and methods). Within that, each group member should briefly summarize one study that was utilized to form the research questions and derive hypotheses, i.e. prior literature on the topic.
Research experiment groups - journal club I
https://giphy.com/gifs/Friends-episode-12-friends-tv-the-one-with-embryos-WOIGpnJ3ye445BUQl4
https://giphy.com/gifs/season-7-the-simpsons-7x20-3o6MbnqpZBwWdWg3gk
Task for y’all: before we start, each group should set up a shared doc with the following parts/headings: general impression, experiment outline/summary, hypotheses derivation & methods and send the link via Discord.
Folks who are not presenting are supposed to provide feedback via this doc.
Research experiment groups - journal club I
Outline for this session
https://twitter.com/OfficeMemes_/status/1298982572848869380/photo/1
https://giphy.com/gifs/Friends-episode-12-friends-tv-the-one-with-embryos-WOIGpnJ3ye445BUQl4
https://giphy.com/gifs/season-7-the-simpsons-7x20-3o6MbnqpZBwWdWg3gk
Task for y’all: provide feedback and impressions gathered so far regarding running your own experiment, initiate discussions, etc., basically consider this section an open mic/stage!
Research experiment groups - general feedback/discussion
Outline for this session
https://twitter.com/OfficeMemes_/status/1298982572848869380/photo/1
Outro/Q&A - Readings/add-on material for this session
Outro/Q&A - Recap for this session
Outro/Q&A - Were your expectations for this session fulfilled?
Outro/Q&A - Questions you could/should ask based on this session
How to weigh pros and cons of online experiments?
How to validate online experiments and responses?
How to situate the interaction between online experiments and different research fields?
If possible, would you consider running only online experiments?
https://giphy.com/gifs/season-17-the-simpsons-17x6-xT5LMB2WiOdjpB7K4o