HRI Human Study Reporting Form
Revision: Jan 13, 2023
This document was created by members of the IEEE P3108 Standards Working Group, “Recommended Practice for Human-Robot Interaction Design of Human Subject Studies”. The goal is to provide a reference for study details such that readers can be confident that it was performed in a technically sound manner. Additionally, we wish to enable reproducibility of studies by collecting detailed information that is not generally included in the main paper text. This form may also be useful as a checklist when formulating your future study.
We intend this form to be submitted with a paper as an appendix or supplemental material, so it should not include new content crucial to understanding the paper (e.g. important technical details not contained within the main text). However, there should be logistical details included here that would be irrelevant or distracting in the main paper text, as the goal is to create a place for that missing information.
The structure and content of this form is not meant to be restrictive; all fields are optional. If you need extra space or entry fields, please add them. If you feel a category of information is missing, please add it as needed. We have recommended a baseline of information that we feel is important, and we hope other HRI researchers can be more fully informed after receiving this extra information.
If you have any feedback about this form, please contact Shelly Bagchi, U.S. National Institute of Standards and Technology, shelly@nist.gov. You can also join the IEEE P3108 Slack here to keep up to date and contribute.
We recommend that authors self-evaluate the reproducibility of your paper after filling out this form. You may wish to consult the ACL Reproducibility Criteria: Review Form – ACL Rolling Review – A new initiative of the Association for Computational Linguistics. We recommend also following the ACL Responsible Research Checklist while writing your paper: responsibleNLPresearch/templates at main · acl-org/responsibleNLPresearch (github.com)
Study Title | Study Setting | City, Country of research | Dates |
e.g. lab, event, field, online / crowdsourcing platform | |||
Author Names | |||
Author 11, Author 21, Author 32, … | |||
Author Institutions | |||
1: Institution 1, 2: Institution 2, … | |||
Institutional / Ethical Review Body Name | If not required by your country/institution, indicate that here | ||
Approval Number (and/or link, if available) | |||
For multiple trials, copy this section and fill it out for every set of participants (along with the Study Design section if different).
# Participants | Recruitment pool | Recruitment method |
e.g. students, MTurk criteria | e.g. mailing list, flyers, direct contact | |
Justification for number of participants (e.g., a-priori power analysis) | ||
Compensation or rewards | e.g. monetary (how much?), class credit | |
Consent type | e.g. form, implied, by guardian | |
Link to consent form or info sheet | ||
Were demographics collected? | Yes, No, N/A | |
Why or why not? Justify for each category | ||
Link to demographics questionnaire or list questions asked here | ||
Demographics of participants (e.g. age range, gender breakdown, etc.): | ||
Category (Insert more rows as needed) | Statistics | |
e.g. age | e.g. range, max/min, distribution | |
What is the target population of your technology? | e.g. children, seniors, all adults (be as specific as possible) | |
Demographics of target population (e.g. age range, gender breakdown, etc.): | ||
Category | Statistics | |
Pre-study inclusion/exclusion criteria, if any | # of excluded participants per criterion | |
e.g. health issues, age, language | ||
Category (Insert more rows as needed) | Justification (why did you need to ask about this category?) | Breakdown within / Statistics of participants | Statistics of target population (if known) |
e.g. age range, gender | e.g. range, max/min, distribution | fill in with corresponding stats | |
Age | To obtain a variety of viewpoints | 18-45 years old; average age 26 | |
Generation | To analyze effect of digitalisation | Equal distribution between X,Y,Z (or) X: 45 (30%) Y: 45 (... | |
Gender | To obtain a representative sample, and to observe if there were any gender-specific effects | 20 female (50%) | |
Apparatus | Control method | Interaction method |
e.g., robot platform, crowdsourcing tool | e.g., autonomous, manual, hybrid, scripted, WoZ | e.g., physical, simulation, remote, video, audio, images |
Structure | Duration / repeats / intervals | # Erroneous runs |
e.g., between/within subjects | ||
Participants per group | See below for table layout if preferred | |
Post-study exclusion criteria (e.g., outliers, breakdown) | # of excluded participants per criterion | |
List of questionnaires used and relevant measure | ||
Description of modifications to existing scales | ||
Independent variables (conditions/design, e.g. 2x3) | ||
Dependent variables (measures used) | Metrics used | |
Control variables (measures used) | Metrics used | |
These are simple examples - create your own structure as needed.
Within Subjects
Group A (# participants) | Group B (# participants) | Group C (# participants) |
Order of conditions: 1, 2, 3 | 2, 3, 1 | 3, 2, 1 |
Demographics of Group A | Demog. of Group B | Demog. of Group C |
Between Subjects
Group A (# participants) | Group B (# participants) | Group C (# participants) |
Condition 1 | Condition 2 | Condition 3 |
Demographics of Group A | ||
Hypotheses | Was it supported? | Statistical tests used for each hypothesis |
1. | Yes / No / Partially (see paper for details) | |
2. | ||
Were your hypotheses pre-registered? If yes, provide a link. If no, why? | ||
Predictions (e.g., measurable differences between conditions) | Was it supported? | |
1. | Yes / No / Partially (see paper for details) | |
2. | ||
Were your predictions pre-registered? If yes, provide a link to the database or previous paper (DOI). | ||
Statistical tests summary (e.g. ANOVA table) | ||
If your paper includes software or datasets, you may wish to self-evaluate the respective sections in the ACL Review Criteria: Review Form – ACL Rolling Review – A new initiative of the Association for Computational Linguistics)
If online release is not possible, authors may point to a filepath within a supplemental archive.
Code availability | e.g. open source, available by request, proprietary/not public, etc. | ||
Link(s) to repository or notebook(s) | e.g. GitHub, Google Colab, Jupyter Notebook, R Markdown files, etc. | ||
Link(s) to data collection script(s) | In addition to code, may also include *verbal* scripts for study administration | ||
Link(s) to data analysis script(s) | |||
Link(s) to online dataset(s) | e.g. Zenodo, DOI link | ||
Link(s) to specific questionnaires used (exactly as administered) | |||