[TM Technical Report]
An Analysis of Question Types

[Author: Low Weilin, Contributions: Arnold Christopher Koroa]

Text

Multiple Choice(Single answer)

Multiple Choice(Multiple answer)

Introduction

TEAMMATES allows instructors to set their own questions in feedback sessions. This report aims to document the different types of questions (e.g. essay, MCQ, rubric etc.) which are used in surveys elsewhere.

These are the aspects investigated:

Names: {possible names we can use to identify this question type to users, especially the names used by other tools}

Purpose: {When would this be used?}

Question presentation: {how is the question presented to the respondent? Show all possible options}

Result presentation:{how should the results be presented to receiver? to instructor?} {what kind of summary/statistics can be generated about responses?}

Set-up: {what data is collected from the instructor in what way?}

Question validation: {any validation to be done during the adding/editing of questions?}

Notes: {Other things to note}

The report is based on question types used in:

[IVLE] IVLE
[GDoc] GoogleDocs/Google Forms
[Monkey] SurveyMonkey
[iPeer] iPeer
[Question Pro] QuestionPro
[SurveyGizmo] SurveyGizmo
[CATME] Comprehensive Assessment for Team-Member Effectiveness
[STEAM] Simple Team Experience Assessment Measure
[PRAZE] PRAZE
[TLA] Team Learning Assistant
[WebPA] WebPA
[Moodle] Moodle

Summaries/screenshots of each platform can be found under Appendix: Question formats from elsewhere

Question Types

Text

Names:

[GDoc] Text , Paragraph text

[SurveyMonkey] Open-Ended Question, Comment/Essay, Single Textbox, Multiple Textboxes

[IVLE] Essay

[SurveyGizmo] Short text, Long Text

[QuestionPro] Open-Ended Text

Purpose:

For open-ended questions

Question Presentation:

A textbox where users can type in their answer.

Plain text or rich text can be used.

Result Presentation:

Results of all: Question text, followed by each answer

Statistics:

Average length
How many answered/skipped
Shortest, longest
Sorted by length
Keyword analysis using wordcloud
Similarity analysis between answers

Individual results: Question text, answer text

Set-up: Type in the question.

Plain text or rich text
“Required” option
Response validation e.g. valid email, Max length, etc.

Examples [GDoc]

Question validation: Non-empty question

Notes:

[GDoc] uses two types: Text (for short answers), Paragraph Text (for longer answers)

[Survey Monkey] uses (single/multiple)textbox for short answers, comment/essay for long answers. Numerical Textboxes is a variation where only numbers are allowed. Demographic information is a group of standard questions for contact information.

In TEAMMATES:

Already supported. Name: Text. No response validation. Plain text.

Multiple Choice(Single answer)

Names:

[IVLE] Multiple Choice, Select List

[GDocs] Multiple Choice, Choose from a list

[SurveyMonkey] Multiple Choice (Only One Answer)

[SurveyGizmo] Multiple Choice, Dropdown Menu

[QuestionPro] Multiple Choice - Select One, - Dropdown Menu

Purpose:

A question with multiple answers in which one answer must be selected.The user can optionally be allowed to specify his own answer to the question by enabling the “other” choice.

Question presentation:

Radio buttons or dropdown box, with a small textbox beside the “other” choice if it is enabled.

Result presentation:

Each option can be displayed, with the number/percentage of users who selected that choice. Visually, a graph of the distribution can be shown if the range of choices is large.

Set-up:

Instructor can define the options available, and choose if respondent can define his own “other” choice, also, define if this is a required question that must be answered before submission.

Question validation:

Question must be non-empty. Option should be non-empty.

Response validation should be done to ensure a valid option is selected.(unless an ‘other’ option is used)

Notes:

Conventionally, display-wise, radio buttons are used for single selection items, while checkboxes are for multiple selections. If there are many options, it would be better to consider using a dropdown list for selection instead, to compact the option list.

Multiple Choice(Multiple answer)

Names:

[IVLE] Multiple Response

[GDocs] CheckBoxes

[SurveyMonkey] Multiple Choice (Multiple Answers)

[SurveyGizmo] Checkboxes

[QuestionPro] Multiple Choice - Select Many

Purpose:

A question with multiple answers in which one or more answers must be selected.The user can optionally be allowed to specify his own answer to the question by enabling the “other” choice.

Question presentation:

Checkboxes, with a small textbox beside the “other” choice if it is enabled.

Result presentation:

Choices made by a particular respondent can be listed. A summary of the data can be shown in the form of number of respondents/selections per option.

Set-up:

Instructor can define the options available, and choose if respondent can define his own “other” choice.

Instructor can also define the minimum/maximum number of choices that the respondent must/can select.

Question validation:

Question must be non-empty. Option should be non-empty.

Response validation should be done to ensure a valid option is selected.(unless an ‘other’ option is used)

Notes:

The minimum number of choices should be >=2. Otherwise, it is the same as using multiple choice(single answer).

Likert Scale

Names:

[IVLE] Likert Scale

[GDocs] Scale (0-1 to 2-10)

[QuestionPro] Rating

[SurveyMonkey] Rating Scale

[SurveyGizmo] Likert Scale, Net Promoter Score(1-10 scale)

[CATME] Unique representation, see under CATME for the rating scales/options that it uses.

Purpose:

Used to obtain feedback with options that form a scale.

Question presentation:

Options are usually presented horizontally.

Common sets of options include:

Number scale(1-5, 1-10)
Agree-Disagree
Best-Worst
Stars

Result presentation:

Similar to multiple choice, with a graph summary and average of the choices can be displayed. The rated persons/teams/options can also be ranked according to their average scores.

In TEAMMATES, we can also display average/frequency per person/team, if the receiver is a person/team.

Set-up:

Instructor enters the question, and type of scale to use.

Question validation:

Question must be non-empty, type of scale must be defined.

Response validation should be done to ensure a valid option(in the scale) is selected.

Notes:

This is a variation of Multiple Choice(Single answer). To prevent implementing too many variations of this question type, using a number scale will be sufficient, as the instructor can qualify in the question(e.g. … one being the worst, 10 being the best.)

Matrix

Names:

[GDoc] Grid

[QuestionPro] Matrix Table

[SurveyMonkey] Matrix of Choices, Matrix of Drop-down menus

[SurveyGizmo] Multiple Choice Table, Table of Checkboxes, Table of Dropdown Menus, Table of Textboxes

Purpose:

Used when multiple similar questions have the same options and can be represented in a more compact table form. (e.g. multiple rating questions)

Question presentation:

Question, as well as header fo options are not repeated and grouped into table form.

Result presentation:

Selection count and frequency can be shown in a table.

Set-up:

Instructor enters the question, and headers for each row and column. The question type is then set(multiple choice(single/multiple answer), text boxes, etc)

Question validation:

Question must be non-empty.

Response validation should be done to ensure valid option(s) is selected.

Notes:

This question type can apply to many other question types, to act as a ‘question group’.

Ordering/Ranking

Names:

[IVLE] Ordinal

[QuestionPro] Ordering

[Survey Monkey] Ranking

Purpose:

A question where given N items, the user is to rank them from 1 to N where 1 is the best and N is the worst using some criteria.

Question presentation:

[Survey Monkey]: Lists options and a drop-down list beside the option to allow user to choose ranking.

Alternative presentation might include dragging and dropping options to the correct positions.

Result presentation:

Summary/averaged ranking can be shown. Alternatively, for each option, the frequency of it being chosen as rank x can be shown.

Set-up:

Instructor provides the options available to rank.

Question validation:

Question must be non-empty.

Response validation required to ensure no two options are ranked the same. All options must be ranked.

Fill-In-The-Blank

Names:

[IVLE] Fill-In-The-Blank

Purpose:

Allows user to enter multiple answers within a question/paragraph.

Question presentation:

The question is presented as a paragraph of text, with text boxes anywhere within the text to be filled in by the user.

Result presentation:

A summary of number of correct answers can be shown.

A wordcloud of the most common answers can be used to show common answers.

Set-up:

The instructor is required to type in some text, with tags to indicate where a blank to be filled in should be. In the case of IVLE, “_BLANK_” is used to indicate where an entry can be made. e.g. if the question is "The capital of South Africa is _____", then put "The capital of South Africa is _BLANK_" in the text box.

Multiple such tags can be placed in the question, and answers to each one defined. In IVLE, answers can be text, with option of being case sensitive, and requiring exact or partial match. Numerical answers can also be used, where the correct answer can be defined to be between two numbers.

Question validation:

Question must be non-empty, and contain at least one blank to fill in.

No response validation required for the text input.

Notes: This is a variation of a text question.

Matching

Names:

[IVLE] Matching

Purpose:

To match pairs from two lists.

Question presentation:

In IVLE, each option from list 1 is listed out, with a drop-down beside each option to pair the answers from list 2.

Alternatively, matching questions can be presented by showing two lists on the left and right, and allowing users to drag lines to match pairs from one side to another.

Result presentation:

For each option on the left, the percentage of each option on the right that are matched to it can be shown.

Set-up: In IVLE, the number of options are selected and defined, along with its corresponding answer.

Additionally(not in IVLE), instructors should also be able to define if this is a (one-one / one-many / many-one / many-many) matching. Some options may not necessarily have a corresponding answer to it, and need not be matched.

Question validation:

Question has to be non-empty.

The number of options on each list should be valid.

Response validation should be done to ensure a valid options are selected.

Notes:

Variations of this question type can be made, with options to allow pairings to be not just one to one.

Constant Sum

Names:

[IVLE] Constant Sum

[SurveyGizmo] Continuous sum

Purpose: Question to enter a numerical value for a set of variables, but requires the values to add up to a pre-specified value.

Question presentation:

Similar to a text question. Where a question is followed by a list of text boxes with labels on the left.

Result presentation:

Statistics: Average/Median/Min/Max/Standard deviation values for each variable

Set-up:

Define the constant sum to be added up to.

Define if the sum is dependant on the number of variables.(for example, 100 points per student, if 5 students, split 500 points among them.)

Define number of variables(values) user has to input

Define labels for each variable.

Question validation:

Question cannot be empty.

Response validation should be done to ensure the sum is correct, and only numerical input is entered.

Notes:

A variation of numerical input, with logic added.

Disable floating point numbers to allow easier summing up of values. (Or allow the precision to be set)

Slider

Names:

[SurveyGizmo] Slider

[QuestionPro] Slider

Purpose:

A variation of likert scale or numerical question, that gives ease in selecting a number, using an interactive element.

Question presentation: A slider element, with question text at the top and labels on the left, right and center of the slider.

Result presentation:

A summary of the average value selected or a graph showing the distribution can be shown.

Set-up:

Maximum/minimum values can be set.

Increment value can be set.

Labels on the left/right/center can be defined.

Question validation:

Non-empty question, maximum value should be larger than minimum value.

Response validation should be done to ensure a a number within the correct range is submitted.

Notes:

Although a slider is easy to control, a textbox input for numbers will give more precise control over the intended value.

Image

Names:

[QuestionPro] Image/Multimedia

[SurveyMonkey] Image

[SurveyGizmo] Image Choice(single), Image Choice(multiple)

Purpose:

Used when choices are image/video/multimedia based instead of text. For example: “Which do you think is the best design?”

Question presentation:

The question is presented as a list of images / videos / multimedia, with radio buttons or checkboxes below/beside each option depending on the type of question.

Result presentation:

Each image/video/multimedia is labeled with a number, and statistics for number/percentage of respondents choosing that option is shown.

Set-up:

Instructor enters question, as well as the image/video/multimedia options.

Question validation:

Question must be non-empty.

Response validation should be done to ensure a valid option is selected.(unless an ‘other’ option is used)

Notes:

A variation of multiple choice question types, where the options are images / videos / multimedia.

File Upload

Names: [SurveyGizmo] File Upload

Purpose:

Used when respondent is required to submit a file. (e.g. pdf/doc/image assignment , resume, etc.)

Question presentation:

Question, followed by a file upload button to allow respondent to select file to upload.

Result presentation:

Instructor is shown a list of files uploaded and have the option to download all, or each one individually.

Set-up:

Instructor enters the question and sets file type to be uploaded.

Question validation:

Question has to be non-empty.

Response validation should be done to ensure a valid file type is submitted.

Date/Time

Names:

Date, Time, Date and/or time

Purpose:

Allows user to conveniently select a specific date or time. Examples of usage could be to arrange a timeslot for meeting.

Question presentation: Usually allows user to select from a popup calendar.

Alternatively, it can be entered in text.

Another option would be to have a timetable styled selection, where students can use to indicate his/her free periods. (could be used to find best matches for scheduling meetings/tutorials/etc) (similar to grid/multiple row/col)

Result presentation:

In GDoc, date/time is presented in text, inside a spreadsheet.

Possible summary statistics could include: most popular date/times, a graph of the date/times selected.

Set-up:

Choose question type(date or time or both)
Select date/time format(DD/MM/YYYY) (Available to choose in survey monkey)
Option to change error message to show when date/time is invalid
Set if question is required
Instructor should be able to set a range of selectable date/times(period when he is free to meet for example)

Question validation:

Question must be non-empty. Range of date/times must be valid.

Response validation should be done to ensure a valid date/time is submitted.

Notes:

Some related question types have ability to allow selection of a common timeslot for people to meet up, or to schedule timetables(see http://doodle.com/). This may be helpful in team forming or as a record to ensure teams actually meet up to do work.

Novel question types

These are not found elsewhere, but could be used in TEAMMATES.

First-come-first-served

Options already chosen cannot be chosen by others. Can possibly be used to book appointments with the instructor during his available times.

Choose from persons

Options are generated from students/teams/instructors in the course. e.g. Which student did the best presentation?

Question Types by Importance

This section lists the various question types to be implemented in order of descending importance. This is based on how common these question types are used.

Essay
MCQ(Single answer)
MCQ(Multiple answers)
Likert Scale
Ordering/Ranking
Constant Sum
Numerical
Date/Time
Matching
Matrix
Fill-In-The-Blank
Slider
Image
File Upload
Email

Other Features

This section presents other features in surveys/assignments/quizzes/etc that applies not only to specific question types.

Admin Features

Opening/Closing Date/Time
Time Limit
Number of Attempts
Resume options
Auto-Marking
Export of results

Question Randomization

Randomizing question and option order may give unbiased feedback. For some questions(MCQ), there can also be multiple correct/wrong answers which can be selectively shown at random such that respondents cannot copy from each other.

Question Logic

Question Logic allows control of question flow in a survey. For example, the instructor may want the respondent to skip question 2 if question 1 was answered as “No”.

[GDoc] Paging - Allows defining of pages and skip to pages depending on answers to MCQ(single answer) questions. If multiple questions have such logic, the last question on the page is used.

Field Constraints

Sometimes, specific information is expected and constraints have to be put on the data entered. The below lists common fields that might be used with constraints.

Text length
Email
Numerical
Alphanumeric
File type - (for file upload)

Media in Questions

Some questions require media(images, etc) in the question or as part of the survey. This section list some common means to do so used by popular online survey tools.

[GDoc] Allows Inserting Image, Video into the form.
Rich Text questions allow inserting formatting as well embedding images/videos into questions.

Dynamic/Generated Options

For TEAMMATES, it would be beneficial to have dynamically generated lists as options to a question. Currently, TEAMMATES supports multiple feedback paths such as from a respondent to his/her team members.

However, these questions are presented separately for each path/person. For example, a feedback path to team members works for text questions such as “How did this member contribute?”, and the same question will be shown for each member. However, we might alternatively want a question “Rank your team members according to their contribution.” as feedback to the team.

Extending these dynamically generated options to integrate into other question types (MCQ, Ordering, Likert Scale, etc) and feedback paths, will provide more consistency and flexibility to TEAMMATES.

See: Choose from persons

Auto Marking

For questions that do have clearly defined answers, auto-marking is a possible feature to include. For TEAMMATES however, there might not be ‘correct’ answers to feedback given.

Progress bar

For long surveys, we could add in a progress bar to let students know how many questions are left.

Example from Google Forms:

Future work

Examine questions types used by MOOC systems such as Coursera
To study:

References

[1] Google Forms: http://www.google.com/google-d-s/createforms.html

[2] Google Gson: https://code.google.com/p/google-gson/

[3] Survey Questions and Answer Types. Retrieved December 13, 2013 from QuestionPro: http://www.questionpro.com/a/showArticle.do?articleID=survey-questions

[4] CATME: https://engineering.purdue.edu/CATME

[5] iPeer: http://ipeer.ctlt.ubc.ca/

[6] STEAM: http://peereval.okstate.edu/beta/WelcometoSteam.html

[7] PRAZE: http://www.lms.unimelb.edu.au/teaching/assessment/praze/

[8] Team Learning Assistant: http://smgworld.bu.edu/teamlearning/assistant/

[9] Moodle: https://moodle.org/

[10] IVLE: https://ivle.nus.edu.sg

[11] SurveyMonkey: https://www.surveymonkey.com/

[12] SurveyGizmo: http://www.surveygizmo.com/

Appendix: Question formats from elsewhere

CATME

CATME stands for Comprehensive Assessment for Team-Member Effectiveness. See details here: https://engineering.purdue.edu/CATME. A sample survey can be found here: https://www.catme.org/login/survey_instructions.

Summary of CATME

CATME is a tool focused on helping instructors assess the effectiveness of teams based on the areas below, through a rating system.

Contributing to the team’s work
Interacting with teammates
Keeping the team on track
Expecting quality
Having relevant knowledge skills and abilities

Based on deviations/conflicts in ratings, it helps to identify some of the following information about teams and its members (this is similar to TEAMMATES where inconsistencies can be detected):

Low—a student who rates him/herself as ineffective and who also receives “ineffective” ratings by teammates.
Overconfident—a student rated as “ineffective” by teammates but rates him/herself as much more effective.
High—a student who is rated as highly effective according to both teammate and self ratings.
Underconfident—a student rated as highly effective by teammates but who under-rates her/himself.
Manipulator—a student who rates him/herself as highly effective and who rates teammates as ineffective in disagreement with teammates. Such a student may be trying to influence the distribution of grades unfairly.
Conflict—a team in which there is considerable disagreement among the various raters about the effectiveness of an individual student.
Clique—a team in which cliques appear to have formed. The ratings show that subsets of the team rate members of their subset high and members of other subsets low.

Means of assessment

CATME uses a vertical rating scale for each member in the team, for each area it assesses.

Some screenshots:

A summary of results is shown below. Ratings of the respondent and expected ratings are shown with blue and red arrows. Green arrows are shown if they match.

iPeer

iPeer is a web-based platform/database independent application (written in PHP) to develop and deliver peer evaluations, review and release student comments, build rubrics and progress report forms online, configure email notifications and etc. It can be found here: http://ipeer.ctlt.ubc.ca/. A demo/sandbox version can be found here: http://ipeer.ctlt.ubc.ca/wiki/Demo.

The evaluations provided by iPeer include the following:

Simple Evaluations: students assess the contributions of their fellow group members by distributing a set number of points (e.g. 100) among them.
Rubrics Evaluations: the evaluation process is similar to a multiple choice questionnaire; each group member rates the others on a scale called "level of mastery". iPeer automates the process of peer evaluation and lowers the workload for both instructors and students. With the benefits of automation students can gain feedback more quickly. Additionally, instructors are less likely to lose evaluations and more likely to encourage peer evaluations.
Mixed Evaluations: The mixed evaluation of two above.

Question types include Sliders, Constant sum, Text boxes and Likert Scales. Some screenshots shown below.

STEAM

STEAM - Simple Team Experience Assessment Measure

Website: http://peereval.okstate.edu/beta/WelcometoSteam.html

Demo: http://peereval.okstate.edu/NewStudentDemo/StudentLogin.asp

Questions and options used in STEAM are highly customizable.

Summary of question types used:

Numerical Text box
Likert Scale
Constant Sum(percentage distribution)
Text boxes/Comments

Screenshots below.

Below: Percentage has to tally up to 100% for each column.

Below: Text comments.

Below: Scores have to sum up to 100%. Note: Overall rating is a compulsory field in the feedbacks.

Results

After evaluation, students are able to see scores of himself given by his team members for each section. An interactive java applet shows a graph distribution in comparison to peers. mean scores and standard deviation are also shown in a table.

Example page: http://peereval.okstate.edu/NewStudentDemo/Part2/studentgrade.asp

PRAZE

Site: http://www.lms.unimelb.edu.au/teaching/assessment/praze/

Guide: http://www.lms.unimelb.edu.au/teaching/assessment/praze/praze_userguides/PRAZE_Creating_Forms.pdf

Summary:

Questions are in rich text, giving option to add in images and other embedded media.

PRAZE offers the following question types:

Short/Long answer questions
Multiple Choice questions(single answer) (maximum of ten options per question)
True/False questions
Yes/No questions
Likert Scale questions
Numeric answer questions

Team Learning Assistant

http://smgworld.bu.edu/teamlearning/assistant/

Summary: Uses Likert Scales on several feedback questions, as well as text boxes for comments for each member.

WebPA

WebPA is an open source online peer assessment tool that enables every team member to recognise individual contributions to group work. Site: http://webpaproject.lboro.ac.uk/. Demo: http://webpaos.lboro.ac.uk/login.php

Summary:

WebPA allows two question types, Likert Scale and Constant Sum(split 100 among team members). Instructors can define Title of assessment, introduction text, and assessment criteria for likert scale and “split 100”