The purpose of this study is to understand how people use crowd-powered conversational agents. Natural language dialog is an important and intuitive way for people to access information and services. However, current dialog systems are limited in scope, brittle to the richness of natural language, and expensive to produce. In response to this situation, we explore to use crowdsourcing to provide instant feedback or supports in an automated dialog system to make it better.
A crowd-powered dialog system contains active components that are run by a group of crowd workers. For instance, in Chorus, a group of crowd workers propose and select (by voting) the best response to answer users question in minutes; in Guardian, crowd workers extract information from a running dialog to assist web API perform a query; in InstructableCrowd, the crowd workers create executable rules based on users verbal instruction.
Humans will always be slower than machines; however, humans are more robust. Today's dialog systems often waste user’s time with errors or when a question is out of scope. We will evaluate our system by using traditional evaluation metrics in the field of dialog systems, e.g., task completion rate, task completion time, number of conversational turn, etc.
For the field deployment study, we will deploy our crowd-powered conversational agent as a web-based application that can be accessed via users' desktops, laptops, and mobile phones. Users will be asked to 1) install the agent on their own devices, 2) bring the agent around with them, and 3) use the agent to solve the real-world problems they encounter (e.g., find restaurants, check weathers, etc.) whenever they think the agent can help.
At the beginning of the study, we will first explain the purposes, requirements, and constraints of the study to you. A tutorial document will be provided to teach you how to use our system via your own devices. If you agreed to participate in this study, we will have you sign the on-line consent form. (You don’t need to physically come to our lab to sign the consent form.)
During the course of the study, one or more experimenter(s) will be available, in person or remotely, to answer your questions and collect data.
A pre-test and post-test survey might be performed to collect your opinions.
[Location] You can use the agent anywhere you wish. You don’t need to physically come to our lab to sign the consent form.
[Time] The recruitment along with survey and consent should take up to 30 minutes. You can use your own devices and interact with the agent as long or as short as you wish. This study will take up to one year. If you wish to remain in the study, they must not uninstall the agent for the first two weeks. The participants who uninstall the agent within the first two weeks will be excluded in our study.
Your data and consent form will be kept separate. Your research data will be stored in a secure location on Carnegie Mellon property. Sharing of data with other researchers will only be done in such a manner that you will not be identified. By participating, you understand and agree that the data and information gathered during this study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to others outside of Carnegie Mellon. However, your name, address, contact information and other direct personal identifiers will not be mentioned in any such publication or dissemination of the research data and/or results by Carnegie Mellon. Note that per regulation all research data must be kept for a minimum of 3 years.
The researchers will take the following steps to protect participants’ identities during this study: (1) Each participant will be assigned a number; (2) The researchers will record any data collected during the study by number, not by name; (3) Any original recordings or data files will be stored in a secured location accessed only by authorized researchers.
If you have questions pertaining to your rights as a research participant; or to report concerns to this study, you should contact the Office of Research Integrity and Compliance at Carnegie Mellon University. Email: firstname.lastname@example.org . Phone: 412-268-1901 or 412-268-5460.