1 of 20

Investigating User Perceptions of Conversational Agents for Software-related Exploratory Web Search

​

Matthew Frazier, Shaayal Kumar, Kostadin Damevski, Lori Pollock

5/11/22

2 of 20

3 of 20

Lookup Vs. Exploratory

Categories of Tasks

  1. Lookup
    • Debugging
    • Syntax
    • Documentation
  2. Exploratory
    • Technology comparisons
    • Choosing a library or framework

[2]

4 of 20

Developers are often unsuccessful in quickly finding useful information for their exploratory needs

5 of 20

Related Work

This research presents the first empirical study for software engineers utilizing a CSA for performing exploratory search. 

​

Nevertheless, the most similar research to ours is:

  • Supporting WoZ studies [1, 5, 6] 
  • Prototype chatbots in software engineering [7, 4, 8]

6 of 20

WoZ Study

7 of 20

WoZ Study

ICSE ’22 NIER Track

Determine efficacy of ODCSA for SE exploratory search:

​

Research Questions

  • RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?
  • RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

8 of 20

WoZ Study

Experimental Setup

  • Agents
    • Commercial ODCSA is BlenderBot2.0
    • Wizard is Slack with canned responses and Internet search
  • Recruited 18 diverse college programmers (ethnicity, gender, 18+ age, etc.)
  • Each session = 60 minutes (2 tasks, 30 mins per task)
    • Both tasks are of the same exploratory search intent
  • Post-study survey after using of each agent

9 of 20

WoZ Study

Post-Study Survey

  • Likert Scale
    • Interaction (Q1 - Q4)
    • Usefulness (Q5 - Q7)
  • Open-ended
  • Demographic data

10 of 20

Results

11 of 20

WoZ Study

Post-Study Survey Analysis (Interaction)

RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?

18/18 👍 “Understand agent”

17/18 👍 “Understand me”

06/18 👍 “Behaved awkwardly”

16/18 👍 “Not too many questions”

12 of 20

WoZ Study

Post-Study Survey Analysis (Usefulness)

RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?

17/18 👍 “Useful information”

17/18 👍 “Agent helped me"

14/18 👍 “Use in the future”

13 of 20

Users of an ODCSA backed by a wizard find the agent helpful in gaining useful information from exploratory search for SE tasks

14 of 20

WoZ Study

Post-Study Survey Analysis (Interaction)

RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

04/18 👍 “Understand me”

15/18 👍 “Understand agent”

05/18 👍 “Behaved awkwardly”

02/18 👍 “Not too many questions”

15 of 20

WoZ Study

Post-Study Survey Analysis (Usefulness)

RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

06/18 👍 “Useful information”

05/18 👍 “Agent helped me”

04/18 👍 “Use in future”

16 of 20

Large gap between the current generation of ODCSAs and what participants want in such conversational agents for SE exploratory search

17 of 20

Future Work

18 of 20

Extended Analysis

IST’22/JSS’22

  • Collect more data
  • Unanalyzed data
    • Search Queries
    • Task Outputs
    • Conversation Data

19 of 20

Questions?

20 of 20

  1. Zachary Eberhart, Aakash Bansal, and Collin Mcmillan. 2020. A Wizard of Oz Study Simulating API Usage Dialogues with a Virtual Assistant. IEEE Transactions on Software Engineering (2020), 1–1. https://doi.org/10.1109/TSE.2020.3040935
  2. Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46.
  3. Nikitha Rao, Chetan Bansal, Thomas Zimmermann, Ahmed Hassan Awadallah, and Nachiappan Nagappan. 2020. Analyzing Web Search Behavior for Software Engineering Tasks. In IEEE International Conference on Big Data, Big Data 2020, Atlanta, GA, USA, December 10-13, 2020 , Xintao Wu, Chris Jermaine, Li Xiong 0001, Xiaohua Hu 0001, Olivera Kotevska, Siyuan Lu, Weija Xu, Srinivas Aluru, ChengXiang Zhai, Eyhab Al-Masri, Zhiyuan Chen 0003, and Jeff Saltz 0001 (Eds.). IEEE, 768–777. https://doi.org/10.1109/BigData50022.2020.9378083
  4. Ricardo Romero, Esteban Parra, and Sonia Haiduc. 2020. Experiences Building an Answer Bot for Gitter. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea) (IC-SEW’20). Association for Computing Machinery, New York, NY, USA, 66–70. https://doi.org/10.1145/3387940.3391505
  5. Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles L. A. Clarke. 2017. Exploring Conversational Search With Humans, Assistants, and Wizards. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 2187–2193.
  6. Andrew Wood, Paige Rodeghero, Ameer Armaly, and Collin McMillan. 2018. Detecting Speech Act Types in Developer Question/Answer Conversations during Bug Repair. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 491–502. https://doi.org/10.1145/3236024.3236031
  7. Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo. 2017. AnswerBot: Automated generation of answer summary to developers’ technical questions. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 706–716. https://doi.org/10.1109/ASE.2017.8115681
  8. N. Zhang, Q. Huang, X. Xia, Y. Zou, D. Lo, and Z. Xing. 2020. Chatbot4QR: Interactive Query Refinement for Technical Question Retrieval. IEEE Transactions on Software Engineering (2020), 1–1. https://doi.org/10.1109/TSE.2020.3016006