1 of 5

​

​

  • Non-exploratory search intents: Dialogue strategies in API usage (Eberhart 2021) & speech act detection in bug repair (Wood 2018 FSE).
  • Closed-domain database tools: GitterAns (Romero 2020) technical question detection & Chatbot4QR (Zhang 2020) interactive refinement.

Wizard of Oz Study Methodology

Related Work

Investigating User Perceptions of Conversational Agents for Software-related Exploratory Web Search

Matthew Frazier matthew@udel.edu Lori Pollock pollock@udel.edu

Shaayal Kumar shaayal@udel.edu Kostadin Damevski kdamevski@vcu.edu

Participants

  • 18 diverse college students
  • Complete post-study survey

​

Agents

  • BlenderBot 2.0
  • Wizard interface in Slack with �canned responses

​

Tasks

  • 60 minute sessions
  • 2 tasks (same intent) - 30 mins/task
  • Learn: Comparison, �Knowledge Acquisition
  • Investigate: Evaluation, Planning/Forecasting

Computer and Information Science

Usefulness

Evaluation Task Example

​

“Assume you are an Operations Engineer tasked with optimizing your software deployment process. Write a short paragraph evaluation of three potential continuous integration and continuous development (CI/CD) solutions.”

Interaction

Post-Study Survey

Conclusions

  • Users of wizard find the agent helpful in gaining useful information from exploratory search for software engineering tasks.
  • Large gap between the current ODCSAs and what users want in such conversational agents for software engineering exploratory search.

We acknowledge the funding support of the NSF program. This paper was published at 44th IEEE International Conference on Software Engineering (ICSE 2022) in the New Ideas and Emerging Results (NIER) Track. Authors on this paper are: Matthew Frazier, Shaayal Kumar, Kostadin Damevski, and Lori Pollock.

Contributions

  • Corpus of 18 Wizard of Oz exploratory search dialogues for software engineers.
  • Post-study survey results, including programmer’s comments, ratings of the simulated virtual assistant, and performance on the task sets.

RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

​

​

​

​

​

​

​

Results

1. Interaction

2. Usefulness

18/18 👍 Understand agent

17/18 👍 Useful information

17/18 👍 Understand me

17/18 👍 Agent helped me

06/18 👍 Awkward

14/18 👍 Use in the future

01/18 👍 Many questions

​

1. Interaction

2. Usefulness

15/18 👍 Understand agent

06/18 👍 Useful information

08/18 👍 Understand me

05/18 👍 Agent helped me

05/18 👍 Awkward

04/18 👍 Use in the future

04/18 👍 Many questions

​

  • Web search is common for software engineers during development: ~45 mins/day, 15% of their workday (Xia 2017).
  • Effective web search for developers is essential for developer productivity.
  • A growing number of difficult web search queries are reformulated into conversation starting questions (Liu 2012).
  • Natural interfaces for exploratory search may be conversations with a conversational agent (Zhang 2018, Kiesel 2018).

Motivation & Background

Problem & Research Questions

Open-Domain Conversational Search Agent (ODCSA)

BlenderBot 2.0 Model Architecture

Conversation Excerpts

Wizard

BlenderBot 2.0

Evaluation Task Recommendation Request

PAPER

RPL

Existing developer web search tools are effective at lookup tasks but are ineffective for exploratory tasks which result in query reformulation.

​

  • RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?
  • RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46.

2 of 5

​

​

  • Non-exploratory search intents: Dialogue strategies in API usage (Eberhart 2021) & speech act detection in bug repair (Wood 2018 FSE)
  • Closed-domain database tools: GitterAns (Romero 2020) technical question detection & Chatbot4QR (Zhang 2020) interactive refinement

Wizard of Oz Study Methodology

Related Work

Investigating User Perceptions of Conversational Agents for Software-related Exploratory Web Search

Matthew Frazier matthew@udel.edu Lori Pollock pollock@udel.edu

Shaayal Kumar shaayal@udel.edu Kostadin Damevski kdamevski@vcu.edu

Participants

  • 18 diverse college students
  • Complete post-study survey

​

Agents

  • BlenderBot 2.0
  • Wizard interface in Slack with �canned responses

​

Tasks

  • 60 minute sessions
  • 2 tasks (same intent) - 30 mins/task
  • Learn: Comparison, �Knowledge Acquisition
  • Investigate: Evaluation, Planning/Forecasting

Computer and Information Science

Evaluation Task Example

​

“Assume you are an Operations Engineer tasked with optimizing your software deployment process. Write a short paragraph evaluation of three potential continuous integration and continuous development (CI/CD) solutions.”

Conclusions

  • Users of wizard find the agent helpful in gaining useful information from exploratory search for software engineering tasks
  • Large gap between the current ODCSAs and what users want in such conversational agents for software engineering exploratory search

We acknowledge the funding support of the NSF program. This paper was published at 44th IEEE International Conference on Software Engineering (ICSE 2022) in the New Ideas and Emerging Results (NIER) Track. Authors on this paper are: Matthew Frazier, Shaayal Kumar, Kostadin Damevski, and Lori Pollock.

Contributions

  • Corpus of 18 Wizard of Oz exploratory search dialogues for software engineers.
  • Post-study survey results, including programmer’s comments, ratings of the simulated virtual assistant, and performance on the task sets.

RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

​

​

​

​

​

​

​

Results

1. Interaction

2. Usefulness

18/18 👍 Understand agent

17/18 👍 Useful information

17/18 👍 Understand me

17/18 👍 Agent helped me

06/18 👍 Awkward

14/18 👍 Use in the future

01/18 👍 Many questions

​

1. Interaction

2. Usefulness

15/18 👍 Understand agent

06/18 👍 Useful information

08/18 👍 Understand me

05/18 👍 Agent helped me

05/18 👍 Awkward

04/18 👍 Use in the future

04/18 👍 Many questions

​

  • Web search is common for software engineers during development: ~45 mins/day, 15% of their workday (Xia 2017).
  • Effective web search for developers is essential for developer productivity.
  • A growing number of difficult web search queries are reformulated into conversation starting questions (Liu 2012).
  • Natural interfaces for exploratory search may be conversations with a conversational agent (Zhang 2018, Kiesel 2018).

Motivation & Background

Problem & Research Questions

Open-Domain Conversational Search Agent (ODCSA)

BlenderBot 2.0 Model Architecture

Conversation Excerpts

Wizard

BlenderBot 2.0

Evaluation Search Task

PAPER

RPL

Existing developer web search tools are effective at lookup web search tasks but are ineffective for exploratory web search tasks (e.g. learn, investigate) which result in query reformulation.

​

  • RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?
  • RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

Usefulness

Interaction

Post-Study Survey

3 of 5

​

​

  • Non-exploratory search intents: Dialogue strategies in API usage (Eberhart 2021) & speech act detection in bug repair (Wood 2018 FSE)
  • Closed-domain database tools: GitterAns (Romero 2020) technical question detection & Chatbot4QR (Zhang 2020) interactive refinement

Wizard of Oz Study Methodology

  • Web search is common for software engineers during development: on average ~45 mins/day, 15% of their workday (Xia 2017).
  • Existing web search tools solve lookup tasks (e.g. information retrieval).
  • Web search for developers completing exploratory tasks (e.g. learn and investigate) result in query reformulation.

Problem & Motivation

Related Work

Investigating User Perceptions of Conversational Agents for Software-related Exploratory Web Search

Matthew Frazier matthew@udel.edu Lori Pollock pollock@udel.edu

Shaayal Kumar shaayal@udel.edu Kostadin Damevski kdamevski@vcu.edu

Participants

  • 18 diverse college students
  • Complete post-study survey

​

Agents

  • BlenderBot 2.0
  • Wizard interface in Slack with canned responses

​

Tasks

  • 60 minute sessions
  • 2 tasks (same intent) - 30 mins/task
  • Learn: Comparison, �Knowledge Acquisition
  • Investigate: Evaluation, Planning/Forecasting

Computer and Information Science

Usefulness

Evaluation Task Example

​

“Assume you are an Operations Engineer tasked with optimizing your software deployment process. Write a short paragraph evaluation of three potential continuous integration and continuous development (CI/CD) solutions.”

Open-Domain Conversation Search Agent (ODCSA)

BlenderBot 2.0 Architecture

Interaction

Post-Study Survey

Conclusions

  • Users of wizard find the agent helpful in gaining useful information from exploratory search for software engineering tasks
  • Large gap between the current ODCSAs and what users want in such conversational agents for software engineering exploratory search

​

Conversation Excerpts

We acknowledge the funding support of the NSF program. This paper was published at 44th IEEE International Conference on Software Engineering (ICSE 2022) in the New Ideas and Emerging Results (NIER) Track. Authors on this paper are: Matthew Frazier, Shaayal Kumar, Kostadin Damevski, and Lori Pollock.

Contributions

  • Corpus of 18 Wizard of Oz exploratory search dialogues for software engineers.
  • Post-study survey results, including programmer’s comments, ratings of the simulated virtual assistant, and performance on the task sets.

Example 1

Example 2

Wizard Comparison Search Task

RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

​

​

​

​

​

​

​

Results

1. Interaction

2. Usefulness

18/18 👍 Understand agent

17/18 👍 Useful information

17/18 👍 Understand me

17/18 👍 Agent helped me

06/18 👍 Awkward

14/18 👍 Use in the future

01/18 👍 Many questions

​

1. Interaction

2. Usefulness

15/18 👍 Understand agent

06/18 👍 Useful information

08/18 👍 Understand me

05/18 👍 Agent helped me

05/18 👍 Awkward

04/18 👍 Use in the future

04/18 👍 Many questions

​

4 of 5

Future Work

  • Customizing existing agents for software engineers towards improving the usefulness of agent responses
  • Reducing ODCSA response time
  • Examining the ODCSA’s long-term memory

Contributions

  • Corpus of 18 Wizard of Oz exploratory search dialogues for software engineers.
  • Post-study survey results, including programmer’s comments, ratings of the simulated virtual assistant, and performance on the task sets.

RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?

​

  1. Interaction

18/18 👍 Understand agent

17/18 👍 Understand me

06/18 👍 Behaved awkwardly

01/18 👍 Too many questions

​

  • Usefulness

17/18 👍 Useful information

17/18 👍 Agent helped me

14/18 👍 Use in the future

​

​

​

​

​

​

​

​

​

RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

​

  1. Interaction

15/18 👍 Understand agent

08/18 👍 Understand me

05/18 👍 Behaved awkwardly

02/18 👍 Too many questions

​

  • Usefulness

06/18 👍 Useful information

05/18 👍 Agent helped me

04/18 👍 Use in the future

​

​

​

​

​

​

Results

Research Questions

  • RQ1: Can open-domain conversational agents help software engineers be effective in performing exploratory search?
  • RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

Problem Statement

Effective web search for developers is essential for developer productivity. Web search for developers completing exploratory tasks (e.g. learn and investigate) result in query reformulation. Existing web search tools solve lookup tasks (e.g. information retrieval).

​

Can open-domain conversational search agents aid developers to effective perform exploratory search?

Conversation Excerpts

Example 1

Example 2

Wizard Comparison Search Task

Effective web search for developers is essential for developer productivity. Developer tools completing lookup tasks in web search retrieve effective information easily. For completing exploratory tasks (e.g. learn and investigate), web search results in query reformulation.

​

Can open-domain conversational search agents aid developers in performing effective exploratory search ?

Problem Statement

  • Web search is common for software engineers during development: on average ~45 mins/day, 15% of their workday (Xia 2017).
  • Web search for developers completing exploratory tasks (e.g. learn and investigate) result in query reformulation.
  • A growing number of difficult web search queries are reformulated into conversation starting questions.
  • Natural interfaces for exploratory search may be conversations with a conversational agent.

Motivation & Background

  • Web search is common for software engineers during development: on average ~45 mins/day, 15% of their workday (Xia 2017).
  • Existing web search tools solve lookup tasks (e.g. information retrieval).
  • Web search for developers completing exploratory tasks (e.g. learn and investigate) result in query reformulation.

Motivation

5 of 5

Open-Domain Conversation Search Agent (ODCSA)

BlenderBot 2.0 Architecture

BlenderBot 2.0 Model Architecture

Evaluation Task Convo. Excerpts

Wizard

Open-Domain Conversational Agent

BlenderBot 2.0 Model Architecture

Open-Domain Convo. Search Agent

BlenderBot 2.0

Developer web search tools completing lookup tasks (e.g. navigation, fact retrieval) retrieve effective information. However, these tools when given exploratory web search tasks (e.g. learn, investigate) result in query reformulation.

​

Can conversational search agents aid software developers to perform effective exploratory search (e.g. reduce query reformulation)?

Problem Statement

Can conversational search agents aid software developers to perform effective exploratory search (e.g. reduce query reformulation)?

​

  • RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?
  • RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

Problem & Research Questions

Developer web search tools are effective at IR tasks.These same tools when given exploratory (e.g. learn, investigate) web search tasks are ineffective and result in query reformulation.

​

  • RQ1: Can ODCSAs help software engineers be effective in performing exploratory search?
  • RQ2: How far does the current generation of ODCSAs go in helping software developers with exploratory search?

Problem & Research Questions