1 of 108

Conversational Agents for�Automated Group Meeting Facilitation�f�A Computational Framework for Facilitating Small Group Decision-Making Meetings

PhD Thesis

Ameneh Shamekhi

Committee:

Timothy Bickmore

Stacy Marsella

Lu Wang

Vera Liao, IBM Research

Relational Agents Group

2 of 108

2

3 of 108

Dissertation Outline

Motivation
Related Work
Automated Group Facilitation Framework
Prototype 1 – Embodiment and Feasibility
Prototype 2 – Automation
Prototype 3 – Disagreement Management
Discussion
Conclusion

3

4 of 108

Motivation�

Why do we need group meeting facilitation?

How can technology help?

4

5 of 108

~30% of employees time and >$1.4 trillion per year in U.S. are spent on meetings

5

~40% of meeting time is wasted costing >$100 million per year in U.S.

https://www.themuse.com/advice

https://blog.lucidmeetings.com/blog

… We all attend many meetings every week. Group meetings either online or co-located are the primary means of communication and collaboration In many organizations, and millions of meeting are held every day.

Resulting in on average a third of employees’ time is spent in meetings.

*Which is all fine, until we realize that despite the time and money spent in meeting, employees and managers do always not find them helpful.

Surveys showed that many of these meetings are not productive. They are considered to be costly, too long, and dissatisfying. And It happens very often that employees are not satisfied with the meeting and feel their time was wasted.

Meeting Analysis: Many reviews and surveys [1, 2, 3, 15, 21, 52, 53, 57, 75, 82] reveal that meetings dominate workers’ and managers’ time and yet are considered to be costly, unproductive and dissatisfying. Studies [13, 52, 53, 56, 57] show that meetings are essential and that the number of meetings and their duration has been steadily increasing. Studies of Managers and knowledge workers [1, 13, 21, 51, 52, 53, 68, 75, 79, 89, 90, 92] reveal that they spend between 25%-80% of their time in meetings, suggesting that meetings are an important part of one’s working life. Estimates of meeting expenses [1, 13, 39, 53] range from costs of $30 million to over 100 million per year to losses between $54 million and 3.7 billion annually!

6 of 108

Small Group Meeting Challenges

Getting off the subject (Digressions)
No goal or Agenda
Hidden Agenda
Distraction
Domination
Waiting to speak
Fear of speaking
Conflict
Not Engaging

6

[Nunamaker Jr, et al. ,1996, Lessons from a Dozen Years of Group Support Systems Research]

6

The meeting quality, group performance, and outcomes are challenged by a variety of dysfunctional behaviors, unproductive social dynamics, and lack of experience in conducting efficient and productive meetings.

I’m sure you all experienced similar situations, that Even In a small group of people with different personalities and backgrounds it often happens that some people tend to speak more and dominate the meeting … not establishing or diverging from a meeting agenda, intragroup conflict, and domination of the discussion by one participant. may cause miscommunications about agenda and criteria. Conflict may happen and so on …Missing information and poorly defined goals

And as the result it happens often that we leave a meeting unproductive and without getting what we wanted from the meeting

7 of 108

7

Enforcing meeting agenda [Viller, 1991]

Eliciting equal participation [Westley and Waters, 1988]

Managing conflicts [Viller, 1991]

Professional Meeting Facilitators

8 of 108

Barriers of Having a Human Facilitator

Training and hiring human facilitators are costly.

Facilitating a group decision making often requires “standing apart”.

A small group may not wish to lose the contributions of a human member just so he/she can facilitate.

8

9 of 108

Automated Group Meeting Facilitation

~30% of employees time and >$1.4 trillion per year in U.S. are spent on meetings
~40% of meeting time is wasted costing >$70 billion per year in U.S.

9

https://www.themuse.com/advice

https://blog.lucidmeetings.com/blog

Meetings

Group FACILITATOR

AUTOMATED Group Facilitation

10 of 108

Virtual Facilitator is an Embodied Conversational Agent (ECA):

Uses speech
Has human-like personification
Has nonverbal behavior & facial expressions
Provides authoritative guidance
Holds neutral point-of-view

10

Automated Group Meeting Facilitation

In my thesis project I suggest to use an embodied conversational agent as a virtual facilitator that attempts to assist the group.

I hypothesized that several capabilities of an embodied conversational agent with humanoid look, such as the ability to interact with people in natural language and displaying nonverbal behaviors, the ability to hold a non-judgmental and neutral point-of-view, and access to information resources, can make a virtual Facilitator an effective meeting facilitator

In order to provide the most natural interaction with group members, I will use an Embodied Conversational Agent (ECA).

ECAs are —virtual or robotic— computer systems designed to interact with users through natural conversations. and have a range of nonverbal conversational behavior, such as gaze and facial display, and has the potential to provide authoritative guidance during decision-making meetings.

11 of 108

Group Decision-Making

11

Problem Solving

Information Sharing

Team Building

Group Meetings

Etc.

Group Decision Making

12 of 108

Example: Hiring Session

Hiring Decision Meeting

2-3 hiring managers reviews 5-6 resumes of candidates
Decision-making process adapted from the ‘Nominal Group Technique’

12

Resumes

13 of 108

Challenges in a Group Decision-Making Session

Lack of Structure

Imbalanced Participation

Intragroup Conflict

13

[Romano, N. & Nunamaker, J. , 2001, Meeting analysis: Findings from research and practice]

14 of 108

Group Facilitation to Help with the Challenges

Structure Management

Keeping the group focused on the objectives
Leading and managing meetings [Ho 2000]
Using the agenda to guide the group [Ho 2000]
Reinforcing meeting agenda and rules [Romano and Nunamaker, 2001]

Participation Management

Ensuring equal member participation [Ho 2000]

Conflict Management

Creating a safe and stable environment for open discussion
Making sure everybody’s voice is head and understood

14

A

B

15 of 108

Related Work�

15

16 of 108

Group Decision Support Systems (GDSS)

16

Interactive, computer-based system
Face-to-face and distance participation
Usually used with a human facilitator

[DeSanctics and Gallupe , 1987, A foundation for the study of group decision support systems.]

GDSS

Facilitator

[Anson R., 1995, An Experiment Assessing Group Support System and Facilitator Effects on Meeting Outcomes]

17 of 108

Technology Driven Group Facilitation

Smart Meeting Rooms

17

[Tennent, H. et al. 2019. Micbot]

[Kim, T., et al. 2008. Meeting Mediator]

[Bhattacharya, I. et al. 2018. A Multimodal-Sensor-Enabled Room]

18 of 108

Automated Detection of Decisions and Conflicts

Decision Detection

Learning about meetings [Kim and Rudin,2024]

Feature vector: shifted bag- of-dialogue-acts
SVM-based model to predict key decision times with F-measure >0.84

Conflict Detection

Verbal (lexical, prosodic and structural) features [Germesin and Wilson2009]
Non-verbal behavior [Bousmalis et al. 2011]

18

19 of 108

Embodied Conversational Agents In Groups

19

[Bohus, D. and Horvitz, E. 2011. Multiparty Turn Taking in Situated Dialog]

Multi Party Interaction

20 of 108

Embodied Conversational Agents In Groups

20

[Bohus, D. and Horvitz, E. 2011. Multiparty Turn Taking in Situated Dialog]

[Matsuyama, Y. et al. 2015. A facilitation robot controlling engagement]

Multi Party Interaction

Participation Facilitation

21 of 108

Embodied Conversational Agents In Groups

21

[Bohus, D. and Horvitz, E. 2011. Multiparty Turn Taking in Situated Dialog]

[Matsuyama, Y. et al. 2015. A facilitation robot controlling engagement]

[Jung, M.F. et al. 2015. Using Robots to Moderate Team Conflict]

Multi Party Interaction

Participation Facilitation

Conflict Facilitation

or even how a robot can moderate team conflicts in a problem solving task.

Through a Wizard-of-Oz st

And lastly in this work Malte and his colleagues designed a robot to manage conflicts in a team-based problem solving scenario, and studied the effect of a robot intervention on affect, perceptions of conflict, perceptions of team members’ contributions, and team performance during a problem-solving task. They and compared different management strategies for the robot.

Contrary to our expectations, we found that the robot’s repair interventions increased the groups’ awareness of conflict after the occurrence of a personal attack thereby acting against the groups’ tendency to suppress the conflict.

robots can aid team functioning by regulating core team processes such as conflict

In a 2 (negative trigger: task- directed vs. personal attack) x 2 (repair: yes vs. no)

udy in which a robot attempts to repair interpersonal violations in 3-member teams

22 of 108

Group performance is impacted by several challenges.
Group facilitation can improve group performance.

Technology can improve meeting performance.
Embodied Conversational Agents have been used in multiparty interactions.

Can an Embodied Conversational Agent guide and facilitate a group meeting to improve meeting performance?

22

?

Context

Existing Approaches

My Research

23 of 108

Framework�

23

24 of 108

Automated Group Facilitation System

24

Automated Group Facilitation System

is designed to support and facilitate a group decision-making session

is guided by a Virtual Facilitator that is an embodied conversational agent

25 of 108

Virtual Facilitator provides three types of facilitation

25

Greeting and Introduction
Set the stage
Provide “ice-breakers”
Build trust and rapport
Discuss desired meeting outcomes
Display understanding, and empathy
Display attentive listening

Social Facilitation

Meeting Facilitation

Decision-Making Facilitation

Inspired by the roles defined for a human facilitator in the management science literature, I defined three categories of facilitation for my virtual facilitator.

Social Facilitation. One of the most important tasks of a group facilitator is to catalyze the social interaction among members of a workgroup [133]. Social facilitation tasks of a group facilitator include: greeting everyone and introducing herself and the members of the group; “setting the stage” by orienting group members to the task and process; providing “ice-breakers” to acquaint group members with each other and begin building trust and rapport; discussing desired meeting outcomes; and reviewing the agenda. For group members to accept the robot in this social role, it needs to be able to engage them in natural, multimodal, face-to-face conversation. In order to build rapport and trust as a social facilitator, it needs to display understanding and empathy using conversational backchannels [11] and “active listening” behavior [97].

26 of 108

Virtual Facilitator provides three types of facilitation

26

Greeting and Introduction
Set the stage
Provide “ice-breakers”
Build trust and rapport
Discuss desired meeting outcomes
Display understanding, and empathy
Display attentive listening

Review the agenda
Enforce meeting structure
Encourage and Balance Participation
Content-based Feedback
Manage the Time
Maintain Authority

Social Facilitation

Meeting Facilitation

Decision-Making Facilitation

In order to alleviate some of the common challenges of group decision-making meetings, such as inefficient and unstructured discussion and imbalanced participation, the AGF system should provide some meeting facilitation

Enforce meeting structure: To avoid off-topic discussions and deviation from the meeting agenda, the system should be able to enforce the meeting structure by offering step-by-step instructions and monitoring the decision-making process. The system needs to know the context and the meeting stage to offer timely feedback, and guide the meeting flow by asking targeted questions.

Encourage and Balance Participation: Dealing with people who tend to dominate the conversation in a meeting is an important function of a group facilitator [108]. To ensure balanced participation, and to create a safe space for all participants, the system needs to detect when some participants are significantly less vocal during a meeting and engages them in the conversation.

Content-based Feedback: A group facilitator should also provide feedback related to the task being discussed by the group. Hence the AGF system needs to recognize some speech content to provide interventions based on the topic of participants’ discussion. For example, the system should track aspects of the task that need to be discussed and remind the group if they forget to discuss them.

Manage the Time: Group facilitators keep meetings on track by watching the time and agenda. The AGF system should also track the time spent on each task in the meeting to both remind team members of the time, if needed, and offer comments based on the time and the stage of the meeting.

Maintain Authority: A facilitator should be perceived as an authorized and intelligent member to enforce participants attendance and adherence to her instructions. As suggested by Justine Cassell, the computer agent’s embodiment helps users to better locate its intelligence and power and improves the perceived trust [26]. Using a conversational robot with a human-like face and nonverbal behaviors such as gaze and facial expression as the group facilitator, allows participants to treat it as an intelligent and authorized social entity with a persona.

27 of 108

Virtual Facilitator provides three types of facilitation

27

Greeting and Introduction
Set the stage
Provide “ice-breakers”
Build trust and rapport
Discuss desired meeting outcomes
Display understanding, and empathy
Display attentive listening

Informational Support
Identify Disagreements
Manage Disagreements

Review the agenda
Enforce meeting structure:
Encourage and Balance Participation
Content-based Feedback:
Manage the Time:
Maintain Authority

Social Facilitation

Meeting Facilitation

Decision-Making Facilitation

28 of 108

Essential Processes to Support Automated Group Facilitation

28

Acceptance of the Virtual Facilitator as a Group Member

Simulate social behaviors
Interact in a natural conversation
Perceived as an intelligent social entity
Perceived as an authorized and trustworthy entity

Multi-Party Conversation

29 of 108

Essential Processes to Support Automated Group Facilitation

29

Acceptance of the Virtual Facilitator as a Group Member

Simulate social behaviors
Interact in a natural conversation
Perceived as an intelligent social entity
Perceived as an authorized and trustworthy entity

Multi-Party Conversation

Understanding the inputs
Show non-verbal behaviors
Turn-taking management

Another important feature of the group decision-making setting is the involvement of multiple humans in the interaction, so a VF should be able to support multi-party conversations. The agent needs to receive and makes sense of verbal and non-verbal inputs from multiple sources, process them to provide the right response.

The robot also needs minimal conversational turn-taking management capabilities, tracking which participant is speaking and indicating attentive behavior by gazing at the speaker, as well as indicating her intention for a specific user to speak by gazing toward them.

In a multi party interaction having non-verbal behavior and facial expression also helps the virtual facilitator in managing turn-taking behaviors. For example, she should know which participant is speaking, and show that she is listening. A VF should indicate her intention to talk and gazes toward each participant when s/he is expected to talk.

30 of 108

Automated Group Facilitation Framework

Layer 1

Layer 2

Layer 3

Audio Data

Meeting Capture

Meeting Recognition

Semantic Process

Dialogue Manager

Visual Data

Rating application

Verbal & Nonverbal Behavior

Time

30

31 of 108

Automated Group Facilitation Framework

Layer 1

Layer 2

Layer 3

Audio Data

Meeting Capture

Meeting Recognition

Semantic Process

Dialogue Manager

Visual Data

Rating application

Verbal & Nonverbal Behavior

Time

31

ASR

Gaze Direction Detection

Participation Detection

Keyword Detection

Discussion Topic Detection

Decision Status

Detection

Disagreement Detection

32 of 108

Automated Group Facilitation Framework

Layer 1

Layer 2

Layer 3

ASR

Gaze Direction Detection

Participation Detection

Keyword Detection

Discussion Topic Detection

Participation Management

Content-Based Recommendation

Audio Data

Meeting Capture

Meeting Recognition

Semantic Process

Dialogue Manager

Turn-Taking Management

Visual Data

Rating application

Rule-Based Temporal Guidance

Decision Status

Detection

Verbal & Nonverbal Behavior

Time

32

Disagreement Detection

Active Listening Behavior and Social Facilitation

Disagreement Management

33 of 108

Research Questions

RQ0- Will members of a face-to-face decision-making meeting accept a CA in the role of a group facilitator?
RQ1- What is the appropriate embodiment for a virtual facilitator in a group setting?
RQ2- To what extent do the members of a group follow the virtual facilitator's instructions and recommendations in the context of group decision-making?
RQ3- How can a virtual facilitator impose and enforce a structure on a group decision-making activity, and ensure that all participants have an opportunity to be heard?
RQ4- Can a virtual facilitator improve conflict management in a group decision-making setting?
RQ5- What would information workers expect from a social agent at their workplace? 5.a) what role and features would they prefer for the robot to hold?

33

34 of 108

Dissertation Outline

Motivation
Related Work
Automated Group Facilitation Framework
Prototype 1 – Acceptance, Feasibility, and Embodiment
Prototype 2 – Automated Meeting Structure Management
Prototype 3 – Disagreement Management
Discussion
Conclusion

34

35 of 108

Dissertation Outline

Motivation
Related Work
Automated Group Facilitation Framework
Prototype 1 – Acceptance, Feasibility, and Embodiment
Prototype 2 – Automated Meeting Structure Management
Prototype 3 – Disagreement Management
Discussion
Conclusion

35

36 of 108

Prototype 1�

Feasibility Study

Exploring Embodiment of and the Social facilitation by the group agent

36

?

[Shamekhi et al., ‘Face Value? Exploring the Effects of Embodiment for a Group Facilitation Agent’, CHI 2018]

37 of 108

Experiment Procedure

37

CASSY; Facilitates a Hiring session

38 of 108

Facilitation Agent Functionalities

38

A

B

Decision-Making Facilitation

Meeting Facilitation

Social Facilitation

Greeting/Intro.

Agenda/Task Setting

Review /Initial Voting

Wrap up-summary

Farewell

Criteria discussion

Elimination

Selection

Final Voting/ deciding

Start

Discussion

Wrap up

End

CASSY initiates each session with greeting and introduction, explain the task and the agenda, and then suggests that the group discuss the hiring criteria. Then she invites the group to go through the candidates and eliminate the unqualified ones. When participants converge on an elimination, she confirms that decision. Finally She summarizes the session, and thanks the group for participating.

And throughout theses steps CASSY has

Social Interaction: The agent exhibits active listening behaviors to express interest and social support. Examples

include: greetings, Ice breaking, short verbal phrases to reflect understanding and support, and facial Expression and gaze.

She also provides meeting facilitation, She enforces a structure and guides the group through the pre-defined six steps. The agent attempts to manage the time and manage participation by trying to engage the less active participant to improve meeting efficiency. managing time, agenda description, turn taking management.

CASSY implicitly supports the decision making:. by tracking decision states, suggesting actions and opinion exchange, summarizing decisions.

39 of 108

User Study 1

Feasibility Study
Effects of embodiment (voice-only vs. embodied)
Preliminary implementation of the framework
Wizard-of-Oz study
Between-subject study
N = 40 (20 user groups)

60% male

39

vs.

40 of 108

Embodiment improved the social perception of the agent

40

41 of 108

41

Embodiment affects in the group setting:

Subjective measures:

social perception of the agent

Objective measures

Group behavior

More equal contribution

More proactive interaction

Decision outcome

42 of 108

Qualitative Results

Virtual Facilitator was well accepted by the groups
Participants found it very helpful and engaging
Embodiment provides additional benefits

Social perception
Locating social intelligence
Perception of task capabilities
Enhanced presence specially good for certain tasks
Engagement

42

I also analyzed the interviews transcripts. And identified themes related to the perception of the agent, the facilitation functions and agent’s embodiment.

Overall Virtual Facilitator was well accepted by the groups

Participants found it very helpful and engaging

About the embodiment.

Consistent with the literature, many of our participants discussed how embodiment helped them to locate attention and emotions, for example to indicate her intention to talk.

TASK CAPABILITIES: Several groups also mentioned human-like face made them feel she better understands them, and she is more capable of certain tasks like answering questions.

Participants commented that embodiment enhanced the agent’s presence, making it more of an active part of the group discussion instead of lurking in the background.

43 of 108

Feasibility Study Conclusion

43

RQ0- Will members of a face-to-face decision-making meeting accept a CA in the role of a group facilitator?

RQ1- What is the appropriate embodiment for a virtual facilitator in a group setting?

44 of 108

Prototype 2�

Automated Group Facilitation Robot

44

[Shamekhi et al., ‘A Multimodal Robot-Driven Meeting Facilitation System for Group Decision-Making Sessions ’, ICMI 2019]

45 of 108

SARAH; Fully Automated Group Facilitation System

45

46 of 108

System Architecture

46

47 of 108

Dialogue Management

47

1. Visual Input

2. Voice Input

3. Decision status

4. Time

Non-Verbal Behavior

Verbal Behavior

Actions

Robot Dialogue Manager

48 of 108

Visual Input from Kinect

48

Sense users

Presence
Enter
Exit
Attend

Identify user location
Estimate their gaze direction using head orientation
Number of users

49 of 108

Voice Input from Microphone

49

Converted to text using Google’s ASR.
The speech signal is used to detect:

Silence vs. speech intervals for each user
Who the current speaker is
User vocal participation from the accumulated speech time of each participant in each meeting stage.

50 of 108

Task status from the tablet application

Data from the application is stored on a Database:

User names
Options’ status
Number of discussed and eliminated options
When initial ranking is submitted
When the active option is updated
When the decision status on each option is updated (kept or eliminated)
When the best option is selected.

50

The third category of input is detailed information about the Decision status collected from a tablet application. I designed and developed a custom tablet application to support the group decision-making tasks. The application is used for individual and group activities during the group-decision making activity. It provides some of the basic features of group decision support systems such as voting and sharing the decision status. The application facilitates the communication and sharing of opinions by providing a discussion interface that reflects the group decision at each stage of the decision-making process. Participants use the tablet application to review and rank candidates individually, and to share ideas during the group discussion.

Sends signals to the dialogue manager on occurrence of each event. The dialogue manager obtains data such as the user names, candidate’s status, and the number of discussed and eliminated candidates from the application. Other input signals are received from the application at the following moments during the session: 1) when initial ranking is submitted during the individual review phase, 2) when the active candidate is updated during the group discussion, 3) when the decision status on each candidate is updated (kept or eliminated), 4) when the best candidate is selected.

51 of 108

Time

Initiate timing signals at several occasions during the interaction:

Reminders
Track the spent time on each stage

51

52 of 108

Dialogue Management

52

1. Visual Input

2. Voice Input

3. Decision status

4. Time

Non-Verbal Behavior

Verbal Behavior

Actions

Robot Dialogue Manager

53 of 108

53

Non-Verbal Behavior

Verbal Behavior

Actions

Attending to the speaker and active listening behavior
Directional gaze to manage the turn-taking in multiparty interaction
Directional gaze when the group members enter and when they are conversing
Smiles, Head nods, confirmation filler words (aha, I see, etc.)

Robot-Initiated Utterances
User Speech-Initiated Utterances
Tablet Application-Initiated Utterances
Time-based Utterances

Extend the time for decision-making task (upon user request)
Move forward to the next task
Update the tablet application interface
Calculate participants’ contribution time to nudge less active participants

NONVERBAL: The robot facilitator performs a range of human-like non-verbal behavior to show active listening behavior and to manage the turn-taking in multiparty interaction. communicative facial displays, smiles, nods her head, gaze orientation, and uses confirmation filler words (aha, I see, that makes sense, etc.).

VERBAL: In each stage of the meeting the robot’s has various verbal output that are (synthetic speech) either initiated proactively by the robot, or by user speech input, time, or inputs from the tablet application.

Besides the verbal and nonverbal behavior the system uses the input to take some Actions. for example to may extend the time allowed for a given decision-making task (upon user request), move forward to the next task, update the tablet application interface, or calculate participants’ contribution time to nudge less active participants.

54 of 108

Social & Meeting Facilitation Services

Social Facilitation
Enforce Meeting Structure
Ensure content coverage
Mange the time
Encourage and balance participation

54

A

B

55 of 108

Task Domain: Hiring Meeting

55

6 Resumes

56 of 108

User Study 2

56

VS.

Fully Automated:

Robot Facilitation + Tablet Application

Control:

Written instructions + Tablet Application

(11 groups, N = 22)

(9 groups, N = 18)

57 of 108

Participants Rated the Robot’s Social and Meeting Facilitation Positively

57

Strongly Disagree

Strongly Agree

Neutral

Meeting Facilitation

Social Facilitation

Knowledgeable

Powerful

Friendly& Warm

58 of 108

Results – Team and Decision Making Satisfaction

Team and decision-making satisfaction
5 items (α=.88)

E.g. The members of this team get along well together.
E.g. I found we reached a decision efficiently.

P=.06

58

Then I compared participants self reports on their team and decision making satisfaction. And I found that participants in the ROBOT condition were relatively more satisfied with their team and decision-making activity, compared to participants in the CONTROL condition.

Cohesion: To what extent do you agree with the following statements? - The members of this team get along well together. Satisfaction: To what extent do you agree with the following statements? - I am pleased with the way my partner and I participated in the decision-making task. Please� indicate to what extent you agree with the following statements about your team� and your partner? - I am very satisfied with working in this team. Please� indicate to what extent you agree with the following statements about your team� and your partner? - I felt my partner and I understood each other. Please� indicate to what extent you agree with the following statements about your� group decision-making meeting? - I found we reached a decision efficiently.

59 of 108

Meeting is More Structured with the Robot

59

60 of 108

Robot is Effective in Balancing the Participation

Balancing the Participation

60

Group discussions mostly are somewhat chaotic. [but] here we were given a fairly good chance to speak about each and every resume and to state what we like and what we do not like ...”

She gave equal opportunities to both of us to speak, that kind of resolved conflict itself because everyone feels their voice is being heard.”

;Mentioned by 90% of the groups

“

61 of 108

Study 2 Conclusion

RQ0- Will members of a face-to-face decision-making meeting accept a CA in the role of a group facilitator?
RQ1- What is the appropriate embodiment for a virtual facilitator in a group setting?
RQ2- To what extent do the members of a group follow the virtual facilitator's instructions and recommendations in the context of group decision-making?
RQ3- How can a virtual facilitator impose and enforce a structure on a group decision-making activity, and ensure that all participants have an opportunity to be heard?

61

62 of 108

Prototype 3�Disagreement Management for Group Facilitation Robot

Disagreement Detection
Disagreement Management
Investigate the needs at real work groups

62

63 of 108

Robot-Driven Disagreement Management in Group Decision-Making Sessions

3.1 Detect Task Conflict/Disagreement

3.2 Manage Task Conflict/Disagreement

63

64 of 108

3.1 Disagreement Detection

64

Datasets

AMI¹

GAP²

Features

Sentences

Speech

Sentiment

Models

LSTM

BERT

Realtime?

For the first part, I developed a machine learning model to detect disagreements based on the verbal communication among members (language features of the utterances).

Although several papers have employed traditional machine learning techniques to detect disagreement using lexical, structural, durational, prosodic, and discourse related features, less work has used state of the art computational approaches in Natural Language Processing (NLP) and deep learning models to detect disagreement by considering the meaning of utterances and the conversation context. One advantage of deep learning models is their ability to learn and extract features as a part of the training process. And This gives them a great potential to be used in real-time systems.

Thus, in my work, I investigated the use of a deep learning approach to detect disagreement in conversations.

Moreover, I used language features to train my models as they don’t require any manual extraction and can be fed to a model in real-time.

65 of 108

Datasets

AMI

100 hours in total
94 (out of 139) sessions has argument annotation
Twente Argument Schema (TAS)

GAP

Winter Survival Task
28 Sessions

65

66 of 108

Disagreement Detection Models

Pre-processing and Up-sampling
LSTM (Long Short Term Memory)

Model and learn dependencies in sequential data
Used Gensim doc2vec for sentence modeling

BERT (Bidirectional Encoder Representation of Transformer)

Powerful language model

66

67 of 108

Results

67

Model	Training Accuracy	Testing Accuracy	F1-score	# of labels
LSTM-3	0.88	0.64	0.63	7
LSTM-4	89	68	67	7
LSTM-5	0.92	0.69	0.69	7
LSTM-6	0.82	0.66	0.65	7
BERT	0.76	0.69	0.68	3
BERT	0.91	0.78	0.71	2

68 of 108

Summary of Disagreement Detection

Proof of concept
Future steps:

Include other features such as audio and sentiment which are known as good indicators of disagreement yet can be automatically extracted.
Use the previous utterances of the same speaker to better understand the context to detect disagreement.

Context-based evaluation

68

69 of 108

Robot-Driven Disagreement Management in Group Decision-Making Sessions

3.1 Detect Task Conflict/Disagreement

3.2 Manage Task Conflict/Disagreement

69

70 of 108

Intragroup Conflict/Disagreement

Task conflict, relationship conflict, process conflict, status conflict
Task conflict is helpful only when there is no relationship conflict

70

Task conflict,

“The effective management of conflict is critical” to optimize its effect.

[Lindred Leura Greer, et al. 2017, Conflict in Teams]

Let me first remind you of the context before describing the robot-driven disagreement management intervention.

The literature in management sciences suggest that Group conflict or disagreements is inevitable in collaborative decision making tasks, and it can be a major source of meeting inefficiency if it is left unmanaged. So the literature suggests the key is effective management of conflict to optimize its effects.

**Also, There are different types of intragroup conflict …. Task conflict, results from disagreement over ideas, and perspective about the group task, while relationship conflicts refers to interpersonal clashes and tensions, And it has been shown that coincidence of task and relationship conflict at the same time could be very destructive for the team [13].

It has been shown that task conflict could be positive for a team but it can only help the group when there is no personal (relational) conflict among the members. So it’s really important to manage the conflict and the group to avoid the negative impacts of conflicts.

71 of 108

Intragroup Conflict Management

Many different conflict management strategies have been investigated

A facilitator can manage the conflict:

Understand the topic of conflict, and involved parties
Make sure the group is emotionally stable
Stay focused on the problem rather than emotional and personal relations
Considering a wide range of alternative solutions
Creating a cooperative atmosphere
Enforcing an organized and orderly process

71

72 of 108

SoFi; a Social Robot for Group Facilitation

72

Social Facilitation
Meeting Facilitation

Structure Management
Balance Participation
Time Management

Decision-Making Facilitation

Disagreement Management
Information Sharing

73 of 108

Disagreement Management Strategies

PASSIVE Disagreement Management

In case of disagreements:

SoFi says: “Well it seems you have different ideas about this item, why don’t you try to work it out?”

ACTIVE Disagreement Management

Robot has a pro-active and explicit verbal intervention to resolve the disagreement.
In case of disagreements:

SoFi guides the group through procedural steps to resolve their conflict

73

74 of 108

ACTIVE Disagreement Management

Robot describes Active Listening before the group decision making.
Acknowledge the Disagreement and Articulate the Emerging Consensus: The robot starts with an agreement to establish a common ground between the parties
Active Listening Confirms and clarifies the source of the disagreement
Review Pros and Cons: Identifies the alternatives and clarifies the pros and cons of each
Review Alternative Options: Asks specific delineating questions to each party (e.g. how much will it cost, what is involved, etc.)
Summarize and wrap-up: Checks with the parties to determine is consensus has been reached, and announces the final decision.

74

In active disagreement management the robot first 1) Acknowledge the Disagreement and states what parties have agreed on to boost the discussion climate.

2) then SoFi guides the group through an active listening practice to clarify the sources of disagreement. SoFi first asks member A to describe what s/he thinks and asks member B to carefully listen and then repeat what person A said in her own words.

3) Review Pros and Cons: Then if they still disagree, SoFi, asks each party to review the pros and cons of their unfavored option. This instruction enables them to actively seek pros for their disliked option.

4) Review Alternative Options: In case the group does not reach to a consensus after step 2 and 3, the robot requests each participant to suggest an alternative option.

5) Summarize and Wrap up: Finally, SoFi again asks each member about their decisions. If they agree on an option, SoFi announces the final decision, thank them, and moves on to the next step. If they still disagree after three rounds of interventions, SoFi tells them that it is fine and moves on to the next task. In step 2, when the group has to decide ranking of items, in case of disagreement, SoFi lets the group know that to make the task easier in the next round, she uses a random number generator to randomly pick one of the two items as the n^th item in the list. SoFi also insists that this won't be counted as points for any of the participants. Table 8 shows sample scripts for each of these steps.

75 of 108

User Study 3

Wizard of Oz
Experiment Task: Winter Survival Exercise

Individual initial ranking of items
Comparing pairs of items, ranking items in group
Individual final ranking of items

N = 26

75

ACTIVE disagreement management

PASSIVE disagreement management

vs.

76 of 108

Task: Winter Survival (modified)

Hypothetical Situation

A small plane crash-landed in the woods of northern Minnesota and southern Manitoba. It is 11:32 A.M. in mid-January. The sun will set before 4 P.M.
While escaping from the plane, your group salvaged the 10 items. Your task is to rank these items according to their importance to your survival.

76

77 of 108

Session Structure

77

A

B

Decision-Making Facilitation

Meeting Facilitation

Social Facilitation

Start

Discussion

Wrap up

End

Greeting/ Intro.Ice Breaking

Agenda/ Task Description

Indiv. Initial Ranking

Group Discussion Step 1

Group Discussion Step 2

Summary & Wrap up

Farewell

Indiv. Final Ranking

Pair comparison on 10 items

Ranking of 5 items

78 of 108

Quantitative Results

78

79 of 108

Participants rated their experience positively

Cohesion [sample items]: - The members of this team get along well together.
Decision-Making Satisfaction [sample items]: I am satisfied with our decision-making process and outcome in the meeting.
Robot Facilitation [sample items]: The robot -Helped managing the meeting structure. -Assured that my partner and I have equal chance to express our ideas. -Could effectively manage the disagreements between my partner and I.

79

*

(Composite scale)

80 of 108

Participants followed Disagreement Management Instructions

Video Analysis of Groups in ACTIVE condition

Participants followed the robot’s disagreement management instructions 100% of the time

Active listening practice
Review pros and cons
Review alternatives

80

81 of 108

Active Listening

81

*p<.05

p=.06

*p<.05

NS

p=.09

Statements about ME

Statements about TEAMMATE

Video Analysis: 100% adhering to robot’s instructions in Active Listening

82 of 108

Robot Evaluation

82

*p<.05

(Composite scale)

(Single item)

83 of 108

Group Performance

No significant differences in:

*Team and Meeting evaluation,* Task outcomes, *Number of Disagreements

83

*P=.05

*P=.06

Disagreement Recall

Use of time

84 of 108

Questions

Why the robot with ACTIVE DM is rated as less friendly and trustworthy?
Why participants felt the time was used less wisely in ACTIVE DM?

Expectations from a robot? Who should the robot represent? What role should the robot play?
What can a robot do to help with the group meeting challenges such as disagreement?
How do people perceive the errors and delays of the robot?
How would a facilitation robot be helpful in different types of groups (size, hierarchy, task)?

84

85 of 108

Focus Group

85

Grant Group at Northeastern University

N = 7
66% male
Average Age: 42.8 (SD = 10.4),
0.5 – 7 years of experience

86 of 108

Qualitative Results

Interviews

Focus Group

86

87 of 108

Group Meeting Challenges

Feedback on the Current Facilitation Functions of the Group Facilitation Robot
Robot in Groups with Hierarchy
Level of Robot’s involvement: Mediation vs. Arbitration
Meeting Efficiency with the Robot
Disagreement Management by the Robot

87

Information

Topic Deviation

Emotional Discussion

Dominance

Trust

No Bias

Robot’s Errors; The impact, Group tolerance, and the Repair Strategy

88 of 108

A holistic Approach for Disagreement Management Prevention vs. Cure

[P11]: “[SoFi] Formalizes the argument more compared to any other just plain human interaction argument… it can be helpful to keep the argument calmer …. there was a structured way to carry out the entire argument. … By helping people hear each other out at the same time they are allowed to raise their opinions and see if it leads to a conflict.”

88

“

One important theme was about “the ho’listic approach to manage disagreement”. Participants believed SoFi could Formalize the argument, and keep the argument calmer by enforcing the structure and helping people clearly express their thoughts and hear each other.

Showing the role of the robot’s meeting facilitations as a preventive care for conflicts in the group meetings.

[P11]: “[SoFi] Formalizes the argument more compared to any other just plain human interaction argument, formalize the arguments. Right. And, at the same time I think having it can be helpful to keep the argument calmer compared to when it's just a human because it allows you to express your ideas. You're allowed to voice your opinions at a certain time. …. So yeah, again, as I mentioned, there was a structured way to carry out the entire argument. … By helping people hear each other out at the same time they are allowed to raise their opinions and uh, see if it leads to a conflict.”

89 of 108

Active Listening Useful

[P14]: “I thought that was really unique and really interesting. And just encouraging us to listen to each other. Repeat back with the other person said, I feel like that made me feel more validated when I was saying, the other person was listening actively to what I was saying and we would be able to work together.”

[P2]: “I liked how she asked us to convey what the other person said in our own words so that we are on the same page. It's easy to misunderstand things. So when you recalculate what you said you can always say this is not what I meant.”

89

“

Another theme is related to the active listening practice, most of participants commented very positively about the active listening practice and found it unique and helpful in staying on the same page, and it makes them feel more validated when the other person was actively listening to them. And preventing misunderstanding

GROUP 9: in the absence of that, the con of that is you end up not listening to that other point of view and do much stick to that. And that dominant person might at least their point of view may get accepted. So that is a con of not having that. So at least she and sure that you listen to that other person, even if you didn't heard the a hundred percent or if not be active, but at least you heard that person and when you repeated that, at least that decision making goes to your mind. It's because you hear it and you [process it]. If you remove that part, you might just listen it just for the sake of listening and just to reply. Not to process it. Yeah

90 of 108

Active Listening Useful but … Long

The intervention is long and becomes repetitive / robot overexplains:

[P19]: “The one thing I thought was that like it got kind of repetitive eventually and I felt like she was just asking the same questions she was asking the whole time. So like there got to a point where I felt like we could continue on with the pattern that she was providing.”

90

“

91 of 108

Participants Felt the Lack of Disagreement Management in PASSIVE condition

[P5]: “… I thought that it was just stating the obvious because I know there is a disagreement, but she just told us that basically you're not on the same page. So it is important to state that you have not reached a decision. But you have to do something about it because it cannot be left unsaid if in a group meeting, if you just leave as certain topics onset or you don't reach a certain conclusion, it's not going to benefit anyone.”

91

“

92 of 108

Robot May Disagree but Should Never Embarrass

ME: Would you be offended if SoFi disagrees with you?

[P9]:“Not really, because I know down the line that it's a robot. in fact I would appreciate it, because this robot is being trained with multiple teams and multiple things, so definitely I'm sure that the robots input is totally unbiased so there's nothing to feel offended about [compared to a human]...”

[P29]: “ … the biggest thing you want to be careful of is embarrassing somebody in front of those other people”.

92

“

Regarding the involvement of the robot during a decision-making, participants expect the robot to provide extra information when they have a disagreement and felt it is fine if the robot disagree with them and they won’t be offended since they know the robot is unbiased, however they felt the robot should be very careful not to embarrass an individual in front of the group.

Full quote:

“Not really because I know down the line that it's a robot. so definitely it's using its own as natural language processing skills that it's coming to some conclusion so definitely in fact I would appreciate it, because this robot is being trained with multiple teams and multiple things, so definitely I'm sure that the robots input is totally unbiased so there's nothing to feel offended about. I mean like for a person saying in something, you may feel that he has some interests with that guy. Maybe they are friends. That's why he's taking his side for the robot's decision would be based simply on the processing, which I can always like any other solution, ignore it if I don't want [inaudible].”

93 of 108

Robot as an Emotion Stabilizer

[P12]: “I think if we were hostile towards each other, I would rather be more angry at the device [robot] than at you. I feel like if I had to yell back at the device then it would kind of take off some pressure of you. That way you don't also become combative at me.”

93

“

Last but not least, some participants found SoFi could be a good emotion stabilizer when they need to express their anger.

They said they would rather be more angry at the device than their colleagues, and it would take off some pressure of them.

***

Emotional discussion: “But I think if we were hostile towards each other, I would rather be more angry at the device than at you. Like I feel like if I had to yell back at the device then it would kind of take off some pressure of you. That way you don't also become combative at me. “

Group 9: "And if I do sense like, Oh, this robot's pointing out a lot more of their points and my points at least like I'm like blaming the robot. It's not like he's like, we need to focus on this. You ignored me on that or like yelling at me for it. The robots kind of like, you ignore this person and I've been at the robot rather than directly at him for speaking up too much. "

Robot won’t be offended

Prefer getting angry at the robot instead of a human

94 of 108

Discussion and Conclusion�

Lessons Learnt

94

95 of 108

Tailored Robot Facilitator

Robot’s Role in Group Meetings - One Role Does Not Fit All
Depending on:

Meeting type
Group hierarchy
Task
Time
Group size

Programmable roles and functions for the robot

95

96 of 108

Human - Robot Relationship Over Time

Individuals and groups build a relationship with a robot facilitator

Longitudinal Consequences of Robot’s Malfunctions

96

97 of 108

Importance of Logistic and Administrative Facilitation

97

Decision-Making Needs

Meeting Needs

Administrative (Documenting) Needs

Logistic (Scheduling) Needs

98 of 108

A Design Framework for Social Robots as Meeting Facilitators

98

99 of 108

Social Interaction�Humanoid Social behavior, Interactive, Attentive Listening, Confirming Responses, social catalyzer

99

Structure management:

Programmable and dynamic agenda, recap and summary

Balance Participation:

Dynamic assignment of time to group members based on their role and responsibilities

Time Management:

Providing passive reminders instead of verbal reminders, Flexible timing

Disagreement Management:

Prevention by managing participation, Providing information, Encourage and remind active listening, Providing weight/metric table for options

Information Support:

Enable Asking Questions, Explain when requested,

Provide information for disagreement management, Access to background domain information, Referring to other groups decision

Meeting Facilitation

Decision Making Facilitation

Logistic Facilitation – Assisting Meeting Moderators

Indicating the importance of social interactions throughout the meeting. Providing dynamic and flexible structure and timing.

paying attention to preventive care in disagreement management verses interventions after a conflict arises.

Allowing people to ask for information , and providing Informational support on demand is essential.

Besides this, providing scheduling facilitation and assisting meeting moderators are other requirements of an ideal group facilitation system.

A group facilitation robot needs to be interactive and provide social support throughout a meeting.

Regarding the meeting facilitation, providing dynamic and flexible structure and timing is very important.

For disagreement management paying attention to preventive care is critical.

Allowing people to ask for information , and providing Informational support on demand is essential.

Besides this, providing scheduling facilitation and assisting meeting moderators are other requirements of an ideal group facilitation system.

Assisting the meeting moderator:

Logistic and meeting Information (meeting stats + behavior analysis of members)

100 of 108

Automated Group Facilitation System

100

Automated Group Facilitation Framework
Acceptance, Feasibility, and Embodiment
Automated Meeting Structure Management
Disagreement Management & Real-world Applications

Prototype1

Prototype2

Prototype3

Framework

101 of 108

Future Work

Integrate NLU and NLP
Support Online Meeting
Add Sentiment Analysis
Support Larger Group and more realistic scenarios
Support Various Conflict Management Approach

101

102 of 108

103 of 108

Other Learned Lessons : )

(more important !)

104 of 108

104

105 of 108

105

106 of 108

Thank you!

106

107 of 108

Future Work

107

Explore

Detect

Develop

Real groups with a collaboration history
Different group sizes
Different group cultures
Different group tasks
Power dynamics and group hierarchies
Information resource
Level of agent involvement
The agent’s role and persona

Group emotion
Individual emotion
Dominance and power dynamics
Relationships between members
Real-time topic extraction

Larger groups
Manage conflict
Task/information assistance
Decision-making assistance
Decision-making guidance
Control dominance

108 of 108

Research Questions

108

1) How would people react to a group facilitation robot? (with meeting, decision-making and social facilitation)

2) How can a robot manage disagreements in a group meeting?

3) What would people expect from a group facilitation robot at real workplaces?

User Study

Focus Group