1 of 38

What makes you change your mind?

An empirical investigation in online group decision-making conversations

Georgi Karadzhov, Tom Stafford, Andreas Vlachos

@G_Karadzhov

gmk34@cam.ac.uk

delibot.xyz

gkaradzhov.com

2 of 38

Group collaboration in psychology

When tested on reasoning tasks, individuals solve the task correctly 10-20% of the time. Groups on the other hand could have up to 70-80% group performance.
Assembly Bonus Effect: “The group is better than the sum of its parts” (Mercier and Sperber, 2011)
Small focused groups perform better than the wisdom of the crowd (Navajas et al, 2018)

3 of 38

What makes a constructive discussion

Advocating for a solution?
Disagreeing?
Considering opposite solutions?
Probing the conversation?

6 of 38

Task-oriented chatbot

“We can implement a task-oriented chatbot - Easy!“

Complete task for the user
Siri/Alexa/customer support bots
The goal is to complete the task as quickly and efficiently as possible
Trivial to implement in the context of the Wason card selection task
BOT: “Hi all, the answer is U and 7. Cheers, byee!”

7 of 38

Wait a minute

If a chatbot solves a task for a group of people - is this really a constructive discussion?
People can lead constructive discussions on a variety of topics, a task-oriented chatbot cannot
A trained human moderator can participate in a wide range of discussions even if they are not expert on the topic

8 of 38

DEliBots - Deliberation Enhancing Bots

9 of 38

Core principles

The goal is to increase the performance of the group
The bot doesn’t know the answer
The bot knows how to ask good questions - probing
Asking for reasoning or suggesting opposite solutions correlate with good performance
So to speak the “Moderator” of the discussion

10 of 38

DEliBots - Deliberation Enhancing Bots

The goal of the dialogue

The dialogue is the goal

Complete a task for the person

Provide a framework that enables people to contribute towards a common goal

https://www.delibot.xyz/

11 of 38

DeliData - Deliberation Dataset

Thanks to the Isaac Newton Trust and Cambridge University Press - without their support this would not be possible

12 of 38

Annotation Schema

14 of 38

Annotated Example

15 of 38

Let’s build the DEliBot

Splendid! We have all these data, let’s just train a dialogue model to say probing utterance!
Uhh, how?

16 of 38

Problem 1 - when a DEliBot should speak

Remember - this is not a two-party conversation!
A skilled moderator knows exactly when to say something
The goal of the bot is to help the group of people, not to overwhelm them with useless prompts

17 of 38

Problem 2 - what a DEliBot should say

Easy solution: just train on all probing utterances
Reality check: Not all utterances are created equal - One may probe the group and ask a question but that is not necessarily helpful

18 of 38

What makes you change your mind?

An empirical investigation in online group decision-making conversations

Georgi Karadzhov, Tom Stafford, Andreas Vlachos

19 of 38

Addressing the timing of DEliBot participation

Time

Performance

Intervention

20 of 38

Finding good things to say

21 of 38

Task definition

Predicting which utterances will make someone change their mind
Online detection - we don’t incorporate any knowledge of what was said after the current utterance

23 of 38

Data

50 annotated conversations
450 unannotated (but with game data!)

24 of 38

Gold and Weakly supervised

26 of 38

Methods and Models

27 of 38

Methods from reliability engineering (Non linguistic)

Hazard function
Positional prior
Bayesian Online Changepoint Detection (Adams and MacKay, 2007)

28 of 38

Text-based models - simple classification

USR_1: What is your reasoning�USR_2: We need to turn a vowel

X = [“What is your reasoning <SEP> We need to turn a vowel”]

Y = [1]

29 of 38

Learning to Rank

Most utterances don’t cause a change of mind -> class imbalance problem
What if we approach the task as a ranking?

“What is your reasoning <SEP> We need to turn a vowel” >>

“Hey how are you doing <SEP> I am well, thank you”

30 of 38

Evaluation - desiderata

Reward exact matches
Deal with cluster matches
Allow for a relaxed evaluation

32 of 38

Results - baselines

Model	Micro AUC	Macro AUC	Cutoff P-R
Everything causes a change of mind	0.07	0.07	N/A
Bag-of-words + Random Forest	0.19	0.20	0.21

33 of 38

Results - Language agnostic

Model	Micro AUC	Macro AUC	Cutoff P-R
Everything causes a change of mind	0.07	0.07	N/A
Bag-of-words + Random Forest	0.19	0.20	0.21
Hazard Function	0.16	0.17	0.16
Positional prior	0.17	0.17	0.20
Bayesian Online CP detection	0.18	0.21	0.22
BOCP + Positional Prior	0.21	0.23	0.26

34 of 38

Results - Text-based

Model	Micro AUC	Macro AUC	Cutoff P-R
Everything causes a change of mind	0.07	0.07	N/A
Bag-of-words + Random Forest	0.19	0.20	0.21
Linguistic Model (Neural)	0.20	0.20	0.23
Learning to Rank	0.23	0.26	0.24

35 of 38

Results

Model	Micro AUC	Macro AUC	Cutoff P-R
Bag-of-words + Random Forest	0.19	0.20	0.21
Bayesian Online CP detection	0.18	0.21	0.22
BOCP + Positional Prior	0.21	0.23	0.26
Linguistic Model (Neural)	0.20	0.20	0.23
Learning to Rank	0.23	0.26	0.24
Learning to Rank + BOCP + Positional Prior	0.25	0.26	0.30

36 of 38

What is next - DeliData 2.0

Explore the current version of DeliData!
By the end of the year we will release a fully annotated DeliData
More possibilities for dialogue analysis and modeling!
Supported from Cambridge Language Sciences, Isaac Newton Trust, and Cambridge University Press

37 of 38

What is next - Deliberation4Good

Come to our Deliberation4Good workshop: delibot.xyz/d4g
We will talk about how people collaborate to make better decisions together
14th of October at Fitzwilliam College in Cambridge
Looking for speakers and attendees!
Reach out if interested!

Opening Up Minds