Flexible Dialogue Management for Virtual Conversational Agents
Using Semantically Rich Event Schemas
Benjamin Kane
University of Rochester
31 March 2023
Overarching Question�How can event representations be used incrementally to process human-computer dialogue?
Schemas
Object schema: (?x banana.n) 🍌
:types
(?y banana-peel.n)
:parts
(?y part-of ?x)
:purposes
((a.d person.n) eat.v ?x)
…
Event schema: ((?x eat.v ?y) ** ?e) 🍴
:types
(?x agent.n) (?y food.n)
:goals
(?x (want.v (that (not (?x hungry.a))))
:preconds
(?x have.v ?y)
…
Related: frames (Minsky 1974)
Related: scripts (Schank & Abelson 1977), templates
Schemas can be used for discourse analysis.
Event schema: ((?a1 buy.v ?x) ** ?e) 💰
:types
(?a1 agent.n) (?a2 agent.n)
(?x object.n) (?m money.n)
:goals
(?a1 (want.v (that (?a1 own.v ?x)))
:preconds
(?a1 own.v ?m)
(?a2 own.v ?x)
:episodes
?e1 (?a1 go.v (to ?a2))
?e2 (?a1 request.v ?a2 (for.p ?x))
?e3 (?a2 acknowledge.v ?e2)
?e4 (?a2 give.v ?a1 ?x)
?e5 (?a1 give.v ?a2 ?m)
E1: John gets out $5.
E2: John goes to Bob.
E3: John says “I’d like to buy a ticket.”
Q: What did John give to Bob?
Q: Why did John go to Bob?
A: John wanted to own a ticket.
A: John gave Bob $5.
Towards Natural Language Story Understanding with Rich Logical Schemas
Lane Lawley and Gene Louis Kim and Lenhart Schubert
Department of Computer Science, University of Rochester
Workshop on Natural Language and Computer Science, ACL, 2019
Learning General Event Schemas with Episodic Logic
Lane Lawley and Benjamin Kuehnert and Lenhart Schubert
Department of Computer Science, University of Rochester
Natural Logic Meets Machine Learning (NALOMA), 2021
Mining Logical Event Schemas From Pre-Trained Language Models
Lane Lawley and Lenhart Schubert
Department of Computer Science, University of Rochester
Student Research Workshop, ACL, 2022
Overarching Question�How can event representations be used incrementally to process human-computer dialogue?
Dialogue between Rob (🤖) and John (👨)
Event schema: ((?a1 buy.v ?x) ** ?e) 💰
:types
(?a1 agent.n) (?a2 agent.n)
(?x object.n) (?m money.n)
:goals
(?a1 (want.v (that (?a1 own.v ?x)))
:preconds
(?a1 own.v ?m)
(?a2 own.v ?x)
:episodes
?e1 (?a1 go.v (to ?a2))
?e2 (?a1 request.v ?a2 (for.p ?x))
?e3 (?a2 acknowledge.v ?e2)
?e4 (?a2 give.v ?a1 ?x)
?e5 (?a1 give.v ?a2 ?m)
E1: John gets out $5.
E2: John goes to Rob.
E3: John says “I’d like to buy a ticket.”
John wants to own a ticket.
Step 1: Rob acknowledge E3
Step 2: Rob give John a ticket
Step 3: John give Rob $5
Context (common ground)
Intentions & Expectations
Schemas can enable hierarchical planning…
((?a1 take.v ?train (to ?loc)) ** ?e)
…
:episodes
?e1 (?a1 go.v (to ?station))
?e2 (?a1 buy.v ?ticket)
?e3 (?a1 board.v ?train)
…
John go to Station3
John take train to NYC
John buy Ticket17
John board Train4
John go to Bob
John request ...
…
… but also reflexive, pattern-driven behavior
((?a1 tell-time.v ?a2) ** ?e)
…
:episodes
?e1 (?a2 ask.v ?a1 (the.d time.n))
?e2 (?a1 check.v (the.d time.n))
?e3 (?a1 tell.v ?a2 (the.d time.n))
…
((?a1 tell-time.v ?a2) ** ?e)
…
:episodes
?e1 (?a2 ask.v ?a1 (the.d time.n))
?e2 (?a1 check.v (the.d time.n))
?e3 (?a1 tell.v ?a2 (the.d time.n))
…
((?a1 tell-time.v ?a2) ** ?e)
…
:episodes
?e1 (?a2 ask.v ?a1 (the.d time.n))
?e2 (?a1 check.v (the.d time.n))
?e3 (?a1 tell.v ?a2 (the.d time.n))
…
((?a1 tell-time-to.v ?a2) ** ?e)
…
:episodes
?e1 (?a2 ask.v ?a1 (the.d time.n))
?e2 (?a1 check.v (the.d time.n))
?e3 (?a1 tell.v ?a2 (the.d time.n))
…
John go to Station3
John board Train4
Alice ask John the time
John check the time
John tell Alice the time
Semantic representation: Episodic Logic
She wants to eat the cake.
(she.pro ((pres want.v) (to (eat.v (the.d cake.n)))))
(E1 at-about.p Now17),
((Chell (want1.v (to (eat1.v Cake3)))) ** E1)
(Schubert & Hwang 2000)
Semantic parsing
Scoping, disambiguation, coref, deindexing
Unscoped logical form (ULF)
Episodic logical form (EL)
D
KA→(D→(S→2))
(D→(S→2))→KA
Case Studies
Blocks World (David)
Spatially-aware QA and concept tutoring
Where is the blue block?
Between the green block and the rightmost red block.
I would like to teach you the concept of an arch. Let’s start with a simple example…
Is that right?
…now put a block to the right of the blue block.
Well, the block should not touch the blue block.
OK.
Where was it in the beginning?
Above the green block.
SOPHIE
Virtual standardized cancer patient
I’m really sorry to tell you this SOPHIE, but it seems that your cancer has spread.
[SAD] Oh no… I don’t understand…
I know this can be difficult, and I’m here to help you with any questions you might have.
What does this mean for me? My grandson is graduating in three months. I can’t miss that.
Dialogue Architecture
Output
Input
Schema Knowledge
Dialogue Context & Episodic Memory
Perception
U L F
Reasoning
U L F
U L F
Planning
U L F
schema
1
2
3
Execution
U L F
Perception
U L F
pattern transduction
Input
Output
Input
Output
Input
Output
Input
Output
VP
V
NP
∀x P(x)
Q(x)
Two stages of text transduction
“Black beans!”
Utterance
“What do you want?”
“Let me see if I can find some.”
“You want to eat black beans.”
“I will try to find black beans.”
“What do you want to eat?”
Gist
(^you ((pres want.v) (to (eat.v (black.a (plur bean.n))))))
(^me ((futr try.v) (to (find.v (black.a (plur bean.n))))))
((^you ((pres want.v) (to (eat.v what.pro)))) ?)
ULF
Context
Context
Reasoning may be used to infer new facts or possible actions
(^you ask.v ^me
((Block1 be.v where.pro) ?))
(Block1 on.p Block2)
(^me tell.v ^you (that
(Block1 be.v (on.p Block2)))
(Block1 above.p Block2)
(Block2 above.p Block3)
(Block1 above.p Block3)
Reasoning may use domain specialist servers
X
y
z
(E1 before.p E2)
(E2 during.p E3)
(Block1 on.p Block2)
(Block2 at-loc.p Coords1)
(dog.n is-a animal.n)
(animal.n is-a organism.n)
Plan representation
?e4: (^me say-to.v ^you
“What is my prognosis?”)
E4: (^me say-to.v ^you
“What is my prognosis?”)
?e5: (^you reply-to.v E1)
?e6: (^me ask.v ^you
(about.p options.n))
?e3: (^me ask.v ^you
(about.p prognosis.n))
?e1: ((set-of ^me ^you)
have-cancer-dialogue.v)
E2: (^me ask.v ^you
(about.p condition.n))
…
Now
What does this mean for my future?
Now
…
(Certainty: 0.9)
E4: (^me say-to.v ^you
“What is my prognosis?”)
?e5: (^you reply-to.v E1)
?e6: (^me ask.v ^you
(about.p options.n))
?e3: (^me ask.v ^you
(about.p prognosis.n))
?e1: ((set-of ^me ^you)
have-cancer-dialogue.v)
E2: (^me ask.v ^you
(about.p condition.n))
…
Now
Unfortunately Sophie, you might have up to a year…
…
E5: (^you reply-to.v E1)
E3: (^me ask.v ^you
(about.p prognosis.n))
?e1: ((set-of ^me ^you)
have-cancer-dialogue.v)
E2: (^me ask.v ^you
(about.p condition.n))
…
E4: (^me say-to.v ^you
“What is my prognosis?”)
E5: (^you say-to.v ^me
“Your prognosis is …”)
?e6: (^me ask.v ^you
(about.p options.n))
Now
…
…
E4: (^me say-to.v ^you
“What is my prognosis?”)
E5: (^you say-to.v ^me
“Your prognosis is …”)
?e6: (^me ask.v ^you
(about.p options.n))
Now
…
Event schema: ((?s ask.v ?h
(about.p options.n) ** ?e)
:goals
(?s (want.v (to (know.v (about.p …)))))
:preconds
(?s (not (know.v (about.p …))))
:episodes
?e1 (?s say-to.v ?h “What are my options?”)
?e2 (?h reply-to.v ?e1)
Abstract steps may be expanded using schemas.
E4: (^me say-to.v ^you
“What is my prognosis?”)
E5: (^you say-to.v ^me
“Your prognosis is …”)
?e6: (^me ask.v ^you
(about.p options.n))
Event schema: ((?s ask.v ?h
(about.p options.n) ** ?e)
:goals
(?s (want.v (to (know.v (about.p …)))))
:preconds
(?s (not (know.v (about.p …))))
:episodes
?e1 (?s say-to.v ?h “What are my options?”)
?e2 (?h reply-to.v ?e1)
?e7: (^me say-to.v ^you
“What are my options?”)
?e8: (^you reply-to.v ?e7)
Now
Abstract steps may be expanded using schemas.
…
…
?e6: (^me ask.v ^you
(about.p options.n))
E7: (^me say-to.v ^you
“What are my options?”)
Event schema: ((?s ask.v ?h
(about.p goals.n) ** ?e)
…
:episodes
?e1 (?s say-to.v ?h “What are your goals?”)
?e2 (?h reply-to.v ?e1)
?e8: (^you reply-to.v ?e7)
Now
Matched schemas may be dynamically inserted into plan.
…
…
Can you tell me about your goals at this point?
?e6: (^me ask.v ^you
(about.p options.n))
E7: (^me say-to.v ^you
“What are my options?”)
Event schema: ((?s ask.v ?h
(about.p goals.n) ** ?e)
…
:episodes
?e1 (?s say-to.v ?h “What are your goals?”)
?e2 (?h reply-to.v ?e1)
?e8: (^you reply-to.v ?e7)
Now
Matched schemas may be dynamically inserted into plan.
…
…
?e9 (^you ask.v ^me
(about.p goals.n))
Now
?e6: (^me ask.v ^you
(about.p options.n))
E7: (^me say-to.v ^you
“What are my options?”)
?e8: (^you reply-to.v ?e7)
…
…
?e9 (^you ask.v ^me
(about.p goals.n))
E10: (^you say-to.v ^me
“What are your goals?”)
?e11 (^me reply-to.v E10)
Event schema: ((?s ask.v ?h
(about.p goals.n) ** ?e)
…
:episodes
?e1 (?s say-to.v ?h “What are your goals?”)
?e2 (?h reply-to.v ?e1)
Matched schemas may be dynamically inserted into plan.
Now
Other plan operations:
Results
Blocks World
Parser accuracy | 74% |
% Incorrect parses due to ASR errors | 53% |
Total # well-formed spatial questions | 329 |
Correct | 66.6% |
Partially correct | 13.7% |
Incorrect | 18.8% |
Parser accuracy | 94% |
% Incorrect parses due to ASR errors | N/A |
Total # well-formed historical questions | 387 |
Correct | 77% |
Partially correct | 3% |
Incorrect | 20% |
Spatial Questions
Historical Questions
A Spoken Dialogue System for Spatial Question Answering in a Physical Blocks World
Georgiy Platonov and Benjamin Kane and Aaron Gindi and Lenhart Schubert
Department of Computer Science, University of Rochester
Special Interest Group on Discourse and Dialogue (SIGdial), 2020
Registering Historical Context for Question Answering in a Blocks World Dialogue System
Benjamin Kane and Georgiy Platonov and Lenhart Schubert
Department of Computer Science, University of Rochester
Text, Speech, and Dialogue (TSD), 2020
Generating Justifications in a Spatial Question-Answering Dialogue System for a Blocks World
Georgiy Platonov and Benjamin Kane and Lenhart Schubert
Department of Computer Science, University of Rochester
Reasoning and Interaction (ReInAct), 2021
SOPHIE pilot experiment
NOTE: benchmarked against DialoGPT fine-tuned on cancer patient transcripts.
Novel Computational Linguistic Measures, Dialogue System and the Development of SOPHIE: Standardized Online Patient for Healthcare Interaction Education
Mohammad Ali and Taylan Sen and Benjamin Kane and Shagun Bose and Thomas M Carroll and Ronald M Epstein and Lenhart K. Schubert and Ehsan Hoque
University of Rochester
IEEE Transactions on Affective Computing, 2021
A Flexible Schema-Guided Dialogue Management Framework: From Friendly Peer to Virtual Standardized Cancer Patient
Benjamin Kane and Catherine Giugno and Lenhart Schubert and Kurtis Haut and Caleb Wohn and Ehsan Hoque
Department of Computer Science, University of Rochester
In Review at IEEE Transactions on Affective Computing, 2023 (arXiv:2207.07276)
Ongoing Work
Using LLMs for few-shot response generation
Context + Memory
Event schema: ((?s tell.v ?h
(about.p pain.n) ** ?e)
:goals
(?s (want.v (to (decrease.v
((?s ‘s) pain.n)))))
:episodes
?e1 (?s paraphrase-to.v ?h
“My pain is located in my chest.”)
?e2 (?h reply-to.v ?e1)
“I’m sorry to hear… where is it?”
“Mostly in my chest, but it’s been spreading.
I’m really worried about it.”
Gist: “Where is your pain?”
Gist: “My pain is located in
my chest.”
Sentence Transformer Retriever
Relevant Knowledge
InstructGPT
N examples
Paraphrase examples
Modeling beliefs, desires, and intentions
Intensional Gaps: Relating veridicality, factivity, doxasticity, bouleticity, and neg-raising
Benjamin Kane and Will Gantt and
Aaron Steven White
University of Rochester
Semantics and Linguistic Theory (SALT), 2022
(?a prefer.v (that ?e))
(?a want.v (that ?e))
Questions?