Meaning Workshop

Social Reasoning and Theory of Mind

Working definitions

Theory of Mind: the ability to ascribe mental states (beliefs, desires, intentions, emotions) to people

Social reasoning: reasoning about people and situations (societal expectations and norms)

Pragmatics: how context contributes to meaning

Toy QA tasks evaluating (false) beliefs

Nematzadeh, Aida, et al. "Evaluating Theory of Mind in Question Answering." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

​

​

Scenarios annotated with beliefs, intents, reactions

Rashkin, Hannah, et al. "Event2Mind: Commonsense Inference on Events, Intents, and Reactions." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.

​

Sap, Maarten, et al. "Social IQa: Commonsense Reasoning about Social Interactions." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

​

Moral alignment with humans

Hendrycks, Dan, et al. "Aligning AI With Shared Human Values." International Conference on Learning Representations. 2020.

Existing Datasets

• Toy tasks evaluating false beliefs, ToM, etc.
• bAbi tasks
• Evaluating Theory of Mind in Question Answering:
• Revisiting the Evaluation of Theory of Mind through Question Answering
• Inferring motivations and outcomes of people in text scenarios
• SOCIAL IQA: Commonsense Reasoning about Social Interactions
• Event2Mind: Commonsense Inference on Events, Intents, and Reactions
• Alignment with humans datasets
• Aligning AI with Shared Human Values
• SCRUPLES: A Corpus of Community Ethical Judgments on 32,000 Real-life Anecdotes:
• Scalar quantity implicatures
• Text-only
• Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
• Harnessing the linguistic signal to predict scalar inferences
• “Was It Good? It Was Provocative.” Learning the Meaning of Scalar Adjectives
• Visually grounded reference games
• Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding
• Learning from omission
• ShapeGlot: Learning Language for Shape Differentiation
• Other
• SCONE (Simpler Context-Dependent Logical Forms via Model Projections)

Dimensions of Datasets

• Purely text-based vs. multi-modal
• Toy vs. “real-world”
• Dataset based on existing psychology / cog-sci datasets vs. crowd-sourced
• Highlighting deficiencies vs. enabling new capabilities
• Target: ToM vs. alignment vs. pragmatics
• Output space / task: classification vs. text generation vs. action

​

​

​

Questions

• What do we want out of a dataset? Why is it hard to get there?
• What is the minimal setup that still captures naturalistic inferences?
• What’s considered ground-truth? (Heterogeneity of responses...)
• What is the relationship between social reasoning and pragmatics?
• What are the behavioral or representational signatures of social/pragmatic reasoning in models?