Semantic Category Induction

Aaron Steven White

University of Rochester

​

Colloquium Talk

Rutgers University

11 November 2022

Slides

aaronstevenwhite.io 

Data + Code 

megaattitude.io 

Overarching Question�In using a particular linguistic expression, what can we mean and what must we mean?

Jo did not grab beer.

Cessation inference

Veridicality inferences

Observation�What we can mean in using an expression is constrained by lexical knowledge.

Jo did not grab beer.

Cessation inference

Veridicality inferences

Observation�What we can mean in using an expression is constrained by lexical knowledge.

in conjunction with structural knowledge

Jo did not grab beer.

Cessation inference

Veridicality inferences

Jo did not grab beer.

Cessation inference

Veridicality inferences

Question�What knowledge undergirds what we must mean in using an utterance?

forget

Language

Concepts

NP

that S

forget

Language

Concepts

NP

that S

Question�What types of concepts does language “see”?

Prior Work�For some areas of the lexicon, we have a solid understanding what language ”sees”.

Generalization #1 Barwise & Cooper 1981�Determiners are conservative: if D is a determiner then D expresses a relation R between sets A and B s.t. R(A, B) iff R(A, A ∩ B).

Every greyhound is happy.

Some greyhound is happy.

Most greyhounds are happy.

Generalization #1 Barwise & Cooper 1981�Determiners are conservative: if D is a determiner then D expresses a relation R between sets A and B s.t. R(A, B) iff R(A, A ∩ B).

Generalization #2 Gärdenfors 2000, Jäger 2010�Color terms express convex regions in color space.

Language only “sees” convex color concepts.

Challenge�As we expand to larger and more open classes of words, generalizations tend to be harder to find.

Reason�Standard methodologies for discovering lexical generalizations do not scale well because we don’t have good sampling methodologies.

White, Aaron Steven. 2021. On Believing and Hoping Whether. Semantics and Pragmatics 14 (6): 1–18.

Proposed Generalization #1 Egre 2008 see also Hintikka 1975�A predicate triggers veridicality inferences…

Jo did not grab beer.

Cessation inference

Veridicality inferences

Proposed Generalization #1 Egre 2008 see also Hintikka 1975�A predicate triggers veridicality inferences iff it takes both declarative and interrogative clauses.

Triggers veridicality inferences

Takes both declaratives and interrogatives

White & Rawlins 2018, White 2021

Proposed Generalization #2 Zuber 1983, Theiler et al. 2017, 2019�If a predicate triggers neg-raising inferences, it does not take interrogative clauses.

Challenge�As we expand to larger and more open classes of words, generalizations tend to be harder to find.

Reason�Standard methodologies for discovering lexical generalizations do not scale well because we don’t have good sampling methodologies.

This Talk�Inducing semantic categories from lexicon-scale acceptability and inference judgment datasets.

forget

Language

Concepts

NP

that S

Part 1: Discovering Inferential Patterns�Which inferential patterns associated with lexical items are attested and “seen” by language?

English

Jo hated that Bo left. ⇝ Bo left.

NP   V           S     ⇝    S    

Veridicality inference

Part 1: Discovering Inferential Patterns�Which inferential patterns associated with lexical items are attested and “seen” by language?

Part 2: Generalizing Inference Patterns�How do we determine which inference patterns are cross-linguistically active and which are only marked in a particular language?

English

Inference Patterns

Approach

1. Cluster predicates based on measures of their inferential properties.

NP V S ⇝ S

Veridicality inference

Doxastic inference

NP V S ⇝ NP believe S

Bouletic inference

NP V S ⇝ NP want S

Approach

1. Cluster predicates based on measures of their inferential properties.
2. Determine optimal # of clusters based on how well particular clusterings predict syntactic distribution.

Roadmap

1. Measuring distribution
2. Measuring inference
3. Discovering inference patterns
4. Investigating inference patterns

Inference Patterns�Measuring Distribution

MegaAcceptability dataset

Acceptability for 1,000 verbs in 50 syntactic frames focused on clause-embedding.

​

​

White & Rawlins 2016, 2020

...

Verbs

Bleaching method

Frame templates (e.g. NP __ that S) instantiated by semantically bleached fillers.

​

​

Someone __ed something happened

Someone __ed that something happened

Someone __ed to someone that something happened

...

...

x

Verbs

Frames

50,000 total items x 5 judgments per item

​

​

MegaAcceptability dataset

Acceptability for 1,000 verbs in 50 syntactic frames focused on clause-embedding.

​

​

White & Rawlins 2016, 2020

Question

Is bleaching a valid method for capturing the acceptability of a verb in a frame?

​

​

Validation Strategy

Compare judgments for bleached items against judgments from trained linguists. 

​

Validation data

1. Select 30 verbs from across Hacquard & Wellwood's (2012) classification
2. Gather judgments for these verbs in all 50 syntactic frames from:
1. trained linguists
2. naïve speakers 

​

Comparison

Correlation between judgments from LI and Sprouse et al.'s (2013) dataset

​

​

Sprouse Linguistic Inquiry

MegaAcceptability

Correlation

Conclusion

Safe to use bleaching to collect acceptabiliy judgments focused on capturing selection.

​

Inference Patterns�Measuring Inference

NP V S ⇝ S

Veridicality inference

Doxastic inference

NP V S ⇝ NP believe S

Bouletic inference

NP V S ⇝ NP want S

Recipe

1. Validate a bleaching paradigm for collecting judgments for an inference type.
2. Select a set of frames of interest.
3. Select predicates acceptable in those frames using MegaAcceptability.
4. Collect judgments using the paradigm.

​

Veridicality task

White & Rawlins 2018

Someone was irritated that a particular thing happened.

Did that thing happen?

no      maybe or maybe not       yes

Veridicality task

White & Rawlins 2018

Neg-raising task

An & White 2020

If I were to say I don’t think that a particular thing happened, how likely is it that I mean I think that that thing didn’t happen?

Neg-raising task

Extremely unlikely

Extremely likely

An & White 2020

If A knew that C happened, how likely is it that A believed that C happened?

Doxastic task

Extremely unlikely

Extremely likely

Kane et al. 2021

If A persudaded B that C happened, how likely is it that B believed that C happened?

Doxastic task

Extremely unlikely

Extremely likely

Kane et al. 2021

If A was appalled that C happened, how likely is it that A wanted C to have happened?

Bouletic task

Extremely unlikely

Extremely likely

Kane et al. 2021

If A apologized to B that C happened, how likely is it that B wanted C to have happened?

Bouletic task

Extremely unlikely

Extremely likely

Kane et al. 2021

Question

Is bleaching a valid method for capturing inferences associated with verb in a frame?

​

​

Validation Strategy #1

Compare judgments for bleached items against judgments from trained linguists. 

Non-neg-raising

Neg-raising

Mean rating of bleached example

Validation Strategy #1

Compare judgments for bleached items against judgments from trained linguists. 

Validation Strategy #2

Compare judgments for bleached items to judgments for more contentful items.

​

Implementation

For each verb-frame pair in validation set, sample five items from corpus.

​

Mean rating of corpus example

Mean rating of bleached example

r = 0.8

(p < 0.001)

Validation Strategy #1

Compare judgments for bleached items against judgments from trained linguists. 

Validation Strategy #2

Compare judgments for bleached items to judgments for more contentful items.

​

Validation Strategy #3

Compare inference judgments for bleached items to acceptability judgments for established distributional diagnostic.

​

Implementation

For each verb-frame pair in validation set, collect acceptability of strong NPI (additive either).

​

Jo didn’t do a particular thing, and…

…I think that Bo didn’t do that thing either.

…I don’t think that Bo did that thing either.

Mean rating of bleached example

Mean acceptability of strong NPI

r = 0.77 

(p < 0.001)

Conclusion

Safe to use bleaching to collect at least these types of inference judgments.

​

Important Point (again)

Be cautious in using this dataset to investigate individual predicates.

​

​

Inference Patterns�Discovering Patterns

Approach

Cluster predicate-frame pairs in inference space using a multiview mixed effects mixture model.

Finding clusters

Fit model to raw that-clause data in MegaVeridicality, MegaNegRaising, and MegaIntensionality using variational inference.

​

Output

1. A distribution over inference patterns for each verb-frame pair.

Output

1. A distribution over inference patterns for each verb-frame pair.
2. Distributions over judgments for each inference type and inference pattern

​

Question

How many inference patterns should we assume there are?

Idea

Only as many as we need to explain syntactic distribution.

Implementation

Select the smallest clustering for which no larger clustering improves prediction of the judgments in MegaAcceptability.

Cluster

Implementation

Select the smallest clustering for which no larger clustering improves prediction of the judgments in MegaAcceptability.

Result

Optimal number of inference patterns is 15.

Interpretation

There are at least 15 distributionally correlated inference patterns.

Important Point #2

Enriching the distributional representation could increase the granularity of the patterns.

​

Important Point #1

Not all inference patterns instantiated by particular predicates will get their own inference pattern.

Inference Patterns�Investigating Patterns

Representiationals

doxastic mental states and mental processes

NP {thought, believed, suspected} that S

Preferentials 

expressions of preference for a (future) situation.

NP {hoped, wished, demanded, recommended} that S[+/-future]​

Positive internal emotives

positive emotional states

A was {pleased, thrilled, enthused} that C happened.

Preferentials 

expressions of preference for a (future) situation.

NP {hoped, wished, demanded, recommended} that S[+/-future]​

Negative emotive miratives

expressions of surprise with negative valence

NP was {dazed, flustered, alarmed} that S[+future].

Negative external emotives

expressions of negative emotion with behavioral correlates

NP {whined, whimpered, pouted} to NP that S[+future].​

Positive external emotives

expressions of positive emotion with behavioral correlates

NP was {congratulated, praised, fascinated} that S.

Positive internal emotives

positive emotional states

NP was {pleased, thrilled, enthused} that S.

Preferentials 

expressions of preference for a (future) situation.

NP {hoped, wished, demanded, recommended} that S[+future/-tense]​

Negative internal emotives

negative emotional states

NP was {frightened, disgusted, infuriated} that S.​

Representiationals

doxastic mental states and mental processes

NP {thought, believed, suspected} that S

Speculatives 

communication of uncertain beliefs.

NP {ventured, guessed, gossiped} that S

Future commitment

expressions of commitment to future action or result.

NP {promised, ensured, attested} S[+future]

​

Weak communicatives

communicative acts with weak doxastic inferences about the source.

NP {reported, remarked, yelped} to NP that S

​

Representiationals

doxastic mental states and mental processes

NP {thought, believed, suspected} that S

Speculatives 

communication of uncertain beliefs.

NP {ventured, guessed, gossiped} that S

Future commitment

expressions of commitment to future action or result.

NP {promised, ensured, attested} S[+future]

Strong communicatives

communicative acts with strong doxastic inferences about the source.

NP {confessed, admitted, acknowledged} that S​

Discourse commitment

communicative acts committing the source to the content’s truth.

A {maintained, remarked, swore} that C would happen.

Negative emotive miratives

expressions of surprise with negative valence

A was {dazed, flustered, alarmed} that C would happen.

Negative external emotives

expressions of negative emotion with behavioral correlates

A {whined, whimpered, pouted} to B that C would happen.​

Positive external emotives

expressions of positive emotion with behavioral correlates

A was {congratulated, praised, fascinated} that C happened.

Positive internal emotives

positive emotional states

A was {pleased, thrilled, enthused} that C happened.

Preferentials 

expressions of preference for a (future) situation.

NP {hoped, wished, demanded, recommended} that S[+/-future]​

Negative internal emotives

negative emotional states

A was {frightened, disgusted, infuriated} that C happened.​

Negative emotive communicatives

communicative acts with broadly negative valence.

A {screamed, ranted, growled} to B that C would happen.​

​

Weak communicatives

communicative acts with weak doxastic inferences about the source.

NP {reported, remarked, yelped} to NP that S

​

Representiationals

doxastic mental states and mental processes

NP {thought, believed, suspected} that S

Speculatives 

communication of uncertain beliefs.

NP {ventured, guessed, gossiped} that S

Future commitment

expressions of commitment to future action or result.

NP {promised, ensured, attested} S[+future]

Strong communicatives

communicative acts with strong doxastic inferences about the source.

NP {confessed, admitted, acknowledged} that S​

Deceptives

actions involving dishonesty, deceit, or pretense.

NP {lied, misled, faked, fabricated} ((to) NP) that S.​

Discourse commitment

communicative acts committing the source to the content’s truth.

NP{maintained, remarked, swore} that S[+future].

Interpretation

There are at least 15 distributionally correlated inference patterns.

Important Point #2

Enriching the distributional representation could increase the granularity of the patterns.

​

Important Point #1

Not all inference patterns instantiated by particular predicates will get their own inference pattern.

Interpretation

There are at least 15 distributionally correlated inference patterns.

Important Point #2

Enriching the distributional representation could increase the granularity of the patterns.

​

Important Point #1

Not all inference patterns instantiated by particular predicates will get their own inference pattern.

Inference Patterns�Discussion�

Discovering Inferential Patterns�Which inferential patterns associated with lexical items are attested and “seen” by language?

Generalizing Inference Patterns�How do we determine which inference patterns are cross-linguistically active and which are only marked in a particular language?

English

Discovering Inferential Patterns�Which inferential patterns associated with lexical items are attested and “seen” by language?

Generalizing Inference Patterns�How do we determine which inference patterns are cross-linguistically active and which are only marked in a particular language?

English

Mandarin

Generalizing Inferential Patterns

Mandarin distribution

Generalizing Inferential Patterns�Mandarin�MegaAcceptability

Verbs

Challenge #1a�No large digital lexicon of predicates in Mandarin.

MegaAcceptability draws verbs from VerbNet (>3,700 predicates)

Mandarin VerbNet is exceedingly small (~100 predicates)

https://github.com/Embedding/Chinese-Word-Vectors

Verbs

Verbs

Challenge #1a�No large digital lexicon of predicates in Mandarin.

Challenge #1b�No direct mapping between English syntactic features and Mandarin syntactic features.

Verbs

Verbs

Challenge #1a�No large digital lexicon of predicates in Mandarin.

Challenge #1b�No direct mapping between English syntactic features and Mandarin syntactic features.

Challenge #1c�Only access to a small pool of Mandarin-speaking participants.

Sprouse Linguistic Inquiry

MegaAcceptability

Correlation

Mandarin distribution

Generalizing Inferential Patterns�Cross-linguistic mapping

Challenge #2�How do we map between English and Mandarin when neither the predicates nor the frames match?

Approach

1. View:
1. verbs as points in frame space
2. the lexicon as a cloud of such points

Challenge #2�How do we map between English and Mandarin when neither the predicates nor the frames match?

Approach

1. View:
1. verbs as points in frame space
2. the lexicon as a cloud of such points
2. Match the shape of those clouds in Mandarin and English by rigidly rotating one of them.

Challenge #2�How do we map between English and Mandarin when neither the predicates nor the frames match?

Approach

1. View:
1. verbs as points in frame space
2. the lexicon as a cloud of such points
2. Match the shape of those clouds in Mandarin and English by rigidly rotating one of them.
3. Use the rotation as the cross-linguistic map.

Mandarin Frames

English Frames

Mandarin distribution

Mandarin distribution

Generalizing Inferential Patterns�Cross-linguistic mapping

Challenge #3�How do we compose our English to Mandarin mapping with our inference pattern mapping?

Approach

1. Create a chimera predicting the acceptability of English predicates in Mandarin frames.
2. Predict those acceptabilities using inference patterns.

Generalizing Inferential Patterns�Discussion�

Discovering Inferential Patterns�Which inferential patterns associated with lexical items are attested and “seen” by language?

Generalizing Inference Patterns�How do we determine which inference patterns are cross-linguistically active and which are only marked in a particular language?

English

Idea

Compose inference pattern-to-distribution mapping with cross-linguistic distribution-to-distribution mapping and assess predictive performance.

Question

Why is the optimal number of inference patterns for English nearly three times that for Mandarin?

Subquestion #1

If we induced inference patterns in Mandarin predicates, would we find a classification of similar granularity to English?

Subquestion #2

If yes to subQ1, how different would the overall structure of the patterns be?

Conclusion

190

Question�What types of concepts does language “see”?

This Talk�Inducing semantic categories from lexicon-scale acceptability and inference judgment datasets.

Discovering Inferential Patterns�Which inferential patterns associated with lexical items are attested and “seen” by language?

Generalizing Inference Patterns�How do we determine which inference patterns are cross-linguistically active and which are only marked in a particular language?

English

Current Directions�How do we discover the underlying representational components?

Subdirection #1�Decomposition of the inference patterns themselves. 

Subdirection #1�Decomposition of the inference patterns themselves. 

Subdirection #2�Decomposition of the relationship between inference patterns and syntactic distribution. 

Relationship between inference patterns and syntax

Subdirection #1�Decomposition of the inference patterns themselves. 

Subdirection #2�Decomposition of the relationship between inference patterns and syntactic distribution. 

Subdirection #3�Decomposition of the relationship between inference patterns and lexical items. 

Possible Unified Approach�Multi-task combinatory categorial grammar induction with structured denotation decoders

forget

Language

Concepts

NP

that S

forget

Language

Concepts

NP

that S

λp.λx. p(w_@) &

[∃i₁.END(i) < t_ref & BELIEVE(x, p, i₁)] &

[∃i₂.BEG(i) = t_ref & ~BELIEVE(x, p, i₂)]

Thanks!

Supported by NSF-BCS-1748969

The MegaAttitude Project: Investigating selection and polysemy at the scale of the lexicon

Appendix A:�Further Validation of MegaAcceptability

Case Study�The vast majority of about-PPs are adjuncts�

Rawlins 2013, 2014

XP₁ V (XP₂) (XP₃) about XP₄

is acceptable

XP₁ V (XP₂) (XP₃)

is acceptable

X

NP _ed  

NP _ed about XP  

Rawlins 2014

NP _ed about XP  

NP _ed  

NP _ed about XP  

NP _ed  

NP _ed  

NP _ed about XP  

NP _ed about XP  

NP _ed  

NP _ed  

NP _ed about XP  

NP _ed about XP  

NP _ed  

Noise variance / acceptability variance 

Proportion violations

Independence

Noise variance / acceptability variance 

Proportion violations

Independence

NP (was) _ed  

NP (was) _ed about whether S  

NP (was) _ed about whether S  

NP (was) _ed  

NP (was) _ed about whether S  

NP (was) _ed  

NP (was) _ed about whether S  

NP (was) _ed  

Acceptability threshold 

Proportion violations

Noise variance / acceptability variance 

Proportion violations

Independence

Acceptability threshold 

Proportion violations

Acceptability threshold 

Proportion violations

Appendix B:�Distribution of Inference Judgments

Appendix C:�Validation of MegaIntensionality

Question

Is bleaching a valid method for capturing doxastic and bouletic inferences associated with verb in a frame?

​

​

Challenge

Doxastic and bouletic inferences are highly sensitive to world knowledge.

​

​

Jo doubts that Bo left. ⇝ Jo doesn't believe that Bo left.

Jo doubts that Bo left. ⇝ Jo wants Bo to have left.

Trump doubts that he won in 2020.

Trump wants to have won in 2020.

Approach

1. Norm scenarios for likelihood of prior belief or desire not conditioned on a previous sentence

​

​

Executives generally want their deals to go through.

Executives generally believe that their deals will go through.

Norming

Approach

1. Norm scenarios for likelihood of prior belief or desire not conditioned on a previous sentence
2. Test those normed schenarios in an inference task focused 24 verbs. 

​

​

Executives generally want their deals to go through.

Executives generally believe that their deals will go through.

Norming

The executive knew that his deal had gone through.

Contentful

Approach

1. Norm scenarios for likelihood of prior belief or desire not conditioned on a previous sentence
2. Test those normed schenarios in an inference task focused 24 verbs. 
3. Compare to bleached variants.

​

​

Executives generally want their deals to go through.

Executives generally believe that their deals will go through.

Norming

The executive knew that his deal had gone through.

Contentful

A knew that C happened.

Bleached

Appendix D:�Number of possible inference patterns

(3 veridicality inferences)^{2 matrix polarities}

x

(3 doxastic inferences)^{2 matrix polarities}

x

(3 bouletic inferences)^{2 matrix polarities}

x

2 neg-raising inferences

=

1,458 inference patterns

​

If any lexical knowledge relevant to any inference type is gradient (and continuous), there are an uncountable number of patterns.

Appendix E:�Principal Component Analysis

95% of variance

1. The polarity of veridicality and doxastic inferences under negation is anti-correlated with neg-raising.
2. The polarity of a belief presupposition about a recipient is correlated with the polarity of a desire presupposition.
3. The valence of an emotive communicative is anticorrelated with veridicality.
4. Bouletic inferences about the source and the target of a communication are anticorrelated with veridicality.
5. Desire inferences about the source in a communication are anticorrelated with belief inferences about the target.
6. Veridicality is correlated with belief inferences in the target of a communication but anticorrelated with desire inferences.

Appendix F:�Inference Patterns to Mandarin Frames