Towards a Science and Mathematics of Language�Models

Sophie Hao

MOL 18

8.17.2025

ChatGPT (2022)

P A R T 0 : I N T R O D U C T I O N

​

2

ChatGPT (2022)

P A R T 0 : I N T R O D U C T I O N

​

3

The Language of LLMs

In this talk, I argue that large language models (LLMs) are the first mainstream technology with a “language” that…

• is productive
• supports displacement
• is learnable by other LLMs
• is interchangeable with many human languages

P A R T 0 : I N T R O D U C T I O N

4

The Language of LLMs

Unlike human language, LLM language…

• is made of “tokens,” with no duality of patterning
• often has no semantic grounding
• does not execute any particular communicative intent

P A R T 0 : I N T R O D U C T I O N

​

5

If we are to understand the language of LLMs, we need to develop a linguistics of LLMs.

THESIS OF THIS TALK�LLMs have a “language” of their own, �which resembles human language, �albeit with fundamental differences.

P A R T 0 : I N T R O D U C T I O N

​

6

AI Interpretability

I work in interpretability and explainability for deep neural models used in natural language processing (NLP).

• Interpretable models are often considered more trustworthy than black-box models. �(Štrumbelj et al., 2009; Doshi-Velez and Kim, 2017; Hao, 2022, Ch. 6)
• Interpretability is thought to help with debugging.�(Zeiler and Fergus, 2014; Ribeiro et al., 2016; Lertvittayakumjorn and Toni, 2021; Hao, 2022, Ch. 6)

P A R T 0 : I N T R O D U C T I O N

​

7

Towards LLM Linguistics

There is no consensus on what interpretability is. My position is that interpretability is LLM linguistics—a science and mathematics of LLM language.

1. What are LLMs?
2. How do we describe LLM language?
3. Is there such a thing as LLM grammar?
4. What does research in LLM linguistics look like?

​

P A R T 0 : I N T R O D U C T I O N

​

8

NLP and LLMs

A Brief Introduction

P A R T 0 1

Natural Language Processing

NLP is the branch of AI that deals with computer processing of human language text.

• Classification
• Generation
• Parsing
• Multimodal tasks
• etc.

​

P A R T 1 : N L P A N D L L M S

​

10

Example: Sentiment analysis pipeline

P A R T 1 : N L P A N D L L M S

​

'i just saw this movie on a local tv station (tv8\'s "big chuck and little john")'

↓�['i', 'Ġjust', 'Ġsaw', 'Ġthis', 'Ġmovie', 'Ġon', 'Ġa', 'Ġlocal', 'Ġtv', 'Ġstation', 'Ġ(', 'tv', '8', "'s", 'Ġ"', 'big', 'Ġchuck', 'Ġand', 'Ġlittle', 'Ġjohn', '")']

↓�[72, 655, 2497, 428, 3807, 319, 257, 1957, 31557, 4429, 357, 14981, …]

↓�{'positive': 1.307, 'negative': 1.215}

11

Deep Learning

Current approaches to NLP are based on deep learning.

• Tokens (“words”) are represented as vectors of uninterpreted features.
• These vectors are aggregated to form sequence representations (≈ a sort of “fancy weighted average”).
• Predictions are made using logistic regression on sequence representations.

​

P A R T 1 : N L P A N D L L M S

​

12

Example: Bag of words model

P A R T 1 : N L P A N D L L M S

​

​

13

Deep learning represents tokens using uninterpreted features.

P A R T 1 : N L P A N D L L M S

​

14

There is a unique feature vector for each token in the vocabulary.

P A R T 1 : N L P A N D L L M S

​

15

Feature vectors are trained (i.e., fit) along with regression weights.

P A R T 1 : N L P A N D L L M S

​

16

The current state of the art in NLP is the Transformer architecture.�(Vaswani et al., 2017)

P A R T 1 : N L P A N D L L M S

​

​

…

…

…

17

Transformers have several layers of intermediate representations.

P A R T 1 : N L P A N D L L M S

​

​

…

…

…

⋮

⋮

18

Language Modeling

Given a sequence of tokens from a document, a language model (LM) predicts the next token in that document.

• LMs can be used as autoregressive text generators by sampling from the softmax distribution:

P A R T 1 : N L P A N D L L M S

​

19

During training, Transformer LMs make all possible predictions in parallel.

P A R T 1 : N L P A N D L L M S

​

​

…

…

…

…

20

This makes it very efficient to train Transformers on very large datasets.

P A R T 1 : N L P A N D L L M S

​

​

…

…

…

…

21

However, Transformer text generation is slow because the input grows in size.

P A R T 1 : N L P A N D L L M S

​

​

…

…

…

…

22

Moreover, Transformers are (arguably) not a realistic model of language.

P A R T 1 : N L P A N D L L M S

​

​

…

…

…

…

23

LLMs as General AI Models

What is the most likely continuation of the following?

• 'The 2025 Mathematics of Language meeting will be held in'
• Large LMs have been found to be so good at next-token prediction that they can apparently solve general AI tasks via autoregressive generation.�(Kaplan et al., 2020; Brown et al., 2020)

P A R T 1 : N L P A N D L L M S

​

24

Why LLM Linguistics?

What is the most likely continuation of the following?

• 'Q: What caused the COVID-19 pandemic? A:'
• 'The 15th digit of (1345 + 2815)/7 is'
• 'Q: How many “r”s are there in “strawberry”? A:'

P A R T 1 : N L P A N D L L M S

​

25

Description

of LLM Language

P A R T 0 2

Descriptive LLM Linguistics

Descriptive linguists describe the structure of human languages.

• LLM language is designed to resemble human language.
• We can describe LLM language by measuring how close this resemblance is.

P A R T 2 : D E S C R I P T I O N

27

Descriptive Methods

Descriptive linguists use a combination of corpus and field methods.

• Minimal pair tests
• Acceptability judgments

P A R T 2 : D E S C R I P T I O N

28

Field methods don’t work with LLMs! �(Hu and Levy, 2023)

P A R T 2 : D E S C R I P T I O N

​

29

LLM Field Methods

An LLM can tell you the probability that it will generate a certain text.

We can use these probabilities to access an LLM’s linguistic knowledge.

P A R T 2 : D E S C R I P T I O N

30

Syntactic Contrasts

Which sentence is an LLM more likely to generate?

• *The cat that the dog saw is happy.
• * The cat that the dog saw are happy.

Minimal pair tests show that LLM English replicates many syntactic contrasts of human English. �(Linzen et al., 2016; Marvin and Linzen, 2018; Gauthier et al., 2020; Warstadt et al., 2020; etc.)

P A R T 2 : D E S C R I P T I O N

31

LLM Acceptability Judgments

Is there such a thing as an “LLM acceptability judgment”?

• In general, an LLM should be less likely to generate unacceptable sentences than acceptable ones.
• Therefore, we might expect probability to reflect acceptability.

P A R T 2 : D E S C R I P T I O N

32

Probability vs. Acceptability

Does probability reflect acceptability? No.

​

Dataset: 1,450 English sentences from Linguistic Inquiry (Sprouse et al., 2013)

LLM: OPT 175B (Zhang et al., 2022)

“Probability” refers to log probabilities.

​

P A R T 2 : D E S C R I P T I O N

33

Why Not?

According to Lau et al. (2017), language model probabilities are influenced by the following factors, which do not apply to acceptability:

• Longer texts have lower probability
• Texts with rare words have lower probability

P A R T 2 : D E S C R I P T I O N

34

MORCELA Model

In Tjuatja et al. (2025), I propose the MORCELA model:

where

P A R T 2 : D E S C R I P T I O N

35

MORCELA Model

In Tjuatja et al. (2025), I propose the MORCELA model:

where

• p = LLM (log) probability

P A R T 2 : D E S C R I P T I O N

36

MORCELA Model

In Tjuatja et al. (2025), I propose the MORCELA model:

where

• p = LLM (log) probability
• u = unigram (log) probability

P A R T 2 : D E S C R I P T I O N

37

MORCELA Model

In Tjuatja et al. (2025), I propose the MORCELA model:

where

• p = LLM (log) probability
• u = unigram (log) probability
• l = sentence length

P A R T 2 : D E S C R I P T I O N

38

MORCELA Model

In Tjuatja et al. (2025), I propose the MORCELA model:

where

• p = LLM (log) probability
• u = unigram (log) probability
• l = sentence length
• β and γ are linear regression weights

P A R T 2 : D E S C R I P T I O N

39

Co-creators of MORCELA

Lindia Tjuatja

CMU CS

Graham Neubig

CMU CS

Tal Linzen

NYU Ling/DS

Sophie Hao

BU Ling/DS�NYU DS

40

Evaluating MORCELA

MORCELA scores have a high correlation with gradient acceptability judgments.

​

Dataset: 1,450 English sentences from Linguistic Inquiry (Sprouse et al., 2013)

LLM: OPT 175B (Zhang et al., 2022)

“Probability” refers to log probabilities.

​

P A R T 2 : D E S C R I P T I O N

41

Predicting Rare Words

LLMs that better predict acceptability via MORCELA have lower values of β.

These LLMs are better at predicting rare words in context.

​

LLMs: Pythia (Biderman et al., 2023)

P A R T 2 : D E S C R I P T I O N

​

42

Part 2 Conclusion

Descriptive LLM linguistics involves comparing LLM language to human language…

• …in terms of syntactic contrasts, through comparison of minimal pair probabilities
• …or in terms of acceptability judgments, through linking models like MORCELA.
• The larger the LLM, the closer its LLM English is to human English.

P A R T 2 : D E S C R I P T I O N

​

43

Grammar

in LLM Linguistics

P A R T 0 3

LLM Grammar

Linguists think of grammar as containing:

• Representations of basic objects
• Rules on how to combine or manipulate objects

What are the representations and rules of LLM grammar?

P A R T 3 : G R A M M A R

45

Can we interpret an LLM’s uninterpreted features?

P A R T 3 : G R A M M A R

46

LLM Features

LLM feature bundles are known to contain features representing…

• Semantic relations (Mikolov et al., 2013)
• Gender (Bolukbasi et al., 2016)
• Sentiment (Hao, 2022)
• Word length (Hao, 2022)
• etc.

P A R T 3 : G R A M M A R

47

To find features, we look for groups of words that are linearly separable.�(Bolukbasi et al., 2016; Dufter and Schütze, 2019; Ravfogel et al., 2020; etc.)

she

woman

sister

mother

queen

hen

he

man

brother

father

king

rooster

actress

actor

P A R T 3 : G R A M M A R

48

To find features, we look for groups of words that are linearly separable.�(Bolukbasi et al., 2016; Dufter and Schütze, 2019; Ravfogel et al., 2020; etc.)

woman

she

sister

mother

queen

hen

he

man

brother

father

king

rooster

actress

actor

P A R T 3 : G R A M M A R

49

It is also possible to find continuously-valued features.�(Hao, 2022)

a

I

an

in

cat

bird

elephant

butterfly

rabbit

cow

woodpecker

whale

dog

crane

pelican

P A R T 3 : G R A M M A R

50

What happens if you edit a feature value?

plane

car

bike

ship

train

bus

cars

bikes

trucks

planes

ships

buses

truck

trains

P A R T 3 : G R A M M A R

51

What happens if you edit a feature value?

plane

car

bike

train

bus

cars

bikes

truck

planes

ships

buses

truck

trains

ship

ship

P A R T 3 : G R A M M A R

52

Editing LLM Features

In Hao and Linzen (2023), I show that editing an LLM’s [±plural] feature causes it to conjugate verbs incorrectly.

• BEFORE EDITING�The cat that the dog saw → is
• AFTER EDITING�The cat that the dog saw → are

P A R T 3 : G R A M M A R

53

LLMs transform feature bundles into other feature representations.�(Vaswani et al., 2017)

The

cat

that

sees

the

dog

P A R T 3 : G R A M M A R

54

Editing one or more of these sometimes reverses verb conjugations.

The

cat

that

sees

the

dog

P A R T 3 : G R A M M A R

55

Editing all of these always reverses verb conjugations.

The

cat

that

sees

the

dog

P A R T 3 : G R A M M A R

56

Use of LLM Features

LLM representations contain semantic and syntactic features, and LLMs use these features to decide what to generate.

• What are the rules for how LLMs can use these features?

P A R T 3 : G R A M M A R

57

Formal Analysis

Why is the following phonological process unattested?

/abcd…/ → […dcba]

• Rules of the form /x/ → [y] / p _ q are finite-state. (Kaplan and Kay, 1994)
• Most phonological processes are finite-state. �(Chandlee, 2014; Chandlee et al., 2015; Hao and Andersson, 2019; Burness and McMullin, 2019; etc.)
• String reversal is not finite-state.

​

P A R T 3 : G R A M M A R

58

Circuit Complexity

Transformers can be thought of as boolean circuits, a model of parallel computation.

P A R T 3 : G R A M M A R

59

A boolean circuit�From Wikipedia

P A R T 3 : G R A M M A R

60

Complexity of Transformers

In Hao et al. (2022), I show that hard-attention Transformers (HATs) can be simulated using boolean circuits of constant depth and polynomial size.

• HATs are in the complexity class AC⁰.
• Equivalent to languages definable in first-order logic
• PARITY, EQUALITY, MAJORITY, and DYCK are not AC⁰

P A R T 3 : G R A M M A R

61

In a Transformer, each layer is a “weighted average” of the previous layer.

P A R T 3 : G R A M M A R

62

Under hard attention, each feature bundle is based on one feature bundle from the previous layer.

P A R T 3 : G R A M M A R

63

Hard Attention Is Interpretable

For a machine translation model�(Bahdanau et al., 2015)

P A R T 3 : G R A M M A R

64

Provers of Hao et al.’s Theorem

Sophie Hao

BU Ling/DS, NYU DS (Fmr. Yale Ling/CS)

Bob Frank

Yale Ling

Dana Angluin

Yale CS

65

Part 3 Conclusion

LLM representations contain features that are similar to linguistic representations.

• LLMs use those features similarly to humans.
• However, the Transformer formalism does not allow for full syntactic processing.
• This is an engineering trade-off designed to make LLMs more computationally efficient.

P A R T 3 : G R A M M A R

66

Analogy: �LLMs as Digital Photographs

Photo: 111-segmented LCD display on an NJ Transit train�Source: Reddit user 1Davide

P A R T 3 : G R A M M A R

67

Towards LLM Linguistics

A Research Program

P A R T 0 4

Why LLM Linguistics?

Mathematical linguists should care about LLM linguistics because…

• …LLMs are based on very different mathematics than generative linguistics.
• …LLMs provide new perspectives on linguistic knowledge and language learning.
• …LLM linguistics present opportunities for broad social impact.

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

69

LLM Language Phenomena

There is little understanding of what linguistic phenomena are unique to LLM language.

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

70

Languages without orthographic word boundaries may exhibit morphological garden path effects.�(Li and Hao, 2025)

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

71

Both sentences have negative sentiment, but 留心 is a positive-sentiment word.

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

72

Morphological garden pathfinders

Wing Li

SFU CS�(Fmr. NYU Eng.)

Sophie Hao

BU Ling/DS�NYU DS

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

73

Analyzing LLM Phenomena

Some LLM phenomena can be explained via linguistic analysis.

• LLM hallucination is usually thought of as a bug.�(Ji et al., 2022; Sahoo et al., 2024; Huang et al., 2025)
• But human language involves guessing (Dorst and Mandelkern, 2021) and bullshit (Frankfurt, 2005; Meibauer, 2016; Deck, 2023; etc.).
• What is the best way to mitigate hallucination?

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

74

Applications of LLMs

LLMs and related technologies have broad potential for application to linguistics, social science, and the humanities.

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

75

Female Objectification

Luo et al. (2024) use LLM representations to analyze female objectification in literature.

• Do the names of female characters have higher values for “appearance”-related features than male characters?
• Yes for novels with male authors, �No for novels with female authors.

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

76

Female objectification co-authors

Cindy Luo

Harvard Psych.�(Fmr. NYU DS)

Cora Mao

NYU DS Alum

Bella Zhang

NYU DS Alum

Sophie Hao

BU Ling/DS�NYU DS

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

77

LLMs, AI, and Society

The AI community is very interested in making sure that LLMs treat all users fairly.

• Example: Hate speech detectors use mentions of a group as evidence that a text is hate speech. �(Hartvigsen et al., 2022)
• There is interest in investigating dialectal bias, but more linguistic expertise is needed.

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

78

You’re Invited!

LLM linguistics has something for all mathematical linguists.

• Exciting new mathematical problems
• Conceptual and philosophical implications
• Unexplored linguistic phenomena
• Tools to support your existing research
• Opportunities to shape the AI revolution

P A R T 4 : T O W A R D S L L M L I N G U I S T I C S

79

Thank You!

uu@bu.edu

notaphonologist.com�

80