The role of structured reasoning�in the age of GPT

Peter Clark

Allen Institute for AI

​

Outline

1. Do we still need structured reasoning / NLI?
2. Entailer
3. Reflex
4. Towards microtheories

Do LLMs create internal World Models?

How we read…

Is a LLM doing anything similar?

What is a World Model?

cup

plate

table

on

on

• entities
• relationships
• inference procedure to update entities/relationships

​

on

What is a World Model?

e5

e89

e45

r43

r43

• entities
• relationships
• inference procedure to update entities/relationships

​

r43

Do all LLMs have World Models?

• No!

• But now…

!!

https://aclanthology.org/P19-1621.pdf

The Main Points

1. Modern LLMs do form internal world models

​

It's true in a sense that models do autocomplete. But if you think about it: to do really good autocomplete, you need to understand what someone's saying. And models do understand what someone's saying - they've learned to understand.

https://www.youtube.com/watch?v=Y6Sgp7y178k

The Main Points

1. Modern LLMs do form internal world models
2. We need interpretable versions of these models, for:
• Communication, explanation, teaching
• Verification
• Interaction, argumentation

​

​

​

Extract

The Main Points

1. Modern LLMs do form internal world models
2. We need interpretable versions of these models
3. These are constructed, well-behaved approximations of the internal world model

​

​

​

Extract

Construct

The Main Points

1. Modern LLMs do form internal world models
2. We need interpretable versions of these models
3. These are constructed, well-behaved approximations
4. We need to choose a formalism for these

​

​

• formal logic?
• probabilistic program?
• NLI!

​

The Main Points

1. Modern LLMs do form internal world models
2. We need interpretable versions of these models
3. These are constructed, well-behaved approximations
4. We need to choose a formalism for these
5. New Architecture:

​

​

LLM Layer

“Rational” Layer

Overall System

Interpretable world model

(“microtheory”)

Outline

1. Do we still need structured reasoning / NLI?
2. Entailer: (very) simple world models
3. Reflex: (better) world models
4. Towards microtheories: (even better) world models

LLM Layer

“Rational” Layer

Overall System

Generating Representations

Can volcanic eruptions cause plants to die?

H: Eruptions cause plants to die

H’: Eruptions do not cause plants to die

?

?

Generating Representations

H: Eruptions cause plants to die

eruptions produce ash clouds.

“Generate a chain of reasoning that supports this fact”

Generating Representations

H: Eruptions cause plants to die

“Generate a chain of reasoning that supports this fact”

reflects the LLM’s knowledge?

is a valid inference?

Generating representations

Rabbits have six legs.

Rabbits have six legs BECAUSE

1. a rabbit is a kind of animal

2. animals usually have six legs

Do animals usually have six legs?

No!

a generation ≠ a belief !

reflects the LLM’s knowledge?

is a valid inference?

Generating representations

Rabbits have six legs.

Rabbits have six legs BECAUSE

1. a rabbit is a kind of animal

2. animals usually have six legs

A rabbit is a kind of animal?

reflects the LLM’s knowledge?

is a valid inference?

Generating representations

Rabbits have six legs.

Rabbits have six legs BECAUSE

1. a rabbit is a kind of animal

2. animals usually have six legs

✘

Animals usually have six legs?

reflects the LLM’s knowledge?

is a valid inference?

✘

Generating representations

Rabbits have six legs.

Rabbits have six legs BECAUSE

1. a rabbit is a kind of animal

2. animals usually have six legs

1 and 2 entail 3?

✘

?

reflects the LLM’s knowledge?

is a valid inference?

✘

Generating representations

Rabbits have six legs.

Rabbits have six legs BECAUSE

1. a rabbit is a kind of animal

2. animals usually have six legs

Rabbits have six legs BECAUSE

1. …

2. …

Rabbits have six legs BECAUSE

1. …

2. …

Generating Representations

Can volcanic eruptions cause plants to die?

H: Eruptions cause plants to die

H’: Eruptions do not cause plants to die

?

?

Eruptions block the sun.

Plants will die without any sunshine

Results

• Faithful, truthful answers
• Improved explanation quality
• New opportunities for interaction (TeachMe)

LLM Layer

“Rational” Layer

Overall System

https://arxiv.org/abs/2204.13074

Outline

1. Do we still need structured reasoning / NLI?
2. Entailer: (very) simple world models
3. Reflex: (better) world models
4. Towards microtheories: (even better) world models

S₁

S₂

S₆

S₃

S₉

S₇

S₅

S₄

xor

R₁

R₂

R₃

R₄

S₈

S₁₀

“Rational” Layer:

LLM

LLM Layer:

xor

S₁

S₂

S₆

S₃

S₉

S₇

S₅

S₄

xor

R₁

R₂

R₃

R₄

S₈

S₁₀

“Rational” Layer:

LLM

LLM Layer:

xor

“Rational” Layer:

LLM Layer:

S₁

S₃

S₇

S₅

S₄

S₂

S₆

xor

R₁

R₂

R₃

R₄

S₈

S₉

S₁₀

LLM

xor

Which animal gives birth to live young?

(A) Shark (B) Turtle (C) Giraffe (D) Spider

A giraffe gives birth to live young

A spider gives birth to live young

A giraffe is a kind of mammal

Mammals give birth to live young.

Animals give birth to live young.

A spider is a kind of animal

A mammal is a kind of animal

Mammals give birth to live young.

Which animal gives birth to live young?

(A) Shark (B) Turtle (C) Giraffe (D) Spider

A giraffe gives birth to live young

A spider gives birth to live young

A giraffe is a kind of mammal

Mammals give birth to live young.

Animals give birth to live young.

A spider is a kind of animal

A mammal is a kind of animal

Mammals give birth to live young.

A spider does not give birth to live young

xor

nand

T

T

T

T

T

T

T

T

Which animal gives birth to live young?

(A) Shark (B) Turtle (C) Giraffe (D) Spider

A giraffe gives birth to live young

A spider gives birth to live young

Mammals give birth to live young.

Animals give birth to live young.

A spider is a kind of animal

A mammal is a kind of animal

A spider does not give birth to live young

A giraffe is a kind of mammal

nand

xor

A mammal is a kind of animal

T

T

T

T

T

T

T

T

Which animal gives birth to live young?

(A) Shark (B) Turtle (C) Giraffe (D) Spider

A giraffe gives birth to live young

A spider gives birth to live young

Mammals give birth to live young.

Animals give birth to live young.

A spider is a kind of animal

A mammal is a kind of animal

A spider does not give birth to live young

A giraffe is a kind of mammal

nand

xor

A mammal is a kind of animal

T

T

T

T

F

F

T

T

Which animal gives birth to live young?

(A) Shark (B) Turtle (C) Giraffe (D) Spider

Beach balls contain (A) water (B) carbon dioxide (C) sunlight (D) beach sand

The Cognitive Architecture

Does a spider give birth to live young?

No! Because

- a spider is an arachnid

- arachnids lay eggs

Solution/

Answer

Reflection /

Rumination

LLM Layer

“Rational” Layer

An “understanding” of the problem

Outline

1. Do we still need structured reasoning / NLI?
2. Entailer: (very) simple world models
3. Reflex: (better) world models
4. Towards microtheories: (even better) world models

Towards Microtheories

• Goal: A theory that covers a set of (related) questions
• More than just the union of all explanation facts

Towards Microtheories

• Goal: A theory that covers a set of (related) questions
• More than just the union of all explanation facts

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

A magnet would attract a (A) Wooden hairbrush (B) Oranges (C) Glassware (D) Zippers

Which won’t a magnet pick up? (A) penny (B) iron pan (C) paperclip

A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic cup (D) a paper plate

What might pick up a heavy steel rod? (A) magnetism (B) heat (C) electricity (D) gravity

….

Questions:

Towards Microtheories

• Goal: A theory that covers a set of (related) questions
• More than just the union of all explanation facts

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

A magnet would attract a (A) Wooden hairbrush (B) Oranges (C) Glassware (D) Zippers

Which won’t a magnet pick up? (A) penny (B) iron pan (C) paperclip

A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic cup (D) a paper plate

What might pick up a heavy steel rod? (A) magnetism (B) heat (C) electricity (D) gravity

….

Questions:

Towards Microtheories

• Goal: A theory that covers a set of (related) questions
• More than just the union of all explanation facts

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

A magnet would attract a (A) Wooden hairbrush (B) Oranges (C) Glassware (D) Zippers

Which won’t a magnet pick up? (A) penny (B) iron pan (C) paperclip

A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic cup (D) a paper plate

What might pick up a heavy steel rod? (A) magnetism (B) heat (C) electricity (D) gravity

….

Questions:

A zipper is made of steel.

Steel is a kind of magnetic metal

A zipper is made of magnetic metal

A magnet will attract magnetic material

A magnet will attract a zipper

Towards Microtheories

• Goal: A theory that covers a set of (related) questions
• More than just the union of all explanation facts

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

A magnet would attract a (A) Wooden hairbrush (B) Oranges (C) Glassware (D) Zippers

Which won’t a magnet pick up? (A) penny (B) iron pan (C) paperclip

A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic cup (D) a paper plate

What might pick up a heavy steel rod? (A) magnetism (B) heat (C) electricity (D) gravity

….

Questions:

Towards Microtheories

• Goal: A theory that covers a set of (related) questions
• More than just the union of all explanation facts

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

A magnet would attract a (A) Wooden hairbrush (B) Oranges (C) Glassware (D) Zippers

Which won’t a magnet pick up? (A) penny (B) iron pan (C) paperclip

A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic cup (D) a paper plate

What might pick up a heavy steel rod? (A) magnetism (B) heat (C) electricity (D) gravity

….

Questions:

A penny is made of copper

Copper is not magnetic

A penny is not magnetic

A magnet will not attract a non-magnetic object

A magnet won’t pick up a penny

Towards Microtheories

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

LLM Layer

“Rational” Layer

Overall System

Towards Microtheories

Theory:

(1) A magnet will attract magnetic material.

(2) A magnet will not attract a nonmagnetic object.

(3) A magnet is made of a ferromagnetic material.

(4) Nickel is a kind of magnetic metal.

(5) Steel is a kind of magnetic metal.

(6) A magnetic metal is a ferromagnetic material.

(7) Copper is not magnetic.

(8) A zipper is made of steel.

(9) A penny is made of copper.

(10) Some metals are magnetic.

(11) A magnet creates a magnetic field.

...

Set of Questions

Theory Enumeration

(Theory-CoT

Prompting)

Answers + Structured Explanations

Search over space of theories

Metrics:

• Consistent
• Concise
• General (high coverage)
• High QA accuracy

An interpretable, well behaved world model

The Main Points

1. Modern LLMs do form internal world models
2. We need interpretable versions of these models
3. These are constructed, well-behaved approximations
4. We need to choose a formalism for these
5. New Architecture:

​

​

LLM Layer

“Rational” Layer

Overall System

Interpretable world model

(“microtheory”)