1

Domain Model Learning in AI Planning

Tutorial in AAAI 2026

Part 5: Beyond Offline Learning of Classical Planning Models

Roni Stern

Ben Gurion University of the Negev

​

Common Setting for Learning Planning Domains

• States include Boolean state variables

​

• Deterministic effects

​

• Single agent

​

• Offline learning

​

​

3

Beyond the Common Problem Setting

• States include Boolean state variables
• States also include numeric state variables
• Deterministic effects
• Stochastic effects, partial observability
• Single agent
• Multiple agents acting concurrenlty
• Offline learning
• Active learning: must act to collect observations

4

Numeric Planning

5

Action: Move(Truck, A, B)

Pre: At(Truck, A)

Eff: At(Truck, B), not(Truck, A)

State:

At(Truck, A)

At(Package, B)

State:

At(Truck, A)

At(Package, B)

Fuel(Truck)=5

Action: Move(Truck, A, B)

Pre: At(Truck, A),

Fuel(Truck)>0

Eff: At(Truck, B), not(Truck, A)

Decrease Fuel(Truck) by 1

Numeric state variables

Numeric preconditions

Numeric effects

PlanMiner [Segura-Muros et al., 2021]

1. Pre-process to extract pre- and post-state datasets per action
2. Search for numerical expressions that fit the effects
3. Search for logical rules over them that fit as preconditions

​

6

• Discovering relational and numerical expressions from plan traces for learning action models. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Applied Intelligence, 2021.
• Learning Numerical Action Models from Noisy Input Data. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Arxiv, 2021.

Move

(A, B)

Dataset for Move(?x, ?y)

Load

(Pkg, C)

Move

(B, C)

Dataset for Load(?x, ?y)

PlanMiner [Segura-Muros et al., 2021]

1. Pre-process to extract pre- and post-state datasets per action
2. Search for numerical expressions that fit the effects
3. Search for logical rules over them that fit as preconditions

​

7

• Discovering relational and numerical expressions from plan traces for learning action models. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Applied Intelligence, 2021.
• Learning Numerical Action Models from Noisy Input Data. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Arxiv, 2021.

Move

(A, B)

Dataset for Move(?x, ?y)

Load

(Pkg, C)

Move

(B, C)

Symbolic Regression over functions and integers

Objective: fit numeric expressions to the post-values

Example: Fuel-1

PlanMiner [Segura-Muros et al., 2021]

1. Pre-process to extract pre- and post-state datasets per action
2. Search for numerical expressions that fit the effects
3. Search for logical rules over them that fit as preconditions

​

8

• Discovering relational and numerical expressions from plan traces for learning action models. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Applied Intelligence, 2021.
• Learning Numerical Action Models from Noisy Input Data. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Arxiv, 2021.

Dataset for Move(?x, ?y)

Learn Rules to Classify Between Pre- and Post- States (NSLV)

Preconditions!

PlanMiner [Segura-Muros et al., 2021]

1. Pre-process to extract pre- and post-state datasets per action
2. Search for numerical expressions that fit the effects
3. Search for logical rules over them that fit as preconditions

​

9

• Discovering relational and numerical expressions from plan traces for learning action models. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Applied Intelligence, 2021.
• Learning Numerical Action Models from Noisy Input Data. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Arxiv, 2021.

Advantages:

• Can handle diverse numeric expression
• Include some handling of missing values
• N-PlanMiner: extended to handle noisy observations
• Can incorporate domain knowledge

​

Limitations:

• Runtime is unbounded (symbolic regression…)
• Unclear how to constant numbers
• No guarantees on soundness or completeness

PlanMiner [Segura-Muros et al., 2021]

1. Pre-process to extract pre- and post-state datasets per action
2. Search for numerical expressions that fit the effects
3. Search for logical rules over them that fit as preconditions

​

10

• Discovering relational and numerical expressions from plan traces for learning action models. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Applied Intelligence, 2021.
• Learning Numerical Action Models from Noisy Input Data. Segura-Muros, José Á., Juan Fernández-Olivares, and Raúl Pérez. Arxiv, 2021.

Learning numeric preconditions

= Learning Boolean functions

Learning numeric effects

= Learning general functions

Intractable to learn ☹

Solution: reasonable assumption ☺

Effects are

linear functions

Preconditions are

conjunctions of linear inequality

Numeric SAM (N-SAM) [Mordoch et al. ’23]

11

N-SAM

Numeric

Discrete

SAM Learning

Learning half-spaces

Linear regression

Preconditions

Effects

Numeric SAM: Learning Effects

12

Pre-state vectors as matrix

Coefficient vector

Next state values

Safety:

Only allow action with a unique fit

Pre-state

Action

Post-state

Numeric SAM: Learning Preconditions

13

13

Observation #1:

Preconditions form a convex hull

Numeric SAM: Learning Preconditions

14

14

Observation #1:

Preconditions form a convex hull

Observation #2:

Sampled pre-states form a “sub” convex hull

Numeric SAM: Learning Preconditions

15

15

Observation #1:

Preconditions form a convex hull

Observation #2:

Sampled pre-states form a “sub” convex hull

Limitation: Linear Dependencies

16

Cannot create a convex hull!

NSAM* [Mordoch et al. ’24]

​

​

​

​

Technical details:

• Ensure only states on this subspace are allowed
• Find basis of orthogonal half-space
• Preconditions are dot product = 0
• Preconditions need to be projected as well

​

17

Cannot create a convex hull!

Project points to the

subspace spanned by them

NSAM* is optimal for learning safe action models

(optimal=cannot learn stronger action model with the same data)

Limitations: Sample Complexity

​

​

​

​

​

​

​

​

• Worst case is linear in the # states ☹
• Open question: does this occurs “in practice”?

18

Reachable states

?

Sampled states

New state

Learned Convex Hull

Experimental Results for N-SAM

19

Benchmark Numeric Planning Problems from the IPC and Scala et al.

Problem Solving Capabilities

20

1. Better than PlanMiner in 4 (sometimes drastically)
2. Often NSAM is sufficient and NSAM* is not needed

​

​

​

​

​

​

• PlanMiner better in 3 domains

​

​

​

​

​

​

• Inconclusive results in 2 domains

​

​

​

​

Note: PlanMiner is not safe

Note: only 80 trajectories!

Learning Numeric Planning Domain

N-PlanMiner

☺ Handles arbitrary numerical expressions (*)

☺ Some support for noise and missing values

☹ No theoretical guarantees, runtime is unbounded

NSAM*

☺ Runtime is polynomial in the input transition

• Guaranteed safety and optimality (sample complexity)
• Limited to polynomial effects and preconditions
• No support for missing observations or noise

​

Many open directions:

• Richer domains: stochastic effects, processes and events,…
• Noise and missing values with guarantees on learned model
• Integration with RL and neural-network-based methods

….

21

That’s it?!

(Juba and Stern ’22)

(Deng and Juba ‘23)

Beyond the Common Problem Setting

• States include Boolean state variables
• States also include numeric state variables
• Deterministic effects and full observability
• Stochastic effects, partial observability
• Single agent
• Multiple agents acting concurrently
• Offline learning
• Active learning: must act to collect observations

22

Multi-Agent

• The “credit assignment problem” for effects
• MA-SAM: learn to disambiguate
• Open question: action interaction, MARL

23

From One to Many [Brafman and Domshlak ‘08]

​

​

​

​

​

​

​

Lammas [Zhuo et al. ‘11],

MA-SAM [Mordoch et al. ’22]

From One to Many [Brafman and Domshlak ‘08]

​

​

​

​

​

​

​

Conflicts between actions?

Collaborative actions?

“Credit assignment”?

Concurrent execution – trajectories of joint actions

From One to Many [Brafman and Domshlak ‘08]

​

​

​

​

​

​

​

“Credit assignment”

Concurrent execution – trajectories of joint actions

Not(Have(apple))

Have(apple)

Have(apple) is an effect of

A11, or A21, or both

From One to Many [Brafman and Domshlak ‘08]

​

​

​

​

​

​

​

“Credit assignment”

Concurrent execution – trajectories of joint actions

From One to Many [Brafman and Domshlak ‘08]

​

​

​

​

​

​

​

“Credit assignment”

Concurrent execution – trajectories of joint actions

Many open directions:

• Richer domains: stochastic effects, processes and events,…
• Handle noise and missing values
• Integration with MARL and neural-network-based methods
• Learning conflicts between actions and collaborative
• Distributed learning

….

Beyond the Common Problem Setting

• States include Boolean state variables
• States also include numeric state variables
• Deterministic effects and full observability
• Stochastic effects, partial observability
• Single agent
• Multiple agents acting concurrently
• Offline learning
• Active learning: must act to collect observations

29

Online Domain Model Learning

​

​

​

​

​

​

30

Action

State

• Choose actions
• Process observations
• Learn/update domain model

Learning Agent

Online

Learning

Active

Learning

Goal-Literal Babbling (Chitnis et al. ‘21)

31

Action

State

• Choose actions
• Process observations
• Learn/update domain model

Learning Agent

• Prioritize novel “goals”
• Filter impossible goals

Online Learning of Action Models [Lamanna et al. ‘21]

​

​

32

Action

State

• Choose actions
• Process observations
• Learn/update domain model

Learning Agent

Online Learning of Action Models [Lamanna et al. ‘21]

33

Online Learning of Action Models [Lamanna et al. ‘21]

34

What is not informative?

and we won’t learn anything new

Online Learning of Action Models [Lamanna et al. ‘21]

35

Action

State

• Choose actions
• Process observations
• Learn/update domain model

Learning Agent

Halt when no such (s,op) pair exists

Online Learning of Action Models [Lamanna et al. ‘21]

☺ Converge to a sufficient domain model

• Sufficient = safe and complete
• Requires very few iterations to halt
• Utilizes failed actions

36

Many open directions:

• Optimize for limited budget
• Choice of returned model
• Richer domains
• Handle noise and missing values

….

Agent Interrogation Algorithm (AIA) [Verma et al. ‘21]

​

​

​

​

• Objective: understand the AI agent
• Approach: learn its capabilities
• Iterate over “pal-tuples” (p=predicate, a=action, l=pre. or eff.)
• Generate a disambiguating plan (to check if pal-tuple is true)
• Reduce this problem to a planning problem
• Filter candidate models as needed

37

Extended to support user-AI vocabulary differences (Verma et al. ‘22)

Optimistic Exploration (Sreedharan & Katz ‘23)

• Maintain an optimistic model
• Goal-oriented exploration
• Encourage exploration by using a diverse planner

38

Optimistic Exploration (Sreedharan & Katz ‘23)

• Maintain an optimistic model
• Goal-oriented exploration
• Encourage exploration by using a diverse planner

39

Used train an off-policy

Model-free RL algorithm

(Q Learning)

Numeric Online Action Model Learning [Mordoch et al. ‘26]

• NOAM: Online learning of a numeric action model
• Maintains safe and optimistic action models
• Goal-oriented exploration

40

Numeric Online Action Model Learning [Mordoch et al. ‘26]

• NOAM: Online learning of a numeric action model
• Maintains safe and optimistic action models
• Goal-oriented exploration

41

Many open directions:

• Scaling to larger domains
• Optimize for cumulative regret
• Richer domains
• Handle noise and missing values

….

Summary and What’s Next?

• Beyond learning classical planning domains - not much work …
• Mostly PlanMiner, NSAM, MA-SAM,learning RDDL models…
• Some work on online learning of domains
• OLAM, AIA, Optimistic Exploration, Glib, NOAM,…
• Common approach:
• Learn an optimistic model
• Plan to do more informative actions

​

Many, many direction for future work

42

Online representation and action model learning

Richer Domains

(H)RL-integrations

(some work on this)

LLM-integrations (some work on this)

Theoretical foundations

Applications: is it really worth it?

What’s Next – Lunch!

43

Thank you!

Link to feedback form:

please help us get better