LLM Agents, Agent Memories, and Harness Technology

Xinyi Fan

COMPUTER SCIENCE

UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN

​

August 9, 2026

1

1

Outline

• LLM Agents: A General Introduction
• Early Agents: LLM External Tools
• LLM Search Agent
• LLM Agent Memory
• Some Recent Research on Agents

​

2

LLM Agents

• LLM agent: A (semi)-autonomous system in which an LLM serves as the central decision-making component, coordinating reasoning, tool use, memory, and actions to achieve goals
• LLM agents represent a major evolution from traditional LLMs. Instead of functioning solely as next-token predictors, agents are designed to perceive, reason, plan, act, and learn while interacting with external environments, tools, humans, and other agents.

• The emergence of LLM agents has transformed LLMs from passive assistants into active problem solvers capable of handling complex, long-horizon tasks.

Figure from Xi, Z. et al. (2023). The Rise and Potential of Large Language Model Based Agents: A Survey.

3

A Brief History of LLM Agents

• Pre-LLM Intelligent Agents: Possessed planning and action capabilities but lacked flexible natural language reasoning.
• Research roots can be traced to: Classical AI planning, Reinforcement learning agents, Belief-Desire-Intention (BDI) agents, Autonomous software agents, Multi-agent systems, Robotics agents
• Tool-Augmented LLMs: Set the foundation of agent architectures
• Early milestones: Toolformer, ReAct, WebGPT
• Key idea: LLMs can Think, Invoke tools, Observe results, Continue reasoning
• Autonomous Agents
• 2023 witnessed the emergence of: AutoGPT, BabyAGI, MetaGPT, CAMEL
• These systems demonstrated: Goal decomposition, Long-horizon execution, Self-reflection, Multi-agent collaboration
• Emerging Research Frontiers (2025–2026)
• Agentic RAG / Reasoning-Acting Integration / Multi-Agent Collaboration / Long-Term Memory Agents / Computer-Use Agents / Embodied Foundation Agents / Self-Improving Agents

​

4

LLM-based Autonomous Agent: Architecture Design Framework

A unified framework for the architecture design of LLM-based autonomous agent.

From: L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin et al., “A survey on large language model based autonomous agents,” arXiv:2308.11432.

• Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al., “The rise and potential of large language model-based agents: A survey,” arXiv:2309.07864, 2023.
• Z. Durante, Q. Huang, N. Wake, R. Gong, J. S. Park, B. Sarkar, R. Taori, Y. Noda, D. Terzopoulos, Y. Choi, K. Ikeuchi, H. Vo, L. Fei-Fei, and J. Gao, “Agent ai: Surveying the horizons of multimodal interaction,” arXiv:2401.03568

Other survey papers on LLM agents

5

Research Challenges in Agent Technology

• Reliability
• Agents still: Hallucinate // Misplan // Misuse tools
• Long-Horizon Tasks
• Performance degrades as: Task Length ↑ Success ↓
• Memory Management
• Problems: Memory pollution // Retrieval errors // Context overload
• Cost and Efficiency
• Current agent systems often require: Many LLM calls // Long reasoning traces // Multiple tools
• Safety and Alignment
• Risks include Autonomous harmful actions // Tool abuse // Security vulnerabilities // Deceptive behavior
• Evaluation Crisis
• The field lacks: Standardized benchmarks // Reproducibility // Real-world evaluation protocols

6

Outline

• LLM Agents: A General Introduction
• Early Agents: LLM External Tools
• LLM Search Agent
• LLM Agent Memory
• Some Recent Research on Agents

​

7

LLM External Tool: ART

B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro, “Art: Automatic multi-step reasoning and tool-use for large language models,” 2023.

• An LLM can access external tools (e.g., external functions or services) to augment its functionality
• Several tool-aware prompting approaches have been developed to make usage of tools more scalable
• Automatic Multistep Reasoning and Tool-use (ART): Combines automated chain of thought prompting with the use of external tools

• Given a task and input, the system first identifies similar tasks from a task library.

• These tasks are then used as examples in the prompt, guiding the LLM on how to approach and execute the current task

• Effective when tasks require a combination of internal reasoning and external data processing or retrieval.

8

LLM External Tool: Toolformer

• Toolformer: Train an LLM to decide what tool to use, when, and how to use it (even what parameters the API needs)
• Toolformer is built upon a pre-trained GPT-J model with 6.7B parameters. Subset of CCNet as language modelling dataset. It incorporates various tools: a calculator, a QA system, a search engine, a translation system & a calendar
• Toolformer enables language models to learn tool usage in a self-supervised manner in 3 steps

T. Schick, J. Dwivedi-Yu, R. Dess`ı, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” 2023

• Sampling API calls using in-context learning
• Executing and filtering these API calls
• Fine-tuning the model on the filtered API calls

Pls see Aditya Sinha’s class presentation for more detail

Examples of using Toolformer

9

Tools Play a Magic Role at Mitigating Hallucination

• Hallucination: LLMs can produce very plausible but untruthful answers

Q: “List the title, venue and authors of highly cited papers on heterogeneous information network”

A: “Heterogeneous Information Network Analysis and Mining: A Comprehensive Survey”, by Jiawei Han, Micheline Kamber, and Jian Pei, KDD 2011 (cited over 4,300 times as of March 2023), ….

• There is no easy way for prompting or chain-of-thought to do it right!
• But ChatGPT gives a perfect and comprehensive answer in June 2026!
• What is the magic?
• Exploring the power of tools like Google Scholar

​

10

Outline

• LLM Agents: A General Introduction
• Early Agents: LLM External Tools
• LLM Search Agent
• LLM Agent Memory
• Some Recent Research on Agents

11

The Landscape of Agentic Tasks: � Where LLM Agents Are Deployed Today

Agentic coding

edit, run, and debug code across a repo

Computer / browser use

click, type, and navigate real GUIs

Search & deep research

gather and synthesize evidence

Data analysis

query, transform, and chart data

Workflow automation

orchestrate tools and APIs

Scientific discovery

propose and test hypotheses

Common thread. Everyone is a loop: The agent gathers context, decides an action, observes the result, and repeats.

12

Anatomy of an Agentic Task: Coding

Coding: THE GATHER → PLAN → ACT → OBSERVE LOOP

Task

“fix the failing test”

Agent (LLM)

Gather

read files, run grep

Plan

decide the next edit

Execute

edit code, run tests

Observe

read output & errors

repeat until the test passes

Environment

file system

shell

test runner

linter

Claude Code

Cursor

Codex

The agent’s real skill is managing context—deciding what to read, keep, and act on at each step (search!)

13

What Fills the Context Window

INSTRUCTIONS

system prompt, rules, tool definitions

CAPABILITIES

tools, MCP servers, sub-agents, skills

STATE

the conversation, files, and a running summary

Context composition of a coding agent (Cursor).

AN AGENT STEP IS MOSTLY CONTEXT ENGINEERING

14

LLM vs. Naive RAG vs. Agent: Three Levels of Access to the World

LLM

Answers from parametric memory only

No external access

Fast, but frozen & can hallucinate

(Naive) RAG

Retrieves once, then answers

Single-shot external context

Better grounding, but static retrieval

Agent

Decides when & what to retrieve, in a loop

Multi-step tool use + reasoning

Gathers, verifies, and acts until done

15

Agentic Search vs. Single-Shot RAG: Search Becomes a Decision, Repeated

Single-Shot RAG

Query

Retrieve (once)

Generate

Answer

Agentic search

Reason

Search?

Retrieve

Read & verify

Answer

loop: search more until the evidence is enough

The rest of this part: how agents learn to run this loop well — first without RL (prompting & reflection), then with RL.

16

ReAct: Reasoning × Acting

Non-RL Foundations · Interleave Thoughts with Tool Actions

LLM policy

Thoughtₜ: reason

Actionₜ: act

Environment

search[entity]

lookup[string]

finish[answer]

action

observation

repeat the Thought → Action → Observation cycle until finish[answer]

Example trace

Thought I need the composer�of the opera…�Action search[opera]�Obs …premiered 1902…�Thought now find the�composer…�Action lookup[composer]�Obs …by Debussy…�Action finish[Debussy]

Key idea. Reasoning and acting in one trace — reasoning plans the next action; observations ground the next thought. Pure prompting, no training.

Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” ICLR 2023. (Original schematic.)

17

Self-RAG: Retrieve, Generate, Critique

Non-RL Foundations · A Model That Reflects With Special Tokens

Input

Retrieve?

(token)

no

generate

directly

yes

Retriever

K passages

for each passage (parallel)

ISREL relevant?

generate segment

ISSUP supported?

ISUSE useful?

Critique-guided

beam search

select best segment

next segment

Four reflection tokens (added to the vocabulary)

Retrieve on-demand ISREL relevance ISSUP support ISUSE utility

Trained offline

GPT-4 labels → critic model → augments corpus → generator learns to emit the tokens itself.

Asai et al., “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” ICLR 2024.

18

FLARE: Forward-Looking Active Retrieval

Non-RL Foundations · Retrieve Only When The Model Is Unsure

Generate a

temporary next

sentence

any token

prob < θ ?

no

accept the

sentence

yes

Form a query

mask low-conf /

ask a question

Retrieve

(BM25 / Bing)

Regenerate

the sentence

continue to the next sentence

Joe Biden attended the University of ____ ← low-confidence span triggers a search

The model’s own look-ahead becomes both the trigger and the query.

Two variants. FLARE-direct uses the sentence itself; FLARE-instruct prompts the model to emit [Search(q)]. Training-free.

Z. Jiang et al., “Active Retrieval Augmented Generation (FLARE),” EMNLP 2023. (Original schematic.)

19

Search-o1: Agentic Search in the Reasoning Chain

Non-RL Foundations · A Reasoning Model That Searches Mid-thought

Reasoning chain

large reasoning

model (o1-style)

hits a knowledge gap

<|begin_search_query|>

Retriever

(Bing)

→ documents

Reason-in-Documents

condense docs using

query + prior reasoning

→ refined knowledge

Inject &

continue

reasoning

<|begin_search_result|>

search as many times as needed, then answer

The novelty: the Reason-in-Documents step distills long retrieved text into a concise fragment, so external knowledge enters the chain without breaking its coherence.

Li et al., “Search-o1: Agentic Search-Enhanced Large Reasoning Models,” EMNLP 2025. (Original schematic.)

20

Non-RL Agentic Search: A Summary

Smarter Loops Over a Frozen Model — Then RL

Method

Core idea

How

Signature

ReAct

interleave thought + action

prompting

search/lookup/finish

Self-RAG

retrieve & self-critique

trained tokens

Retrieve / ISREL / ISSUP / ISUSE

FLARE

retrieve when unsure

prompting

confidence threshold θ

Search-o1

search inside the reasoning chain

prompting (LRM)

Reason-in-Documents

The limitation. All of these make a frozen model search more cleverly — the model never gets better at searching.

Next: use reinforcement learning to train the policy itself — so the agent learns search strategies that transfer.

21

Two Ways to Make an Agent Self-Evolve: �Change the Weights, or Accumulate Experience

Parametric Evolution

update θ

Post-train the policy (RL, DPO, SFT) so the agent gets durably better at the task.

Capability baked into the weights

Strong, persistent gains — the engine of search agents

Costly; risks catastrophic forgetting

DeepRetrieval · Search-R1 · s3 · Harness-1

Experience I/O

memory & skills

Keep weights frozen; write & read an external store of experience the agent grows over time.

Capability lives outside the weights

Cheap, modular, continual — no retraining

Bounded by the frozen model’s ability

dynamic stores · reflective memory · skill libraries

Framing: Jiang et al., “Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills,” 2026.

22

Parametric Evolution: The Post-Training Pipeline

Where Reinforcement Learning Enters

Pre-training

next-token prediction on web-scale text

broad knowledge,

no task alignment

Supervised

Fine-Tuning

imitate curated demonstrations

follows instructions,

learns formats

Alignment:

RL / Preference

optimize against a reward signal

RLVR — our focus

Key idea. Pre-training and SFT teach broad ability; RL then optimizes behavior against an outcome — which is exactly what lets us train an agent to search well.

Ouyang et al., “Training LMs to follow instructions with human feedback,” NeurIPS 2022.

23

From RLHF to RLVR: Replace the Reward Model with a Verifiable One

RLVR reward

A rule-based, checkable reward (is the answer correct? does the code pass?) — no reward model needed.

DeepSeek-R1-Zero. Pure RL with verifiable rewards elicited emergent multi-step reasoning — the recipe behind RL search agents.

Why it matters for search. Retrieval recall and answer correctness are verifiable — perfect RLVR rewards.

Guo et al., “DeepSeek-R1,” 2025. RLHF → RLVR

24

The RL Algorithms: PPO & GRPO: How the Policy Is Actually Updated

PPO — actor + critic

Clipped objective with a learned value network (critic); stable but heavier.

GRPO — critic-free

Drop the critic: normalize rewards across a group of samples. Lighter; popularized by DeepSeek-R1.

For search agents: either optimizer works — what matters is the reward. What reward makes an agent search well?

Schulman et al., “PPO,” 2017. Shao et al., “DeepSeekMath (GRPO),” 2024

25

Why RLVR Matters to Retrieval? �― Retrieval Is Verifiable; Rewriting Is Exploration

Key idea. Recall is a checkable reward, and rewriting the query is an exploration problem, not a supervised one — reward rewrites by how well they retrieve.

P. Jiang, et al., “DeepRetrieval,” COLM 2025.

26

DeepRetrieval: Learn to Search by Exploration―Method

The LLM reasons, then emits an augmented query; the retrieval metric (recall / NDCG) is the RL reward.

No golden queries. A 3B policy discovers search strategies by trial-and-reward against the real engine.

P. Jiang, et al., “DeepRetrieval: Hacking Real Search Engines and Retrievers with LLMs via RL,” COLM 2025.

27

DeepRetrieval: Learn to Search by Exploration―Results

Recall on literature search (left) and across QA benchmarks (right) — the 3B policy leads.

P. Jiang, et al., “DeepRetrieval: Hacking Real Search Engines and Retrievers with LLMs via RL,” COLM 2025.

28

Search-R1: End-to-End RL for Reasoning + Search―Method

Reward = answer EM. The policy interleaves reasoning and search; retrieved tokens are masked from the loss.

B. Jin, et al., “Search-R1: Training LLMs to Reason and Leverage Search Engines with RL,” COLM 2025.

29

Search-R1: Accuracy Across QA Benchmarks―Results

Search-R1 (RL) beats inference, RAG, and SFT baselines on single- and multi-hop QA.

B. Jin, et al., “Search-R1: Training LLMs to Reason and Leverage Search Engines with RL,” COLM 2025.

30

A Wave of RL Search Agents

Reasoning × Search

R1-Searcher

two-stage RL to incentivize autonomous search

Song et al., 2025

ReSearch

learn to reason while searching via RL

Chen et al., 2025

DecoupleSearch

separate reasoning & search policies

2025

Search Environments

ZeroSearch

train against a simulated LLM search engine

Sun et al., 2025

DeepResearcher

end-to-end RL in real web environments

Zheng et al., 2025

WebDancer / Sailor

autonomous deep-web research agents

Tongyi, 2025

Reward & Process

StepSearch

step-level process rewards for multi-hop

2025

Atom-Searcher

fine-grained atomic-thought rewards

2025

Survey → AI Search

a fuller map of RL-based search

survey, 2025

See references for the full landscape.

31

s3: Optimize the Searcher, Freeze the Generator―Method

Gain Beyond RAG (GBR)

Reward the searcher only when its context helps a frozen generator beat naive RAG.

Why not just EM (Exact Match)?

EM penalizes correct-but-paraphrased answers and entangles search quality with the generator. GBR is robust.

P. Jiang, et al., “s3: You Don’t Need That Much Data to Train a Search Agent via RL,” EMNLP 2025.

32

s3: Optimize the Searcher, Freeze the Generator―Results

P. Jiang, et al., “s3: You Don’t Need That Much Data to Train a Search Agent via RL,” EMNLP 2025.

With ~70× less training data, s3 leads on general-domain QA and transfers to medicine with no medical training.

Decoupling + a help-based reward = efficiency and transfer — you don’t retrain for every new domain.

33

The Long-Horizon Search Problem

One policy is asked to do everything

Plan — a multi-step search over many turns

Remember — every document it has retrieved

Rank — which findings actually matter

Compress — so the context does not overflow

Verify — whether each claim is supported

Stop — decide when the evidence is enough

As the transcript grows…

the model spends its attention rebuilding its own memory every turn instead of deciding what to do next.

For RL, the reward becomes poorly conditioned — one diluted signal stretched across 40+ turns is too weak to learn from.

The fix. Move the bookkeeping out of the model and into the environment.

P. Jiang, et al., “Harness-1: A Stateful Harness for Training Long-Horizon Search Agents,” 2026.

34

Formalism: The (PO)MDP (What “Environment” Means Precisely)

Markov Decision Process

States 𝒮, actions 𝒜, transition P, reward R, discount γ

Markov property: the next state depends only on the current state and action

Goal: a policy maximizing long-run reward

Partial observability (POMDP)

The agent sees an observation oₜ, not the full state

Must infer & remember — a belief over states

LLM agents live here: the context window is their observation, and memory is how they cope

Why it matters. Long-horizon search is a POMDP — the agent never sees “all the evidence” at once, so what it remembers becomes the bottleneck.

35

The Idea: Split the Two Jobs (Stateful Cognitive Offloading)

POLICY

the LLM

keeps the decisions

what to search for

what to keep

what to verify

when to stop

HARNESS

the environment

keeps the state

candidate pools

curated evidence

entity links & graph

verification, history & budget

split

The paper calls it stateful cognitive offloading: free-form reasoning stays in the model; bookkeeping moves to the harness.

P. Jiang, et al., “Harness-1,” 2026. Policy: gpt-oss-20b.

36

The Harness: Working Memory

P

Candidate pool

every document retrieved so far

C

Curated set

kept docs, tagged by importance · cap 30

G

Evidence graph

entities bridging documents

V

Verification

claims checked against source text

Z

Compression

observations deduplicated & shrunk

B

Budget render

kept within the context limit

The model keeps only the decisions → search · keep · verify · stop

P. Jiang, et al., “Harness-1,” 2026 (harness working memory).

Harness: Search State, Kept by the Environment

37

One Turn: the Policy Decides, the Harness Remembers

(State, Action) → (State′, Observation)

POLICY

20B model

reasons + decides

emits 1 action / turn

HARNESS

environment-side working memory

Candidate pool

all retrieved

Curated set

cap 30 · evict lowest

Evidence graph

bridges + hops

Verification

yes / no vs claim

Compression

top-4 BM25 · dedup .85

Budget render

30,720 tokens

action — curate{add, importance:high }

observation—working memory + recent turns

8 actions / 5 classes:

Retrieve

fan_out · search · grep

Inspect

read · review

Curate

add / remove · tag

Verify claim

End

submit set

P. Jiang, et al., “Harness-1,” 2026. Free-form reasoning stays in the model; bookkeeping moves to the harness.

38

What the Model Sees: Not A Transcript — A Structured State

== Working Memory · summarizing turns 0–12 ==

Query�“Which Brussels synagogue, completed in 1878,� was designed by Désiré De Keyser?”��Curated Set (14 / 30) ↺ auto�very_high 22816 Grande Synagogue ✓ verified�high 91442 Désiré De Keyser, architect�high 62390 Brussels synagogue register�fair 88114 19th-c Brussels architecture�low 30119 Belgian heritage evicts first at 30��Document Pool — 31 total · 17 uncurated�[ ] 99012 synagogues of Belgium [ ] 50441 …

[Evidence Graph] bridges ↺ auto�Brussels → 22816, 62390, 88114, 91442�1878 → 22816, 91442, 62390�De Keyser → 22816, 91442�bridge docs: 22816, 91442 · 5 singletons��Verification ↺ auto�claim: 1878 · De Keyser · Brussels synagogue�↳ 22816 yes — states 1878, De Keyser�↳ 62390 no — architect unnamed��Search History ↺ auto�T9 fan_out_search → 11 new · +6 curated�T10 grep_corpus “De Keyser” → 3 hits�T11 verify → 1 yes, 1 no��[Context 21,402 / 30,720]

Six kinds of state — extracted, ranked, verified, deduped, budgeted — re-derived for the model every single turn.

P. Jiang, et al., “Harness-1,” 2026 (rendered working-memory state).

39

The Recipe: Make Search Trainable � (SFT Teaches the Interface · RL Teaches the Decisions)

1 · Supervised warm-start teacher acts in the harness → 899 demos (recall ≥ 0.10)

2 · On-policy RL CISPO, terminal reward, 40-turn cap → 3,453 queries

01

Warm-started curation

The first good search seeds the set with its top 8 — the model is always editing, never starting from a blank page.

02

Compact state

Importance tags, the evidence graph, and verification records — all rendered small enough to fit the budget.

03

Diversity incentives

Reward a rhythm, not just discovery: search → curate → review → verify.

The reward shapes a small, complete, verified curated set (recall weighted 4×) — and rewards the search rhythm, not just raw discovery.

P. Jiang, et al., “Harness-1,” 2026 (training recipe & reward).

40

Results: Evidence Recall on 8 Hard Benchmarks

AVERAGE CURATED-EVIDENCE RECALL (%)

Opus-4.6 frontier

76.4

Harness-1 20B · ours

73.0

GPT-5.4 frontier

70.9

Sonnet-4.6 frontier

68.8

Kimi-K2.5 frontier

64.7

Tongyi DR 30B · best open

61.6

Context-1 20B

60.3

GPT-OSS 120B

49.6

Search-R1 32B

28.9

GPT-OSS-20B our base

26.2

+11.4

vs. best

open agent

A 20B model matches the frontier. Only Opus-4.6 scores higher; the harness + RL add +46.8 recall over the untrained base (26.2 → 73.0).

P. Jiang, et al., “Harness-1,” 2026. Average over 8 difficult evidence-retrieval benchmarks.

41

Trained on About 4k Examples: Need Much Less Training Data

Training examples

Harness-1

4,352

A typical search agent (Search-R1)

221,328

≈50×�less data

Most of the behavior lives in the interface, not in the weights. The harness does the remembering; RL only has to learn the decisions.

P. Jiang, et al., “Harness-1,” 2026. 4,352 = 899 SFT + 3,453 RL.

42

Transfer to Unseen Environments

The Biggest Gains Appear Where It Never Trained

Recall improvement over the base model (pts)

Source-family benchmarks

+7.9

Held-out transfer benchmarks

+17.0

2.2 × larger improvement on unseen tasks than Context-1

It didn’t memorize domains — it learned a reusable search workflow: plug in your corpus, retriever, and verifier, and the trained policy operates the same interface.

The Lesson Learned from Harness-1

Don’t just train a bigger brain on a thin interface. Shape the interface itself — move the bookkeeping out, and the learned search behavior transfers to new tasks and environments.

small model

20B

+

stateful harness

working memory

=

frontier search

0.730 recall

P. Jiang, et al., “Harness-1: A Stateful Harness for Training Long-Horizon Search Agents,” 2026.

43

Experience I/O: Memory & Skills

Self-evolving without Touching the Weights

Agent

(frozen weights θ)

act in the

environment

Experience store (memory + skills)

read it back next time — the agent improves by accumulating experience

Memory

what the agent remembers

Facts, past episodes, and self-reflections

An external store the agent reads & writes

Grows continually across tasks

Skills

what the agent can reuse

Reusable procedures & workflows

Distilled once from experience, reused often

Accumulate into a growing skill library

Framing: “Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey,” 2026.

44

A Taxonomy of Agentic Memory: How Agents Store & Reuse Experience

Working / short-term

The context window: what the agent is actively attending to right now.

Long-term stores

Read / write an external memory the agent manages itself.

e.g. MemGPT, Mem0

Episodic & reflective

Store outcomes and self-reflections to improve later attempts.

e.g. Reflexion, Generative Agents

Semantic / structured

Organize memory as a graph, tree, or database for retrieval.

Parametric / hybrid

Fold memory into the weights, or mix parametric + external.

Test-time curation

Decide on the fly what to keep, and how important it is.

e.g. Harness-1

Categories follow “Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey,” 2026. (Original schematic.)

45

Memory as a Learnable Skill: Curate The Memory — Don’t Just Store It

Memory operations are decisions

What to write, keep, evict, and retrieve are choices the agent makes — so they can be learned, not hand-coded.

write: what is worth remembering?

evict: what can be forgotten?

retrieve: what to surface for this step?

Test-time curation

The agent decides during the task what enters memory and at what importance — keeping it small and high-signal.

Bounded memory → fits the context budget

Importance tags rank what survives

Self-editing: memory the agent rewrites

We have already seen this

Harness-1’s curated set with importance tags is exactly a learned, test-time-curated memory: the policy chooses what to curate, and importance eviction is a memory-management skill trained by RL.

Connects Harness-1’s learned curation to the memory-as-skill view.

46

Agent Skills & Skill Libraries: Reusable Know-how the Agent Accumulates

A skill = a reusable procedure the agent distills once and reuses many times. Skills accumulate into a growing library the agent can search and apply — capability without retraining.

Voyager

An LLM agent in Minecraft writes executable code skills and stores them in a self-growing library it reuses for harder tasks.

Wang et al., 2023

Agent Workflow Memory

Induces reusable workflows from past trajectories, then applies them to new tasks — memory of how, not just what.

Wang et al., 2024

Claude Skills

Procedural skill files loaded into context on demand — exactly the ‘Skills’ slice of the context budget we saw earlier.

Anthropic, 2025

Full circle. On the context slide, Skills was a slice of the prompt budget — here is what fills it: distilled, reusable procedures.

Examples representative; framing follows the memory & skills survey, 2026.

47

Outline

• LLM Agents: A General Introduction
• Early Agents: LLM External Tools
• LLM Search Agent
• LLM Agent Memory
• Some Recent Research on Agents

​

48

A Multidimensional View of Agent Memory Research

• Substrates
• External memory
• Internal memory
• Cognitive Mechanism
• Sensory memory
• working memory
• episodic memory
• semantic memory
• procedural memory
• Subjects
• User-centric
• Agent- centric

Wei-Chieh Huang, et al, "Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey", ArXiv:2602.06052

• Operational Architectures
• Single-agent
• Multi-agent
• Learning Policy
• Prompt-based
• Fine-tuning
• Reinforcement Learning
• Evaluation
• Accuracy-based
• Similarity-based
• LLM-as-a-Judge

49

Adapted from Wei-Chieh Huang, et al, "Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey", ArXiv:2602.06052

50

Memory Substrates: External vs. Internal Memory

• External memory: Store the knowledge, info, and experience outside the agent model’s parameter/state
• Types: vector index, text-record, structural store, and hierarchical store
• Adv: A clear separation between LLM’s internal parameters and the external knowledge
• Enable scalable, easy-to-update, cross-session retention of knowledge and interaction history
• Can preserve past info in its original form, helping reduce hallucinations or knowledge cutoff
• Drawbacks: Retrieval can be unreliable; costs of storage and indexing, need transformation
• Internal memory: Store directly within the model’s architecture (in parameters/working states)
• Knowledge/experience embedded in parameters (pre-training, post-training, or parm editing)
• Weight updates by continual learning, model editing, and distillation
• Latent-State: the environment holds not only parameters but also intermediate states in memory
• Use KV Cache: Cache per-layer attention keys and values from previous tokens during decoding
• Compress the KV cache: e.g., balance “heavy hitters” with recently generated tokens, identify consistent attention patterns, retain only the most informative vectors

​

51

Memory Cognitive Mechanisms

• Cognitive psychology distinguishes multiple interacting systems that explain how information is perceived, maintained, and reused
• Five atomic systems: Sensory, working, episodic, semantic, and procedural memory

Long-Term Memory

Short-Term Memory

52

Memory Operation Mechanism

53

Memory Subjects: User-Centric vs. Agent-Centric

• Memory subjects characterize who the memory is primarily modeling and serving for
• User-centric memory: An abstraction of user-specific facts and preferences
• Ex. Memory management in dialogues; persistent user simulation; long-term personalization; privacy-preserving memory.
• Agent-centric memory: distilled knowledge, skills, and operational task priors; supports long-context, long-horizon, and long-running tasks across real-world environments.
• Ex. Long-Horizon Tasks (key intermediate states); Domain-Specific Long-Tail Solutions (retention of the rare insights and knowledge); Cross-Task Knowledge Transfer (task-agnostic cognitive knowledge from diverse interactions); Strategy and Skill Learning (Retain environment-grounded procedural memories)
• Working memory, procedural memory, and sensory memory are predominantly agent-centric, reflecting their roles in supporting many real-world downstream tasks
• Semantic and episodic memory appear in both user- and agent-centric settings
• For users, they encode stable preferences and interaction histories
• For agents, they support world modeling and experience accumulation.

54

Memory Learning Policy

Learning policy refers to how an agent learns to manage memory, what to store, when to store it, how to represent it, when to retrieve or discard it, and where to store or retrieve, rather than relying on fixed, hand-crafted heuristics. Such policies are typically optimized from data or feedback (e.g., supervised signals, reinforcement learning, or self-improvement)

55

Memory Learning Policy: Prompt-based, Fine-Tuning vs. RL

• Prompt-based Memory Learning: Parameterizes the memory policy as natural language prompts.
• Static Prompt-based Control: Human-designed rules, invariant during execution
• Design targets: Static memory OS and organization; single-agent vs. multi-agent systems
• Dynamic prompt-based control: Adaptable at test time based on experience and feedback
• Methods on memory usage policies; memory representations (but lack explicit credit assignment, limiting their capacity for long-term policy optimization compared to fine-tuning and RL-based)
• Fine-Tuning: Parameterized Memory Policies
• Policy Internalized into parameters: Parameterized policy stabilization and boundary control; parameterized policy efficiency and retrieval refinement
• Reinforcement Learning for Memory Policies: Optimized through interaction and reward feedback
• Step-Level Memory Decisions: based on the immediate or short-horizon impact on task reward
• Trajectory-Level Memory Representation: View memory as part of the agent’s Markov state, whose quality is assessed through downstream decision performance
• Cross-Episode and Multi-Agent Memory: Distill higher-level decision-relevant knowledge; extending across agents or representation spaces

56

Applications of the Foundation Agent Memory System

Memory transforms LLMs into dynamic, persistent agents, representing a fundamental shift in recent research.

​

When implemented in complex real-world scenarios, agentic memory has emerged not merely as a storage utility, but as the cognitive substrate that enables continuity, learning, and personalization, bridging an agent’s past experiences with its future actions.

​

Recent work has broadly investigated memory-enabled capabilities in LLM agents where the ways of storing, operating, and managing memory vary significantly.

​

We summarize recent representative works across education, scientific research, gaming and simulation, robotics, healthcare, dialogue systems, software engineering, and workflow automation.

57

Future Directions and Challenges of Agent Memory Systems

58

Outline

• LLM Agents: A General Introduction
• Early Agents: LLM External Tools
• LLM Search Agent
• LLM Agent Memory
• Some Recent Research on Agents

​

59

ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory

• A key limitation of LLMs on continuous streams of tasks: failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors
• ReasoningBank, a memory framework, distills generalizable reasoning strategies from an agent’s self-judged successful and failed experiences (store high-level strategies and reasoning hints)
• At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time
• MaTTS (memory-aware test-time scaling) accelerates and diversifies this learning process by scaling up the agent’s interaction experience

Siru Ouyang, et al., “ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory” ArXiv: 2509.25140

• Learning from past experiences as memory
• Developing self-evolving agent systems

60

Overview of ReasoningBank

• Experiences are distilled into structured memory items with a title, description, and content
• Title: a concise identifier summarizing the core strategy or reasoning pattern
• Description: a brief one-sentence summary of the memory item
• Content: The distilled reasoning steps, decision rationales, or operational insights extracted from the past
• For each new task, the agent retrieves relevant items to interact with the environment and constructs new ones from both successful and failed trajectories
• These items are then consolidated into ReasoningBank, forming a closed-loop memory process

Integration of ReasoningBank with Agents: recall effective insights, avoid previously observed pitfalls, and adapt more robustly to unseen queries via (i) memory retrieval, (ii) memory construction, and (iii) memory consolidation

61

MaTTS: Memory-aware Test-Time Scaling

Memory-aware Test-Time Scaling: Translate more experiences into greater Improvements

• Vanilla TTS (direct combination): No aggregation ,no leverage of inherent contrastive signal
• MaTTS (parallel): Generate multiple trajectories for the same query under the guidance of retrieved memory items: Identify consistent reasoning patterns while filtering out spurious solutions
• MaTTS (sequential): Iteratively refine its reasoning within a single trajectory after the initial completion, following the principle of self-refinement. During this process, the intermediate notes generated in self-refinement are also used as valuable signals for memory, since they capture reasoning attempts, corrections, and insights that may not appear in the final solution.

62

Experiment: ReasoningBank on Web Arena Benchmark

• ReasoningBank consistently outperforms baselines across LLM backbones on all datasets

Success rate (SR ↑)

# of steps (Step ↓)

3 backbone LLMs

63

SkillOS: Learning Skill Curation for Self-Evolving Agents

Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, Maohao Shen, Vishy Tirumalashetty, George Lee, Jiawei Han, Tomas Pfister, Chen-Yu Lee, "SkillOS: Learning Skill Curation for Self-Evolving Agents", arXiv:2605.06614

• Traditional LLM-based agents are often one-off problem solvers that do not learn from past experiences, limiting their long-term adaptability and performance in streaming or sequential task settings.
• SkillOS: Enable agents to self-evolve by accumulating, refining, and reusing procedural knowledge (skills) from experience, thus improving future task performance
• SkillOS pairs a frozen Agent Executor with a trainable Skill Curator. The executor retrieves relevant skills from SkillRepo to act; the curator edits the repo (insert/update/delete) based on the resulting experiences
• Skill Representation: Each skill is stored as a Markdown file with: (1) YAML Frontmatter: contains the skill name and a concise, actionable description; (2) Markdown Body: provides instructions, workflows, constraints, and reusable heuristics.

​

​

64

SkillOS: Learning Skill Curation for Self-Evolving Agents

SkillOS training pipeline:

Each step samples a group of related tasks and initializes an empty SkillRepo. 𝜋S is optimized with composite rewards, enabling self-evolution.

• Experience-Driven RL Training: The skill curator is trained using RL, with feedback signals derived from the executor's performance on future related tasks.

Action Guidelines

1. Analyze the agent trajectory and its result. Identify what went well and what didn't.

2. If the trajectory is correct, extract reusable knowledge or skills. If it is incorrect, identify the failure point and extract skills that can help fix the issue.

3. Compare the extracted skills with past skills. Determine whether to insert a new skill, update an existing skill, or delete an existing skill using the following tools.

• Grouped Task Streams: Training instances are constructed as groups of related tasks, so skills learned from earlier tasks are evaluated by their impact on later tasks in the group
• Composite Reward Function: Task outcome reward; function call reward; compression reward; content quality reward

​

65

SkillOS: Performance Study

Experiment on ALFWorld benchmark. Success rate (SR ↑) and the number of steps (Steps ↓) are reported on 6 subsets with 3 different frozen executors.

Further performance study shows:

​

Effectiveness: SkillOS outperforms both memory-free and memory-based baselines in success rate and efficiency across multiple benchmarks (e.g., ALFWorld, WebShop, mathematical reasoning tasks).

Efficiency: Achieves higher task success with fewer interaction steps, indicating more targeted and actionable skill use.

Generalization: The trained skill curator transfers well across different executors and task domains, showing modularity and robustness.

Skill Evolution: Over time, SkillRepo evolves to contain more structured, higher-level meta-skills and actionable strategies, not just verbatim task trajectories.

66

PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents

• Existing memory design: Task-specific (non-transferable) or task-agnostic but less effective
• Observation: Decision-relevant info is concentrated as abstract knowledge
• PLUGMEM: A task-agnostic plugin memory module that can be attached to arbitrary LLM agents without task-specific redesign
• Representation: Structure episodic memories into a compact, extensible knowledge-centric memory graph that explicitly represents propositional and prescriptive knowledge

• Efficient memory retrieval and reasoning over task-relevant knowledge, rather than verbose raw trajectories
• Treat knowledge as the unit of memory access and organization instead of entities or text chunks
• Evaluation: Across three heterogeneous benchmarks
• PLUGMEM consistently outperforms task-agnostic baselines and exceeds task-specific memory designs, while achieving the highest info density under a unified information-theoretic analysis.

Ke Yang, Zixi Chen, Xuan He, Jize Jiang, Michel Galley, Chenglong Wang, Jianfeng Gao, Jiawei Han, ChengXiang Zhai, “PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents”, in ICML, July 2026

67

PlugMem Transforms Raw Episodic Memory into Structured, Knowledge-dense Representations

• Effective agent memory should actively transform raw episodic memory into structured, knowledge-dense representations
• Episodic memory (detailed records of experience) vs. knowledge-level memory (semantic memory (knowing that; factual propositions) & procedural memory (knowing how; action-oriented prescriptions))
• PlugMem operates on knowledge units (i.e., propositions and prescriptions) which form the fundamental units of memory access and manipulation
• vs. GraphRAG (operating on entities and relations)

​

​

PLUGMEM performs memory-to-knowledge abstraction & supports the unified management of multiple key memory types across agentic tasks

PlugMem consists of (1) a structuring module that standardizes heterogeneous raw memories and extracts propositional and prescriptive knowledge through hierarchical abstraction, organizing them into a memory graph; (2) a retrieval module that selects task-relevant subgraphs; and (3) a reasoning module that further adapts and compresses retrieved knowledge for the base agent

68

The Structuring Module of PlugMem

Methods:

• Standardize heterogeneous interaction traces into a unified episodic representation

​

​

​

​

The structuring module transforms heterog. memory into a formalized knowledge-dense memory graph: From (o: observation) to (g: subgoal, s:state, a:action, r:reward, s’)

• Structuring module: Standardize heterog. episodic memories and induce propositional and prescriptive knowledge
• Retain episodic traces as verifiable evidence

• Rep. a raw interaction trace as a sequence of observ.-action pairs: (o, a), then standardized as e_t = (o_t, s_t, a_t, r_t, g_t)
• Aggregating all standardized steps yields an episodic memory sequence [e_t]₁^T
• Induce propositional (semantic) & prescriptive knowledge that can be independently indexed and reused across tasks
• Use an LLM to extract a set of atomic propositions that describe salient facts implied by the interaction
• Each proposition is accompanied by a set of associated concepts, which serve as semantic tags for indexing
• Ex. “Tam Sventon, known in Swedish as Ture Sventon, is a fictional private detective based in Stockholm.”
• → {Tam Sventon, fictional private detective, Stockholm}
• To ensure extraction quality, use coreference resolution, proposition deduplication, and length control
• The extracted propositions and concepts are stored in a semantic graph GS

69

The Retrieval Module of PlugMem

• Procedural Memory: Extract reusable action strategies from episodic trajectories to support future decision making
• Segment the trajectory into coherent sub-trajectories by detecting boundaries
• For each trajectory segment, using LLM induce a compact (intent, prescription) pair

​

• The Structuring stage constructs an episodic graph G^E, a semantic graph G^S, and a procedural graph G^P
• Both G^S and G^P maintain explicit provenance links to G^E, enabling verifiable grounding info.
• Given a task description or query Q, an LLM-based retriever first determines which memory types to emphasize: episodic, semantic, or procedural.
• Retrieval begins by encoding Q into an embedding q and scoring it against all low-level nodes (i.e., proposition or prescription nodes) to initialize a candidate set C₀

Knowledge-centric memory graph design and the standard graph operations

Ex. of prescription: “To identify the lowest price of an item, search for the item using the search bar, sort the results by price and verify the minimum across variants.”

• Intent: the objective pursued within the segment; prescription: an environment agnostic action workflow that captures the key steps

​

70

PlugMem: The Reasoning Module and Overall Operations

• The reasoning module leverages the LLM to aggregate and condense such info into a compact, task-aligned representation, distilling the shared signal across messages into a single actionable summary
• Summary: PLUGMEM standardizes episodic memory, extracts semantic and procedural knowledge, organizes them into structured memory graphs, and enables retrieval and reasoning over stored experience to support downstream decision making
• At the system level, PLUGMEM supports 4 basic memory graph operations: i) create, which inserts newly observed episodic experience into structured memory, ii) retrieve, which retrieves relevant semantic, procedural, or episodic memory given a task or query, iii) update, which revises existing memory entries when new evidence becomes available, and iv) delete, which removes obsolete or low-utility memory.

• Reasoning module: Transform retrieved memory into immediately actionable guidance for the playing agent
• Retrieved memory may contain multiple overlapping or verbose descriptions of past interactions that are individually relevant but collectively redundant for the current decision

71

PlugMem: Performance Study

PLUGMEM consistently achieves a more favorable utility-cost trade-off, dominating prior approaches by providing higher decision-relevant utility under smaller memory budgets across benchmarks

Results on LongMemEval. #TokAvg. is the average length of memory tokens. Experiments use NV-Embed-v2 (abbreviated as NVE) as the embedding model for retrieval, and Qwen2.5-32B (Q32)/gpt-4o (4o) as base LLMs for structuring and reasoning.

Results on HotPotQA

Ablation Study on HotPotQA

Ablation Study on LongMemEva

72

Can Agent Memory Systems Track Evolving State?

• Existing memory benchmarks focus on recall-shaped tasks. But as facts, constraints, and decisions are revised over a long interaction, answers must reflect the current state, not a superseded one
• State tracking: maintaining currently operative values of facts, rules, and derived quantities under cross-session revision — a capability separable from recall
• State drift: the relevant fact is present in the assembled context, but the agent acts on a stale or incomplete version of it — the obstacle is not retrieving the fact, but tracking its relevance to current state

• Drift is already prevalent — and persists even under perfect retrieval
• On LongMemEval oracle (recall = 1.0 by construction), 44.4% of confirmed failures are state drift — the largest failure category under guaranteed evidence
• Drift leads the failure distribution on MemoryArena-shopping (63.5%) and τ²-bench-Z
• Enabling the reasoning trace does not help (84.0%→76.0%): drift is a state failure, not a reasoning shortfall

Xinyi Fan*, Miri Liu*, Ruozhen Yang, Siru Ouyang, Jiawei Han, “Can Agent Memory Systems Track Evolving State?”, arXiv preprint, 2026

73

StateMemBench: A Benchmark Targeted at State Tracking

• 234 multi-session scenarios (322 graded probes) across three domains — research, shopping, personal finance — in two length regimes: Set A (~165 turns) and Set B (~600 turns, three trap threads fused)
• Failure modes defined mechanically as policy divergence: replay a symbolic event program vs. “lazy reader” policies → status, salience, sequence, compound, and anti-trap modes
• Closed-pool grading: the superseded (drift) answer is a distinct scored outcome, separating badly-tracked answers from totally-incorrect ones; anti-update controls penalize an “always-prefer-the-latest” heuristic

Xinyi Fan*, Miri Liu*, Ruozhen Yang, Siru Ouyang, Jiawei Han, “Can Agent Memory Systems Track Evolving State?”, arXiv preprint, 2026

74

StateMem: A State-First Memory Method

• Represent conversational memory as structured state units: (id, content, priority, source, deps), with typed links (derived_from, coupled_with) recorded at ingestion
• Ingestion: a TurnEncoder parses each turn into state units and flags units at risk of supersession (one LLM call per turn)
• Update: apply supersessions; a deterministic Rechecker traverses the dependency graph and marks derived units needs_recheck — O(|E|), no LLM calls

• Test time: deterministically assemble the valid state, then one answer call with recompute guidance
• The deterministic layer decides which units are relevant and which are stale; the LLM only recomputes the answer given those flags
• Superseded units stay in the store for auditability but are withheld from answering

Xinyi Fan*, Miri Liu*, Ruozhen Yang, Siru Ouyang, Jiawei Han, “Can Agent Memory Systems Track Evolving State?”, arXiv preprint, 2026

75

StateMem: Performance Study

• On StateMemBench: 1.8× over the strongest same-backbone baseline on DeepSeek-V4-Flash (0.205→0.363) and 1.6× over the strongest memory system on Qwen-3.5-9B (0.149→0.233) — beating same-backbone long-context on both (0.149)
• Competitive on recall benchmarks (LongMemEval, LoCoMo): state tracking does not cost recall
• Ablations: supersession marking is the largest single step (0.174→0.298); recompute guidance adds ~6 pp

• StateMemWrapper: the same state approach as a single-call, answer-time wrapper lifts every backend it is applied to (+32 to +67 points across six memory / retrieval backends)
• A length- and cost-matched control attributes +15 to +32 of those points to state structure rather than added context
• Why systems drift: a capable answerer converts would-be drifts into gold once the operative value is explicit — StateMem’s drift-rate falls from 64.3% to 49.1%

Xinyi Fan*, Miri Liu*, Ruozhen Yang, Siru Ouyang, Jiawei Han, “Can Agent Memory Systems Track Evolving State?”, arXiv preprint, 2026

76

Meta-Harness: End-to-End Optimization of Model Harnesses

• Harness determines what info to store, retrieve, and present to the LLM model
• Yet still designed largely by hand
• Harness engineering: Refine the code around an LLM to improve the overall system’s performance
• Meta-Harness, an outer-loop; search over harness code for LLM applications
• Use an agentic proposer; access source code, scores, and execution traces of all prior candidates through a filesystem

Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn, “Meta-Harness: End-to-End Optimization of Model Harnesses“, arXiv: 2603.28052

Meta-Harness search loop: (1) An agent reads a filesystem containing all prior candidates’ source code, execution traces, and scores, and proposes a new harness; (2) we evaluate the proposed harness on evaluation tasks; and (3) all logs (proposed code, reasoning traces, evaluation scores) are stored in the filesystem, and the loop repeats.

• Meta-Harness: An agentic harness for optimizing harnesses via end-to-end search.
• Its proposer is a coding agent, i.e., a language-model-based system that can invoke developer tools and modify code.
• Its key design choice is to expose full history through a filesystem, enabling selective diagnosis of raw prior code and execution traces rather than optimization from compressed per-candidate summaries.

77

Meta-Harness: Performance Study

• These results show that richer access to prior experience can enable automated harness engineering

On text classification, Meta-Harness outperforms the best prior hand designed harnesses (ACE) and existing text optimizers (TTT-Discover, OpenEvolve), matching the next-best method’s final accuracy after just 4 evaluations.

On TerminalBench-2, Meta-Harness outperforms all reported Claude Haiku 4.5 harnesses.

78

OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

(Top) The fast loop lets the agent interact with environments and a four-level memory hierarchy;

(Down) the slow loop converts outcome-calibrated hindsight into on-policy self-distillation signals

• OPD-Evolver, a slow-fast co-evolution framework that cultivates an agent evolver through on-policy self-distillation
• Fast loop: interact with a 4-level memory hierarchy to read, use, write & maintain experience for rapid test-time evolution
• Slow loop: Outcome-calibrated memory attribution and privileged hindsight distill the 4 abilities into the deployable policy

Guibin Zhang, Xun Xu, Yanwei Yue, Zikun Su, Wangchunshu Zhou, Xiaobin Hu, Shuicheng Yan, “OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation”, ArXiv: 2606.17628

79

OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

• Task: Systematically transforms interaction history/feedback into persistent future improvements
• Unlike memory-augmented agents (store trajectories, reflections, tips, or lessons and inject them into later prompts); skill-augmented agents (distill experience into reusable strategies, tools, or procedures); or optimize only one fragment (e.g., retrieving experience)
• A qualified agent evolver should have 4 inter-coupled capabilities:

❶experience selection identifies useful memories from a growing and noisy repository;

❷ experience-grounded execution converts selected experience into effective multi-turn actions;

❸ experience writing extracts reusable knowledge from new trajectories and feedback; and

❹ experience management scores, consolidates, updates, and retires memories over time.

• Fast evolution loop: operates over a 4-level memory substrate of trajectories, tips, skills, and tools: it selects task-relevant mem before execution, acts with them, then writes and periodically maintains mem from trajectories, rewards, and feedback. Convert into supervision for what should be selected, used, written, and preserved.
• Slow evolution loop: On-policy self-distillation: Observes attribution-enriched evidence unavailable at deployment (candidate memory value, trajectory snippets, future utility of written memories, and repository-level diagnostics), enabling OPD-Evolver to acquire transferable lifecycle-level competence for self-improvement.

​

80

OPD-Evolver: Performance

Task success metric across self-evolving agent benchmarks. For AMA-Bench, CI: Causal Inference, SU: State Updating, SA: State Updating.

Comparison with training-based agent improvement methods

Ablation study on InterCode (Bash/CTF/SQL) with OPD-Evolver-4B. “Writing Distill.” denotes the exclusion of self-distilling experience writing capability

81

References (I)

• Aadharsh Aadhithya, Sachin Kumar, and KP Soman, “Enhancing Long-Term Memory using Hierarchical Aggregate Tree for Retrieval Augmented Generation”, arXiv:2406.06124 (2024)
• Petr Anokhin, Nikita Semenov, Artyom Sorokin, Dmitry Evseev, Mikhail Burtsev, and Evgeny Burnaev, “AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents”, arXiv:2407.04363 (2024)
• Anthropic, 2026, “How Claude remembers your project”, https://code.claude.com/docs/en/memory (accessed 2026-04-26)
• A. Asai, et al., “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” ICLR 2024
• DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL,” arXiv 2025
• Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su, “From RAG to Memory: Non-Parametric Continual Learning for Large Language Models, in ICML’2025, July 2025
• Wei-Chieh Huang, et al, "Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey", ArXiv:2602.06052
• Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, and Alex Pentland, “MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks”, arxiv:2602.16313 (2026)
• Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo, “HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model”, in ACL 2025, July 2025
• Wei-Chieh Huang, et al, "Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey", ArXiv:2602.06052
• P. Jiang, et al., “DeepRetrieval: Hacking Real Search Engines and Retrievers with LLMs via RL,” COLM 2025
• P. Jiang, et al., “s3: You Don’t Need That Much Data to Train a Search Agent via RL,” EMNLP 2025
• P. Jiang, et al., “Harness-1: A Stateful Harness for Training Long-Horizon Search Agents,” arXiv 2026
• P. Jiang, et al., “Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills,” arXiv 2026
• Z. Jiang, et al., “Active Retrieval Augmented Generation (FLARE),” EMNLP 2023
• B. Jin, et al., “Search-R1: Training LLMs to Reason and Leverage Search Engines with RL,” COLM 2025
• Zixuan Ke, Yifei Ming, Austin Xu, Ryan Chin, Xuan-Phi Nguyen, Prathyusha Jwalapuram, Jiayu Wang, Semih Yavuz, Caiming Xiong, and Shafiq Joty, “Mas-orchestra: Understanding and improving multiagent reasoning through holistic orchestration and controlled benchmarks”, arXiv:2601.14652 (2026)

​

82

References (II)

• Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn, “Meta-Harness: End-to-End Optimization of Model Harnesses”, arxiv:2603.28052
• Xiang Lei, Qin Li, and Min Zhang, “D-smart: Enhancing LLM dialogue consistency via dynamic structured memory and reasoning tree,” arXiv:2510.13363 (2025)
• X. Li, et al., “Search-o1: Agentic Search-Enhanced Large Reasoning Models,” EMNLP 2025
• Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie, “Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks”, NeurIPS’2024, Dec. 2024
• Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto, “s1: Simple test-time scaling”, arXiv:2501.19393 (2025)
• OpenAI, 2026. Harness engineering: leveraging codex in an agent-first world, openai.com/index/harness-engineering (accessed: 2026-04-26)
• L. Ouyang, et al., “Training Language Models to Follow Instructions with Human Feedback,” NeurIPS 2022
• Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, Tomas Pfister, “ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory”, in ICLR 2026
• Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, Maohao Shen, Vishy Tirumalashetty, George Lee, Jiawei Han, Tomas Pfister, Chen-Yu Lee, "SkillOS: Learning Skill Curation for Self-Evolving Agents", arXiv:2605.06614
• C. Packer, et al., “MemGPT: Towards LLMs as Operating Systems,” arXiv 2023
• J. S. Park, et al., “Generative Agents: Interactive Simulacra of Human Behavior,” UIST 2023
• O. Press, et al., “Measuring and Narrowing the Compositionality Gap (Self-Ask),” Findings of EMNLP 2023
• Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef, “Zep: a temporal knowledge graph architecture for agent memory”, arXiv:2501.13956 (2025)
• Alireza Rezazadeh, Zichao Li, Wei Wei, and Yujia Bao, “From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs”, in ICLR’2025, April 2025

​

​

​

​

​

83

References (II)

• J. Schulman, et al., “Proximal Policy Optimization Algorithms,” arXiv 2017
• Z. Shao, et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning (GRPO),” arXiv 2024
• N. Shinn, et al., “Reflexion: Language Agents with Verbal Reinforcement Learning,” NeurIPS 2023
• R. S. Sutton, A. G. Barto, “Reinforcement Learning: An Introduction,” MIT Press, 2018
• Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Rajan Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, and Tomas Pfister, “In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents”, in ACL’2025
• Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein, “ChemAgent: Self-updating Memories in Large Language Models Improves Chemical Reasoning”, in ICLR 2025
• H. Trivedi, et al., “Interleaving Retrieval with Chain-of-Thought Reasoning (IRCoT),” ACL 2023
• Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. “Voyager: An Open-Ended Embodied Agent with Large Language Models”, in Transactions on Machine Learning Research (2024)
• Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig, “Agent Workflow Memory”, in ICML 2025
• Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma, “Effective Long-Context Scaling of Foundation Models”, in NAACL’ 2024
• Ke Yang, Zixi Chen, Xuan He, Jize Jiang, Michel Galley, Chenglong Wang, Jianfeng Gao, Jiawei Han, ChengXiang Zhai, “PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents”, in ICML 2026
• Ruozhen Yang, Yucheng Jiang, Yueqi Jiang, Priyanka Kargupta, Yunyi Zhang, Jiawei Han, "Grounding Agent Memory in Contextual Intent", in ACL 2026
• S. Yao, et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” ICLR 2023
• Guibin Zhang, Muxin Fu, Kun Wang, Guancheng Wan, Miao Yu, and Shuicheng Yan, “G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems”, in NeurIPS 2025
• Guibin Zhang, Xun Xu, Yanwei Yue, Zikun Su, Wangchunshu Zhou, Xiaobin Hu, Shuicheng Yan, “OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation”, ArXiv: 2606.17628
• Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, and Wenya Wang. “Memskill: Learning and evolving memory skills for self-evolving agents”, arXiv:2602.02474 (2026)

​

84

Course Coverage

• A Brief Introduction to Large Language Models
• Retrieval and Retrieval Augment Generation (RAG)
• Reasoning with Structures for LLMs
• LLM Agents and Agent Memory Technology
• Look Forward to the Future and Q&A

85