CSCI-SHU 376: Natural Language Processing

Hua Shen

Course Agenda: 2026 Spring-NLP-[CSCI-SHU-376]-Class Schedule

2026-04-21

Spring 2026

Lecture 16: LLM Agents

Contents adapted from EMNLP 2024 Tutorial on Language Agents

Today’s Plan

• Language Agents Overview
• Foundations: Reasoning, Memory, Planning
• Data, Application, Evaluation

​

​

​

What is Agent?

3

Recap: Reinforcement Learning

Modern Agent

• LLM + external environment?

Two Competing Views

• LLM first view: make LLM as an agent
• I.e. LLM perceives from environment and acts accordingly
• Just prompt engineering?

​

• Agent first view: Integrate LLM into AI agents so they can use language for reasoning and communication
• We re-examine the challenges (e.g., perceptron, reasoning, communication) in the new lens of LLMs

What’s different now?

• LLM agent can use language for reasoning and communication

Revisit Classic view of agents

Language Agents Definition

• AI agents capable of using language for reasoning and communication are called “language agents”.

Today’s Plan

• Language Agents Overview
• Foundations: Reasoning, Memory, Planning
• Data, Application, Evaluation

​

​

​

Key Concepts for Language Agents

• Action Space
• Reasoning: Update short-term memory
• Retrieval: Update long-term memory

​

• Planning:
• Choose action(s) from the action space

Recap: Reasoning

Why Reasoning is helpful for Agents?

ReAct

• Combine reasoning and acting with LMs

The flexibility of “Act”

The flexibility of “Act”

Why reasoning is special for agents?

• Action space is infinite
• LLMs learn reasoning priors by imitating human reasoning tracks

Reasoning: Take-aways

• Reasoning as internal actions for agents
• No external feedback, just change internal context
• Reasoning guides acting, acting updates reasoning

Recap: Retrieval-based Language Models (RALM)

Inference

• Retrieval-based LM = Retrieval + LMs (or commonly referred as RAG)
• It retrieves from an external datastore

19

Agent Memory

Inference

• Not only others’ knowledge, could be own experience as well
• Also be able to write to it!

20

Generative Agents

Inference

21

Generative Agents

Inference

• The need for memory
• Context window cannot possibly hold all event streams
• Even if possible, long-context modeling might be challenging

22

Generative Agents

Inference

23

Implementing memory

Inference

• Naïve Implementation
• Read: off-the-shelf retriever
• Write: Append to text corpus

​

• Some interesting new implementations
• Read: online retrieval
• Write: index augmentation with referrel

24

Memory: Take-aways

• Language agents interact with external environments and internal memories
• Short-term memory: reasoning
• Long-term memory: (event logs, codebase etc), retrieval

​

​

Planning: definition

• General trends in planning:
• Increasing expressiveness in goal specification: in natural language instead of formal language
• Open-ended action space
• Increasing difficulty in automated goal test

Language agent planning: web agents

Language agent planning: travel planning

Planning: take-aways

• Language agents are expanding into new planning scenarios
• Expressive but fuzzy goal specifications, open-ended action spaces, etc

​

• Language for reasoning also enables new planning abilities
• The best planning strategy is dependent on the LLM

​

• Still a pretty open-ended question

​

Today’s Plan

• Language Agents Overview
• Foundations: Reasoning, Memory, Planning
• Data, Application, Evaluation

​

​

​

Agent Applications

Coding Agents

• API calling for tool use
• Environment: software system such as database, app services
• Observation space: API docs, system info
• Action space: function calls, etc

​

​

Coding Agents

• Project-level coding tasks
• Environment: project code repos, filesystems
• Observation space: code files, exe outputs, docs etc
• Action space: code updates, files search etc

​

​

Gaming Agents

• Digital Games
• Environment: game world
• Observation space: screenshot of game states, inventory etc
• Action space: game controls

​

​

Web agents

• Web / app use
• Environment: web browsers / apps
• Observation space: screenshots, DOM trees, HTML, historical actions
• Action space: browser / app controls

​

​

Computer use agents

• Computer use for universal digital tasks: OSWorld
• Environment: desktop operating systems
• Observation space: desktop screenshots, historical actions
• Action space: keyboard / mouse controls

​

​

Computer use agents

Robotic agents

• Robotics for physical interaction
• Environment: physical world spaces
• Observation space: visual inputs, sensor, physical states
• Action space: motor controls

​

​

Agent Data

Agent Data Example

• Task goal aligned trajectories

​

​

Unifying digital agent data

• Digital agent data unification
• Observation: screenshots
• Actions: universal computer mouse and keyboard controls

​

Agent Evaluation

Recap: What is a good benchmark?

• Difficulty
• Diversity
• Usefulness
• Reproducibility
• Data Contamination

​

​

Good Agent benchmark?

• Natural and challenging tasks
• Good agent evaluation framework
• Realistic agent environment
• Automatic task setup
• Automatic task evaluation

​

​

Recap: Chatbot Arena

• https://lmarena.ai
• Ask human to vote which response is better!

Computer Agent Arena

• An open evaluation platform where users can compare AI-agents performing real-world computer use tasks