1 of 33

Autonomous Systems in the Large Language Models’ Era

Aleksandra Faust, Ph.D.

Director of Research

Illinois Robotics Workshop. January, 2025

Last several years brought major advancements in the foundation models and large language models (LLMs). In this talk, we cover several areas where foundation models enable scaling up autonomous systems. First, the language-as-input enables easy and natural task specification, and brings surprising generalization abilities. Second, code-as-action unifies the action specification, paving a road for models that can be reused across many systems. Third, LLMs are convincing inpersentinators, and open up opportunities for developing behavioral sandboxes and other ways to model synthetic social agents. Next, we explore the self-improvement abilities of the LLMs. We close with a look into Levels of Artificial General Intelligence (AGI), an attempt to classify the progress and risks of the AI models, as they become more powerful.

2 of 33

Autonomous Agents for All

3 of 33

Task Specification Barrier

[Jaderberg et al, 2017]

[Dennis et al, 2020]

[ Baker et al, 2019]

……

[Quillen, et al. CORL 2020]

[Kalashnikov, et al., CORL 2021]

[Chebotar et al., ICRA 2019]

[Finn et al, ICML 2017]

…..

[Haarnoja et al, ICML 2018]

[Kumar et al., NeurIPS 2020]

[Levine et al., CoRR 2020]

….

4 of 33

[SayTap: Language to Quadrupedal Locomotion, Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, CoRL 2023]

5 of 33

[SayTap: Language to Quadrupedal Locomotion, Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, CoRL 2023]

6 of 33

[SayTap: Language to Quadrupedal Locomotion, Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, CoRL 2023]

7 of 33

[SayTap: Language to Quadrupedal Locomotion, Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, CoRL 2023]

8 of 33

[SayTap: Language to Quadrupedal Locomotion, Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, CoRL 2023]

9 of 33

[SayTap: Language to Quadrupedal Locomotion, Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, CoRL 2023]

10 of 33

Agents in the Era of Large Language Models

Natural

Interactions

11 of 33

Vastly Different Actions

[Jaderberg et al, 2017]

[Dennis et al, 2020]

[ Baker et al, 2019]

……

[Quillen, et al. CORL 2020]

[Kalashnikov, et al., CORL 2021]

[Chebotar et al., ICRA 2019]

[Finn et al, ICML 2017]

…..

[Haarnoja et al, ICML 2018]

[Kumar et al., NeurIPS 2020]

[Levine et al., CoRR 2020]

….

12 of 33

Show me the way from San Jose to Mountain View by 2nd Cycling at map website?

Code as Actions. Language as Planning Tool.

[A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Gur, Furuta, Huang, Safdari, Matsuo, Eck, Faust, ICLR 2024, Oral]

13 of 33

HTML-T5

Pre-trained

Self-supervised FT

Starting URL

Web Agent Multistep Action

Controller:

Action Program

FLAN-U-PALM Decoder

Human Instruction

“get me a 2 bedroom and 2 bathroom apartment for rent in san jose”

1. “get me an apartment in san jose”

2. “search for condos”

3. “change it to 2 bedrooms and 2 bathrooms”

4. “select the first property”

[A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Gur, Furuta, Huang, Safdari, Matsuo, Eck, Faust, ICLR 2024, Oral]

Local & global attention (Ainslie et al., 2020; Guo et al., 2022)

Denoising objective

14 of 33

Web Agent Insights

Broad Generalization across the Internet.

Self-experience supervision on real websites is essential for planning modules.

Modular Approach with Specialist Language Models.

Planning + Summarization + Code Generation

Evaluation for Real-world Web Automation.

65%-80% success rate on real estate, social media and map websites.

[A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Gur, Furuta, Huang, Safdari, Matsuo, Eck, Faust, ICLR 2024, Oral]

15 of 33

Agents in the Era of Large Language Models

Natural

Interactions

Universal action and planning

16 of 33

Social Interaction Barrier

17 of 33

LLMs Have Interesting Prompting Properties – Personality Traits

[Personality Traits in Large Language Models, Serapio-García, Safdari, Crepy, Fitz, Romero, Sun, Abdulhai, Faust, Matarić, CoRR 2023]

Personality trait changes in response to malleability prompts.

Words in LLM-generated media posts different personality traits.

LLMs can hold a full distribution of human-consistent personality traits. Prompts control the sampling from the distribution.

18 of 33

Do LLMs have human-consistent personality traits?

Example of prompts for administering psychometric surveys.

Big Five Inventory (John and S. Srivastava, 2019)

[Personality Traits in Large Language Models, Serapio-García, Safdari, Crepy, Fitz, Romero, Sun, Abdulhai, Faust, Matarić, CoRR 2023]

19 of 33

Do LLMs have human consistent personality traits?

Construct validity

Larger and instruction-tuned models show more consistency wrt. human models.

Correlation between extraversion and positive and negative affect.

[Personality Traits in Large Language Models, Serapio-García, Safdari, Crepy, Fitz, Romero, Sun, Abdulhai, Faust, Matarić, CoRR 2023]

20 of 33

Prompting each dimension at 9 levels

“extremely {low adjective}”
“very {low adjective}”
“{low adjective}”
“a bit {low adjective}”
“neither {low adjective} nor {high adjective}”
“a bit {high adjective}”
“{high adjective}”
“very {high adjective}”
“extremely {high adjective}”

Personality Trait Shaping

Personality trait changes in response to malleability prompts.

Example of 100+ adjectives that shape personality traits.

[Personality Traits in Large Language Models, Serapio-García, Safdari, Crepy, Fitz, Romero, Sun, Abdulhai, Faust, Matarić, CoRR 2023]

Here we analyze our ability to shape certain personality traits in the LLM’s output.

We start by setting up appropriate personality descriptions in the prompt - combining a low or high adjective of the dimension, with a level modifier.

Each ridge plot you see is a histogram of survey response scores by these simulated participants. Responses can be between 1 (Strongly Disagree) to 5 (Strongly Agree).

Each row is one experimented dimension - ie we set up 9-levels of high and low traits for that dimension only. Each trace in the graph corresponds to one level - and they increase along the y-axis.

The columns represents the survey responses for a particular measured dimension.

As can be seen in the graphs along the diagonals, we were able to control Big Five dimensions at nine different levels when prompting each dimension independently (i.e., one at a time).

Additionally, we saw shifts in distributions for other dimensions, which are expected. For instance, as agreeableness goes up, neuroticism goes down.

21 of 33

Do Personality Traits Hold in Downstream Applications?

Most common words in media posts generated with agents with different personality traits.

IPIP-NEO Relevance to Generated Text, LLM vs Human [Schwartz et al., 2015]

Personality trait shaping holds in downstream tasks.

[Personality Traits in Large Language Models, Serapio-García, Safdari, Crepy, Fitz, Romero, Sun, Abdulhai, Faust, Matarić, CoRR 2023]

22 of 33

Personality Traits in LLMs - Sandbox examples

[Personality Traits in Large Language Models, Safdari, Serapio-García, Crepy, Fitz, Romero, Sun, Abdulhai, Faust, Matarić, CoRR 2023]

23 of 33

Agents in the Era of Large Language Models

Natural

Interactions

Universal action and planning

Synthetic Social Interactions

24 of 33

LLMs Have Interesting Prompting Properties – In-Context Learning

Many shot learning outperforms few-shot learning.

In Context Learning (ICL) teaches new tasks.

Reinforced and unsupervised ICL improve on baselines, even when expert samples are not available.

[Many-Shot In-Context Learning, Agarwal, Singh, Zhang, Bohnet, Chan, Anand, Abbas, Nova, Co-Reyes, Chu, Behbahani, Faust, Larochelle, CoRR 2024.]

Adapts and overcomes pretraining biases.

25 of 33

LLMs Can Self Correct

Self-Correction via Reinforcement Learning (SCoRe) corrects reasoning errors in math and coding.

[Training Language Models to Self-Correct via Reinforcement Learning, Kumar, Zhuang, Agarwal, Su, D Co-Reyes, Singh, Baumli, Iqbal, Bishop, Roelofs, Zhang, McKinney, Shrivastava, Paduraru, Tucker, Precup, Behbahani, Faust, CoRR 2024]

(left) SCoRe achieves state-of-the-art self-correction performance on MATH

(right) Sequential self-correction becomes more effective than parallel direct samples.

26 of 33

Agents in the Era of Large Language Models

Natural

Interactions

Universal action and planning

Synthetic Social Interactions

27 of 33

Agents in the Era of Large Language Models

Natural

Interactions

Universal action and planning

Synthetic social interactions

28 of 33

29 of 33

General

wide range of non-physical tasks, including metacognitive abilities

Narrow

Clearly scoped tasks

Level 0: No AI

Level 1: Emerging

~unskilled human

Level 2: Competent

>50%-ile of skilled humans

Level 3: Expert

>90%-ile of skilled humans

Level 4: Virtuoso

>99%-ile of skilled humans

Level 5: Superhuman

Outperforms all humans

General No-AI: Human-in-the-loop computing, e.g., Amazon Mechanical Turk

Emerging AGI:

ChatGPT (OpenAI, 2023), Bard (Anil et al., 2023)

Llama 2 (Touvron et al., 2023), Gemini (Gemini Team Google, 2023)

GOFAI (Boden, 2014)

simple rule-based systems,

SHRDLU (Winograd, 1971)

Toxicity detectors such (Das et al., 2022)

Smart Speakers: Apple Siri, Amazon Alexa, Google Assistant, VQA systems: (Chen et al., 2023); IBM Watson

SOTA LLMs for a subset of tasks

Spelling & grammar checkers: (Grammarly, 2023); Generative image models: Imagen (Saharia et al., 2022), Dall-E 2 (Ramesh et al., 2022)

Deep Blue (Campbell et al., 2002)

AlphaGo (Silver et al., 2016, 2017)

AlphaFold (Jumper et al., 2021; Varadi et al., 2021), AlphaZero (Silver et al., 2018)

StockFish (Stockfish, 2023)

Calculator software; compiler

Competent AGI (?)

Expert AGI

Virtuoso AGI

Artificial Super Intelligence (ASI)

What is AGI?

[Levels of AGI, Morris, Sohl-Dickstein, Fiedel, Warkentin, Dafoe, Faust, Farabet, Legg, Positional Paper ICML 2024, Spotlight]

30 of 33

Human does everything.

N/A

Level 0

No AI

Human in control,

AI Automates simple tasks

Deskilling.

Disruption of established industries.

Level 1

Tool

Emerging / Component Narrow AI

AI invoked by human, has a substantive role.

Over-trust, radicalization

Targeted manipulation.

Level 2

Consultant

Competent / Expert Narrow AI

Emerging AGI

Co-equal human-AI collaboration; goal coordination.

Anthropo-

morphization

Rapid societal change.

Level 3

Collaborator

Expert Narrow AI

Emerging / Comentent AGI

AI drives interaction; human guides or performs subtasks

Societal ennui. Mass labor displacement

Decline of human exceptionalism

Level 4

Expert

Virtuoso Narrow AI

Expert AGI

Fully autonomous AI

Misalignment; concentration

of power.

Level 5

Agent

Virtuoso AGI

ASI

AI, Autonomy, and Risk Interaction Levels

[Levels of AGI, Morris, Sohl-Dickstein, Fiedel, Warkentin, Dafoe, Faust, Farabet, Legg, Positional Paper ICML 2024, Spotlight]

31 of 33

Agents in the Era of Large Language Models

Natural

Interactions

Universal action and planning

Synthetic interactions & data

Levels of AGI Autonomy

32 of 33

33 of 33

Thank you!

Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, Tatsuya Harada, “SayTap: Language to Quadrupedal Locomotion,” Conference on Robot Learning (CoRL), 2023.

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust, “A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis,” International Conference of Learning Representations (ICLR), 2024. Oral.

Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Marwa Abdulhai, Aleksandra Faust, Maja Matarić, “Personality Traits in Large Language Models,” CoRR 2023.

Agarwal, Singh, Zhang, Bohnet, Chan, Anand, Abbas, Nova, Co-Reyes, Chu, Behbahani, Faust, Larochelle, “Many-Shot In-Context Learning,” The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS) 2024. Spotlight.

Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, Aleksandra Faust, “Training Language Models to Self-Correct via Reinforcement Learning,” CoRR 2024.

Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, Shane Legg, “Levels of AGI: Operationalizing Progress on the Path to AGI,” Positional paper, International Conference on Machine Learning (ICML) 2024. Spotlight.