3 of 25

Challenge

Usually involving the recruitment of human users to experience and provide feedback, iteration of such systems can be time-consuming and costly,

iteration 1

iteration 2

…

user

feedback

user study participants

budget

time

challenge in recruiting targeted users

Rapid Prototyping Needs

Agenda

4 of 25

What If …?

Human Proxy Agent

1. Collecting User Experiences for Design Insights

2. System Usability Test

3. Conducting User Study for Evaluation

4. System Automation

Our work may boost your working efficiency in these tasks:

Digital Assessment System

Agenda

5 of 25

Recent Progress

Autonomous Web Agent^[1]

refers to systems that complete complex tasks on web interfaces without human intervention;

it usually consist of profile, plan, action, and memory modules in its architecture

Task

(e.g., "book a flight to Vienna")

Web Application

Vision-Language Models

Autonomous Web Agent

execute action

observation

ask for action

plan action

memory

[1] Chevrot, Antoine, et al. "Are Autonomous Web Agents Good Testers?." Proceedings of the ACM on Software Engineering 2.ISSTA (2025): 206-228.

Agenda

6 of 25

Recent Progress

Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies. arXiv preprint arXiv:2505.22981 ( Siyang Liu, et al. , 2025).

Generative Agent Simulations of 1,000 People. arXiv preprint arXiv:2411.10109 (Joon Sung Park, et al., 2025)

Role-play Agents

The (relative) success of language agents simulating human behaviors and collecting user experiences at scale

Agenda

7 of 25

Research Questions

1. Can we leverage agent capabilities to simulate participants and prototype with digital assessment systems?

2.How to simulate student role in this educational context?

3.How does it perform on a real-world case - prototyping ETS's conversation-based assessment?

Agenda

8 of 25

Scenario

Conversation-based Assessment System

Weather Task:

1. Understanding the formation of thunderstorms

2. Learning how to predict thunderstorms and set appropriate alert levels for a given weather station

3. Writing a thunderstorm warning forecast

Key Interactive Scenes:

1. Responding to questions from Dr.Garcia and peer Art

2. Learning storms from a video

3. Filling the concept map

4. Calculate the TT score

5. Selecting answers for multi option quizzes

6. Writing storm alert notification

Click Here for Demo

Example of a Conversation-based Assessment System - Weather Task

Agenda

9 of 25

Solution

1. An Agent Architecture with Modular Student Role and Web Task Performer � A hierarchical architecture that separates individual decision-making processes from low-level execution to support user experience engineering.�

2. Student Role Agent Design in Educational Settings� Each agent is equipped with a student profile, memory, planning, and intent-based action, simulating personalized decision-making.�

3. Deployed in ETS Weather Task Assessment� Applied to ETS’s conversation-based assessment (weather task), demonstrating superior performance in realistic educational settings.

Agenda

10 of 25

Demonstration in Weather Task Scenario

A demo video

Agenda

11 of 25

Solution - How it works?

1. An Architecture with Modular Student Role and Web Task Performer � A hierarchical architecture that separates individual decision-making processes from low-level execution to support user experience engineering.

Agenda

12 of 25

OpenAI^[1]

Updated Web Page

HTML Elements, Interactive Elements and ScreenShot

①

②

Reasoning and Action (Text)

Screenshot and

Text Description

Executable Action

(Javascripts)

③

④

Prompting

③

②

Prompting

PlayWright^[2]

Javascripts for Execution

Page Parser

①

④

Internal Support

Communication Arrow

Legend

①

Workflow Stage

Lite Web Agent^[3]

Web Application

Student Role Agent

[1]LiteWebAgent (https://github.com/PathOnAIOrg/LiteWebAgent)

[2]Openai-python(https://github.com/openai/openai-python)

[3]PlayWright (https://github.com/microsoft/playwright-python)

Agenda

13 of 25

Solution - How it Works?

2. Student Role Agent Design in Educational Settings� Each agent is equipped with a student profile, memory, planning, and intent-based action, simulating personalized decision-making.�

Agenda

14 of 25

Student Role Agent Design

Autonomous Web Agent

Interactive dialogue systems that can process a query and produce a human-like conversation

(Chien & Yao, 2020)

Application URL

OpenAI

Vision, Interactive Element Descriptions

Intent, DOM, Vision Signals

Updated Web Page

DOM, UI, ScreenShot

①

Intent

("go to next section")

Executable Action

Screenshot, Text Description

Executable Action

(click ('393'))

Autonomous Web Agent

Parses DOM, renders elements, summarizes UI

Maps intent to DOM actions

Web Application & Environment

(e.g., conversation-based assessment system)

Run executable scripts,

Return new web page information

④

Perception Layer

Autonomous Web Agent

1.Student Profile

2.Observation

4.Action

3. Memory

Log of Interactions with Web Agent

Screenshot of the Current Page

Interactive Element Description

[16] img ''

[31] input ''

[35] input ''

Demographics

Personal Traits

Learning Background and Preferences

Assessment Behavior

Reasoning

Intent

[User Experience] This step looks straightforward and clear.

[Previous Step Analysis]

[Action] Type the introduction message: "Hello, I\'m Samantha. I am a 15-year-old student from …

Agenda

15 of 25

Profile

Demographics

Name: Samantha
Gender: Female
Age: 15
Ethnicity: Hispanic
Primary Language: Spanish
Technology Access: High (have Wi-Fi at home; play online games for fun; use the internet every day)

Personal Traits

Big Five: High Openness, Moderate Conscientiousness, Low Extraversion, High Agreeableness, Moderate Neuroticism
Examination Anxiety Level: Moderately high (Scored at 3.0 in Westside Test Anxiety Scale)
Mindset: Growth mindset – views effort and feedback as learning opportunities

Learning Background and Preferences

Grade Level: 10th grade
Science Performance: Level A (90% or higher)
Math Performance: Level B (80% or 89%)
Concept Mastery (NGSS):

MS-ESS2-1. Develop a model to describe the cycling of Earth’s materials and the flow of energy that drives this process. Moderate.
MS-ESS2-3Analyze and interpret data on the distribution of fossils and rocks, continental shapes, and seafloor structures to provide evidence of past plate motions. High

Cognitive Load Tolerance: Medium – benefits from chunked or scaffolded instruction
Pacing Preference: Moderate – prefers time to reflect before responding

Assessment and Expression Behavior

Misspelling Tendency: Low
Concreteness in Writing: High
Response Comprehensiveness: Sometimes partial – may overlook one part of a multi-part task

Afzal, Arfa, et al. "Addressing the digital divide: Access and use of technology in education." Journal of Social Sciences Review 3.2 (2023): 883-895.

Mustafa, Sohaib, et al. "Digital students’ satisfaction with and intention to use online teaching modes, role of big five personality traits." Frontiers in psychology 13 (2022): 956281.

Cassady, Jerrell C., and Betty E. Gridley. "The effects of online formative and summative assessment on test anxiety and performance." The Journal of Technology, Learning and Assessment 4.1 (2005).

Deno, Stanley L. "Relationships Among Simple Measures of Spelling and Performance on Standardized Achievement Tests." (1980).

Agenda

16 of 25

Observation

Interactive Elements

[78] img ''

[87] button 'Previous'

[88] button 'Next'

[95] img ''

[96] img ''

[101] img ''

[102] img ''

[107] img ''

[108] img ''

[114] textarea ''

[115] input ''

[323] img ''

Screenshot

Agenda

17 of 25

Action

[User Experience] I feel good about writing notes because it helps me focus on key ideas and prepare for later questions. This fits with my high agreeableness and growth mindset, embracing effort and reflection.

[Previous Action Analysis] Everything looks good so far

[Action] I will type my answer to Dr. Garcia\'s question in the chat box based on the video and my notes: "The warm air at Earth\'s surface moves upward because it is lighter than the air above. As it rises, it cools and forms clouds, which leads to storm development." Then I will click \'Send.\''

actions to parse: ["click('114')"]

web agent's parser

Agenda

18 of 25

Memory

Previous Observation, Reasoning, and Action

[

{

"role":"system",

"content": "[observation]"

{

"role":"assistant",

"content": "[reasoning] [action]"

…

]

Agenda

19 of 25

Solution - How it Works Cont'd

3. Deployed in Weather Task � Applied to ETS’s conversation-based assessment (Weather Task), demonstrating superior performance in realistic educational settings.

Agenda

20 of 25

How to use it in ETS for Other Tasks?

The architecture is generalizable for all testing needs as long as in an educational setting.

What you need to do is to customize below:

(1) URL of your application

(2) Config student profile and prompts based on your needs

(3) Launch the agent

Agenda

21 of 25

Summary

2. Student Role Agent Design in Educational Settings� Each agent is equipped with a student profile, memory, planning, and intent-based action, simulating personalized decision-making.�

3. Deployed in ETS Weather Task Assessment� Applied to ETS’s conversation-based assessment (weather task), demonstrating superior performance in realistic educational settings.

Agenda

22 of 25

Future Work

1. Analyze the simulations and compare to real user logs.

2. Quantitatively and qualitatively evaluate the architecture.

3. Paper writing

Agenda

23 of 25

Acknowledgements

Jessica Andrews Todd

Yang Jiang

Dr. Carol Forsyth

Diego Zapata-Rivera

Liang Zhang

Chunyi Ruan

Jeremy Lee

Mike Suhan

Jamie Mikeska

Michael Flor

Patrick Kyllonen

Edith Aurora Graf

Kevin M. Williams, Ph.D.

Nathan Lederer

Beata Beigman Klebanov

I thank everyone listed for their guidance, insights, and conversations during my internship experience.

Also thanks Nimon Joan and HR team to make this internship organized and enriching!

Agenda

24 of 25

Any questions?

Agenda

25 of 25

How to Use It in ETS for Other Tasks?

The architecture is generalizable for all testing needs as long as in an educational setting.

What you need to do is to customize below:

(1) URL of your application

(2) Config student profile based on your production needs

(3) Launch the agent

Agenda