Human-Centered Evaluation of Coding Agents

Valerie Chen

Brief intro 👋

“Software is eating the world”

Marc Andreessen (2011)

Source: Evans Data Corporation

Recently, a proliferation of AI tools for code

Most frequent responses based on

survey in Fall 2024 (n=170)

A ripe opportunity to study human-AI interaction!

Coding assistants are evolving

A growing ecosystem

This was 7 months ago

Increased autonomy, increased risk

These concerns are not limited to SWE

So how do we design the next generation of (coding) agents?

It starts with evaluation!

Human-Centered Evaluations

Evaluating copilots

Prior Evaluations

In practice, models don’t work on their own

​

GitHub Copilot

How do we pick which LLM to use?

​

What about benchmark performance?

We test this hypothesis by running a user study where people program with models of varying benchmark performance.

We create a web interface to evaluate two forms of AI support

Chatbot

The interface is part of an end-to-end pipeline to evaluate LLMs for coding

Our results highlight the importance of human-in-the-loop evaluation

• 213 participants assigned to 7 conditions
• Gaps in benchmark performance do not match differences in human performance
• The benefits that a human gets from LLM support vary by task type (e.g., data science vs interview-style problems)

Notice that none of these models are used anymore for code!

What’s the best way to do human-in-the-loop evaluation?

Realistic usage, but not scalable!

What’s the best way to do human-in-the-loop evaluation?

As of September 2025, the platform has collected over 3.5 million head-to-head votes across more than 400 models

Demo

https://lmarena.ai/

​

What’s the best way to do human-in-the-loop evaluation?

Chatbot Arena has introduced a coding category to evaluate models with coding capabilites.

Scalable, but not realistic!

Existing evaluations are flawed

Realistic

Scalability

Copilot Arena aims to achieve the best of both worlds

Realistic

Scalability

Copilot Arena

Copilot Arena is a real VSCode Extension

Copilot Arena workflow

What models does Copilot Arena support?

How are chat models performing code completions?

Need to be able to “fill-in-the-middle” (FiM)

Not trained to do FiM!

What’s unique about code?

34

Slide from CMU Neural Code Generation (11-891)

Training models to fill in the middle

35

Encode code in the same way during inference

https://arxiv.org/pdf/2207.14255

Many chat models struggle with correct formatting

Correct Outcome

Instead of generating completions, we ask models to generate code snippets

Generate the entire snippet

Correct Outcome

We post-process the generated snippet to remove overlaps

Correct Outcome

Remove any overlapping text

Simple solution to allow chat models to perform FiM!

• Performance on benchmarks like HumanEval infilling drastically improves model performance (and reduces indentation or formatting issues).

Copilot Arena so far

• >30k battles
• >4.5k users contribute votes
• Dozen models in the arena
• We even help evaluate new models before launch!

Leaderboard Computation

Copilot Arena Leaderboard

Real-world “data distribution”

Comparison to prior evals

Which models best align with user preferences?

• Downstream task significantly affects user preference, while programming languages have little effect.
• Smaller models tend to perform better on data similar to static benchmarks.

TLDR

• Existing evaluations do not necessarily correlate well with in-the-wild preferences.
• Model performance is affected by task, context, and code structure. No model that is “one-size-fits-all.”
• Diverse and realistic human preference data is essential for effective code generation models.
• We also now have a lot of interesting data to dig into!

Evaluating agents

A lot of work has been done to understand copilot usage

Current understanding of agent usage

Scope of tasks has increased

Demo

https://app.all-hands.dev/

Demo

Agents still require humans in the loop

Edit files

Now, developers have options

What if we put them head to head?

Comparing copilots and agents

We recruit participants who are regular users of GitHub Copilot

Study design

Summary of Findings

• On average, participants with agents are more productive than with copilots (a 35% increase in task correctness).
• For correctly solved problems, we find a significant difference in user effort between the time spent using copilots and agents (25.1 vs 12.5 minutes).

Summary of Findings

​

There is room for improvement!

What should we change about the agent?

Edit files

Case studies in agent design

Measuring quality of agent work

Prediction-powered User Label Synthesis & Evaluation

Step 1: Collect ratings

Users are prompted to provide feedback after each work segment

​

A work segment = actions between user command and the agent returning to “stopped” state

​

We collect a dataset of N=1747 labeled user trajectories where the average rating is 4.07

​

Step 2: Train rating predictor

Features based on the user:

• Sentiment of user messages
• Number of user messages

Features based on the agent:

• Type of task

Features that show task progression:

• Git actions

User sentiment and git push were the most predictive features!

Step 2: Train rating predictor

Supervised learning approaches using our features outperform naive LLM-as-a-Judge baseline

Step 3: Compute effect size

Augment with Infilled Labels

Overview of Users

Over the course of multiple months, we ran our 3 case studies on over 15k users of the OpenHands SaaS platform.

Results

Results

• Changes to the LLM model has the largest differences on user satisfaction

​

+5.9% difference between claude-3.7-sonnet and claude-4-sonnet

-7.8% difference between claude-4-sonnet and gpt-5

​

vs.

​

+3.1% difference between no plan and planning

Results

• No statistically significant in memory case study actually shows how we can reduce cost while preserving user satisfaction

​

​

• Our results also show how PULSE can lead to more conclusive results (reducing confidence intervals by up to 40%)

​

​

Comparison to benchmarks

Static benchmarks don’t tell the full story!

TLDR

• We are seeing a shift towards more autonomous workflows in AI coding assistants
• Evaluations in these multi-turn settings pose unique challenges compared to the copilot setting
• However, benchmarks do not always correlate with user satisfaction, requiring the use of efficient human-in-the-loop approaches

​

​

Future Directions

Current benchmarks

Alternatively, focus on collaboration

Current Usage

• Since agents may be editing multiple files and making many changes, it can become difficult for users to understand why the agent made certain changes.
• In the post-study feedback, we found that participants wanted ways to understand quickly what the agent did and why changes were necessary.
• We also see this in the user messages, where one participant asked “Did you delete most of the functions in [filename]? If so, explain why did you do this.”

Future Usage

• Prior literature has studied how users consume model explanations has largely focused on ML models and more recently LLMs.
• However, there is a need to propose explanations of agent actions. Recent work introduced a way for agent developers to view counterfactual roll-outs, but this is not necessarily user-friendly for end users (e.g., developers).
• Future work should consider how to improve the transparency of agent behaviors

Current Usage

• Many participants observed, or even complained, about how OpenHands would take more actions than necessary.
• One participant wrote in a message to the agent, “could the code have been simplified, I did not expect 10 files to be created with more than 1000 lines each.”

Future Usage

• Agents should be better calibrated in terms of their confidence about whether it has completed the user’s request.
• Recent work on UI agents has explored proactively pausing agents at task boundaries, identifying such boundaries in the software engineering settings may be a fruitful direction to improve user perception of agent actions.

Current Usage

• Human effort can be measured in many ways, including the amount of time spent interacting with the agent.
• On this front, many participants noted that the “the generation time is slower” for OpenHands than GitHub Copilot and sometimes “im just kind of sitting there”.

Future Usage

• User experience can be improved by explicitly optimizing for latency when engaging in back-and-forth with the user and providing more direct ways for users to steer agent behaviors.
• Additionally, developers will increasingly need to multi-task to be most productive in agentic workflows, though prior work has characterized the cognitive cost of doing so.

Acknowledgements and paper links

Papers mentioned in this talk

• The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers. TMLR, 2025.
• Copilot Arena: A Platform for Code LLM Evaluation in the Wild. ICML, 2025
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows. Under Review.
• How can we assess human-agent interactions? Case studies in software agent design. Under Review.

The projects presented in this talk are in collaboration with

Aditya Bharat Soni, Aditya Mittal, Ameet Talwalkar, Anastasios Angelopoulos, Calvin Smith, Chris Donahue, David Sontag, Dennis Wei, Graham Neubig, Hoang H Tran, Hussein Mozannar, Ion Stoica, Juan Michelini, Manish Nagireddy, Mohammed Alsobay, Naman Jain, Prasanna Sattigeri, Robert Brennan, Rohit Malhotra, Sebastian Zhao, Subhro Das, Tianjun Zhang, Wayne Chi, Wei-Lin Chiang, Xingyao Wang, Xuhui Zhou

From

CMU, MIT, UC Berkeley, IBM, OpenHands, LM Arena

Thank you! Questions?