CrowdLLM: A Synthetic Crowd Simulator for Crowdsourcing with LLM Workers

Augmented with Lightweight Generative Model

Feng (Ryan) Lin¹, Hanming Zheng², Keyu Tian², Congjing Zhang¹, Li Zeng², Shuai Huang¹

¹University of Washington ²City University of Hong Kong

Collaborators

Ryan Lin

Congjing Zhang

Keyu Tian

Hanming Zheng

Dr. Li Zeng

Agenda

• Introduction

​

• Methodology

​

• Experiments

​

• Conclusion

Outline

• Introduction

​

• Methodology

​

• Experiments

​

• Conclusion

Crowd-based Decision-making

• A variety of systems builds on the participation of humans

​

Crowdsourcing

Voting

Recommender System

2

Digital Population

Real Population

Decision-making

Can LLMs build a realistic Digital Population?

Real humans are costly, hard to recruit, …

Limitation of LLM-based Population

[1] Yang, Joshua C., et al. "LLm Voting: Human choices and ai collective decision-making." Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 2024.

[2] Y. Gao, D. Lee, G. Burtch, & S. Fazelpour, “Take caution in using LLMs as human surrogates”, PNAS, 2025.

Heuristic, lack clear analytical formulation

Voting

Outline

• Introduction

​

• Methodology

​

• Experiments

​

• Conclusion

2

Our Proposed CrowdLLM

LLM

Real Population

Digital Population

Decision-making

Generative Model

Profile

Individual Belief

Reference

Decision

2

Architecture of CrowdLLM

2

Architecture of CrowdLLM

2

Architecture of CrowdLLM

1. Reference Generation

• Reference decision is a purely LLM-based decision

2. Profile Generator

We generate realistic profiles of the digital population by adopting the state-of-the-art generative models.

Easy-to-sample

distribution

Target population

Personalized

Bias Distribution

3. Belief Generator

Semi-Implicit VAE

Encoder

Sample

Latent Space

4. Blender

• We can correct the bias through a blender :

Training of CrowdLLM

Reconstruction Loss

Decision-making loss

Individual Belief

LLM Backbone

Profiles

Outline

• Introduction

​

• Methodology

​

• Experiments

​

• Conclusion

Experimental Studies

Real-World Case Studies

Simulation Studies

Ablation Study

Data efficiency

Cost effectiveness

Voting

Amazon Product Reviews

Crowdsourcing

Belief Diversity

Case I: Voting

State-of-the-art

Our CrowdLLM

Case II: Amazon Product Reviews

Beauty (448 products, 10957 users)

Music (629 products, 12396 users)

Case II: Amazon Product Reviews

Case II: Amazon Product Reviews

Real Human

CrowdLLM

LLMs

Case III: Crowdsourcing

Experiments on multiple datasets:

• Offensiveness
• QA Difficulty
• Politeness
• Mitigating Biases
• Differences in Fairness

Experimental Studies

Real-World Case Studies

Simulation Studies

Ablation Study

Data efficiency

Cost effectiveness

Voting

Amazon Product Reviews

Crowdsourcing

Belief Diversity

Data Efficiency of CrowdLLM

​

CrowdLLM shows great superiority over other LLM-based methods with only 1% training data (human data).

Outline

• Introduction

​

• Methodology

​

• Experiments

​

• Conclusion

Conclusion

• CrowdLLM creates digital populations through a combination of LLMs and generative models.

​

• The digital population generated by CrowdLLM can capture the diversity of real human population.

​

• CrowdLLM provides a promising solution to practical issues in crowd-based decision-making involving real human population.

​