SIMULATIONS FOR LEARNING�STEERABLE AI SYSTEMS

Anil Nelakanti�IIIT Hyderabad

Presented at the First Agent-based Modeling India Workshop�Indian Institute of Management Kozhikode�iimk.ac.in/apps/ABMWorkshop25

INTERACTIVE SYSTEMS

• Search and retrieval
• Web-search product-search job-search content-search (music, movies, etc.)
• Recommender systems
• Content-curation, personalization
• Slate recommendations: content-carousels, news-feed
• Smart-homes: play music, set blinders and increase AC, etc.
• Advertising systems
• Sponsored-ads retargeting-ads
• Varied spaces [1]
• Discrete like ranking | ads | slates | etc.�Continuous like pricing | inventory quantity | etc.

​

​

1. Bram van den Akker, Olivier Jeunen, Ying Li, Ben London, Zahra Nazari, Devesh Parekh. Practical Bandits: An Industry Perspective

RANK BY RELEVANCE

• Principle: {rank by relevance} and {measure user response}
• Relevance = similarity | preference | skill (TrueSkill) | etc.�Response = clicks | purchases | dwell time | adoption rate | likes | shares | etc.
• Say, user viewed, { , and .

​

​

​

• Rank the catalogue { , , etc.} by probability of relevance [1,2]

​

​

​

1. S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, 1977.
2. P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender systems. In Conference on Recommender systems, 2011.

RANK AND MEASURE

• Principle: rank items by relevance and measure response
• Practice: tailor to applications
• Product: rank by context
• Ad placement: rank by auction
• Music search: rank by interests
• Reality: large multi-stage system
• Too many moving parts
• Engineering & Algorithmic | Policy & Regulatory | UX choices

Impression

Click / no-click

LARGE SPACE OF DESIGN CHOICES

1. S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, 1977.
2. P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender systems. In Conference on Recommender systems, 2011.

User

Item Ranking

ACT AND MEASURE

• Incrementally modify the moving parts A/B test interventions [1,2]
• Expensive-interventions | Not easy to ensure assumptions hold
• Choosing from Action A vs Action B?
• Product changes: with feature vs without feature
• Technical changes: algorithmic changes, ML/AI models
• Creative changes: red-button vs blue-button, email-once vs email-twice
• Policy changes: second-price auction vs first-price auction,
• Regulatory changes: impact from implementation

WHAT IS THE COST OF A/B TESTING INTERVENTIONS?

Display output

Measure metrics �of B relative to A

Control A

Treatment B

Ranking A

Ranking B

1. Donal B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 1974.
2. D. Tang, A. Agarwal, D. O’Brien, and M. Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In KDD 2010.

MEASURE OFFLINE THEN ONLINE

• Very high cost of A/B testing interventions
• Significant resources for production deployment
• Sub-optimal allocation of live traffic: large opportunity costs�Risk anywhere from revenue loss and full extinction
• Peek into the results with offline tests
• Measure proxies before A/B test like mean average precision�reciprocal rank | cumulative-gain-based | etc.
• Plenty of logged data: evaluate on it, like�- Does B rank clicked items higher than A in logs?�- Does B filter more irrelevant items than A in logs?�- Position bias and importance of being at the top

​

1. E. M. Voorhees. The trec-8 question answering track report. In Trec, 1999.
2. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management, 2000.
3. K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 2002.
4. N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. Intl. conference on web search and data mining, 2008.

Measure proxy

metrics before

A/B test?

Ranking A

Ranking B

INTERACTIVE AI SYSTEM LIFECYCLE

SHOOTING IN THE DARK?

• Winner’s curse: expected gains not realized?�What could go wrong?

LONG TURNAROUNDS WITH LOWER SUCCESS RATES: HOW MUCH BETTER THAN HIPPO?

Not got money?

FALSE DISCOVERIES I

BAD TESTING WITH POOR METRICS OF MODELS TRAINED ON POORER DATA

1. Brian Christian. The A/B test: Inside the technology that’s changing the rules of business, April 2012.
2. Y. Xu, N. Chen, A. Fernandez, O. Sinno, and A. Bhasin. From infrastructure to culture: A/B testing challenges in large scale social networks. In KDD 2015.
3. Alex Deng, and Xiaolin Shi. Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned. In KDD 2016.
4. Meta Analytics Team. Notifications: why less is more. Meta 2022.
5. Timothy Chan. Beyond Simple A/B Testing: Advanced Experimentation Tactics.

​

​

FALSE DISCOVERIES II

BAD TESTING WITH POOR METRICS OF MODELS TRAINED ON POORER DATA

1. K. Krauth, S. Dean, A. Zhao, W. Guo, M. Curmei, B. Recht, and M. I. Jordan. Do offline metrics predict online performance in recommender systems? Arxiv 2020.
2. C. A. Gomez-Uribe and N. Hunt. The netflix recommender system: Algorithms, business value, and innovation. ACM TMIS, 2015.
3. A. V. Bodapati. Recommendation systems with purchase data. Journal of marketing research, 2008.
4. Jeonghee Yi , Ye Chen, Jie Li, Swaraj Sett, Tak W. Yan. Predictive model performance: offline and online evaluations, In KDD 2013.
5. Xiaojie Wang, Ruoyuan Gao, Anoop Jain, Graham Edge, Sachin Ahuja. How well do offline metrics predict online performance of product ranking models? In SIGIR 2023.

DATA-FEEDBACK LOOPS

• System-user interaction loop
• System recommends 🡪 Users respond 🡪 System logs 🡪 Model learns 🡪 System …
• Cycles reinforcing biases: leaving the unexplored unobserved
• Users influence and are influenced [1, 2]
• Reactively identify issues that bubble up
• Fix by explicitly tailoring the model = learning + data
• Limited scope and flexibility

Log responses

1. Seth Flaxman, Sharad Goel, and Justin M Rao. Filter bubbles, echo chambers, and online news consumption. In Public Opinion Quarterly, 2016
2. Sarah Dean, Evan Dong, Meena Jagadeesan, and Liu Leqi. Accounting for AI and users shaping one another: The role of mathematical models. On Arxiv 2024.

EMERGENT BEHAVIOR

• Emergent behaviour and unforeseen outcomes like
• Popularity bias [10] 🡪 lower diversity, click-baiting, etc. [4]�Performance paradox 🡪 homogenization [1] | filter-bubble [2,3]�Preferential attachment 🡪 steeper visibility barrier [5]�Echo chambers 🡪 opinion polarization [6,7]�Evolving users 🡪 influence on user preferences [8,9]

STATIONARY USERS 🡪 NON-STATIONARY USERS 🡪 INTERDEPENDENT INTERACTIONS

1. A. J. B. Chaney, B. M. Stewart, and B. E. Engelhardt. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. In RecSys 2018.
2. T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, and J. A. Konstan. Exploring the Filter Bubble: The Effect of Using Recommender Systems on Content Diversity. In WWW 2014.
3. G. Aridor, D. Goncalves, and S. Sikdar. Deconstructing the Filter Bubble: User DecisionMaking and Recommender Systems. In RecSys 2020.
4. S. Fortunato, A. Flammini, F. Menczer, and A. Vespignani. Topical interests and the mitigation of search engine bias. PNAS 2006.
5. Soumen Chakrabarti, Alan Frieze, Juan Vera. The Influence of Search Engines on Preferential Attachment. IM 2006.
6. Yingqiang Ge, Shuya Zhao, Honglu Zhou, Changhua Pei, Fei Sun, Wenwu Ou, and Yongfeng Zhang. Understanding echo chambers in e-commerce recommender systems. On Arxiv 2020.
7. Derek O’Callaghan, Derek Greene, Maura Conway, Joe Carthy, and P´adraig Cunningham. Down the (white) rabbit hole: The extreme right and online recommender systems. Social Science Computer Review, 2015.
8. Sarah Dean, Evan Dong, Meena Jagadeesan, and Liu Leqi. Accounting for AI and users shaping one another: The role of mathematical models. On Arxiv 2024.
9. Seth Flaxman, Sharad Goel, and Justin M Rao. Filter bubbles, echo chambers, and online news consumption. Public Opinion Quarterly, 2016.
10. Anastasiia Klimashevskaia, Dietmar Jannach, Mehdi Elahi, and Christoph Trattner. A survey on popularity bias in recommender systems. User Modeling and User-Adapted Interaction, 2024.

OUT OF THE LOOP

• What we want?
• Emerging phenomena and response curves.�Explore policies with desirable properties.
• What we have? Responses for logging policies.�What we need? Responses + learning of unexplored policies.
• What are our options? Play them all out.
• On live traffic: on-policy (bandit) evaluation + learning�On simulated traffic: off-policy (counterfactual) evaluation + learning�- Flexibility-fidelity trade-off

PLAY LIVE / LEARN FROM LOGS / EXPLORE IN SIMULATIONS

ON-POLICY: FIX IN PROD

• Bandits: sample-efficient exploration with partial-information
• Multi-armed: Stateless action space with stationary rewards | few arms, e.g. colour, font�Contextual bandits: Larger action spaces with correlations | enables personalization�Reinforcement Learning: long-term dynamics with temporal impact of actions
• What could go wrong?
• Evaluation: blind deployments | limited policies with bad-apple risk | could go off the rails �Training: requires online updates | action space complexity | infeasible offline eval�Infra limitation: engineering (scalable robust system) and time (warm-up) costs

​

​

LET’S TAKE-OFF FOR NOW AND GO FIGURE UP THERE!

Bandit feedback

OFF-POLICY: COUNTERFACTUALS

*WITH A LOAD OF BIAS IN DATA

1. T. Schnabel, A. Swaminathan, A. Singh, N. Chandak, and T. Joachims. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning, PMLR, 2016.
2. A. Sharma, J. M. Hofman, and D. J. Watts. Estimating the causal impact of recommendation systems from observational data. ACM Conference on Economics and Computation, 2015.
3. Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, Simon Dollé. Offline A/B testing for Recommender Systems. In WSDM 2018.

SIMULATING HYPOTHETICALS

• Various applications and interventions: use behavioural models for accurate simulations
• Online-news –SIREN [2] | Retargeting advertising –RecoGym [5] | Search and retrieval –AESim [6]�Marketplaces –Mars-GYM [4] | e-retailing –Virtual-Taobao [7] | Impact of local recommendations [3]�Incremental updates for fairness [1] | Causal inferences [8]
• Various phenomena and response curves: empirically replicate collective behaviour and study their effects
• Accordion for long-term dynamics | RecSim uncertainty modelling | T-RECS to study societal impact�ML-fairness-gym for fairness evaluation | Exposure bias | Bias amplification from feedback loops�Measuring system effects | RecLab for offline-online divergence | Longitudinal impact | evolution of user choices�Homogeneity effects [11,12]

​

1. G. K. Patro, A. Chakraborty, N. Ganguly, and K. Gummadi. Incremental fairness in twosided market platforms: on smoothly updating recommendations. In AAAI 2020.
2. D. Bountouridis, J. Harambam, M. Makhortykh, M. Marrero, N. Tintarev, and C. Hauff. SIREN: A simulation framework for understanding the effects of recommender systems in online news environments. In FAT 2019.
3. G. K. Patro, A. Chakraborty, A. Banerjee, and N. Ganguly. Towards safety and sustainability: Designing local recommendations for post-pandemic world. In RecSys 2020.
4. M. R. O. Santana, L. C. Melo, F. H. F. Camargo, B. Brand˜ao, A. Soares, R. M. Oliveira, and S. Caetano. Mars-gym: A gym framework to model, train, and evaluate recommender systems for marketplaces. In ICDMW 2020.
5. D. Rohde, S. Bonner, T. Dunlop, F. Vasile, and A. Karatzoglou. RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising. On Arxiv 2018.
6. Y. Gao, G. Huzhang, W. Shen, Y. Liu, W.-J. Zhou, Q. Da, and Y. Yu. Imitate theworld: A search engine simulation platform. On Arxiv, 2021.
7. J.-C. Shi, Y. Yu, Q. Da, S.-Y. Chen, and A.-X. Zeng. Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning. In AAAI 2019.
8. Y. Wang, D. Liang, L. Charlin, and D. M. Blei. Causal inference for recommender systems. In RecSys, 2020.

​

​

SIMULATING HYPOTHETICALS

• Various applications and interventions: use behavioural models for accurate simulations
• Online-news –SIREN [2] | Retargeting advertising –RecoGym [5] | Search and retrieval –AESim [6]�Marketplaces –Mars-GYM [4] | e-retailing –Virtual-Taobao [7] | Impact of local recommendations [3]�Incremental updates for fairness [1] | Causal inferences [8]
• Various phenomena and response curves: empirically replicate collective behaviour and study their effects
• Accordion [7] for long-term dynamics | RecSim [9] uncertainty modelling | T-RECS [10] to study societal impact�ML-fairness-gym [4] for fairness evaluation | Exposure bias [3] | Bias amplification from feedback loops [5]�Measuring system effects [6] | RecLab [8] for offline-online divergence | Longitudinal impact [1] | evolution of user choices [2] �Homogeneity effects [11,12]

​

1. A. Ferraro, D. Jannach, and X. Serra. Exploring Longitudinal Effects of Session-based Recommendations. In RecSys 2020.
2. N. Hazrati, M. Elahi, and F. Ricci. Simulating the Impact of Recommender Systems on the Evolution of Collective Users’ Choices. In HT, 2020.
3. A. Banerjee, G. K. Patro, L. W. Dietz, and A. Chakraborty. Analyzing ‘near me’ services: Potential for exposure bias in location-based retrieval. In BigData 2020,
4. A. D’Amour, H. Srinivasan, J. Atwood, P. Baljekar, D. Sculley, and Y. Halpern. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In FAT 2020.
5. M. Mansoury, H. Abdollahpouri, M. Pechenizkiy, B. Mobasher, and R. Burke. Feedback Loop and Bias Amplification in Recommender Systems. In CIKM’20.
6. S. Yao, Y. Halpern, N. Thain, X. Wang, K. Lee, F. Prost, E. H. Chi, J. Chen, and A. Beutel. Measuring Recommender System Effects with Simulated Users. In Workshop on FATESW, 2020.
7. J. McInerney, E. Elahi, J. Basilico, Y. Raimond, and T. Jebara. Accordion: A trainable simulator for long-term interactive systems. In RecSys 2021.
8. K. Krauth, S. Dean, A. Zhao, W. Guo, M. Curmei, B. Recht, and M. I. Jordan. Do offline metrics predict online performance in recommender systems? Arxiv, 2020.
9. M. Mladenov, C.-W. Hsu, V. Jain, E. Ie, C. Colby, N. Mayoraz, H. Pham, D. Tran, I. Vendrov, and C. Boutilier. RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems. On Arxiv 2021.
10. E. Lucherini, M. Sun, A. Winecoff, and A. Narayanan. T-recs: A simulation tool to study the societal impact of recommender systems. On Arxiv, 2021.
11. Allison J B Chaney, Brandon M Stewart, and Barbara E Engelhardt. 2018. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. In RecSys 2018.
12. Daniel M Fleder and Kartik Hosanagar. 2009. Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. Management Science 2009.

​

​

AD PLACEMENT: AUCTION SETTING

• Repeated real-time auctions | First price auctions | second price auctions | variants like reserve prices
• Limited budgets with varied objectives | bidders’ valuation unclear | information asymmetry
• Goal: learn to bid with bandit-feedback

1. Olivier Jeunen, Sean Murphy, and Ben Allison. Learning to Bid with AuctionGym. AdKDD 2022.
2. Olivier Jeunen, Sean Murphy, and Ben Allison. Off-Policy Learning-to-Bid with AuctionGym. SIGIR 2023.

​

…seller…

LEARN TO BID

1. Olivier Jeunen, Sean Murphy, and Ben Allison. Learning to Bid with AuctionGym. AdKDD 2022.
2. Olivier Jeunen, Sean Murphy, and Ben Allison. Off-Policy Learning-to-Bid with AuctionGym. SIGIR 2023.

​

Responds C

Observes W/C

Seller

*BATCH LEARNING WITH BANDIT FEEDBACK

SIMULATION AND LEARNING

1. Olivier Jeunen, Sean Murphy, and Ben Allison. Learning to Bid with AuctionGym. AdKDD 2022.
2. Olivier Jeunen, Sean Murphy, and Ben Allison. Off-Policy Learning-to-Bid with AuctionGym. SIGIR 2023.

​

WELFARE ACROSS PARTIES

1. Olivier Jeunen, Sean Murphy, and Ben Allison. Learning to Bid with AuctionGym. AdKDD 2022.
2. Olivier Jeunen, Sean Murphy, and Ben Allison. Off-Policy Learning-to-Bid with AuctionGym. SIGIR 2023.

​

BUSINESS KPI

1. Olivier Jeunen, Sean Murphy, and Ben Allison. Learning to Bid with AuctionGym. AdKDD 2022.
2. Olivier Jeunen, Sean Murphy, and Ben Allison. Off-Policy Learning-to-Bid with AuctionGym. SIGIR 2023.

​

ANALYZING REGRETS

1. Olivier Jeunen, Sean Murphy, and Ben Allison. Learning to Bid with AuctionGym. AdKDD 2022.
2. Olivier Jeunen, Sean Murphy, and Ben Allison. Off-Policy Learning-to-Bid with AuctionGym. SIGIR 2023.

​

EMERGING DYNAMICS

• Beyond response curves and behaviour: sellers adapt to changing competition
• Evolution of bidding strategies and resulting phenomena is impossible to capture without simulations
• All sides in a market tend to adapt: does the equilibrium have desirable properties?�SEO for webpages | Taste communities in music/videos (Spotify/Netflix) [1,2,3]�Content moderation on social media (Birdwatch) [4,5] | Auctioneers changing policies
• Markets are often multi-sided.
• Traditional recommendation systems tend to take two-sided view
• Advertisers – Ad Exchange – Consumers | Employers – Job Markets – Candidates�Drivers – Cab Aggregators – Riders | Content Creators – Publishing Platform – Viewers
• Does equilibrium of underlying dynamics have desirable properties? Can we drive it?
• Better-response dynamics [6] | Driving with incentives [7]

1. Josef Adalian. Inside the binge factory, 2018.
2. Glenn McDonald. On netflix and spotify, algorithms hold the power. but there’s a way to get it back. 2019.
3. Meena Jagadeesan, Nikhil Garg, Jacob Steinhardt. Supply-Side Equilibria in Recommender Systems. In NeurIPS 2023.
4. Stefan Wojcik, et al. Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation. Arxiv 2022.
5. Sameer Borwankar et al. Democratization of Misinformation Monitoring: The Impact of Twitter’s Birdwatch Program. In Information Systems Research, 2024.
6. Omer Ben-Porat, Itay Rosenberg, and Moshe Tennenholtz. Content provider dynamics and coordination in recommendation ecosystems. In NeurIPS 2020.
7. Arpita Ghosh and Patrick Hummel. Learning and incentives in user-generated content: multiarmed bandits with endogenous arms. ITCS 2013

METRICS AND LEARNED POLICIES DO NOT CAPTURE OVERALL WELFARE

STEERABLE AI SYSTEMS

1. Kianté Brantley, Zhichong Fang, Sarah Dean, Thorsten Joachims. Ranking with Long-Term Constraints. In WSDM 2024.
2. Jamie Morgenstern. Ensuring equity in online advertising for employment, housing, and credit, 2021.
3. Michael Jordan. Action AI Summit, 2025. https://www.youtube.com/watch?v=PrTd7yKPZJk

LEARN POLICIES WITH DESIRABLE PROPERTIES

[3]

QUESTIONS?

AI Economist: gather-and-build simulation to empirically�validate economic theories and design policies.

1. Stephan et al. The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies. On Arxiv 2020.
2. Alexander Levin et al. Simulation Intelligence: Towards a New Generation of Scientific Methods. On Arxiv 2022.