Keep the Positive in the Loop:�A Field Experimental Study of the Impact of Solicited Collective Feedback in Content Recommendation Systems

​

Changrong Xiao^a

xcr21@mails.tsinghua.edu.cn

​

Joint work with Yilin Li^b, Chong Alex Wang^c, Sean Xin Xu^a, and Jiayin Zhang^a

^a Tsinghua University, School of Economics and Management

^b Peking University, Guanghua School of Management

^c City University of Hong Kong, College of Business

Two Typical Cases

The inconsistency between engagement and preference

2

More engagement ≠ More satisfaction/perceived value

Just watch one more! Just one more!

Let’s stop here and do something else!

Satisfaction, retention, but user engagement may not be optimal

Engagement, but users may suddenly leave the platform

Introduction

The inconsistency between engagement and preference

• The recommender systems are optimized based on user behavior data reflecting short-term engagement,
• but this data does not necessarily align with users’ true preferences.

​

Severe problems:

• Algorithm learned biased user preference, and some preferences remain undiscovered
• Creators get the wrong signal and produce more clickbait content, which pollutes the platform content environment and makes it harder for users to find satisfaction in their engagement,
• Harming users’ feelings and retention, and the platform may finally lose these users.

3

Introduction

Aim: Enhance the alignment between recommender systems and user preferences

Challenge: unable to directly observe user preferences

Proposed solution:

4

1. Solicit user preference feedback

2. Aggregate group opinions

3. Design two recommendation strategies

Distributing pop-up preference surveys on popular videos

Calculate user-video pairwise preference level that identify videos as high- and low-preference

Boosting and filtering high- and low-preference videos in a field experiment

Findings: when carefully designed, the group preference information can complement personalized preference to achieve higher user engagement and retention.

Design & Experiments

5

Design

• We collaborate with a leading Chinese short video platform
• Featured page:
• Recommend popular contents based on users’ historical behavior;
• Full-screen with interactive side buttons;
• Scrolling up and down to navigate through videos

​

• Step 1: Preference surveys
• Randomly distributed with popular videos in the featured page;
• Pop-up during video play;
• "Satisfied," ”Unsatisfied," "Uncertain," or ignore the questionnaire
• Stop when enough responses were collected (> 200 per video)

​

Advantages of survey:

• Users cannot see the response of others, reducing social influence bias.
• Creator manipulation of results is prevented.
• Include negative/mutual options.

Advantages of targeting on popular videos:

• Draw more attention; more impactful;
• More representative survey responses, ensuring statistically reliability

​

​

6

Design

• Step 2: Aggregate group opinions → represent overall user-video pairing preference
• “Wisdom of the crowd”
• Under the influence of recommender system, users who view the same video exhibit similarities

​

• User-video pairwise preference level
• Straightforward metrics: total votes, positive votes, negative votes, uncertain votes, and respective rates

​

​

7

Design

8

Random grouping: Boost, filter, or control

Filter

Boost

Control

Ranking

Boost

Output

Filter

Retrieve

Retrieve

Retrieve

Filtering: Removing the “bad”

Remove the low-preference videos from the retrieval set

Boosting: Enhancing the “good”

After ranking, adjust the video order to prioritize the high-preference videos.

Step 3: Boosting and Filtering strategies

Experiment

Conduct two independent field experiments on device level.

• Boosting experiment
• launched on May 27, 2022, with one week buffer time.
• Pre-treatment period: May 20-26
• Post-treatment period: June 3-9
• Filtering experiment
• launched on July 1, 2022, with one week buffer time.
• Pre-treatment period: June 24-30
• Post-treatment period: July 8-14
• Control group
• Keeping the recommendation logic unchanged before and after the experiment.

9

Analysis & Findings

10

Variables and Descriptive Statistics

• Dependent Variables: Engagement + Retention
• Content consumption: click, dwell time
• Responses: like, comment, share, collect
• Activeness and Retention: average active ratio, seven-day activeness in post-treatment period; average return ratio, seven-day retention in subsequent period
• Control Variables:
• Demographic device characteristics: gender, age range, country region, and city level
• Social features of devices: fans, follows, and friends

11

Sample Selection

• Employ a modified intention-to-treat (mITT) analysis: excluding devices unaffected by the treatment
• Boosting: sample selection + matching
• Choose devices that have high-preference video exposure in the treatment group during the post-treatment period
• Match with the control group during the pre-treatment period to obtain comparable samples
• Other options: only sample selections? No, underestimate the treatment effect!
• Filtering: sample selection
• choose devices that have low-preference video exposure in both treatment and control group during the pre-treatment period

12

Sample Selection

• Boosting: sample selection + matching
• PSM: balance check √
• Filtering: sample selection
• Balance check √

13

DID Analysis

Model:

• Difference-in-Difference: with control variables and individual-fixed effect.
• Treatment: Boost or Filter (independently)

​

​

• Parallel trend test: √
• Manipulation check: √
• Boosting: Exposure concentration of high-preference videos increased
• Filtering: No low-preference videos in treatment group
• Both provide recommendation sets of higher user preference levels

​

14

DID Analysis

Boosting experiment:

• Engagement: Content Consumption increased, Responses increased
• Activeness and Retention increased

​

​

​

15

DID Analysis

Boosting experiment:

• Engagement: Content Consumption increased, Responses increased
• Activeness and Retention increased

​

Filtering experiment:

• Engagement: Content Consumption decreased, Responses decreased
• Activeness and Retention unchanged

​

​

​

16

Additional Analysis

Content Diversity

• Variables:
• Exposed and played content category coverage and entropy
• Exposed and played newly introduced content categories
• Results
• Boosting: increased
• Filtering: decreased

​

​

​

17

Additional Analysis

Content Diversity

• Results
• Boosting: increased
• Filtering: decreased
• Implications
• Boosting: prioritize high-preference videos → introduce additional video categories in users’ undiscovered preferences → more engagement and retention
• Filtering: remove low-preference videos → exclude diverse and niche preferences learned by personalized RS → harm user engagement and retention

​

​

​

18

Additional Analysis

Evidences from Consumptions of High- and Low-Preference Videos

• High-preference videos
• User consumption behaviors positively correlated with positive voting rates
• Low-preference videos
• The correlation is insignificant
• Lower negative voting rates do not always indicate less preferences
• Implications

​

​

​

19

“All happy families are alike; each unhappy family is unhappy in its own way.”

Conclusion & Discussion

• Contributions
• Addressed the inconsistency between user engagement and preferences, providing empirical validation of theoretical derivations by Kleinberg et al. (2024) through field experiments
• Designed and implemented a practical approach to capture group opinions and apply them as actionable recommendation strategies in real-world platform
• Investigate the mechanisms behind the different impacts of high- and low-preference content on user behaviors, providing insights for future researchers and industrial practitioners
• Limitations
• Solicited user feedback can still be biased
• Filtering strategy can be further optimized

20

THANK YOU!