Off-policy Learning over Heterogeneous Information for Recommendation

UTS CRICOS 00099F

Xiangmeng Wang ¹, Qian Li ², Dianer Yu ¹, Guandong Xu ¹

​

1 Data Science and Machine Intelligence (DSMI) Lab, School of Computer Science, University of Technology Sydney

2 School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University

Presentation Outline

• Background: The Bias issue in off-policy learning setting
• Motivation: Debiasing with context information
• Methodology
• Evaluation
• Conclusion

• Main goal
• Take dynamic user feedback on actions to train RS (i.e., agent) for learning an optimal recommendation policy

​

• Advantage
• Unlike static methods (e.g., matrix factorization[1]), it dynamically learns recommendations based on user interactions (i.e., rewards) with the agent
• Challenge
• The data collection requires the agent to interact online with users, which is infeasible and expensive

​

​

​

Background: Reinforcement learning (RL) based-Recommendation

​

Content Discovery

User dynamics

Long-term User Satisfaction

Recommender

Rewards (click?)

Action (rec items)

[1] Koren Y, Bell R, Volinsky C. Matrix factorization techniques for recommender systems[J]. Computer, 2009, 42(8): 30-37.

• Off-policy Learning

​

​

​

​

​

• Achieves policy learning through logged user feedback data, without interactive access to the online environment

Background: Off-policy Learning and the Bias Issue

User feedback collection

Policy deployment

• Bias issues in off-policy learning
• Logged data provide partial information while the rewards for rare actions are missing.

Presentation Outline

• Background: The Bias issue in off-policy learning setting
• Motivation: Debiasing with context information
• Methodology
• Evaluation
• Conclusion

• The Inverse Propensity Scoring methods[2]:
• Re-weight each sample via the propensity score, i.e., assign low probabilities to rare actions, to force the target policy NOT to visit rare actions.

​

• The Poor get the poorer phenomenon
• e.g., a headset will not be recommended simply because it never appeared in the logging policy.

​

​

​

​

​

​

Motivation: Existing Bias Correction Works

[2] Williamson E J, Forbes A. Introduction to propensity scores[J]. Respirology, 2014, 19(5): 625-635.

• Our motivation
• Real-world context information could be useful to infer the missing rewards for rare actions.

​

​

​

​

​

​

​

​

​

• Our investigation

​

• Incorporate context information in HIN to empower off-policy learning in the recommendation
• Develop an unbiased learning objective for achieving unbiased off-policy learning

​

​

Motivation: Debiasing with Context Information

Recommend

action

Presentation Outline

• Background: The Bias issue in off-policy learning setting
• Motivation: Debiasing with context information
• Methodology
• Evaluation
• Conclusion

Off-policy Learning over Heterogeneous Information for Recommendation (HINpolicy)

​

Main component

• Context Representation Learning: co-attentive mechanism on state, action, and context representations

​

• HIN-enhanced policy learning: aggregates context representations to learn the target policy

​

• Counterfactual Risk Minimization to correct the discrepancy between target policy and logging policy

​

Methodology: Model Framework

Methodology: Context Representation Learning

• Context Representation initialization:
• Sample meta paths that relate to users and items from the given HIN; pass CNN layers to learn meta path embeddings

• Context Representation Learning:
• Learn user-action interaction-specific context representations by co-attentive on State, Action, and Context representations

​

Methodology: HIN-enhanced policy learning

Methodology: Counterfactual Risk Minimization

• The clipped estimator (cIPS):
• Caps the propensity ratios (i.e., importance weight) between logging policy and target policy to a maximum value to pursue the unbiased policy optimizing

​

​

​

• Use the cIPS for unbiased policy learning

​

Presentation Outline

• Background: The Bias issue in off-policy learning setting
• Motivation: Debiasing with context information
• Methodology
• Evaluation
• Conclusion

Evaluation: Recommendation performance

• Experiment
• Task: Recommendation performance

​

• Our proposed HINpolicy consistently yields the best performance among all datasets on both evaluation metrics

​

• All baseline models perform the worst on the sparsest Douban Book, our HINpolicy still outperforms other models since HINpolicy can handle data sparsity

​

The SOTA model

Table. Performance comparison with baselines.

Evaluation: Ablation Study

• Task: Study of HIN
• HINpolicy augmented with HIN outperforms the counterpart without HIN

​

• HINpolicy-w/ HIN outperforms HINpolicy-w/o HIN in different user groups as denoted in the Figures.

​

• HIN information has a critical effect on the sparse dataset - Douban Book, especially on the sparsest user group (i.e., ratings <500)

​

​

​

​

​

​

​

​

​

Presentation Outline

• Background: The Bias issue in off-policy learning setting
• Motivation: Debiasing with context information
• Methodology
• Evaluation
• Conclusion

Conclusion

• Our contributions:
• We are the first to leverage contextual information in HIN to provide high-quality target policy learning for correcting the bias in off-policy recommendations.

​

• We develop a new end-to-end framework HINpolicy, which achieves counterfactual risk minimization in an explicit manner under the co-attention mechanism.

​

• Empirically, we generate an online environment using simulators to carry out experiments on two benchmark datasets. Extensive results show that our methods outperform the state-of-the-art methods.

​

​

​

​

​

​

Questions?

UTS CRICOS 00099F

Get in touch:

​

Xiangmeng Wang (xiangmeng.wang@student.uts.edu.au)

Dr. Qian Li ( qli@curtin.edu.au)

Prof. Guandong Xu (Guandong.Xu@uts.edu.au)