Predicting Familiarity in Recommender Systems

Group 7

Tavvi Taijala，Haiwei Ma，Wei Chen

Problem

RQ: How does familiarity impact the recommender system experience?

• Do users prefer to be recommended more familiar items?
• Do users prefer to be recommended less familiar items?
• Does this preference vary between users?
• Does this preference vary over time?

To answer this question, we need a way to manipulate familiarity experimentally.

Our approach was to build a model that could predict familiarity (accurately).

Dataset

Collected a gold standard familiarity dataset on MovieLens users.

• 1,000 MovieLens users
• 40,000 (user, movie, familiarity) data points

Familiarity was measured on a 6-point ordinal scale, condensed to a 3-point scale.

1 2 3 4 5 6

​

Low Med High

6 Classes:

​

3 Classes:

Never

heard of

Only heard

the name

Heard a

little about

Heard some

about

Heard a

lot about

Seen it

Approach

BPR

Models and Algorithms

​

Matrix Factorization Models

• BPR
• AdaptiveBPR
• WRMF
• FunkSVD

Classification Models

• OrdRec
• AdaBoost
• LogitBoost
• RandomForest
• Ordinal Random Forest
• Deep Neural Networks

​

Ordinal�Regressor

Serve as a

baseline:0.511

AdaBoost

100 trees: 0.598

LogitBoost

100 trees: 0.5976

Random�Forest

100 trees: 0.60674

Deep Neural�Network Classifier

3 layers[10,20,10]: 0.625532

4 layers[10,20,20,10]: 0.632290

5 layers [40,20,20,20,10]: 0.7602594

​

Ordinal�Random�Forest

100 trees: 0.623

K-folder Cross Validation Results

Random Forest Vs Ordinal Random Forest

• Accuray
• Non-ordinal random forest: 0.61454
• Ordinal random forest: 0.60674
• Confusion matrix
• Non-ordinal random forest: Ordinal random forest:

​

Experimental Setup

​

Evaluation Metrics

• Accuracy
• Per-class accuracy
• Confusion matrices

Crossfold Validation Approaches

• K folds
• K stratified folds
• K out of sample folds

​

Out of Sample Validation Results

Next Steps

Neural Networks

• Pretraining
• Ordinal Classification

Synthetic Training Data

Thanks!