Trait Study of Semantic Aware Shilling Attacks And Detection
for Recommender System
Presented by:
Farhana Khan (1705100)
Sumaiya Azad (1705048)
1
Supervisor:
Dr. Mahmuda Naznin
Professor
Department of CSE, BUET
Introduction
&
Motivation
2
Recommendation System
3
Which movie do I watch next?
Types
Content-based
Filtering
Collaborative
Filtering
User’s own information
Other users information
Focus of our work
Recommender System
4
Shilling Attack
5
Shilling Attack types
Push
Nuke
To Increase Popularity
To Decrease Popularity
Semantic Aware Shilling Attack (SAShA)
6
6
The Structure of a Shilling Profile (the Fake Users)
- Selected items
- Using the adversary’s knowledge on the system data
- Unrated items
- Items not rated
Target Item
Pushed or Nuked
- Filler items
- Items randomly selected to rate
Knowledge graph addition:
Attack strength increases
Knowledge Graph
7
Represents a network of real-world entities (objects, events, situations, or concepts)
Illustrates the relationship between them
Christopher Nolan
Inception
Interstellar
Directed by
Directed by
Similar item
Shilling Attack Traits
8
Attack size
Filler size
Higher attack size
Expensive attack
Harder for the attacker
Attackers target: high impact with small attack size
Higher filler size
Expensive and conspicuous attack
Harder for the attacker, easier to detect
Attackers target: high impact with small filler size
Motivation
9
- Collaboration of knowledge graphs and recommendation systems
- Currently attracting interests of researchers in this field
- Public Knowledge graph is used to launch stronger shilling attacks
- lack of study of defence
- lack of study of attack traits of semantic aware shilling attack
- cost effectivity
Related Work
10
SAShA: Semantic-Aware Shilling Attacks on Recommender Systems Exploiting Knowledge Graphs [1]
Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, Felice Antonio Merra ESWC 2020
11
Semantic-Aware Shilling Attacks on Recommender Systems Exploiting Knowledge Graphs
12
Problems Addressed :
Findings:
Gaps:
Preventing Shilling Attacks in Online Recommender Systems [2]
Chirita, Paul-Alexandru, Wolfgang Nejdl, and Cristian Zamfir
Proceedings of the 7th annual ACM international workshop on Web information and data management. 2005.
13
Preventing Shilling Attacks in Online Recommender Systems
14
Problems Addressed :
Findings:
Gaps:
Problem Domain
15
Our Challenges
We tried to answer these following research questions:
16
Attack Size
Filler Size
Detection Algorithms performance
Dataset density
Our Challenge 1: Reach of Target Items in Various Attack Sizes
In our experiment we have -
17
Our Challenge 2: Influence of Different Filler Length
In our experiment we have -
18
Our Challenge 3: Effectiveness on Different Density Datasets
19
Yahoo Movies
Dense
High user-item interaction. (a lot of users have rated a lot of items)
Small Library Things
Sparse
Low user-item interaction
| Yahoo Movies | Small Library Things |
dataset trait | dense | sparse |
Number of users | 860 | 1357 |
Number of items | 629 | 701 |
Total ratings | 6919 | 7652 |
Average number of items rated by each user | 8 | 5 |
Average number of users rated an item | 11 | 10 |
Our Challenge 4: Performance of Detection Methods
2 Widely used Detection methods
20
Number of Prediction Difference
(NPD) [2]
Principal Component Analysis
(PCA) [3]
Our Challenge 4: Performance of Detection Methods
= number of recommendation prediction changes, removing u from the system
21
Small Library Things (sparse)
Yahoo Movies (dense)
Our Challenge 4: Performance of Detection Methods
22
Red marked profiles are shillers
Our Solution Approach: Dataset Preprocessing
23
Our Solution Approach: Attack Models
24
Base Attacks
Semantic Aware Attacks
Our Solution Approach: Recommendation Systems
25
Memory Based
(Classical)
User Based
Model Based
(Popular)
Matrix Factorization
Collaborative Filtering Recommendation System
Item Based
- Decomposes the user-item interaction matrix into latent factors
- Latent factors captures hidden characteristics of users and items
User-user similarity
Item-item similarity
Our Solution Approach: Performance Measuring Parameter
26
Hit Ratio
Accuracy
Reach of Target Item
On the Top k Recommendations
Detected Fake Profiles
Number of Real Users Marked True
Our Solution Approach: Summary View
27
KG Data
(public)
RS Data
(partially public)
Step 1: List
Target items
Step 2: Generate pre-attack recommendations
Step 4: Generate post-attack recommendations
Step 3: Generate
fake profiles
Step 5: HR@k vs attack traits
Challenges 1, 2, 3
Step 6: Detect fake profiles
Step 7: Detection Accuracy Challenge 4
Results:
Attack Traits
28
Our Findings: Attack Traits
29
Slower decrease
In Matrix Factorization RS, higher the number of similar filler items, stronger the semantic attacks
Our Findings: Attack Traits
30
Our Findings: Attack Traits
31
Dataset RecSys | Dense (Yahoo movies) | Sparse (Small library things) | Inference | ||
| Base Attacks | Semantic Attacks | Base Attacks | Semantic Attacks | |
Matrix Factorization | filler size ↑ HR@k ↓ | filler size ↑ HR@k slow ↓ | filler size ↑ HR@k ↓ | filler size ↑ HR@k slow ↓ | in both datasets, adding higher number of similar filler items is causing the semantic attack to be stronger |
Item Based | attack size ↑ HR@k ↓ | attack size ↑ HR@k ↑ | attack size ↑ HR@k ↓ | attack size ↑ HR@k ↑ | in both datasets, adding higher number of similar users is causing the semantic attack to be stronger |
User Based | all attack traits are same, infact baseline attacks perform better than semantic attacks | cannot leverage item similarities information from knowledge graph | |||
Table: Summary Table of Our Findings
Our Findings: Attack Traits
32
Increased performance
In Item based RS, higher the number of similar fake users, stronger the semantic attacks.
Our Findings: Attack Traits
33
This is reflected in the overall performance of the semantic attacks
Our Findings: Attack Traits
In Item based RS, higher the number of similar fake users, stronger the semantic attacks
34
Our Findings: Attack Traits
35
Dataset RecSys | Dense (Yahoo movies) | Sparse (Small library things) | Inference | ||
| Base Attacks | Semantic Attacks | Base Attacks | Semantic Attacks | |
Matrix Factorization | filler size ↑ HR@k ↓ | filler size ↑ HR@k slow ↓ | filler size ↑ HR@k ↓ | filler size ↑ HR@k slow ↓ | in both datasets, adding higher number of similar filler items is causing the semantic attack to be stronger |
Item Based | attack size ↑ HR@k ↓ | attack size ↑ HR@k ↑ | attack size ↑ HR@k ↓ | attack size ↑ HR@k ↑ | in both datasets, adding higher number of similar users is causing the semantic attack to be stronger |
User Based | all attack traits are same, infact baseline attacks perform better than semantic attacks | cannot leverage item similarities information from knowledge graph | |||
Table: Summary Table of Our Findings
In User based, HR@k vs Attack size
Our Findings: Attack Traits
36
In User based, HR@k vs Filler size
Our Findings: Attack Traits
37
In User based, semantic attacks performs just as good as base attacks, if not worse
Our Findings: Attack Traits
38
Our Findings : Summary
39
Dataset RecSys | Dense (Yahoo movies) | Sparse (Small library things) | Inference | ||
| Base Attacks | Semantic Attacks | Base Attacks | Semantic Attacks | |
Matrix Factorization | filler size ↑ HR@k ↓ | filler size ↑ HR@k slow ↓ | filler size ↑ HR@k ↓ | filler size ↑ HR@k slow ↓ | in both datasets, adding higher number of similar filler items is causing the semantic attack to be stronger |
Item Based | attack size ↑ HR@k ↓ | attack size ↑ HR@k ↑ | attack size ↑ HR@k ↓ | attack size ↑ HR@k ↑ | in both datasets, adding higher number of similar users is causing the semantic attack to be stronger |
User Based | all attack traits are same, infact baseline attacks perform better than semantic attacks | cannot leverage item similarities information from knowledge graph | |||
Table: Summary Table of Our Findings
Results:
Detection Methods
40
Our Hypothesis: Detection Methods
41
Semantic aware shilling profiles are crafted to be more like real users
Should be better at fooling the detection methods
Accuracy of detection methods
Our Findings: Detection Methods
42
PCA performed better on Semantic aware attacks
e.i,
Contradicts hypothesis
Our Findings: Detection Methods
43
Our Hypothesis Based on Results
In sparse dataset semantic aware attacks are easily detected by detection methods based on profile similarity trait.
Our Result Explanation: Detection Methods
44
In sparse dataset
Less user-item interaction
Probability of having rated common items by two real users is small
Similarity between real users is very small
Our Result Explanation: Detection Methods
45
Filler items are randomly chosen
What happens in base attacks?
No similarity between items
Our Result Explanation: Detection Methods
46
What happens in Semantic attacks?
Our Result Explanation: Detection Methods
47
Target item
Filler items are sampled from similar items
Filler items are similar to each other
Generate items similar to target using KG
Increases similarity between fake profiles
Conspicuous profile similarity
Our Result Explanation: Detection Methods
48
Clustered into 2 groups based on profile similarity
Green : detected fake profiles
Red : fake profiles
PCA of datasets
Our Findings: Detection Methods
49
Even though semantic awareness is strengthening the attack,
Conclusion
50
Future Work
51
Future Work
52
Reference
53
Thank You
Q/A
54