On the True Risks of

Poisoning Attacks on Federated Learning

Amir Houmansadr

2

…

Central Server

The poisoning adversary

owns/controls multiple compromised clients

The adversary leverages its compromised clients to

corrupt the global model,

by sending malicious model updates

Poisoning adversary

FL’s Achilles’ heel!

FL’s Secret Sauce!

Lack of Trust

This Talk!

• Discuss the significance of poisoning attacks to production FL systems
• Relevant attack variants, threat models
• Performance
• Overview attack and defense mechanisms

3

Outline

• Understanding FL poisoning
• Types
• Systemization
• Our poisoning techniques
• Data and model poisoning
• Defenses
• Federated Supermask Learning
• Evaluations for production FL
• Various datasets, models, and techniques

​

4

Outline

• Understanding FL poisoning
• Types
• Systemization
• Our poisoning techniques
• Data and model poisoning
• Defenses
• Federated Supermask Learning
• Evaluations for production FL
• Various datasets, models, and techniques

​

5

Types of FL Poisoning: Goal

​

• Targeted attacks
• Backdoor attacks
• Semantic Backdoors
• Artificial Backdoors
• Untargeted attacks

6

Types of FL Poisoning: Goal

7

Targeted Attacks

Misclassify a small set of inputs

These inputs need not have any specific properties

Correctly classify all the other inputs

Sample universe

Types of FL Poisoning: Goal

8

Backdoor Attacks – Semantic Backdoors

Correctly classify all the other inputs

Misclassify a small set of inputs

These inputs should have specific properties naturally present inside them

Sample universe

Types of FL Poisoning: Goal

9

Backdoor Attacks – Artificial Backdoors

F

+

Misclassify any input when modified in a certain fashion

These inputs have specific backdoor trigger that is manually added to them

Correctly classify all the other inputs

Sample universe

Types of FL Poisoning: Goal

10

Untargeted Attacks

Degrade classification performance on arbitrary inputs without modifying the inputs

Sample universe

Three Classes of FL Poisoning

11

Which one we should care about more

for production FL?

Untargeted Poisoning �Impacts More Participants

12

Affects the entire sample space, and hence, affect all FL clients!

Affect only a tiny fraction of sample space, and hence, does not hurt most FL clients

Outsized impact factor (OIF) =

(# of successful backdoors)/(# of adversary’s data points)

​

Previous work: 0.002 to 0.02 (attacks are ineffective for larger OIFs)

Untargeted

Targeted/backdoor

Impact factor: https://www.youtube.com/watch?v=poOgbgGeySk&list=PLSIUOFhnxEiCJS8q6SYdc0944xlV_6Jbu&index=15

Untargeted Poisoning�Is More Difficult to Detect

13

Misclassification

Confidence reduction

Reduce overall accuracy by small percentages and remains undetected

Still it can impact all samples (FL clients) to large extents

Benign

Poisoned

Model accuracy

Non-suspicious reduction in accuracy

Model confidence

Impact on individual samples

Impact from the server’s point of view

Untargeted Poisoning �Is More Challenging to Succeed

14

Model confidence

Model confidence

Targeted/backdoor attacks should manipulate only a few test inputs that are already vulnerable or are made vulnerable (e.g., by adding a patch)

Untargeted attacks should manipulate all the test inputs that are not altered, and hence, not vulnerable in any way

Test inputs

Test inputs

Types of FL Poisoning: Mechanism

15

Adversary’s Capabilities

Model poisoning

Data poisoning

break-in to access these parts of device

• Can directly manipulate model updates
• Sophisticated but few compromised clients
• Highly impactful poisoned update

• Can indirectly manipulate model updates
• Naïve but many compromised clients
• Poisoned updates with relatively less impact

No break-ins required

Systematization of FL Poisoning

16

Adversary’s Knowledge of the Global Model

Whitebox access

Nobox access

• Complete access to device and model
• Few compromised clients
• Can directly manipulate model updates

• No access to device and model
• Many compromised clients
• Only indirectly manipulate model updates

Systematization of FL Poisoning

17

Mode of Poisoning Attack

Online attack

Offline attack

• Repeatedly and adaptively poison the model updates
• Requires whitebox access

• Poison the model updates only once
• Requires just nobox access

Outline

• Understanding FL poisoning
• Types
• Systemization
• Our poisoning techniques
• Data and model poisoning
• Defenses
• Federated Supermask Learning
• Evaluations for production FL
• Various datasets, models, and techniques

​

18

Defending Against Poisoning in FL

19

Server

…

Key Idea of defenses:

aggregate updates from clients

​

- To attenuate malicious updates

- With minimal impact on model utility

The Goal of Poisoning

• Circumvent the deployed FL aggregation rule (AGR) to reduce model’s accuracy
• Various AGR algorithms (defenses) exist

​

​

​

​

20

High-level Intuition of Poisoning Attacks

21

21

Aggregate

Benign updates

Find malicious updates in the space of possible updates which maximize the distance between benign and malicious aggregates

Attacks are tailored to given AGR: Constraints of AGR decide the space of benign updates

Space of updates

(for a specific AGR)

Malicious update

Our (Untargeted) Model Poisoning

22

Exploit: Access to global model

Intuition: 1. Increase the loss on benign training data via stochastic gradient ascent (instead of decent)

Intuition: 2. Scale the update to circumvent detection by given AGR

V. Shejwalkar, A. Houmansadr. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning, NDSS 2021

Our (Untargeted) Data Poisoning

23

Exploit: Server has no visibility into the data on client devices

Intuition: 1. updates computed using more mislabeled data have higher losses and larger norms

Mislabeling strategies: Static label flip (SLF) and dynamic label flip (DLF)

Intuition: 2. Adjust size of mislabeled data to circumvent given AGR

V. Shejwalkar, A. Houmansadr, P. Kairouz, and D. Ramage. Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Federated Learning, https://arxiv.org/pdf/2108.10241.pdf

A Peek at the Performance

24

25% reduction

in accuracy!

Performance depends on various factors, e.g., ratio of malicious clients, AGR techniques, FL algorithm, number of rounds, etc.

Outline

• Understanding FL poisoning
• Types
• Systemization
• Our poisoning techniques
• Data and model poisoning
• Defenses
• Federated Supermask Learning
• Evaluations for production FL
• Various datasets, models, and techniques

​

25

High-level Intuition of Poisoning Attacks

26

26

Aggregate

Benign updates

Find malicious updates in the space of possible updates which maximize the distance between benign and malicious aggregates

Attacks are tailored to given AGR: Constraints of AGR decide the space of benign updates

Space of updates

(for a specific AGR)

Malicious update

​

​

Promising defense strategy:

Shrink the

space of acceptable updates

(with minimal impact

on model accuracy)

to reduce adversary’s choices

​

e.g., reduce the dimension of updates

Not all space reduction is equal!

27

Malicious update

Malicious update

The shrunken space excludes many benign updates

(hurts utility)

The shrunken space includes many malicious updates

(not robust)

• Two dimensionality reduction approaches

​

• Using knowledge transfer
• Distill local model’s information through public data
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer https://arxiv.org/pdf/1912.11279.pdf
• Using gradient ranks
• Train gradient ranks, not gradient values
• FSL: Federated Supermask Learning https://arxiv.org/abs/2110.04350

​

28

Federated Supermask Learning (FSL)

• Reduces the space of model updates by sending parameter ranks instead of gradient updates
• To Improve robustness
• But also improves communication efficiency

29

H. Mozaffari, V. Shejwalkar, and A. Houmansadr, FSL: Federated Supermask Learning, https://arxiv.org/abs/2110.04350

Federated Supermask Learning

30

Server

…

• Server sends some ranking for model parameters
• Each client sends back some updated ranks (based on local data)

Key Technique: Supermasks

Lottery Ticket Hypothesis [Frankle et al. ICLR 2019]:

• Modern neural networks are generally overparametrized
• A fully-trained neural network contains sparse subnetworks that can be trained from scratch and achieve good performance close to trained original network.

Supermask [Zhou et al. NeurIPS 2019]:

• Instead of training weights, find a subnetwork in randomly initialized neural network.
• A binary mask of 1’s and 0’s that is superimposed on the random neural network to obtain a final subnetwork.

​

31

Server Communications

• Initialization: create a random network (random weights and scores) using a SEED
• Round 1: Broadcast random weights and random scores
• Through the SEED
• Weights and scores remain fixed throughout training
• Round t: Broadcast the current global ranking

32

Client Computations (each round)

• Assign initial scores to edges based on their global ranking
• Update the scores using Edge-PopUp Alg. [Ramanujan et al. CVPR 2020], [Wortsman et al. NeurIPS 2020]:
• Assign a score to each edge in the network
• Train on data, find best edges to minimize the loss
• If adding an edge to the subnetwork increases the loss, reduce its score

33

Client Updates (each round)

• Each client computes local edge ranks (based on updated scores)
• Local ranks are sent to the server

34

Server Aggregation (each round)

• The server aggregates client ranks through a Voting mechanism
• Assign reputation for each vote
• Sums the reputations, and sort them to obtain global ranking for next round

35

Overview of FSL

Instead of weight training, we ask the clients to rank the edges of a random neural network.

​

• FSL Server instructs each client to:
• Start from global ranking that the server announced for this round
• Train a subnetwork with size k% of the original network
• Order the importance of the edges for her local data
• Send back a local ranking of the edges

​

• Server aggregates ranks to obtain the global ranking for next round

36

FSL’s Communication Costs

• FSL achieves similar performance as FedAvg with lower communication cost
• Reduces costs from n*32 (bits) to n*log(n) (bits)
• By sending and receiving rankings instead of model updates
• On CIFAR10, distributed non-iid over 1000 clients, we achieve 85.3% accuracy while reducing communication cost by ~35%.

​

37

FSL’s Robustness

38

Superior robustness

thanks to

smaller update space

Check the paper for bounds on robustness

Takeaways From FSL

• Ranking-based FL is a promising space reduction technique
• Improves robustness
• Reduces communication overhead
• There are robust aggregation mechanisms with provable guarantees for ranking data
• E.g., Simple voting
• Other nice features of a ranking-based approach
• Can be used to personalize models, achieve fairness
• Can be used to unlearn malicious clients in the future
• Future work can develop more effective ranking and ranking aggregation mechanisms

39

Outline

• Understanding FL poisoning
• Types
• Systemization
• Our poisoning techniques
• Data and model poisoning
• Defenses
• Federated Supermask Learning
• Evaluations for production FL
• Various datasets, models, and techniques

​

40

Types of FL Poisoning

​

• Targeted attacks
• Backdoor attacks
• Semantic Backdoors
• Artificial Backdoors
• Untargeted attacks

41

What type is

the most relevant to

production FL?

What is Production FL Anyways?

42

Cross-device FL

Cross-silo FL

• Number of clients from thousands to billions
• Small fraction of clients processed per round
• Examples: Gboard (all Android users), Apple Siri (All iOS users)

• Number of clients up to few hundreds
• All clients processed per round
• Examples: Banks, hospitals

The Gap Between Theory and Practice

43

Existing works use unrealistic ranges while evaluating their attacks (and defenses)

Unrealistic Percentages of Compromised Clients

Not all combinations are practical!

44

Practicality of Threat Models

Cross-silo FL + Model poisoning

Cross-silo FL + Data poisoning

• Silos are large organizations, e.g., banks
• Have sophisticated security measures
• Impossible to break-in without detection

Nobox Online Data poisoning

Nobox Offline Data poisoning

Experimental Setup

• Cross-device FL
• FEMNIST + LeNet + 34,000 clients
• CIFAR10 + VGG9 + 1,000 clients
• Purchase + 1-layer FC + 5,000 clients
• Cross-silo FL
• FEMNIST, CIFAR10 + 50 clients
• Client Data Distribution
• FEMNIST: Naturally non-iid
• CIFAR10 and Purchase: Non-iid using Dirichlet distribution

45

Attack Impact: Reduction in accuracy due to the attack, compared to the FL setting without any compromised clients

�Existing Attacks Are Not Quite Impactful!

46

No impact

No impact

No impact

 Even the simple, low-cost robust AGRs are enough to protect production FL against untargeted poisoning.

�Simple Countermeasures May Be Enough!

47

 Enforcing a limit on the size of the dataset contributed by each client can act as a highly effective (yet simple) defense against data poisoning

No impact even with 10% compromised clients

Evaluating Non-Robust FL

48

Practical % for model poisoning

Practical % for data poisoning

Cross-device FL with (the naive) Average AGR converges with high accuracy, i.e., is highly robust to poisoning attacks for practical percentages of compromised clients.

Robustness Over Time

49

Robustness of AGRs persists even when compromised clients consistently poison cross-device FL for large number of rounds.

Evaluating Robust FL

50

Norm-bounding is more robust

 Understanding the robustness of AGRs in production FL requires a thorough empirical assessment of AGRs, on top of theoretical analysis.

Evaluating Cross-silo FL

51

No impact even with 10% compromised clients

Against Data Poisoning Attacks

Model poisoning is not practical in Cross-silo FL

 For cross-silo FL, model poisoning attacks are not practical and state-of-the-art data poisoning attacks have no impact even with non-robust Average AGR

No impact even with non-robust Average AGR

Summary

• Poisoning can be a major obstacle to the adoption of FL at scale
• But needs to be evaluated for realistic settings
• Existing defenses must be tailored to the unique constraints of production systems
• Heterogenous devices
• Limited/sparse connectivity
• Compression requirements
• Other contending issues
• Privacy
• Fairness

​

52

Related Papers

• V. Shejwalkar, A. Houmansadr, P. Kairouz, and D. Ramage. Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Federated Learning, https://arxiv.org/pdf/2108.10241.pdf
• H. Mozaffari, V. Shejwalkar, and A. Houmansadr, FSL: Federated Supermask Learning, https://arxiv.org/abs/2110.04350
• H. Chang, V. Shejwalkar, R. Shokri, A. Houmansadr. Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer, https://arxiv.org/pdf/1912.11279.pdf
• V. Shejwalkar, A. Houmansadr. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning, NDSS 2021
• V. Shejwalkar, A. Houmansadr. Towards Optimized Model Poisoning Attacks Against Federated Learning, SpicyFL 2020

53