1 of 23

Federated Learning Benchmark for Domain Generalization

athul-s.raj@polymtl.ca

marc-antoine.provost@umontreal.ca

Quebec

Artificial

Intelligence

Institute

2 of 23

Agenda

  • Introduction
    • Federated Learning
    • Properties of FL
    • OOD in FL
      • Data heterogeneity in FL setting
  • Objective
  • Related work
    • DomainBed
    • Existing OOD gen algos in FL-GMA, FED-IRM
  • Federated datasets
  • FeDomainBed
  • Results
  • Discussion
  • Questions

2

3 of 23

Agenda

  • Introduction
  • Objective
  • Related work
  • Background
  • Results
  • Discussion

3

4 of 23

Machine Learning as we know it

Computer

Model

Data

4

5 of 23

Centralized Machine learning

Data

5

6 of 23

Is there a problem?

6

7 of 23

It’s a lot worse..

7

8 of 23

Can we solve this?

Perhaps, share the model..?

And not data!

8

9 of 23

Federated Learning!

Model

Repeat!

9

10 of 23

10

11 of 23

FedAVG

Average the weights

Server model

Client models

11

12 of 23

Homogenous data distribution a.k.a. IID

Average the weights

FedAVG

Generalization problem

Personalization problem

12

13 of 23

non-IID data distribution

Average the weights

FedAVG

Generalization problem

Personalization problem

Non-IID setting takes longer

13

14 of 23

Objective

To understand the performance of OOD gen algorithms in a non-i.i.d. federated learning setting when the algorithms are used for client model training.

14

15 of 23

DomainBed

​

​

15

16 of 23

Existing OODGen in FL

  • FedGMA
    • OOD Gen on server aggregation of models
  • CausalFed-IRM
    • Split federated learning setting - also, OOD gen at the server

16

17 of 23

Federated Datasets

  • Fed-CMNIST
    • Label distribution skew - random selection of labels(at max 3)/classes per client
    • Feature distribution skew - random colors(at max 3) on each client dataset, and test dataset

17

18 of 23

Federated Datasets

18

19 of 23

Federated Datasets

19

20 of 23

FeDomainBed

  • Train clients using OOD gen algorithms from DomainBed
    • Algos: ERM, IRM, VREx and a few more
    • FedAVG to aggregate client models to a global model
  • Evaluate global model on test dataset OOD to all client train datasets

20

21 of 23

Results

  • ERM(=FedAVG) performs better than all the OOD gen algos at client side

​

21

22 of 23

Inference & Discussion

  • Federated learning generalization objective is to learn mechanisms invariant across clients.
  • OOD Gen algorithms when applied at client, learns invariant mechanism within clients. But, these mechanisms may differ across clients.
  • So it may not be a good idea to perform OOD gen on client side.

22

23 of 23

Questions?

Quebec

Artificial

Intelligence

Institute