1 of 37

Split Learning (Variants)

& Theoretical Advances Towards its Privacy

�

Praneeth Vepakomma

MIT

https://praneeth.mit.edu

2 of 37

How is this incentivized?

3 of 37

Server

Client1

Client2

Client3 ..

Federated Learning

4 of 37

Split Learning

Server

Client1

Client2

Client3 ..

Smasher

Smashed Data

Back Prop

5 of 37

Hybrids

6 of 37

Sequential SL

Parallel SL

SL Variants

Evolution of Split Learning

7 of 37

Server-Client Update Imbalance Problem

Client Model Detachment Problem

Privacy Problem

Guiding Principles

8 of 37

AdaSplit

Subset of clients

(UCB)

Global update frequency

Sparse updates

No comm of

server to client

gradients!

9 of 37

We propose Adaptive Split Learning (AdaSplit) to give trade-offs

9

Improves performance under limited resources and adapts to variable resource budgets.

Chopra et al (arxiv 2022). AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learning

10 of 37

11 of 37

Mix-up + SL + Transformers

11

Client₁

Client₂

Mixer

Server

Mixed Smashed Data

CutSmashed data

Upload

Smashed Patch CutMix

Common Pseudo Random Sequence

0	1	0	1	1	0	1

1	0	1	0	0	1	0

Pseudo Random Sequnece

0	1	0	1	1	0	1

CutSmashed data

Smashed data

12 of 37

Amplification

Renyi DP

Private Mix-up

+ SL

+ Transformers

13 of 37

14 of 37

Results

15 of 37

Supervised Manifold Learning Query

Activation Query Privacy

16 of 37

Privatizing the SMLQ

A query on a dataset can be privatized by adding controlled noise.

17 of 37

Privatizing the SMLQ

18 of 37

Client and server-side alignment

Pre-alignment

Post-alignment

19 of 37

20 of 37

Boomerang Split Learning

Front

Center

Back

Front

Center

Back

Split Learning with DenseNet

Split Learning with U-Net

Multiple medical imaging benchmarks

21 of 37

Label Privacy

Differentially Private Label Protection in Split Learning,

Yang, Sun, Yao, Xie, Wang (ByteDance)

22 of 37

Topologies

Vepakomma, Swedish, Gupta, Dubey, Raskar 2018

23 of 37

Post-hoc privacy

How can an informally private prediction model

be made to provide formal privacy guarantees?

�

24 of 37

25 of 37

Main Problem

26 of 37

Overall Pipeline

Informal Privacy

27 of 37

28 of 37

Metric DP as an antidote to single instance encoding

Semantic neighborhood privacy when restricted to

Ref 1: Impossibility result of instance encoding (Carlini et al., 2020)

Ref 2: Metric DP (Chatzikokolakis et al., et al., 2013)

29 of 37

Propose-Test-Release (Dwork, Lei STOC 2009)

30 of 37

What’s an alternative that could be estimated from reasonably sized NN’s?

Exactly Computing the Local Lipschitz Constant of ReLU Networks, Jordan & Dimakis, NeurIPS 2020

31 of 37

32 of 37

Splintering: A resource efficient scheme for distributed linear algebra

33 of 37

Is there a resource efficiency benefit?

34 of 37

What about softmax?

Home work problem!

35 of 37

Matrix Inverse & Multiplication:

Splintering shines

Client only performs +/-, scaling & multiplication with diagonal matrices and inverse of below k by k matrix as opposed to the original n by n matrix with k<<n.

Computational Savings

36 of 37

Matrix Inverse & Multiplication:

Splintering shines

37 of 37

‪Clément Canonne

‪Mohammad Mohammadi Amiri

‪Nina Miolane

Suat Evren

Abhishek Singh

Ramesh Raskar

Alex Pentland

Questions & Discussion

Thanks to all collaborators

Ayush Chopra