1 of 41

What is inside a ML Platform?

Scheduler & Federated Learning Intro

​

2 of 41

Background

  • GPU Scheduler - ML Platform

​

  • Federated learning - ML Platform Security

3 of 41

About

  • Why Scheduler?
  • ​
  • Why GPU?
  • ​
  • Why federated learning?
  • ​
  • Why no details?

4 of 41

About

Cost

5 of 41

Content

  • GPU Scheduler - ML Platform

​

  • Federated learning - ML Platform Security

6 of 41

Non Scheduler

7 of 41

Non Scheduler

I am Reco ->

I am Ads ->

I am video ->

I am NLU ->

​

CUDA_VISIBLE_DEVICES

8 of 41

But…

BERT-large -> 340M

GPT-2 -> 1.5B

Megatron -> 8B

DALL-E -> 12B

GPT-3 -> 175B

9 of 41

???

10 of 41

Problems

  • Long running machine learning jobs
  • ​
  • Isolation
  • ​
  • Scalability for -
    • Training
    • Inference
  • ​
  • Heterogeneous architecture - CPU, GPU, TPU, FPGA…
  • ​
  • Resource utility
  • ​
  • Security

11 of 41

Solution

Scheduler for

  • Resource efficiency
  • Isolation
  • Heterogeneous
  • …

Scheduler itself should not too slow

12 of 41

Cluster Scheduler

  • SLURM
  • ​
  • Yarn
  • ​
  • Mesos
  • ​
  • Kubeflow
  • ​
  • Kserve

13 of 41

Cluster Scheduler

  • ​

14 of 41

Is Cluster Scheduler Good Enough?

  • ​

15 of 41

Bottlenecks?

  • ​

16 of 41

Bottlenecks?

  • ​

17 of 41

Bottlenecks?

  • Is Network the bottleneck?
  • Is Storage the bottleneck?
  • Is PCIe the bottleneck?
  • Is CPU the bottleneck?
  • Is Memory the bottleneck?
  • Is xxx the bottleneck?

18 of 41

Bottlenecks

Operations

  • Computing
  • Communication

​

Bottleneck is relative

  • Single node
  • DMA
  • Distributed
  • RDMA
  • NVLink
  • …

​

19 of 41

Bottlenecks

  • ​

20 of 41

Bottlenecks

IO improved a lot!

​

IO is still a bottleneck.

21 of 41

Solution

Preload

Pipeline

Stream

Low-cost nodes

Pruning

Precision

Distill

…

22 of 41

Solution

23 of 41

Solution

24 of 41

Industry

25 of 41

Industry

26 of 41

Content

  • GPU Scheduler - ML Platform

​

  • Federated learning - ML Platform Security

27 of 41

Security Cost

Hard restrictions

  • GDPR
  • CCPA

Soft restrictions

  • Hack
  • Leak
  • …

28 of 41

Security is hard in ML

  • ​

29 of 41

Security is hard in ML

  • ​

30 of 41

Industry Practise

  • TEE
  • ​
  • MPC

31 of 41

Hack case study

DB.getPassword.equals(inputPwd)

32 of 41

Differential privacy

33 of 41

Federated Learning

34 of 41

Federated Learning

35 of 41

Why Federated Learning?

36 of 41

Why Federated Learning?

37 of 41

Why Federated Learning?

38 of 41

IS federated Learning silver bullet?

  • Data exposure
  • Data collection

39 of 41

IS federated Learning silver bullet?

  • User consent
  • ​
  • Client hacking
  • ​
  • Async parameter update
  • ​

​

40 of 41

IS federated Learning silver bullet?

41 of 41

Thanks!