1 of 38

Differential Privacy �as a privacy-preserving �technique in �Federated Learning

Final Presentation

Master Thesis No. : MT- 3506

Institute of Industrial Automation

and Software Engineering

Karthik Rajendran

Master in Electrical Engineering

2 of 38

Motivation

Distributed Machine Learning

Clients send data to a central server for model training, and the server returns the trained model to clients.

Clients share their sensitive data which raises concerns about data privacy.

Federated Learning

Models are trained locally with local client data. Server aggregates local weights and returns global weights.

Regeneration of local data via model inversion from external threats.

Adding noise via Differential Privacy to protect local weights.

University of Stuttgart, IAS

2

10/4/2023

How to train ML models in a distributed manner while keeping sensitive information safe from external threats ?

Motivation:

Ask a question,
How to train ML models in a distributed manner while keeping sensitive information/ protecting privacy from external threats ?

However the challenge, data regenerated(main)
Traditional ML approach requires to send private data to a server for train a model, however if we send user data, personal information is revealed, in order to overcome this issue, a novel approach called federated learning is proposed by google, in FL local data is not sent rather models are trained on each clients, only parameters that are sent are the model updates.
However recent studies show that it is possible to regenerate private data from model updates through model inversion. To solve this problem there are multiple techniques, promising one is differential privacy
In DP, in this technique the noise is added to model parameters in order to protect them. However when we add noise then we sacrifice accuracy, it is a challenging task to determine the amount of noise added to both protect local data from third parties and obtain high accuracy.
In this regard, the following question need to be answered.
The challenge is, as we increase noise, the accuracy reduces.

**Paste the question**
** verbally talk about the tradeoffs between privacy and accuracy **
That’s y it’s a challenging task and say on how to train

3 of 38

Early Pioneers of Federated Learning

University of Stuttgart, IAS

3

10/4/2023

Source: Huili Chen, Amazon.science

Source: Brendan McMahan and Daniel Ramage, Research Scientists, Google Research

Amazon uses Federated Learning to improve their smart devices.

Google uses Federated Learning to improve keyboard suggestions.

4 of 38

Contents

University of Stuttgart, IAS

4

10/4/2023

Problem Statement

Developed Concept

Implementation

Results and Evaluation

Summary and Future Scope

5 of 38

Federated Learning

FL trains a prediction model without sending the client’s data to the cloud.
The server has an untrained model

University of Stuttgart, IAS

5

10/4/2023

Central Server

Client1

Client2

Client3

6 of 38

Federated Learning

Server sends a copy of that model weights to the clients
The clients have the untrained model

University of Stuttgart, IAS

6

10/4/2023

Client1

Client2

Client3

Central Server

7 of 38

Federated Learning

The clients have data on which to train their model
Each client trains the model to fit the data they have

University of Stuttgart, IAS

7

10/4/2023

Client1

Client2

Client3

Central Server

8 of 38

Federated Learning

Each client sends a copy of its local weights to the server.
One complete iteration of the training process is called a Round.

University of Stuttgart, IAS

8

10/4/2023

Client1

Client2

Client3

Central Server

9 of 38

Federated Learning

The server averages the local weights and sends the global weights to the clients.

University of Stuttgart, IAS

9

10/4/2023

Clients avoid sharing their training data which improves privacy conditions.

Client1

Client2

Client3

Client1

Central Server

10 of 38

Security Threat

Attackers try to infer sensitive information about the training data on individual clients by analyzing the local weights. This could lead to privacy violations.

University of Stuttgart, IAS

10

10/4/2023

GAN

Hacker gets the local weight

Client local weights

Model Inversion Attack

Client1

Client2

Client3

Central Server

11 of 38

Differential Privacy

Safeguards individual privacy by ensuring that specific information about individuals cannot be deduced.

University of Stuttgart, IAS

11

10/4/2023

Outcome is approximately the same

12 of 38

Differential Privacy Integration

University of Stuttgart, IAS

12

10/4/2023

Data Collection

Data Preprocessing

Data Labelling

Model Training

Model evaluation and Testing

Data Sharing and Collaboration

Querying the Model

Model Deployment

Data Reporting and Summarization

Post Processing

Data

Model

Inference

13 of 38

Differentially Private Stochastic Gradient Descent (DPSGD)

University of Stuttgart, IAS

13

10/4/2023

Traditional Model Training

Differentially Private Model Training

14 of 38

Find the compromise between privacy and model accuracy

University of Stuttgart, IAS

14

10/4/2023

„

15 of 38

Contents

University of Stuttgart, IAS

15

10/4/2023

Problem Statement

Developed Concept

Implementation

Results and Evaluation

Summary and Future Scope

16 of 38

System Architecture

University of Stuttgart, IAS

16

10/4/2023

Model Training

Server

Driver

2

Sensors

1

Driver Behaviour Data

2

Client System

3

Local weights

4

Global weights

Data Protection

3

4

. . . . . . . . . . . . . . . . .

5

Server

17 of 38

Driver Behaviour Data

In order to test our approach, we use the Driver Profile Identification task that analyzes driving patterns using sensor measurements in the vehicle.

University of Stuttgart, IAS

17

10/4/2023

1

18 of 38

Model Training

Training takes place in rounds on client devices until specific accuracy criteria is reached.

University of Stuttgart, IAS

18

10/4/2023

2

19 of 38

Server

Multiple clients collaboratively train a global model by sharing model updates while preserving data privacy on the individual clients using Federated Average.

University of Stuttgart, IAS

19

10/4/2023

w₁

w₂

w₃

5

Central Server

Local weights

Global

weights

G_w

20 of 38

System process

University of Stuttgart, IAS

20

10/4/2023

1

2

3

4

5

Initialization

Initialization of a global model in the server

Client Selection

Subset of clients is selected to participate

Local Private Model Training

Clients train on the global model

Model Aggregation

Server aggregates the client weights

Model Update Distribution

New global model is distributed back

6

7

8

9

10

Iterations

Repeated for multiple rounds

Convergence

Global model converges to high accuracy

Stopping Criteria

Training rounds continue until criteria is met

Final Global Model

Global model can be used for inference

Model Evaluation

Final global model is evaluated

21 of 38

Contents

University of Stuttgart, IAS

21

10/4/2023

Problem Statement

Developed Concept

Implementation

Results and Evaluation

Summary and Future Scope

22 of 38

Dataset

The data was recorded on a recent model of KIA Motors and processed from in-vehicle CAN data [1]
The dataset includes 14 sensor features collected over an average of 1200 seconds of driving sessions, with four driver label outputs.

Driving Dataset from OCSlab, Korea University

University of Stuttgart, IAS

22

10/4/2023

Intake air pressure	Fuel consumption	Engine torque	Time (s)	. . . . . .	Driver
33	268.8	5.5	1		1
40	243.2	7	2		1
41	217.6	7	3		1

23 of 38

Dataset Preprocessing

The dataset comprises of driver sequences recorded during driving sessions.
Multivariate 100 second sensor data is the input feature and driver label as the output.

University of Stuttgart, IAS

23

10/4/2023

24 of 38

Architecture Implementation

Raspberry Pi devices are utilized as automotive clients, conducting model training

and testing within a Dockerized container utilizing the Flower framework.

University of Stuttgart, IAS

24

10/4/2023

Central Server

Local weights

Global weights

25 of 38

Architecture Components

University of Stuttgart, IAS

25

10/4/2023

Model

DPSGD Optimizer

Privacy Accountant

26 of 38

Architecture Components

University of Stuttgart, IAS

26

10/4/2023

Model

DPSGD Optimizer

Privacy Accountant

*DPSGD (Differentially Private Stochastic Gradient Descent)

Gradient

Gradient Clipping

Noise Addition

Clipping Bound

DPSGD Gradient

27 of 38

Architecture Components

Privacy metrics are calculated for each epoch within a round to assess model data interaction frequency and privacy reduction.

University of Stuttgart, IAS

27

10/4/2023

Model

DPSGD Optimizer

Privacy Accountant

28 of 38

Privacy Utility Optimization

University of Stuttgart, IAS

28

10/4/2023

Conduct a correlation analysis on the train accuracy for various noise levels.

Noise

Privacy

Accuracy

Optimal value

Select optimal privacy with slight accuracy loss.

29 of 38

Experiments

The Evaluation metric used

Confusion Matrix

Details of the Hyperparameters:

Training and Testing

University of Stuttgart, IAS

29

10/4/2023

Timesteps	100
Rounds	100
Batch Size	32
Optimizer	DPSGD
Learning rate	0.001

Clipping Bound	Noise Multiplier
5	0.1
5	0.5
5	0.8
5	1
5	5
5	10

30 of 38

Contents

University of Stuttgart, IAS

30

10/4/2023

Problem Statement

Developed Concept

Implementation

Results and Evaluation

Summary and Future Scope

31 of 38

Accuracy Across Training Rounds and Noise Levels

University of Stuttgart, IAS

31

10/4/2023

🡪 Increasing noise, reduces accuracy

Clipping Bound = 5

32 of 38

Accuracy, Privacy Budget and Noise Comparison

University of Stuttgart, IAS

32

10/4/2023

Client 1

Client 2

Client 3

🡪 As the noise increases, the accuracy reduces and total privacy improves.

33 of 38

Confusion Matrix on Validation Data

University of Stuttgart, IAS

33

10/4/2023

Noise = 0.1

Noise = 10

🡪 Model prediction across class reduces as noise increases.

34 of 38

Correlation Matrix of Train Accuracy

University of Stuttgart, IAS

34

10/4/2023

Noise	Client Averaged Train Accuracy	Privacy Accountant
0.8	65%	774
1.0	63%	523

Selecting a noise level of 1.0 over 0.8 ensures a 32% increase in model privacy with minimal impact on accuracy.

Noise

35 of 38

Contents

University of Stuttgart, IAS

35

10/4/2023

Problem Statement

Developed Concept

Implementation

Results and Evaluation

Summary and Future Scope

36 of 38

Summary and Future Scope

Federated learning model was developed for Driver Identification Task.
Implemented Differentially Private Stochastic Gradient Descent within a Federated Learning framework.
Conducted a correlation analysis to select the most privacy - preserving model without sacrificing accuracy.

Implement a secure multi-party computation aggregation algorithm on the central server.
Conduct additional experiments with an increased number of clients and varying levels of noise.

University of Stuttgart, IAS

36

10/4/2023

37 of 38

References

[1] ocslab.hksecurity.net/Datasets/driving-dataset

University of Stuttgart, IAS

37

10/4/2023

38 of 38

Karthik Rajendran

Institute for Industrial Automation and Software Engineering

Prof. Dr.-Ing. Dr. h.c. Michael Weyrich

st176764@stud.uni-stuttgart.de

e-mail

University of Stuttgart

Thank you!