FT-PrivacyScore�Privacy Scoring for Machine Learning Participation

Ethan Gu, Jiajie He, Keke Chen

(Extended presentation based on a demo at ACM CCS 2024)

Outline

• Research problem
• Membership inference attack as a privacy scoring method
• Preliminary result
• Summary

​

Human data contributors in the machine learning loop

Data collecting/

integration

Data curator/Model builder

Data contributor

Modeling

Model Deployed

Model users

Controlled access scenario

• Fully preserve data utility and performance of processing

​

• Example: NIH All of Us project
• Aim to recruit one million volunteers
• Share their electric health records, physical measurements, surveys, genomic data, and wearable data with the research program
• Researchers access the data via ”Researcher Workbench”

​

• with strict access control mechanisms
• But data contributors have no control over the specific use of data

Known privacy issues with machine learning models

• Deep neural networks learn more than what is expected
• Not just learn for the target task, e.g., a classification model
• large number of parameters
• Learn representations (e.g., foundational models)

​

​

• Great risks of leaking private information in the training data
• Model inversion (model 🡪 training data recovery)
• Membership inference (likelihood of an instance is used in training)
• Property inference (feature distribution, etc)

​

Data contributors’ privacy rights

• Examples: GDPR (Europe), CCPA (California), etc

https://dataprivacymanager.net/ccpa-vs-gdpr/

- Right to know potential

privacy risks!

- Right to object to processing

How do contributors learn or control privacy risks?

• So far, the most well-known method: differential privacy (via epsilon privacy budget setting)
• Don’t need to know what the actual risk is
• Specify the “level of acceptable risk or privacy loss”
• Model builders implement the guarantee

​

• Not applicable in “controlled access”
• DP leads to significant data utility loss

​

What existing techniques can do (with different threat model assumptions)

• (somewhat) trusted model builder, untrusted model users
• Global differential privacy – sacrifice model utility
• Injecting noises, gradient clipping, etc.

​

• Untrusted model builder, untrusted model users
• local differential privacy + federated learning

​

• Untrusted model builder, (somewhat) trusted model users
• Confidential computing in modeling/model application

​

• (Somewhat) trusted model builder and trusted model users
• Controlled access -- probably most common in practice so far

​

​

​

What we can do with the controlled access setting

Data samples are not equally sensitive!

​

• Data contributors should have the right to know the privacy risks in advance, before they participate in a machine learning task

​

• Why is knowing privacy risks useful?
• Exercise their privacy right: the right to be informed; decide to participate in a particular task or not
• Negotiate with the model builder (potential compensations for privacy loss)

​

Contributions of FT-PrivacyScore

• The first work addressing the privacy scoring issue in the controlled access scenario
• An efficient high-quality scoring method for large models (with solid theoretical justifications)

Background: definition of privacy risk

• Well-accepted differential privacy

in the context of machine learning and a sample x

Prob (model used x)

Prob (model did not use x)

Requirements for our privacy scoring method

• The privacy risk estimation should be
• specific to data samples (up to the sample level, or the user level)
• specific to the modeling task
• Have a solid theoretical justification, e.g., linking to differential privacy

​

Privacy risk estimation with membership inference (MIA)

• Most relevant to our needs
• The likelihood-ratio membership inference attack (LiRA)
• Sample-specific
• Model-specific

​

• Estimate Prob (model used x) and Prob (model did not use x) directly.

Determine which one is more likely via hypothesis testing

​

​

Intuition: the more accurate the MIA attack, the higher the privacy risk

Membership inference method – how it works

• Basic idea of membership inference

Targeted sample x

(provided by some contributor)

Training with random samples of dataset and x

Training with random samples without x

Shadow modeling stage: repeat the above

for several times!

Purpose:

• Extract features that can distinguish

with or without x cases

Collecting enough IN-training and OUT-training examples

Learn a classification model

Or a decision rule

Out

In

What LiRA does

• find that logit scaling of pred. conf. is the best feature (for classification)

is the prediction confidence level at the label y

(the modeling task is classification)

Consider this is just one way to extract a feature of IN/OUT samples! There are certainly more.

p

Bird

Airplane

Truck

Dog

label y = airplane

What LiRA does

1. Collecting samples to estimate

the out-training phi distribution (blue) and

the in-training phi distribution (red);

both are approximately normal distributions

​

2. For a new sample, calculate its phi level’s

likelihood ratio via hypothesis testing

​

conf_obs

Benefits and challenges of LiRA

• Much more accurate than the previous MIA methods
• Applies to the sample level -- sample- and model- specific!

​

• But expensive:

ImageNet data: 32 IN models (for red) and 32 OUT models (for blue) for determining the MIA risk for ONE sample

Offline LiRA to reduce cost (with slightly worse accuracy)

1. Train OUT models only

with public domain data (not specific to the tested sample x)

​

2. one-side hypothesis testing

Privacy Scoring using LiRA

Question: How well an IN sample x is correctly identified (OUT cases are not interesting)

​

1. Prepare offline LiRA (establish the OUT distribution)

​

• Training N models on random sample sets of training data.

* sampling procedure: each sample x is selected with prob. 0.5

* then for each sample x, we have something like (model1, x IN), (model2, x OUT)…

​

• Scoring: Apply offline LiRA to the above N cases 🡪 estimate the attack success rate for x IN cases; repeat this for each sample

was first proposed by the “privacy onion effect” paper (NeurIPS 22)

Challenges with the privacy scoring

• Training N models are expensive (e.g., N=200,000 was used)
• Almost impossible for large models

​

Our contribution – FT-PrivacyScore

• Using fine-tuning to speed up the scoring process
• More practical for large models

​

• Question: whether fine-tuning with LiRA also works

​

• How fine-tuning is applied:
• Fine-tuning to generate examples for estimating the OUT distribution for offline LiRA
• Fine-tuning with random batches of samples in scoring

A scoring service

The initial result is promising

• Not using large data/models yet
• ResNet-18
• CIFAR-10 dataset
• Tested 100 random samples’ privacy scores

​

• The expensive (none-FT) method – the baseline
• Use offline LIRA
• 500 normally trained models for scoring

​

• FT-PrivacyScore
• Use fine-tuning-based offline LIRA
• 500 fine-tuning-based models for scoring

Some promising results

• Quality is mostly preserved!

Much more efficient!

• Excluded the offline LiRA preparation stage, only for the online scoring part
• Nvidia V100 GPUs were used

Ongoing work

• Extend the experiments to large models and datasets (e.g., ViT models and ImageNet, text data andmodels)
• Challenges: difficult to generate the baseline (i.e., the expensive method) for validation

Summary

• We propose privacy scoring in the controlled access setting for data contributors to understand their privacy risks
• Many potential applications of the scoring results

​

• FT-PrivacyScore for fine-tuning large models and scoring privacy risks
• The initial results look promising: good quality and highly efficient