Protecting JPEG images against
adversarial attacks
Data Compression Conference �2018
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo and James Storer�Brandeis University
Image consumption
Image consumption
Adversarial Attack
Predicted: Indian Elephant (99.7%)
Original Image
Adversarial Attack
Predicted: Guacamole (99.9%)
Predicted: Indian Elephant (99.7%)
Original Image
Adversarial Image
Adversarial Attack
- With cleverly designed additive noise the classification of the image can be changed�- It is easy to find noise which makes the classifier predict any given class - Targeted attack
Predicted: Indian Elephant (99.7%)
Predicted: Guacamole (99.9%)
Original Image
Perturbations
Adversarial Image
CNNs are used in critical applications
Standard Convolutional Neural Network
Backpropagation
- Find the gradient (∇) with respect to the loss function, which is a measure of true vs predicted values.
Gradient Descent
- Update the parameters with small value (η) in the direction which decreases the loss
Gradient ascent attack
- Update the image with small value (ϵ) but in the direction which increases the loss
Gradient ascent attack
- Update the image with small value (ϵ) but in the direction which increases the loss
Various attack models
- Gradient Attack
- Fast Gradient Sign Method
- Iterated Gradient Sign Method
- Deep Fool
- JSMA
- L-BFGS
(and many more since we wrote our paper)
Various types of adversarial perturbations
One of the most popular attack is - Fast Gradient Sign Method (FGSM)
It is very efficient and but requires higher ϵ value compared to other techniques.
Iterative version of this method is called Iterated Gradient Sign Method (IGSM)
For every iteration the image is clipped to be within ±ϵ, thus the process is slow but generates more robust adversarial images.
Details of other attacks like JSMA and DeepFool are included in our paper.
Defending against adversarial attacks
Defending against adversarial attacks
Make models harder to attack�(robust classifiers, detectors, adversarial-training)
Better Model
Defending against adversarial attacks
Make models harder to attack�(robust classifiers, detectors, adversarial-training)
Remove perturbations from adversarial images
Remove perturbation
Defending against adversarial attacks
Make models harder to attack�(robust classifiers, detectors, adversarial-training)
Remove perturbations from adversarial images
Image transformations�(crop,reconstruct, pixel-deflection)
Denoising�(quantization, smoothing, shrinkage)
Remove perturbation
Defending against adversarial attacks
Make models harder to attack�(robust classifiers, detectors, adversarial-training)
Remove perturbations from adversarial images
Denoising�(quantization, smoothing, shrinkage)
Image transformations�(crop,reconstruct, pixel-deflection)
Remove perturbation
JPEG as a possible defense
JPEG as a possible defense
JPEG as a possible defense
Predicted: Indian Elephant (99.7%)
Original Image
Adversarial Image
Adversarial Image after JPEG
Guacamole (99.9%)
Indian Elephant (21.1%)
But JPEG has its limitations
80%
20%
75
50
25
JPEG Quantization Level
Classification Accuracy
But JPEG has its limitations
80%
20%
75
50
25
JPEG Quantization Level
Classification Accuracy
JPEG is not the best defense
Better than JPEG
Why do we care about JPEG?
- Fast decoding available in virtually all devices
- Deep Learning datasets already store images as JPEG � �
- JPEG defenses provide adequate visual quality
Attacks are agnostic to object location
FGSM
IGSM
DeepFool
low
high
Average location of adversarial perturbations
Object localization
GAP�Global Activation Pooling
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Semantic contents are localized
low
high
Where are the attacks located?
FGSM
IGSM
DeepFool
Natural Images
low
high
Key to improving JPEG's ability to defend
JPEG helps
Attacks are content blind
Semantic JPEG
Key to improving JPEG's ability to defend
JPEG with semantic quantization - DCC 2017
Soon to be available in Firefox browser
MSROI - DCC 2017
MSROI Map
is learned for every class c and for layer ‘d’
where
CAM
Advantage
Disadvantage
Boy's face and hand (only captured by our method)
Boy's face and hand (only captured by our method)
Making MSROI secure against adversary
MSROI on �Clean Image
MSROI on �Adversarial Image
Making MSROI secure against adversary
MSROI on �Clean Image
MSROI on �Adversarial Image
Various perturbations with Augmented MSROI
Making MSROI secure against adversary
MSROI on �Clean Image
MSROI on �Adversarial Image
Aug-MSROI on �Adversarial Image
Various perturbations with Augmented MSROI
Making MSROI secure against adversary
MSROI
Augmented MSROI
where, ∆ is random perturbation
- Aug MSROI is average of maps over several perturbations.
- Random perturbation helps overcome changes in the activation due to adversarial input
- Classifiers are robust enough that small perturbations do not change the overall class of the image
JPEG Encoder
Qhigh
JPEG Encoder
Qlow
JPEG Encoder
JPEG Encoder
JPEG Encoder
Qmedium
JPEG Encoder
Qfinal
Variable ‘Q’ JPEG
Bins
Bins
Aug MSROI - Accuracy
Predicted: Indian Elephant (99.7%)
Guacamole (99.9%)
Indian Elephant (21.1%)
Indian Elephant (70.3%)
Original Image
Adversarial Image
Adversarial Image �after JPEG
Adversarial Image �after Aug-MSROI
Aug MSROI - Accuracy
Aug MSROI - Image quality vs Accuracy
PSNR
Accuracy
Aug-MSROI JPEG�Standard JPEG
Summary
Thank You
Code: github.com/iamaaditya/protecting-jpeg
Contact: aprakash@brandeis.edu