1 of 49

Anomaly Detection

Prof. Seungchul Lee

Industrial AI Lab.

2 of 49

Anomaly

  • Anomalies and outliers are essentially the same thing
    • Objects that are different from most other objects
    • Something that deviates from what is standard, or expected (one classification)

​

2

3 of 49

Causes of Anomalies

  • Data from different class of object or underlying mechanism
    • disease vs. non-disease
    • fraud vs. not fraud

​

  • Data measurement and collection errors

​

  • Natural variation
    • tails on a Gaussian distribution

3

tail

tail

4 of 49

Anomaly Detection

  • Finding outliers

4

5 of 49

Applications of Anomaly Detection

  • Security & Surveillance

​

​

​

​

​

​

​

  • Biomedical Applications

​

​

5

  1. Patrikar, D.R., Parate, M.R. Anomaly detection using edge computing in video surveillance system: review. Int J Multimed Info Retr 11, 85–110 (2022).
  2. van Hespen, K.M., Zwanenburg, J.J.M., Dankbaar, J.W. et al. An anomaly detection approach to identify chronic brain infarcts on MRI. Sci Rep 11, 7714 (2021).

6 of 49

Applications of Anomaly Detection

  • Industrial Damage Detection

​

​

​

​

​

​

  • Machinery Defects Diagnostics
    • Diagnosis of machinery conditions
    • Early alarm of malfunctioning

​

6

  1. Wirtz, Sebastian F. et al. Frequency-based damage detection of spur gear using wavelet analysis. (2016).

7 of 49

Difficulties with Anomaly Detection

  • Scarcity of Anomalies
    • It is not easy to get anomaly data, because anomaly rarely happens
    • Overfitting issue occurs when there is only small number of data

​

  • Diverse Types of Anomalies
    • There are so many causes of anomalies
    • At the training stage of neural network, we cannot have all possible anomalies as input data

7

8 of 49

Use of Data Labels in Anomaly Detection

  • Supervised Anomaly Detection
    • Labels available for both normal data and anomalies
    • Similar to classification with high class imbalance

​

  • Semi-supervised Anomaly Detection
    • Labels available only for normal data

​

  • Unsupervised Anomaly Detection
    • No labels assumed
    • Based on the assumption that anomalies are very rare compared to normal data

8

9 of 49

Output of Anomaly Detection

  • Label
    • Each test instance is given a normal or anomaly label
    • Same as the typical output of classification-based approaches

​

  • Score
    • Each test instance is assigned an anomaly score
      • Allows outputs to be ranked in the order of anomaly scores
      • Requires an additional threshold parameter

​

​

9

10 of 49

Variants of Anomaly Detection Problem

  •  

10

11 of 49

Statistical Anomaly Detection

11

12 of 49

Statistical Anomaly Detection

  • Anomalies (outliers) are objects that are fit poorly by a statistical model

​

  • Estimate a parametric model describing the distribution of the normal data

​

  • Apply a statistical test that depends on
    • Properties of test instance
    • Parameters of model (e.g., mean, variance)
    • Confidence limit (related to number of expected outliers)

12

13 of 49

Statistical Anomaly Detection

  • Univariate Gaussian Distribution
    • Outlier defined by Z-score > threshold

13

14 of 49

Statistical Anomaly Detection

  • Multivariate Gaussian Distribution
    • Outlier defined by Mahalanobis distance > threshold

14

A

B

15 of 49

Statistical Anomaly Detection

  • Pros
    • Statistical tests are well-understood and well-validated
    • Quantitative measure of degree to which object is an outlier

​

  • Cons
    • Data may be hard to model parametrically
      • multiple modes
      • variable density
    • In high dimensions, data may be insufficient to estimate true distribution

15

16 of 49

Deep Learning-based Anomaly Detection

16

17 of 49

Deep Learning-based Anomaly Detection

  • Train autoencoders only with normal data
    • Trained autoencoders will only capture features of normal data

​

  • Test with (normal + anomaly) data

17

Train

Test

18 of 49

Convolutional Autoencoder (CAE)

  • CAE is trained to compress/decompress normal images to/from the latent space
  • CAE compresses the input image into few latent variables
  • CAE decompresses (reconstruct) the image that includes some error compared to the original
  • For anomalous data, the reconstruction error (anomaly score) would be greater than that of normal images

​

​

18

Training Data

(Normal Data)

Test Data

(Normal Data)

Test Data

(Anomaly Data)

Reconstructed Data

(Normal Data)

Reconstructed Data

(Anomaly Data)

Reconstruction Error

Reconstruction Error

19 of 49

Anomaly Detection Process with CAE

  • Training using only normal data

​

​

​

​

​

  • Anomaly detection with test data
    • Use the CAE that was trained with normal data

​

19

Training Data

(Normal)

CAE

Reconstruction Loss

Test Data

(Normal + Anomaly)

CAE

Anomaly Score

Decision Making

20 of 49

Anomaly Score – Reconstruction Error

  • Root mean squared error (RMSE)

20

Flatten

 

Test Data

Reconstructed Data

Reconstructed Error

21 of 49

Anomaly Detection with CAE in TensorFlow

21

22 of 49

CAE Implementation

  • Fully convolutional
  • Note that no dense layer is used

​

22

23 of 49

Train and Test Dataset

  • Train Dataset: 7 digits only (normal images)

​

​

​

​

​

​

​

  • Test Dataset: digit 7 and digit 5 (normal + anomaly images)

23

24 of 49

CAE Implementation

  • Anomaly Detection: reconstruction error > threshold
  • Normal data (digit 7) in the test dataset gives low reconstruction error
    • Anomaly data (digit 5) in the test dataset gives high reconstruction error

​

  • Encoder

24

25 of 49

CAE Implementation

  • Decoder

​

​

​

​

​

​

​

​

​

​

25

26 of 49

Latent Space and Reconstruction Result

  • Latent space
  • Reconstruction results
    • Normal

​

​

​

​

​

​

​

    • Anomaly

​

​

​

​

​

​

​

​

26

27 of 49

Reconstruction Error and Threshold

  • Reconstruction error and threshold

​

​

​

​

​

​

​

​

​

​

​

  • Anomaly Detection: reconstruction error > threshold

27

28 of 49

Anomaly Detection with GAN

28

  • AnoGAN
  • f-AnoGAN

29 of 49

AnoGAN: GAN-based Anomaly Detection

  • AnoGAN (Anomaly Detection with GAN)
    • Train only normal (healthy) data, no abnormal data

29

Generated (Fake)

Real (Normal)

Real

Fake

Generator

Discriminator

Input two images (Real, Fake)

Noise z

30 of 49

AnoGAN: GAN-based Anomaly Detection

  •  

30

Randomly generated data

Target data

Loss

Update

31 of 49

AnoGAN: GAN-based Anomaly Detection

  •  

31

z update

Normal data

Trained data

Abnormal data

unseen data

anomaly score: 0.027

anomaly score: 0.067

z update

32 of 49

f-AnoGAN: Fast GAN-based Anomaly Detection

  • f-AnoGAN
    • AnoGAN requires an iterative procedure to find the latent z that generates the target data.
      • Key idea: Let’s make this process faster
    • Train only normal (healthy) data, no abnormal data

​

​

32

Generated (Fake)

Real (Normal)

Real

Fake

Generator

Discriminator

Input two images (Real, Fake)

Noise z

33 of 49

f-AnoGAN: Fast GAN-based Anomaly Detection

  • f-AnoGAN
    • Train an additional encoder model to predict latent z from images
    • Generator is fixed
      • Pixel reconstruction loss (discriminator feature loss can also be used)

​

33

Generator

(fixed)

Noise z

Encoder

Encoded

 

 

Loss

Regenerated

34 of 49

f-AnoGAN: GAN-based Anomaly Detection

  • f-AnoGAN
    • Query data is regenerated directly through the encoder and generator
      • If data is normal (trained), data will be regenerated well.
      • Otherwise, anomaly score will be high

34

Encoder

Encoder

Generator

Generator

z

z

Normal data

Trained data

Abnormal data

unseen data

Regenerated data

Regenerated data

Anomaly score: 0.183

Anomaly score: 1.053

35 of 49

Anomaly Detection with GAN in TensorFlow

35

36 of 49

Data Description

  • Train Dataset: digit 2 only (normal images)

​

​

​

​

​

​

  • Test Dataset: digit 2 and digit 6 (normal + anomaly images)

​

36

37 of 49

Generator

  • Build generator and discriminator
    • Train with only normal data (digit 2 only)

37

Generator

Discriminator

Real

Fake

Noise z

Generated

Real

38 of 49

Discriminator

  • Build generator and discriminator
    • Train with only normal data (digit 2 only)

38

Generator

Discriminator

Real

Fake

Noise z

Generated

Real

39 of 49

GAN Training and Results

  • Normal data (digit 2) is well generated

39

40 of 49

Encoder Implementation for f-AnoGAN

  • Build Encoder
    • To predict latent z from image

​

40

Generator

Noise z

Regenerated

Real

Encoder

41 of 49

f-AnoGAN Results

  • Input normal and abnormal data and calculate anomaly score
    • Score: MSE of input and regenerated images

41

42 of 49

Anomaly Detection with LSTM

42

43 of 49

Time Series NN based Anomaly Detection

  • Predict and calculate anomaly score
    • Train with normal data
    • Good performance on periodic signals

​

​

43

LSTM model

Input data

Predicted

ECG data

44 of 49

Anomaly Detection with LSTM in TensorFlow

44

45 of 49

Data Description

  • NASA bearing data
    • 200 rpm by an AC motor (radical load: 6000 lbs)
    • 1-second vibration signal snapshots at 10 minutes intervals (sampling rate: 20 kHz)
      • Absolute mean value for the feature
    • At the end of the test-to-failure experiment, outer race failure occurred in bearing 3.

45

Test rig setup

46 of 49

Data Description

  • NASA bearing data
    • Bearing 3 data is selected
    • Train: normal signal
    • Test: abnormal signal included

46

Test rig setup

47 of 49

LSTM Implementation

  • Model parameters

47

n_output (prediction length)

n_input

(signal length)

Step (# of chunk)

48 of 49

Results

  • Prediction Results
    • Training data

48

Present signal

Real future signal

Predicted signal

49 of 49

Results

  • Predict and calculate anomaly score
    • Difference between predicted and real (measured) signal

49

Abnormal event

above the threshold