1 of 103

Biometric Systems��8.3 Face Recognition

v.2016/1

Xavi Giró, Verònica Vilaplana, Ferran Marqués

2 of 103

Approaches: Feature extraction and recognition

  • Face recognition is a multi-class classification problem: solved using techniques from pattern recognition.

​

  • Features
    • Features from global appearance
      • Principal Component Analysis (PCA)
      • Linear Discriminant Analysis (LDA)
      • Independent Component Analysis (ICA)
    • Features from local regions
      • Local Feature Analysis (LFA)
      • Gabor Wavelet
      • Local Binary Patterns (LBP), SIFT, etc.

​

  • Classification (Recognition)
    • Euclidian Distance
    • Template Matching
    • Neural Networks
    • Elastic Graph Matching
    • Support Vector Machines
    • …

​

2

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Detection

Feature

Extraction

Recognition

3 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
      • Convolutional Neural Networks (CNN, ConvNets)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

3

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

4 of 103

Still Image: Holistic: Embeddings

4

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Low-dimensional neighbourhood

preserving for high-dimensional input data.

Fig: Roweis, Saul (2000)

5 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
      • Convolutional Neural Networks (CNNs, ConvNets)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

5

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

6 of 103

Still Image: Holistic: PCA

  • Concept: Face images (that are normalized wrt scale, translation, rotation and present a common pose) are very redundant. The face data lies on a lower dimensional manifold. The PCA (or KL) gives a global compact representation of this class of images: Face subspace.

6

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

3D space

2D subspace

Eigenvectors

In the case of faces, the usual way to work is:

    • Face images of size N = mxn.
    • Use of PCA to reduce the representation space M << N
    • Eigenvectors are usually referred to as eigenfaces.
    • A linear combination of eigenfaces can describe any face image.

7 of 103

Still Image: Holistic: PCA

  • Computation of the face class (Ω): Study of the eigenfaces and of the associated eigenvalues. Those of lower energy are said to describe the noise and rejected.

7

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  • Similar concept as:
    • the Karhunen-Loève Transform (KLT).
    • Singular Value Decomposition (SVD)

…

M. Turk and A. Pentland, “Eigenfaces for Recognition.” Journal of. Cognitive Neuroscience 3:1, 1991

8 of 103

Still Image: Holistic: PCA

  • Dimensionality reduction: The raw face vector x is approximated in a new space of dimension M.

8

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

vi sorted by maximum variance in the input space

9 of 103

Still Image: Holistic: PCA

9

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Raw pixels from the face with the substracted mean average.

Face projected in the subspace of lower dimension M.

10 of 103

Still Image: Holistic: PCA

10

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

System Input

y0

+

+

+

+

…

Representation of the individual after projection in the face space.

​

x

x

x

x

y1

y2

yM

mx

v1

v2

vM

11 of 103

Still Image: Holistic: PCA

How to recognize faces with PCA

​

11

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  1. Find the eigenvectors from the input data space.

​

  • Build the projection matrix W for a desired dimension M.
  • Project all faces into the learned subspace.
  • Train a classifier where each class corresponds to an ID.
  • Test the classifier.

12 of 103

Still Image: Holistic: PCA + k-NN

How to recognize faces with PCA

​

12

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Unrecognized face

Example: k-Nearest Neighbours with the Euclidean distance.

13 of 103

Still Image: Holistic: PCA + k-NN

How to recognize faces with PCA

  • Euclidean distance: Extension to k Nearest Neighbors (k-NN):
    • To find the k-NN to the individual sample and decide the class with the largest number of representatives among the k-NN.
    • The high dimensionality of the data leads to costly implementations:
      • Use of partial distances that calculate the distance in a subset of dimensions.
      • Use of search trees that pre-structure the searching space.
      • Use hashing functions for fast indexing with binary codes and Hamming distances.

​

13

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

In that example k = 7:

    • Neighbors of class X : 5
    • Neighbors of class X : 2

14 of 103

Still Image: Holistic: PCA + k-NN

14

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Practical exercise 2: Principal Component Analysis for Faces.

Deadline: December 13, 2015.

15 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
      • Convolutional Neural Networks (CNNs, ConvNets)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

15

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

16 of 103

Still image: Holistic: Bayesian PCA

Concept: Replace the Euclidean distance for a Probabilistic measure on a 2-class problem between pairs of faces, instead of the Euclidean distance.

​

16

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

2 classes for pairs of faces

Class intrapersonal

The two faces correspond to the same person

Class interpersonal

The two faces correspond to different persons.

17 of 103

Still image: Holistic: Bayesian PCA

A different PCA is estimated for each class.

​

Different values of the new subspace dimensions MI and ME have been reported:

  • Initial papers: MI = 10 and ME = 30.
  • FERET test: MI = ME = 125.

17

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Class

intrapersonal

Class

interpersonal

WE

Wi

PCA

PCA

18 of 103

Still image: Holistic: Bayesian PCA

18

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

The usual way to work is:

​

    • Classes are generated from differences of images Δ = I1 – I2.
    • Both classes are assumed to be Gaussian distributed.
    • Two faces are said to belong to the same individual using a MAP estimation:

P(Ωi |Δ) > P(ΩE|Δ).

​

Face 1

Face 2

-

WI

WE

19 of 103

Still image: Holistic: Bayesian PCA

19

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

B. Moghaddam, T. Jebara, A. Pentland, “Bayesian Face Recognition.”, Pattern recognition, 2002

20 of 103

Still image: Holistic: Bayesian PCA

Simpler form using Maximum Likelihood (ML):

​

20

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

-

WI

-

WI

I2: Model image

0.85

0.30

21 of 103

Still image: Holistic: Bayesian PCA

Eigenface similarity VS Probabilistic similarity

​

21

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

B. Moghaddam, T. Jebara, A. Pentland, “Bayesian Face Recognition.”, Pattern recognition, 2002

22 of 103

Still image: Holistic: Bayesian PCA

  • The Bayesian approach outperforms the basic PCA approach:

​

    • It was tested in the FERET dataset.

​

  • However, it presents two main problems:

​

    • Strong dependency with the correct alignment of the face images:

​

      • It is necessary to accurately detect the facial features and to precisely scale the images.

​

    • The enrollment of new individuals is more complex:

​

      • In the Euclidean distance, a new enrollment requires the recomputation of W with as many new xi as provided exemplars.

​

      • In the Bayesian approach, the new set of differences have to be computed, so a larger processing effort is required.

22

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

23 of 103

Still image: Holistic: Bayesian PCA + Parzen

Probabilistic technique: Estimation of the pdf: use of Parzen classifiers:

​

    • To impose on every training sample a window function to get a probability density estimate for each class. Typically, Gaussian windows are used.
    • It is time consuming and requires storing all training samples:
      • Use of reduced Parzen classifiers to select a subset of representative samples.

23

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  • Other approaches have been proposed:
    • Neural Networks, Supporting Vectors Machines, Self Organizing Maps, …

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

24 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

24

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

25 of 103

Still image: Holistic: Fisher LDA

25

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  • Problem: PCA searches for the largest variance in an unsupervised manner. That is, it does not consider any labeling of the data. This may drive into bad projections for classification.

​

M. Welling. “Fisher Linear Discriminant Analysis", University of Torornto.

PCA

Example: PCA projection from 2D to 1D

Bad projection for classification, even if indicates direction of maximum variance.

26 of 103

Still image: Holistic: Fisher LDA

  • Concept: To create a space (in a supervised manner) in which the transformed samples from a given class are as close as possible, while being as far away as possible when belonging to different classes. That is, LDA maximizes the distance among classes.

​

26

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

M. Welling. “Fisher Linear Discriminant Analysis", University of Torornto.

Example: LDA projection from 2D to 1D

LDA

Good projection for classification.

27 of 103

Still image: Holistic: Fisher LDA

27

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Source: http://www.dtreg.com/lda.htm

Example:

28 of 103

Still image: Holistic: Fisher LDA

​

  • Two concepts are used:

​

    • [INTER] The Between-class Scatter Matrix (SB)

​

    • [INTRA] The Within class Scatter Matrix (SW)

28

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Source: http://www.dtreg.com/lda.htm

29 of 103

Still image: Holistic: Fisher LDA

​

  • Two concepts are used:

​

    • [INTER] The Between-class Scatter Matrix (SB), representing the scatter of the class mean vectors around the overall mean vector.

​

​

​

​

​

    • [INTRA] The Within class Scatter Matrix (SW), showing the average scatter of the sample vectors in the different classes.

29

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

…and Nc is the number of samples in class c.

Where:

30 of 103

Still image: Holistic: Fisher LDA

30

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

​

  • Fisher LDA considers maximizing function J(w)
    • Maximizing the total scatter of data (SB)
    • Minimizing the within scatter of the classes (SW)
  • The solution of the previous minimization problem is reached when:

Details: M. Welling. “Fisher Linear Discriminant Analysis", University of Toronto.

31 of 103

Still image: Holistic: Fisher LDA

31

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  • To ensure that the matrix Sw is not singular (can be inverted), there should be at least N+C samples (N: dimension of feature space, C: number of classes):

​

    • Usually, this is a very large value and cannot be achieved.

​

    • Typically, the problem is solved by first performing a reduction of the space by means of a PCA transform.

32 of 103

Still image: Holistic: Fisher LDA

  • The LDA approach has been shown to outperform the basic PCA approach:

​

    • A complete analysis has not been carried out.

​

  • However, it presents two main problems:

​

    • When the training data set is too small, PCA improves its performance:

​

      • Combinations of both techniques have been proposed.

​

    • The enrollment of new individuals is complex:

​

      • When a new individual is enrolled, the within-class scattering matrix has to be recomputed and, therefore, the global solution changes.

​

32

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

P.N. Belhumeur, J.P. Hespanha, D.J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection”,

IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, July 1997, pp. 711-720

33 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
      • Convolutional Neural Networks (deep learning)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

33

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

34 of 103

Still image: Holistic: ConvNets: FaceNet

34

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Faces

​

Euclidean space where distances correspond to face similarity

FaceNet

Schroff, Florian, Dmitry Kalenichenko, and James Philbin. "FaceNet: A Unified Embedding for Face Recognition and Clustering." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815-823. 2015

​

Extended summary slides by Xavier Giro on the ReadCV seminar.

35 of 103

Still image: Holistic: ConvNets: FaceNet

35

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

End-to-end learning of an embedding (distance metric learning)...

Weinberger, Kilian Q., and Lawrence K. Saul. "Distance metric learning for large margin nearest neighbor classification." The Journal of Machine Learning Research 10 (2009): 207-244

36 of 103

Still image: Holistic: ConvNets: FaceNet

36

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

...by means of well chosen triplets, using curriculum learning.

Bengio, Yoshua, Jérôme Louradour, Ronan Collobert, and Jason Weston. "Curriculum learning." In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009

37 of 103

Still image: Holistic: ConvNets: FaceNet

37

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

38 of 103

Still image: Holistic: ConvNets: FaceNet

38

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

ReadCV 07/04/2015 (Slides by Xavier Giró-i-Nieto) Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." In Computer Vision–ECCV 2014, pp. 818-833. Springer International Publishing, 2014

​

Architecture 1 (NN1):

​

Modified AlexNet according to NYU’s work on visualization.

39 of 103

Still image: Holistic: ConvNets: FaceNet

39

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

ReadCV 07/04/2015 (Slides by Xavier Giró-i-Nieto) Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." In Computer Vision–ECCV 2014, pp. 818-833. Springer International Publishing, 2014

​

Architecture 1 (NN1):

​

Modified AlexNet according to NYU’s work on visualization.

40 of 103

Still image: Holistic: ConvNets: FaceNet

40

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

ReadCV 10/11/2015 (Slides by Elisa Sayrol): Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going Deeper With Convolutions." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9. 2015.

​

Architecture 2 (NN2):

​

Inception

41 of 103

Still image: Holistic: ConvNets: FaceNet

41

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

ReadCV 10/11/2015 (Slides by Elisa Sayrol): Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going Deeper With Convolutions." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9. 2015.

​

Architecture 2 (NN2):

​

Inception

​

...four more variations:

NN3, NN4, NNS1, NNS2

42 of 103

Still image: Holistic: ConvNets: FaceNet

42

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

43 of 103

Still image: Holistic: ConvNets: FaceNet

43

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

LBW: 99.63% (new record)

YouTubeFaces DB: 95.12%

44 of 103

Still image: Holistic: ConvNets: FaceNet

44

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Software implementation: OpenFace

45 of 103

Still image: Holistic: ConvNets: VGG Face

45

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Parkhi, Omkar M., Andrea Vedaldi, and Andrew Zisserman. "Deep face recognition." Proceedings of the British Machine Vision 1, no. 3 (2015): 6.

46 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

46

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

47 of 103

Still image: Local Features

  • Problem: Eigenfaces (PCA) and Fisherfaces (LDA) require at least 8 images per person for a reasonable performace… in good lighting settings !

​

​

47

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Source: http://docs.opencv.org/trunk/modules/contrib/doc/facerec/facerec_tutorial.html#local-binary-patterns-histograms

48 of 103

Still image: Local Features: LBP

  • Concept: Analyze images with local features.

​

Local Binary Patterns (LBPs)

  • Binary code with the relation of each pixel with its local neighborhood.
    • 0 if greater or equal, 1 if minor
    • If 8-connectivity is considered, 28=256 combinations of binary features are possible.

​

48

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

T. Ahonen, A. Hadid, M. Pietikainen, “Face Recognition with Local Binary Patterns”. ECCV 2004

49 of 103

Still image: Local Features: LBP

49

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Source: Yale Facedatabase A/B

Local Binary Patterns are robuts to changes of illumination.

T. Ahonen, A. Hadid, M. Pietikainen, “Face Recognition with Local Binary Patterns”. ECCV 2004

50 of 103

Still image: Local Features: LBP (II)

Local Binary Patterns Histograms (LBPH)

  • Face images are divided in KxK rectangular windows of equal size.
    • Best results obtained with windows of 18x21 pixels.
  • One histogram is computed for each window.
  • Histograms are concatenated to generate a feature vector for each image.

50

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Figure: Matti Pietikäinen

51 of 103

Still image: Local Features: LBP (III)

Local Binary Patterns Histograms (LBPH)

​

  • Recognition based on Nearest Neighbor (NN), different options for similarity.
    • Weights according to the relevance of the window (eg. eyes are more important).
    • Best results with Chi square statistic.

​

​

51

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

T. Ahonen, A. Hadid, M. Pietikainen, “Face Recognition with Local Binary Patterns”. ECCV 2004

52 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

52

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

53 of 103

Still image: Local Features: Graph Matching

  • Concept: To take advantage of the common topological structure of human faces. Faces are represented as labeled graphs:

​

  • Graphs are formed as:

​

    • Nodes, located at fiducial points (e.g.: eyes, mouth, …), contain a set of descriptors of the texture around the point.

​

    • Edges associated to vectors containing 2D distance information between fiducial points.

53

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Local descriptors are commonly associated to local wavelet analysis:

    • A set of wavelet coefficients (typically, Gabor coefficients) computed for different scales and rotations from a given image form a vector (jet).

54 of 103

Still image: Local Features: Graph Matching

54

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

L. Wiskott, J.-M. Fellous, N. Krueuger, C. von der Malsburg,

“Face Recognition by Elastic Bunch Graph Matching”, IEEE Trans. on

Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp. 775-779, 1997,

Illustration of the Elastic Graph Matching concept:

​

55 of 103

Still image: Feature: Graph Matching

  • The image graph is extended to a face bunch graph by keeping the same topology but allowing multiple jets in the same node.
    • Aims at covering wide range of variations in the appearance of faces:
      • Local diversity in eyes, mouth, noses…
      • Global diversity in age, gender, race…

55

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Face Bunch Graph

L. Wiskott, J.-M. Fellous, N. Krueuger, C. von der Malsburg,

“Face Recognition by Elastic Bunch Graph Matching”, IEEE Trans. on

Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp. 775-779, 1997,

56 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models (AAM)
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

56

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

57 of 103

Still Image: Hybrid Methods (I)

  • Concept: To combine the global and local approaches in a single technique.

​

    • Graph matching approaches can already be seen as hybrid methods.

​

  • There are several approaches to the problem:

​

    • Extension of the eigenface concept to that of eigenfeature. Combination of recognition using eigenfaces and eigenfeatures (eigenmouths, eigeneyes, …).

​

    • Use of Active Appearance Models for face recognition. AAM is an integrated statistical model which combines (i) a model of shape variation with (ii) a model of the appearance variations in a shape-normalized image.

57

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

The AAM matching to an image implies :

    • Finding the model parameters which minimize the difference between the image and a synthesized model example projected into the image.
    • Combining the information of:
      • Shape parameters
      • Local profile model parameters: Gray level models (perpendicular to the shape).
      • Shape-free model parameters: PCA approach.

58 of 103

Still Image: Hybrid: Active Shape Models

Active Shape Models (ASM)

    • Statistical models of shapes of objects which iteratively deform to fit to an example of the object in a new image.
      • Look along normals through each model point to find the best local match for the model of the image appearance at that point.
      • Update the pose and shape parameters to best fit the model to the found points.
      • Repeat until convergence.
    • Shapes are constrained by the Statistical Shape Model to vary only in ways seen in a training set of labeled examples.

​

​

​

​

​

​

​

​

      • Active Shape Models (ASM): T.F. Cootes and C.J. Taylor and D.H. Cooper and J. Graham (1995). "Active shape models - their training and application". Computer Vision and Image Understanding (61): 38–59.

​

​

​

​

​

​

​

58

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

59 of 103

Still Image: Hybrid: Active Shape Models

Statistical Shape Models (I)

    • Each shape in the training set is represented by a set of N labeled landmark points, which must be consistent from one shape to the next.
      • Example: In the hand example, the 7th point may always correspond to the tip of the thumb.

​

​

​

​

​

​

​

​

​

​

    • Training dataset required an individual labeling of each landmark point.

​

Source:http://personalpages.manchester.ac.uk/staff/timothy.f.cootes/Models/faceasm0.gif

59

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

60 of 103

Still Image: Hybrid: Active Shape Models

Statistical Shape Models (II)

    • Each shape can be represented by a vector x of dimension 2*N (2D per point).
    • The aligned training set is a cloud in a 2*N hyperspace.
    • PCA is applied to pick up the main axes in the cloud, and model only the first T ones.
    • Each shape is represented by a feature vector of T << N.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Source:http://personalpages.manchester.ac.uk/staff/timothy.f.cootes/Models/faceasm0.gif

60

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

61 of 103

Image: Hybrid: Active Shape Models

Statistical Shape Models (III)

    • New examples can be generated by varying the resulting shape parameters.

​

​

​

​

​

​

​

​

​

​

​

​

​

Source:http://personalpages.manchester.ac.uk/staff/timothy.f.cootes/Models/faceasm0.gif

61

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Example after variations of the first 3 parameters

b1, b2 and b3

62 of 103

Image: Hybrid: Active Appearance Models

Active Appearance Models (AAM)

    • Generalisation of ASM, using the information in the image region, rather than just near the modelled edges.
      • Active Appearance Models (AAM): T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active Appearance Models", IEEE PAMI, Vol.23, No.6, pp.681-685, 2001

​

  • “Interpretation through synthesis”
  • Explain a new example in terms of the model parameters
  • Example: Multiresolution search from displaced position using face model.

​

​

​

62

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

63 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models (AAM)
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

63

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

64 of 103

Face recognition on Video Sequences

  • Research on face recognition from video has intensified in the last decade
  • Work generally focused on achieving accurate recognition results in degraded viewing conditions.
  • There is a gap between actual application needs and the current state of the art.

​

  • Videos afford three useful properties that can aid in recognition:
    • a set of observations, multiple images of the same face (in different conditions)
    • temporal dynamics
    • 3D information (face seen from different angles, may contain 3D geometric information)
  • But nuisance factors arise in unconstrained face recognition applications
    • pose variation
    • illumination variation
    • expression variation
    • scale variation
    • motion blur
    • occlusion

​

64

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

65 of 103

Face recognition on Video Sequences

  • The approaches can be broadly categorized into two groups, depending on which video properties they exploit:

​

​

​

​

​

​

​

​

​

J. Barr, K, Bowyer, P. Flynn, S. Biswas, “Face recognition from video: a review”, Journal of Pattern Recognition and Artificial Intelligence, 2012.

  • Set-based approaches: they treat videos as unordered collections of images and take advantage of the multitude of observations
  • Sequence based approaches: explicitly use temporal information to increase efficiency or enable recognition in poor viewing conditions

​

65

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

66 of 103

Face recognition on Video Sequences

Set based approaches: they differ in terms of whether they fuse information over the observations before or after the matching.

​

​

​

​

​

​

​

​

​

66

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

67 of 103

Face recognition on Video Sequences

  • Fusion before matching: The data or features extracted from each face image are aggregated together prior to recognition. Drawback: sensitivity to the number of faces and the order in which the vectors are concatenated.

​

​

​

​

​

​

​

​

​

67

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

68 of 103

Face recognition on Video Sequences

68

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  • Super resolution techniques attempt to recover high frequency image content from the aggregated frames with the objective of constructing high resolution images.

69 of 103

Face recognition on Video Sequences

69

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

​

  • 3D modeling techniques leverage the potentially wide range of views contained within a set of frames to recover the 3D structure of face, which can aid in attaining invariance to pose changes.

70 of 103

Face recognition on Video Sequences

70

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

      • Manifold modeling techniques: the entire set of observations can be represented with linear subspaces or nonlinear manifolds, constructed with well-defined metrics that measure distances between sets or the variations that they share in common

71 of 103

Face recognition on Video Sequences

71

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

      • Frame selection: Processing video can be computationally demanding, a task that can be reduced with a selection of frames.
        • Low quality frames are discarded.
        • Diversity is desirable.

Berrani and Garcia, “Enhancing Face Recognition from Video Sequences using Robust Statistics”, 2005

72 of 103

Face recognition on Video Sequences

  • The approaches can be broadly categorized into two groups, depending on which video properties they exploit:

​

​

​

​

​

​

​

​

​

J. Barr, K, Bowyer, P. Flynn, S. Biswas, “Face recognition from video: a review”, Journal of Pattern Recognition and Artificial Intelligence, 2012.

​

72

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Sequence based approaches: explicitly use temporal cues during recognition; can improve recognition performance in degraded conditions where portions of the faces are temporarily deformed, occluded or obscured.

73 of 103

Face recognition on Video Sequences

​

  • Spatio-temporal techniques: the most popular class, use both appearance and motion cues to attain a recognition decision, improving recognition in unconstrained contexts. Spatiotemporal recognition methods can be roughly divided into two categories:
    • Methods that split tracking and recognition into separate tasks performed serially (HMM + tracker)
    • Methods that unify tracking and recognition.

73

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

74 of 103

Face recognition on Video Sequences

​

  • Temporal techniques: have begun to arise more recently. This group of techniques employs facial movements as identifying biometric characteristics. They use the ways facial muscles contract or extend when a person speaks or expresses emotions as a biometric characteristics.

​

74

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Survey articles:

  • Video based face recognition: a Survey, H. Wang, Y. Wang and Y. Cao, World Academy of Science, Engineering and Technology, vol 60, pp 293-302, 2009.

75 of 103

Face Recognition: Outline

  • Still images:

​

    • Holistic: Use the whole face region as the input to the recognition system

​

      • Principal Component Analysis (PCA)
      • Bayesian PCA
      • Fisher Linear Discriminant Analysis (LDA)
    • Feature based: Local features such as eyes, nose or mouth are first extracted and their location and local statistics are used.

​

      • Local Binary Patterns.
      • Graph matching methods
    • Hybrid Methods: Combination of the two previous ones.

​

      • Active Appearance Models (AAM)
  • Video sequence:

​

    • Extensions: Consecutive frames are jointly used. No behavioral analysis.
  • 3D analysis:

​

    • Multiple views: Combination of several views, under little variations, of the same person.
    • Holistic based 3D: extension of holistic 2D or surface matching
    • Feature based 3D: local 3D regions, profiles

​

​

75

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

76 of 103

3D Face recognition

  • 3D face recognition is an emerging technology, due to the availability of improved 3D imaging devices and processing algorithms.
  • For such techniques, 3D images of facial surfaces are acquired using 3D acquisition devices
  • Advantages of 3D facial images over 2D facial images:
    • The pose can be easily corrected by rigid rotations in 3D space
    • They provide structural information about the face (e.g. surface curvature and geodesic distances), which cannot be obtained from a single 2D image.
    • 3D face recognition algorithms have shown to be robust to variations in illumination conditions during image acquisition
  • Disadvantages of 3D facial images over 2D facial images:
    • Adversely affected by variable facial expressions
  • Techniques
    • 3D appearance based
    • Surface matching based
    • Model based
    • Local feature based

​

​

​

​

76

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

77 of 103

3D Face recognition: Approaches

77

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

78 of 103

3D Face recognition: Approaches

78

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

79 of 103

3D Face recognition: Appearance-based

3D appearance-based techniques:

  • They are straightforward extensions of 2D appearance based techniques to range images
  • A 3D facial image is an instance in an N dimensional space (N number of points in the image). Faces lie in a linear subspace built from an ensemble of facial range images. Facial images are projected into the subspace and compared by distance metrics.
  • Statistical subspace projection methods:
    • Principal Component Analysis (“eigensurfaces”) [Chang et al, 2005]
    • Linear Discriminant Analysis (“fishersurfaces”) [Heseltine et al, 2004].
  • Results:
    • 3D PCA performs well for less than 100 subjects,

poorly for larger datasets.

    • 3D LDA outperforms 3D PCA.
  • Pros:
    • Easy to implement, fast response
  • Cons:
    • Requires accurate alignment of probe and gallery images

(may be achieved by localizing facial features)

    • Sensitive to pose variations and facial expressions

​

​

79

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

80 of 103

3D Face recognition: Approaches

80

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

81 of 103

3D Face recognition: Surface matching based

Surface matching based techniques:

  • Pairs of 3D facial surfaces are rigidly or non-rigidly registered and compared
  • Use of local curvature features, which are rotation invariant
  • Use of point-to-point matching
    • The most successful are techniques based on the Iterative Closest Point (ICP) algorithm [Besl,1992], where one 3D model is rotated and translated iteratively in space until its distance from other model converges to a minimum [Russ et al, 2005].
  • Metrics:
    • Mean Squared Error (MSE), point to closest point MSE, point to closest MSE, Hausdorff distance.
  • Results:
    • ICP-based algorithms are robust to variable face poses

and illumination conditions during acquisition.

    • They perform better than 3D PCA
  • Cons:
    • ICP based registration is not guaranteed

to converge to a global minimum

    • Computationally expensive
    • It is affected by variable facial expressions

(non rigid deformations of the facial surface)

    • Sensitive to image noise.

​

​

​

​

81

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

82 of 103

3D Face recognition: Approaches

82

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

83 of 103

3D Face recognition: Model / Local feature based

Model based

  • Techniques:
    • 3D parametric-morphable model [Blanz2007]
    • Non-parametric 3D model [Kakadiaris2007]

​

Local feature based

  • Employ structural properties of local regions of the 3D face
  • Techniques
    • Various facial profile curves are rigidly aligned and compared [Zhang et al, 2006]
    • Central vertical facial profile is effective at uniquely identifying individuals
    • Hierarchical graph matching 2D+3D, combining scores of 2D EBGM and 3D EBGM (EBGM applied to facial range images) [Hüsken, 2005]
  • Results:
    • Hierarchical graph matching was one of the top performers at FRGC2006
  • Cons:
    • Adversely affected by variable facial expressions
    • Choice of facial landmarks ad hoc or extension of local 2D to range images

83

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

[Zhang et al, 2006]

84 of 103

3. Testing Conditions: Test Campaigns

Test Campaigns

​

  • Since the mid 90th there are several companies on the market and sell face recognition systems, several techniques proposed by research groups.
    • Is face recognition solved?
    • How to evaluate recognition systems?

​

  • Since 1993 a series of tests have been performed funded though various US government agencies ( NIST, DARPA, DoD) or EU sponsored research projects.
    • 1993 – 1996 FERET Face Recognition Technology
    • 2000 XM2VTS http://www.ee.surrey.ac.uk/Research/VSSP/xm2vtsdb
    • 2002 FRVT Face Recognition Vendor Test
    • 2003 - 2006 FRGC Face Recognition Grand Challenge
    • 2004 BANCA http://banca.ee.surrey.ac.uk.
    • 2006 FRVT
    • 2006 CHIL http://chil.server.de
    • 2010 MBE Multiple Biometrics Evaluation (Still Face)
    • 2012 FRVT new!

84

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

http://www.nist.gov/itl/iad/ig/face.cfm

85 of 103

3. Testing Conditions: Test Campaigns

85

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Since 1993, the error rate of automatic face-recognition systems has decreased by a factor of 272. The reduction applies to systems that match people with face images captured in studio or mugshot environments.

​

Progress is quantified from the 1993 evaluations to MBE 2010. Improvement is reported at five key milestones. For each milestone, the false rejection rate (FRR) at a false acceptance rate (FAR) of 0.001 (1 in 1,000) is given for a representative state-of-the-art algorithm. For each milestone, the year and evaluation are provided.

​

Beginning with the FRVT 2002, the evaluations switched to a benchmark dataset provided by the US Department of State (DoS), which is comparable to the FERET dataset. (Image courtesy of Jonathon Phillips, NIST)

86 of 103

3. Testing Conditions: Databases

Data Bases & Protocols

​

  • FERET: Face Recognition Technology (Department of Defense)
    • http://www.frvt.org/FERET/
    • http://www.frvt.org/EvalMethod.htm (Evaluation Methodology)

​

  • FRVT: Face Recognition Vendor Test (Extension of FERET)
    • http://www.frvt.org/FRGC (Face Recognition Grand Challenge)
    • High resolution still images, 3D images, and multi-images of a person.

​

  • XM2VTS: Extended M2VTS: Multi Modal Verification for Teleservices and Security applications
    • http://www.ee.surrey.ac.uk/Research/VSSP/xm2vtsdb
    • 295 subjects taken over a period of four months. Each recording contains a speaking head shot and a rotating head shot.

​

  • BANCA: Biometric Access Control for Networked and e-commerce Applications
    • http://www.ee.surrey.ac.uk/CVSSP/banca/
    • Face and voice in 4 languages with high and low quality microphones and cameras. Three different scenarios, controlled, degraded and adverse over 12 different sessions spanning three months. In total 208 people were captured, half men and half women.
    • http://www.ee.surrey.ac.uk/CVSSP/banca/
    • http://www.ee.surrey.ac.uk/CVSSP/banca/documentation/messer-avbpa03.pdf (Protocol)

​

  • The Ouru University Physics-based Face Database

​

    • http://www.ee.oulu.fi/research/imag/color/pbfd.html

​

​

  • http://www.face-rec.org/databases/

86

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

87 of 103

3. Testing Conditions: Databases

​

  • MBGC Face Dataset

​

    • Multiple biometric grand challenge database, 3763 videos, 821 subjects
    • Frontal and off angle poses, people walking ,performing activities, conversing.
    • Unconstrained illumination, pose, movement

​

​

​

​

​

​

​

​

​

​

​

​

​

​

87

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

88 of 103

3. Testing Conditions: Databases

​

  • CMU Motion of Body database

​

    • Subjects walking on a treadmill
    • 150 videos of 25 subjects

​

​

​

​

​

​

​

88

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

89 of 103

3. Testing Conditions: Databases

​

  • Honda and UCSD database

​

    • 75 videos of 20 subjects+ 30 videos of 15 subjects
    • Indoor, wide range of poses

​

​

​

​

​

​

​

​

​

89

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

90 of 103

3. Testing Conditions: Databases

​

  • YouTube Celebrities

​

    • 1910 noisy videos of 47actors and politicians.
    • Low-resolution, highly compressed videos
    • Uncontrolled pose, expression, illumination

​

​

​

​

​

​

​

​

​

90

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

91 of 103

3. Testing Conditions: Databases

​

  • Kinect Face Dataset

​

    • Color , depth and 3D, different facial expressions, lighting and occlusion conditions
    • http://rgb-d.eurecom.fr/

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

91

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

92 of 103

3. Testing Conditions: Databases

​

  • Labeled faces in the wild (wild-a)

​

    • Face images for face recognition in unconstrained images (wild-a aligned version)
    • 13000 images collected from the web
    • http://vis-www.cs.umass.edu/lfw/index.html

​

​

92

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

93 of 103

3. Testing Conditions: Databases

​

  • YouTube Faces DB

​

    • 3425 videos of 1595 people,
    • Face videos for unconstrained face recognition in videos
    • http://sss.cs.tau.ac.il/~wolf/ytfaces

​

​

​

​

​

93

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

94 of 103

4. Commercial systems

Example: Israeli company face.com was acquired by Facebook on June 2012 for $55-60M

  • Online API for third-party developers.
  • Estimation of age, gender, mood and attributes.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

94

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Source: http://techcrunch.com/2012/06/18/facebook-scoops-up-face-com-for-100m-to-bolster-its-facial-recognition-tech/

95 of 103

4. Commercial systems

Given the continuous evolution of this market area, rather than listing a set of

products, we give a few links to web pages that follow the evolution of this

technology:

​

​

  • http://www.face-rec.org/vendors/
    • "Facial Recognition Vendor Test" reports
    • links to vendors developing face recognition technology

​

​

  • http://www.biometrics.org/
    • Biometric Consortium
    • Research http://www.biometrics.org/research.php
    • Publication http://www.biometrics.org/publications.php
    • Vendor information http://www.biometrics.org/vendors.php

​

​

95

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

96 of 103

5. Conclusions

  • Face recognition has reached a certain level of maturity after more than 30 years of research:

​

    • Mainly in controlled environment and 2D cases.
    • Especially if used together with other biometrics

​

  • Still no biometric commercial system working robustly under unconstrained conditions

​

  • There are several scenarios and approaches:

​

    • The choice of an appropriate method should be based on the specific requirements of a given task.

​

  • There is still a considerable research activity in the area:

​

    • 3D and video
    • Combination of different face modalities or other biometrics
    • Increase the robustness in uncontrolled scenarios: varying illumination (outdoor), varying pose (naturally), aging, makeup, low quality (surveillance video), compressed images
    • Combination with soft biometrics (provide some information about an individual): ethnicity, skin color, hair color, tattoos, birthmarks,…

​

  • As in all classification problems, a crucial issue is the availability of public databases and protocols to evaluate and compare the methods.

96

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Bowyer et al.: “Face Recognition Using 2-D, 3-D, and Infrared: Is Multimodal better than Multisample?” Proceedings of the IEEE, Vol. 94, No. 11, Nov. 2006

97 of 103

6. References

In addition to the list of references that we have given during the lecture, here we list a few web pages where an updated list of papers and useful links is maintained:

​

  • http://www.face-rec.org/

Face recognition: General references, algorithms, databases, source code, vendors, conferences, research groups

​

  • http://www.facedetection.com/

Face detection: General references, algorithms, databases, source code, vendors, conferences, research groups

​

  • http://www.cs.colostate.edu/evalfacerec/news2010.php

Evaluation of Face Recognition algorithms

​

​

​

97

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

98 of 103

7. Learn more

98

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Predict the lower face with the upper part.

Source: Scikit-learn, “Face Image Completion with multi-output estimators” (2014)

99 of 103

7. Learn more

99

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Data science challenges on Kaggle.

Eg. Facial keypoint detection.

100 of 103

7. Learn more

100

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Summer seminar on Deep learning for computer vision.

  • Temptative dates: 4-8 July 2015
  • Temptative schedule: 3pm-7pm
  • Hand on lectures included !!

101 of 103

7. Learn more

101

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

Computer Vision Reading Group

​

  • ECTS for TELECS.
  • In partnership with Universitat de Barcelona.
  • Fall 2015: Tuesdays at 12:10pm @ D5-003.
  • Spring 2015: To be decided

102 of 103

7. Learn more

102

IMAGE BASED BIOMETRICS

FACE RECOGNITION

BIOMETRIC SYSTEMS

  • Mobility opportunities (ask me the slides if interested)

103 of 103

Biometric Systems��Image Based Biometrics�Face Recognition

Xavier Giró i Nieto, Verònica Vilaplana, Ferran Marqués