TechTalks:�Person Re-Identification

By Manideep Kolla

Introduction

The Questions:

How can we associate people across unobserved regions?

OR

How can we identify a person at different times recorded on multiple cameras at different angles and places?

Person Re-Identification

Person Re-identification is the task of re-identifying a target person across multiple non overlapping cameras.

​

Key Challenges in Person Re-Identification

ViewPoint Change

​

Illumination variation

Partial Occlusion

Change in camera ViewPoint

Change in Illumination

Objects blocking the image view

Current Approaches

Feature map Similarity Estimation - A Siamese network

• A siamese network few convolution layers will be used for which the inputs are the reference image and the checking image
• Two sets of feature maps will be generated each from each input.
• A similarity estimation layer like Cross Correlation layer, Euclidean distance or neighbourhood difference layer will be used between these two sets of feature maps.
• And then after a couple of convolution layers and a flat layer we output a binary value from a softmax layer. With 1 for similar person and 0 for different person.

Current Approaches

Feature map Similarity Estimation - A Siamese network

Current Approaches

Re-Identification through Pose Estimation

• Generating heat maps of the persons based on their pose.
• Fourteen keypoints are considered for every person representing different parts of the body.
• Generate heat maps or feature maps by training a pose estimation network and incorporate these heat maps into another network which does re-identification task.
• The network will output a n-dimensional vector or a feature map for each of reference and checking image and a distance metric is used to compare them.

Current Approaches

Re-Identification through Pose Estimation

Spindle Net

Current Approaches

Re-Identification through Pose Estimation

What I have been doing...

The Architecture

• Used a Siamese network with shared weights network.
• The output feature maps are then passed into a similarity estimation layer, in this case a Cross-input Neighborhood difference layer.
• And then after three convolution layers, we will get a binary softmax prediction.
• The motivation behind taking differences in a neighbourhood is to add robustness to the positional differences in the corresponding features of the two input images.

Training Scheme:

• The network is trained by passing a pair of images. This pair is selected such a way that to every positive pair a random negative pair will be chosen.
• Adam optimizer is used and the loss function used is the sparse categorical cross entropy loss

The Architecture

The Results

The Results

Confusion Matrices

Testing on CUHK01

The Results - FUJITSU Data

The Results

Loss Plot

Accuracy Plot

The Results - True-Positives

The Results - True-Negatives

The Results - False-Positives

The Results - False-negatives

Challenges I have faced

• Wasn't able to implement the Normalized Cross-correlation layer in simple python instead of CUDA. The simple python implementation would take about 0.5 seconds to forward pass a single pair of feature maps of shape (37, 12, 25) with which its impossible to train the model.
• When using a very deep convolutional network instead of a similarity estimation layer, the learning rate should be very less for the network to learn otherwise it’s predicting every pair as similar.
• There was as lot of overfitting initially due to which i have introduced L2 regularizers and Dropouts in the network.
• The model I have build right now is good enough, but Introduction of features extracted from a pose estimation network will considerably improve the results as discussed earlier. But keeping the duration in mind I haven't tried this.

​

Next steps

• I have tried using hard negative mining, but the results are not very good compared to normal training. Need to work on this and fix it.
• The model I have built for person re-identification can be used to differentiate between a pair of images whether they are similar or different but cannot directly track and re-identify a person in different cameras like I have shown earlier. So, this network should be remodeled to do live person tracking and Re-identification.
• Pose estimation is not used in the network I have built. But using pose estimation will improve the current results. Using pose estimation, the network can learn about misaligned and moving body parts.

References

1. An Improved Deep Learning Architecture for Person Re-Identification
2. Deep Neural Networks with Inexact Matching for Person Re-Identification
3. Spindle Net: Person Re-identification with Human Body Region Guided Feature Decomposition and Fusion
4. Attention-Aware Compositional Network for Person Re-identification
5. Person Re-identification Datasets (Information about all the datasets)

Thank you.