Detecting Anomalies in Object Appearance and Motion Dynamics

​

�Mazen Alotaibi

1

2

https://loc.gov/pictures/resource/var.1680/

What is Common Sense?

Object Permanence:�“Understanding that items and people �still exist even when you can't see or �hear them.” - webmd

3

https://loc.gov/pictures/resource/var.1680/

4

https://www.youtube.com/watch?v=rVqJacvywAQ

Why Machine Common Sense is important?

• Self-driving cars

​

5

Why Machine Common Sense is important?

• Self-driving cars
• Machines don’t learn common �sense automatically
• Teaching all of cases where �common sense knowledge is �applied is hard

6

Benchmark^[1]

• Measures the performance of an agent perceiving the world without any interaction on many common sense reasoning tasks.
• Object Permanence
• Gravity
• Expects from agent to detect anomalies.

7

[1] Ricochet et al. “IntPhys 2019: A Benchmark for Visual Intuitive Physics Understanding”

Prior work

• Utilize a physical simulation module to predict future object states from current beliefs.
• Develops beliefs by answering two questions without �update^[1].
• Develops initial beliefs for object states but update when�it is reasonable^[2].

8

[1] Battaglia et al. “Simulation as an engine of physical scene understanding”

[2] Smith et al. “Modeling Expectation Violation in Intuitive Physics with Coarse Probabilistic Object Representations”

Table of Content

• MCS Challenge
• Our Approach
• Evaluation
• Future work

9

MCS Challenge

10

Problem Statement

• Can a machine predict whether a scene is plausible or implausible based on a set of expectations?

11

Scene

Plausible Y/N

Machine

12

Collisions (COLL)

Focus objects can't change their �motion or appearance without �an explanation.

13

Gravity Support (GRAV)

A dropped focus object follows �gravity.

14

Object Permanence (OP)

Focus objects can't appear �or disappear from the scene �without an explanation.

15

Shape Constancy (SC)

Focus objects can't change �their appearance.

16

replace with video

Spatiotemporal Continuity (STC)

A thrown focus object needs �to have a continuous motion

17

Input

• RGB images
• Ground truth segmentation masks
• Depth images

18

Problem Statement

• Can a machine predict whether a scene is plausible or implausible based on a set of expectations?

19

SEG

RGB

Depth

Plausible Y/N

Machine

Approach

20

System - Pipeline

21

Mask to 2dbox

Tracker

SEG

RGB

Depth

Learning

component

Reasoning�Agent

Plausible Y/N

Rule-base

component

Role Assigner

3D Amodal Detector

Algorithm

Mask to 2dbbox

• Given ground truth segmentation masks
• Converts masks to 2D bounding boxes coordinates

22

Mask to 2dbox

Tracker

Reasoning

Agent

Role Assigner

3D amodal detector

Tracker

• Given 2D bounding boxes, create tracklets

23

Mask to 2dbox

Tracker

Reasoning

Agent

Role Assigner

3D amodal detector

Tracker

2D �Bounding Boxes

Tracks

RGB

Tracker - Pipeline

24

1st�Stage

Learning

component

Algorithm

Voting

Candidates�pairing

Merging

RGB

Tracks

2D �Bounding �Boxes

2nd Stage

Tracker - 1st Stage

• Bi-Linear LSTM model^[1] (out-of-the-box)
• Online tracker
• Relies on motion and appearance cues.
• Given 2D bounding boxes, outputs tracklets of continuous motion.

25

[1] “Discriminative appearance modeling with multi-track pooling for real-time multi-object tracking”, � Kim et al. - CVPR 2021

1st Stage

Tracklets

2D �Bounding Boxes

RGB

Tracker - 1st Stage

Objects before and after�occlusion have different�object IDs.

26

Tracker - 2nd Stage

• Components:
• Candidates pairing
• Voting
• Merging

27

2nd Stage

Tracks

Continuous�Tracklets

RGB

Tracker - 2nd Stage (Candidates Pairing)

• Creates a candidate pair that exists after each other without being visible in any of the frames that they exist on.

28

Tracker - 2nd Stage (Voting)

• Runs a similarity scoring network on multiple pairs from two different tracklets to generate a set of similarity scores.

​

• If the average of the similarity scores is above a threshold, both tracklets be considered as potential merge candidates.

29

Tracker - 2nd Stage (Voting)

30

Tracker - 2nd Stage (Voting)

• Siamese Network with pre-trained ResNet18 as backbone
• Relies on appearance cues only
• Given two images, outputs similarity score of two images.

31

Similarity�Score

Network

Tracker - 2nd Stage (Merging)

• Merges tracklets based on the voting score.

32

Tracker - Pipeline

33

1st�Stage

Learning

component

Algorithm

Voting

Candidates�pairing

Merging

RGB

Tracks

2D �Bounding �Boxes

2nd Stage

Role Assigner

• Pre-trained ResNet18
• Given tracklets, for each, assign a role of
• Focus object (solid)
• Environmental object (dashes)

34

Mask to 2dbox

Tracker

Reasoning

Agent

Role Assigner

3D amodal detector

Role�Assigner

Focus Object�Y/N

Tracks

RGB

3D amodal detector

• CenterTrack^[1] (out-of-the-box).
• Given 2D bounding box of an object and its depth, estimate the 3D bounding cube.

35

[1] “Tracking Objects as Points”, Xingyi Zhou et al. - ECCV 2020

Mask to 2dbox

Tracker

Reasoning

Agent

Role Assigner

3D amodal detector

3D�Amodal�Detector

3D �Tracks

2D Tracks

Depth

Rule-based Reasoning Agent

• Given tracklets, predict if the scene is plausible or implausible.
• For each scene type, we have developed rule-based VOE classifiers.

36

Mask to 2dbox

Tracker

Reasoning

Agent

Role Assigner

3D amodal detector

Reasoning�Agent

Plausible Y/N

3D

Tracks

Rule-based Reasoning Agent - GRAV

• Assumptions for fallen focus �object:
• Has only gravity force applied �to it.
• Is supported by the support �object if:
• Lands on top the support object.
• Well-balanced.

37

Rule-based Reasoning Agent - GRAV

• Given the first position of the fallen focus object and the position of the support object.
• Estimate the last relative position of the fallen object to set an expectation.

38

Rule-based Reasoning Agent - SC

• Assumptions for focus �objects:
• Don’t change appearance.
• Enter the scene from the �edges of the frame only.

39

Rule-based Reasoning Agent - SC

• Check if a new track was initialized in the middle of the scene.

40

Rule-based Reasoning Agent - OP

• Assumptions for focus �objects:
• Enter the scene from the �edges of the frame only.
• Once they leave the scene, �they can’t re-appear.

41

Rule-based Reasoning Agent - OP

• Objects can only enter and leave the sense from the sides.
• When an object leaves, the object shouldn’t re-appear after
• When an object doesn’t leave and was occluded, we expect it to be behind the occluder wall.

42

Rule-based Reasoning Agent - STC

• Assumptions for focus �objects:
• Enter the scene from the �edges of the frame only.
• Have continuous motion.

43

Rule-based Reasoning Agent - STC

• When a focus object has a gap in its track
• Check if the gap has occluder walls placed throughout the entire distance where the object was absent.

44

Rule-based Reasoning Agent - COLL

• Assumptions for focus �objects:
• Enter the scene from the �edges of the frame only.
• Two objects can only interact �if they share depth and �intersect.

45

Rule-based Reasoning Agent - COLL

• Two objects can collide if they share the same depth and path.

46

Evaluation

47

Evaluation

• Evaluation set:
• For each scene type, randomly sample 100 scenes of plausible and implausible.
• Comparing:
• Every component against baselines.
• The system against a baseline system.

48

Tracker

• Baseline:
• ResNet18 pre-traiend on ImageNet, not fine-tuned on MCS data
• Normalization then dot product
• Metrics:
• Number of Identity Switch (IDSW)
• Multiple Object Tracking Accuracy (MOTA)

49

Tracker

50

Role Assigner

• Baseline:
• Decision Tree model to classify roles based on object area and position.
• Metric:
• F1-score, positives are Focus objects

51

Role Assigner

52

Reasoning Agent

• Baseline:
• Learned Reasoning Agent trained on plausible scenes.
• Metric:
• F1-score, positives are implausible scenes

53

Reasoning Agent - Ground-Truth Tracks

54

Reasoning Agent - Actual Tracks

55

Future

56

Improvement

• Data generation: more variation and better quality.
• Tracker
• 1st Stage: introduce a heuristic to split tracklets when object changes its color without occlusion.
• 2nd Stage: train on hard-examples - similar shapes with same colors.
• Reasoning Agent: design rules that are more robust to noise in the pipeline.

57

Questions?

58

Thank you!

59

Appendix

60

Number of Identity Switch (IDSW↓)

• Counts the number of emergences when a ground truth target i is matched to hypothesis j and the last known assignment was k (k != j)

61

Leal-Taixe et al. MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

Multiple Object Tracking Accuracy (MOTA↑)

• Compute a set of measures per frame
• Perform matching between predictions and ground truth
• FP = False positives
• FN = False negatives (missing detections)
• IDSW: identity switches

​

62

Leal-Taixe et al. MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking