October 20^th, 2021

Improving Situation Awareness by Detecting Objects of Interest

Marielle Mokhtari PhD

Defence Scientist

C2I, DRDC Valcartier Research Centre

DG Sci Eng

UNCLASSIFIED

Outline

1. Context & Problem

Urban Ops – Reality Modelling – 3D Objects to avoid modelling

• 3D Object Detection – Lidar Data
• Case Study #1 – Aerial Lidar

Process – Results – Discussion

4. Case Study #2 – Terrestrial Lidar

Process – Results – Discussion

5. Conclusion

​

2

UNCLASSIFIED

1. The Context

3

• Trends in demographics – Continuous & significant increase in both the number & size of urban environments throughout the world
• Urban ops – Military ops of the future
• Characterization of urban environments
• Intricate topography & Large concentration of population
• Multi‐faceted, diverse & highly dynamic environments
• Defined by physical terrain; human terrain; interactions & interdependencies that connect the physical & human terrains
• In Urban Ops → An important element of SA is the complexity of the physical environment (natural terrain & man-made structures)
• 2D data (maps or imagery) has always been the primary means for commanders to understand the terrain and improve SA Introduction of 3D data (3D representations of urban environments) as a new source of information

​

UNCLASSIFIED

1. The Problem – Useless Data

4

UNCLASSIFIED

• Lidar data & imagery (aerial data) Digital 3D models of urban environments
• Make the 3D urban environment model more realistic – e.g., to compensate for the lack of details on the ground → Adding additional geographic information (terrestrial data – static or mobile)
• Filter out useless 3D objects – Specific objects (e.g., cars & humans) should be removed from the 3D model

→ Detect & classify objects of interest in 3D datasets before 3D modelling

2. 3D Object Detection

5

UNCLASSIFIED

• Goal of object detection : locate & classify specific objects appearing in a dataset
• Object detection = Object localization (i.e., identifying the location of one or more objects) + Object classification (i.e., predicting the class of detected objects)
• Canadian National Defense program “Innovation for Defence Excellence and Security” (IDEaS) → Competitive Projects → Challenge “Detect and Classify Objects of Interest” (84 proposals – 13 chosen for Phase 1 – 6 chosen for Phase 2 – 1 chosen for Test drive & 2 for Additional development)
• All proposals focused on the detection & classification of objects of interest in images → How to detect objects of interest in a 3D referential system ? How to detect 3D objects in a 3D dataset?

2. 3D Object Detection �In Lidar Data

6

UNCLASSIFIED

• Classifying 3D datasets into objects of interest composing the physical terrain
• is of prior importance to 3D scene understanding & 3D reconstruction
• allows the segmentation of specific objects in order to model them separately
• Lidar-based point clouds could be classified into classes by the provider.
• But … this option is often expensive.
• It becomes essential to have a way to classify any lidar point cloud

​

• Automatic point cloud classification → Highly active research area
• Point cloud processing, mining & knowledge discovery → Highly active research areas

2. 3D Object Detection �In Lidar Data

7

UNCLASSIFIED

• Aerial lidar (scanning of vast urban areas in relatively short time) + Terrestrial lidar (– static or mobile – scanning of specific urban areas) = Better reality modelling
• Due to number of points & their different spatial resolution, aerial & terrestrial lidar need to be classified in a separate way
• Aerial lidar can be classified using classic Machine Learning algorithms in less time than by applying more traditional methods
• Terrestrial lidar, which usually requires more learning to segment the objects, is better classified by using Deep Learning algorithms

3. Case Study #1 – Classification of Aerial Lidar�Process

8

UNCLASSIFIED

• Objective: Classify aerial lidar in order to extract the points belonging to specific objects → Help 3D scene understanding & Ease 3D modelling of urban env.
• ML classification on 3D point clouds
• Process:
• Analysis of input data
• Feature computation based on data analysis
• Definition of a set of classes (rural vs urban vs forest) → 6 classes
• Labeling, Classifier definition & Training
• Classification computation
• Assessment : Definition of a set of metrics to assess the quality of the model

3. Case Study #1 – Classification of Aerial Lidar Results – Montreal

9

UNCLASSIFIED

Ground, Vegetation, Roof, Façade, Car, Unclassified

3. Case Study #1 – Classification of Aerial Lidar Results – Montreal

10

UNCLASSIFIED

Ground, Vegetation, Roof, Façade, Car, Unclassified

3. Case Study #1 – Classification of Aerial Lidar Results – New-York

11

UNCLASSIFIED

Ground, Vegetation, Roof, Façade, Car, Unclassified

3. Case Study #1 – Classification of Aerial Lidar Discussion

12

UNCLASSIFIED

• Limitations:
• Vegetation is often mixed with roofs
• Some façade points are classified as roof
• Some trucks & trains are classified as roof
• Misclassification of ground points with other classes such as roof & car
• Overall classification is good → The classes building, ground & vegetation can be confidently extracted & modeled separately
• Improvement in 3D scene understanding & consequently in 3D scene representation

4. Case Study #2 – Classification of Terrestrial Lidar�Process

13

UNCLASSIFIED

• Objective : Classify terrestrial lidar in order to extract the points belonging to façades → Augment existing 3D urban environment models that lack detail on the façades
• DL classification on 3D point clouds
• Challenge: Need a lot of data to have interesting results. Search for labeled datasets to train, validate and test the classification
• Process:
• Point analysis – Ground vs non-ground points
• Definition of a set of classes → 8 classes
• Labeling, Training
• Metrics definition for assessment

4. Case Study #2 – Classification of Terrestrial Lidar�Classes chosen to classify the 3D Point Clouds

14

UNCLASSIFIED

4. Case Study #2 – Classification of Terrestrial Lidar�Results

15

UNCLASSIFIED

Man-made terrain is blue,

Natural terrain is dark green,

High vegetation is green,

Low vegetation is olive,

Building is light green, Hardscape is yellow,

Scanning artefacts are orange

Car is red.

4. Case Study #2 – Classification of Terrestrial Lidar�Results

16

UNCLASSIFIED

Man-made terrain is blue,

Natural terrain is dark green,

High vegetation is green,

Low vegetation is olive,

Building is light green, Hardscape is yellow,

Scanning artefacts are orange

Car is red.

7. Conclusion

17

UNCLASSIFIED

• Classification of lidar data into 3D objects of interest in order to
• understand the operational environment & to support decisions
• establish the foundations of the 3D modelling process of the operational environment.
• Aerial lidar + ML algorithm → Automation + Free-of-use tool
• Terrestrial lidar + DL algorithm

18

UNCLASSIFIED

Thanks for your attention!

Q & A