Open-World Panoptic LiDAR Segmentation

Students: Meghana Ganesina, Anirudh Chakravarthy

Advisors: Aljosa Osep, Deva Ramanan, Shu Kong

MSCV capstone project overview, January 2022

Mobile robot perception

2

LiDAR-based mobile robot perception

3

Motivation

• Prior work
• Static models, do not adapt their behaviour over time
• New data: we would need to manually label and restart the training process
• Would this setting generalize to the real-world?

Our work

Continual learning for LiDAR panoptic segmentation via object discovery

• Robot drives around and collects streams of LiDAR sensory data
• At the end of the day: re-consolidate this data (offline).
• Did we observe any “novel” object classes that we do not recognise?
• If so: ask humans about this class (human-in-the-loop) / pseudo-labels (clustering)
• Incrementally update the system

Open Set Vs Closed Set

6

Datasets

• Semantic KITTI, Panoptic NuScenes

4D Panoptic LiDAR segmentation

t

t+1

t+2

Point sampling

4D Semantic + Instance Predictions

Aygun et al, 4D Panoptic LiDAR Segmentation, CVPR 2021.

LiDAR Panoptic Segmentation (single-scan)

Single-scan LiDAR Panoptic Segmentation (Behley et al., ICCV’19, ICRA’21)

Semantic Segmentation

RangeNet(++) - Millioto et al., IROS’19

KPConv - Thomas et al., CVPR’19

Object Detection

PointPillars - Lang et al., CVPR’19

Towards Open World Object Detection

K J Joseph, Salman Khan, Fahad Shahbaz Khan, Vineeth N Balasubramanian

​

CVPR’21

Open world recognition

• Bendale et al., Towards open world recognition, CVPR’16
• Key idea:
• Recognize `novel` classes
• Label and re-train

​

​

​

​

​

​

Open world detection

The premise:

• Detector will detect and classify “known” object classes
• The remaining anchor boxes will be detected/classified as “unknown” objects or rejected as “stuff” or “background” classes
• Those detected “unknown” objects will be labeled by human annotators and used to (incrementally) re-train the detector

12

Method

• Maximize discrimination between latent representations in the feature space
• How? Contrastive learning!
• Intuition: by “pushing apart” features for (distinct) “known” classes and “unknown” classes, it will become easier to identify unknown classes as novel

​

​

​

​

Experiments

Method

• Base: Faster R-CNN
• Add additional (contrastive) loss that “pulls apart” features in the embedding space (N-known + unknown)

​

15

Exemplar-based Open-Set Panoptic Segmentation Network

Jaedong Hwang, Seoung Wug Oh, Joon-Young Lee, Bohyung Han

​

CVPR’21

Open-world panoptic segmentation

• Panoptic segmentation: instance + semantic segmentation
• Open-world panoptic segmentation: remove % of labels for certain classes and call them ‘unknown’

Method

• Based on Panoptic FPN two-stage top-down approach [Kirrilov et al., CVPR’19]
• Sample proposals
• Criterion: > 50% of the area in the ‘void’ area.
• Based on the ‘objectness’ score, extract features (1024-dim)
• Clustering
• Every 200 iter, k-means
• Pick ‘good’ clusters: high average objectness score, small average cosine distance between centroid and elements -> store them
• Mining
• Find exemplars in the incoming mini-batch: compute cosine distance between proposals and exemplars (clusters) -> unknowns (use for supervision)
• Standard cross-entropy loss + “negative supervision” term
• Just treat mined classes as GT

Experiments

• COCO dataset (COCO stuff)
• Declare % of classes as ‘unknown’
• 5%: car, cow, pizza, toilet
• 10%: boat, tie, zebra, stop sign
• 20%: table, banana, bicycle, cake, sink, cat, keyboard
• Observation: due to “dense” GT, objects always somehow well delineated

Results

• Utilizing the “void” boxes

​

​

​

• Baselines

20

Roadmap

• Instance mining - Jan 26
• Define task sets, train models (supervised)
• Instance segmentation (we are here)
• Instance tracking
• Instance grouping / discovery - Mar 15
• Implement evaluation set-up (assume clusters are given)
• Simple baseline (embeddings from a pre-trained network + clustering)
• Self-supervised embedding learning + clustering
• End-to-end model
• Model re-training - Apr 15
• Transfer labels clusters -> point clouds
• Training from scratch, incremental update with replay

1st milestone

2nd milestone

3rd milestone

1. Supervised Learning

• Dataset: SemanticKITTI, Behley et al., ICCV’19, KITTI RAW (1.5h of “raw” recordings), Geiger et al., CVPR’12

• Supervised Learning

• We can do K-way classification reasonably well! (SemanticKITTI validation split)

2. Instance mining

• Goal: stream of raw sensory data => a set of potential objects
• Per-scan proposal generation
• Hu et al., Learning to Optimally Segment Point Clouds, RAL’19
• Instead of the learned regressor average “objectness score” (estimated by the network) across each segment

24

Thank you!

25