Efficient Spatio-Temporal Processing of Event Data

​

Kexin Shi, Yifei Liu�

Mathias Gehrig, Nico Messikommer, Davide Scaramuzza

​

Master Project�Robotics and Perception Group

Department of Informatics, University of Zurich

Institute of Informatics – Institute of Neuroinformatics

Kexin Shi & Yifei Liu – Robotics and Perception Group

1

Motivation

picture: https://github.com/TimoStoff/event_utils

Ways of processing events:

​

(a) Point-based

(b) Voxel-based

(c) Others: spiking neural networks, recurrent networks, etc.

(a) point-based

(b) voxel-based

(c) spiking neural network

Kexin Shi & Yifei Liu – Robotics and Perception Group

2

Related Work

a) Point-based:

EventNet[1]

​

b) Voxel-based:

Video to Events[2] → classification, segmentation

​

c) Other methods:

E2VID[3] → video reconstruction

E-RAFT[4] → optical flow

​

​

​

[1] Y. Sekikawa, IEEE/CVF 2019 [2] D. Gehrig, CVPR 2020 [3] H. Rebecq, IEEE PAMI 2019 [4] M. Gehrig, 3DV 2021

Kexin Shi & Yifei Liu – Robotics and Perception Group

3

Overview of Models

• Voxel-based 2D models.

e.g. 2D CNN (2D Resnet, 2D Unet)

​

• Voxel-based 3D models.

e.g. Volumetric CNN (3D ResNet, 3D Unet)

​

• Point-based 3D models.

e.g. PointNet, PointNet++.

​

• Mixing voxel features and point features.

e.g. Point-Voxel CNN (PVCNN) [5]

[5]Liu, Zhijian, Haotian Tang, Yujun Lin and Song Han. “Point-Voxel CNN for Efficient 3D Deep Learning.” NeurIPS (2019).

Kexin Shi & Yifei Liu – Robotics and Perception Group

4

Point-Voxel CNN

Kexin Shi & Yifei Liu – Robotics and Perception Group

5

Task 1: Object Classification

​

Dataset: N-Caltech101.

Label: barrel

Label: anchor

https://www.garrickorchard.com/datasets/n-caltech101

Kexin Shi & Yifei Liu – Robotics and Perception Group

6

Task 2: Optical Flow Regression

Dataset: DSEC

Input

(events visualized as an image)

Optical Flow

https://dsec.ifi.uzh.ch/

Kexin Shi & Yifei Liu – Robotics and Perception Group

7

Research Questions

Kexin Shi & Yifei Liu – Robotics and Perception Group

8

RQ1: How to downsample effectively?

​

Kexin Shi & Yifei Liu – Robotics and Perception Group

9

Challenge: number of points

An anchor in N-Caltech101

200,000 events

A chair in ModelNet40

1,024 points

Kexin Shi & Yifei Liu – Robotics and Perception Group

10

RQ1: How to downsample effectively?

Kexin Shi & Yifei Liu – Robotics and Perception Group

11

RQ1: How to downsample effectively?

10 temporal bins,

threshold p>1

50 temporal bins,

threshold p>1

raw

random

10 temporal bins,

threshold p>2

50 temporal bins,

threshold p>2

Kexin Shi & Yifei Liu – Robotics and Perception Group

12

RQ1: How to downsample effectively?

Classification:

Regression:

Observation:

1) The less points, the faster speed, but also the lower performance.

2) With similar number of points, data-dependent downsample performs better than random sample.

Kexin Shi & Yifei Liu – Robotics and Perception Group

13

RQ2: Performances of different methods?

Kexin Shi & Yifei Liu – Robotics and Perception Group

14

RQ2: Performances of different methods?

RQ2: Performances of different methods?

Classification:

Regression:

Kexin Shi & Yifei Liu – Robotics and Perception Group

15

RQ2: Performances of different methods?

Classification:

Regression:

Kexin Shi & Yifei Liu – Robotics and Perception Group

16

RQ3: Is mlp part really useful?

Kexin Shi & Yifei Liu – Robotics and Perception Group

17

RQ3: Is mlp part really useful?

Classification:

Regression:

RQ3: Is mlp part useful?

Classification:

Regression:

Observation: Point-features generated by MLP is useless, even some negative effects. MLP structure can not extract useful information from events data.

Kexin Shi & Yifei Liu – Robotics and Perception Group

19

RQ3: Is mlp part useful?

Kexin Shi & Yifei Liu – Robotics and Perception Group

20

RQ4: Is devoxelization necessary?

​

Kexin Shi & Yifei Liu – Robotics and Perception Group

21

RQ4: Is devoxelization necessary?

​

Original

Remove Voxelization

Remove Voxelization and Devoxelization

Classification:

Kexin Shi & Yifei Liu – Robotics and Perception Group

22

RQ4: Is devoxelization necessary?

​

Regression:

W/o devoxelization

With devoxelization

Kexin Shi & Yifei Liu – Robotics and Perception Group

23

RQ4: Is devoxelization necessary?

​

Classification:

Regression:

Observation: Passing point-based features between different resolutions performs better than passing voxel-based features.

Kexin Shi & Yifei Liu – Robotics and Perception Group

24

RQ5: Dense vs Sparse?

Kexin Shi & Yifei Liu – Robotics and Perception Group

25

RQ5: Dense vs Sparse?

Kexin Shi & Yifei Liu – Robotics and Perception Group

26

RQ5: Dense vs Sparse?

Classification:

Regression:

Observation: By substituting the 3D convolution by sparse convolution, speed increases, but performance drops.

Kexin Shi & Yifei Liu – Robotics and Perception Group

27

Research Questions

Kexin Shi & Yifei Liu – Robotics and Perception Group

28

Conclusion

Outlook

1. Point-based models are not suitable for events

2. Point-Voxel: good performance, but slow

3. 3D voxel-based: either slower or worse performance

4. 2D voxel-based: a reasonable trade off between speed and performance

1. Finding a better way of downsample is necessary.

2. Use sparse convolution to increase speed.

​

Kexin Shi & Yifei Liu – Robotics and Perception Group

29

Take-Away Message

• RQ1: How to downsample effectively?
• The less points, the faster speed, but also the lower performance.
• With similar number of points, data-dependent downsample performs better than random sample.
• RQ2: What’s the performances of different methods?
• Point-based method is inaccurate and slow.
• Point-Voxel method has the best performance with sacrificing some speed.
• Voxel-based method (3D method) is either bad at performance or speed. The most time-consuming part is 3D convolution.
• Frame-based method (2D method) has a good trade-off between performance and speed.

Kexin Shi & Yifei Liu – Robotics and Perception Group

30

Take-Away Message

• RQ3: Is MLP part useful?
• No. MLP can only bring noises even we increase the number of layers.
• Point features generated by MLP are not useful for events data.
• RQ4: Is devoxelization necessary?
• Yes. Compared with passing voxel-based features between different resolutions, passing point-based features has a distinct improvement in performance.
• RQ5: Dense convolution or Sparse convolution?
• Sparse convolution is faster, but with worse performance (within the pvcnn structure).

​

Kexin Shi & Yifei Liu – Robotics and Perception Group

31

Thanks!

Code:

https://github.com/uzh-rpg/master_project_kexin_shi_yifei_liu

Kexin Shi & Yifei Liu – Robotics and Perception Group

32

Backup Slide

Kexin Shi & Yifei Liu – Robotics and Perception Group

33

RQ1: How to downsample effectively?

Classification:

Kexin Shi & Yifei Liu – Robotics and Perception Group

34

RQ1: How to downsample effectively?

Regression:

Observation:

1) With similar number of points, data-dependent downsample performs better than random sample.

2) The less points, the faster speed, but also the lower performance.

Kexin Shi & Yifei Liu – Robotics and Perception Group

35

RQ2: Point vs. Voxel vs. Fuse Models

Classification: PVCNN

Kexin Shi & Yifei Liu – Robotics and Perception Group

36

RQ2: Point vs. Voxel vs. Fuse Models

Observation:

1. Pure point-based method is slow and inaccurate.
2. Point-Voxel method gets the best performance in all un-pretrained models.

Classification:

Kexin Shi & Yifei Liu – Robotics and Perception Group

37

RQ2: Point vs. Voxel vs. Fuse Models

Regression: PV-Unet

Kexin Shi & Yifei Liu – Robotics and Perception Group

38

RQ2: Point-based vs Voxel-based vs Point-Voxel

2D Unet:

Kexin Shi & Yifei Liu – Robotics and Perception Group

39

RQ2: Point-based vs Voxel-based vs Point-Voxel

2D Unet Half:

Kexin Shi & Yifei Liu – Robotics and Perception Group

40

RQ2: Point-based vs Voxel-based vs Point-Voxel

3D Unet:

Kexin Shi & Yifei Liu – Robotics and Perception Group

41

RQ2: Performances of different methods?

Regression:

• NPE: 1-pixel-error, the percentage of ground truth pixels with optical flow magnitude error > N. N is either 1, 2 or 3.

​

• EPE: Endpoint error. The average of the L2-Norm of the optical flow error.

​

• AE: Angular error.

Kexin Shi & Yifei Liu – Robotics and Perception Group

42

RQ3: Is mlp part really useful?

Classification:

Kexin Shi & Yifei Liu – Robotics and Perception Group

43

RQ3: Is mlp part useful?

Regression:

Kexin Shi & Yifei Liu – Robotics and Perception Group

44

RQ4: Is devoxelization necessary?

​

Original

Remove Voxelization

Remove Voxelization and Devoxelization

Classification:

Kexin Shi & Yifei Liu – Robotics and Perception Group

45

RQ4: Is devoxelization necessary?

​

With devoxelization:

Without devoxelization:

Regression:

Kexin Shi & Yifei Liu – Robotics and Perception Group

46

RQ4: Is devoxelization necessary?

​

Classification:

Regression:

Observation: Passing point-based features between different resolutions performs better than passing voxel-based features.

Kexin Shi & Yifei Liu – Robotics and Perception Group

47

RQ5: Dense vs Sparse

Classification:

Regression:

Observation: By substituting the 3D convolution by sparse convolution, speed increases, but performance drops.

Kexin Shi & Yifei Liu – Robotics and Perception Group

48

Questions that could be potentially asked

1. How does pointnet++ work? (how does it group points, how does it combine features between groups?)
2. explain why pointnet++ is so slow when faced with so many points. (k-nearest-neighbor search)

Kexin Shi & Yifei Liu – Robotics and Perception Group

49