Visual Odometry with Deep Learning

2022.8.19

Introduction

Geometric Method

Deep Learning Method

Absolute Pose Generate

In the VO tasks, every image in the dataset corresponds to 3×4 [R | t], R denotes 3×3 rotation matrix, which represents the rotation matrix of the left camera coordinate system relative to the first frame of the scene. t denotes 3×1 translation matrix, which denotes the position of the left camera coordinate system in the first frame.

1. Transformation matrix:

Pose of first frame

Rotation matrix in 3D space:

Absolute Pose Generate

In result, absolute pose can be denoted as 6-D vector:

Translation Vector

Relative Pose Generate

Relative Pose Schema

2. Euler angles:

Relative pose of two adjacent frames can be denoted as 6-D vector:

displacement vector

Network Architectures

1. CNN Layers:

Use FlowNetSimple pretrained model to learn the geometric features between two adjacent frames.

2. LSTM Layers:

Let the network learn the relationship between multiple successive poses. Because sometimes the difference between adjacent poses is small.

Cost Function

KITTI VO Benchmark

Autonomous Driving Platform

KITTI Odometry sequence 00-10

Trajectories

Ground Truth of sequence 00-10

Trajectories

nuScenes

nuScenes car setup

nuScenes schema

Use inverse of pose matrix to transform data from global coordinate to camera coordinate.

Trajectories

Ground Truth of scenes 0-849

Experiments

KITTI Loss

NuScenes Loss

Experiments

Test on KITTI Sequences

Experiments

Test on KITTI Sequences (Epoch=10)

Experiments

Comparison between K=50 with K=100 (Epoch=10)

Experiments

Trajectories on test sequences 00-10 for KITTI

Experiments

Experiments

Trajectories on test scenes 0-849 for NuScenes

Conclusion

1. The pose calculation is carried out on KITTI VO benchmark and NuScenes dataset in the VO field. The rotation theory is used to solve the absolute pose of a single image in the scene and the relative pose between two adjacent images. Euler angles and rotation matrices are used respectively, which provides data labels for subsequent deep learning training experiments.
2. A DeepVO estimation method is proposed. This method needs to input continuous multiple frames of RGB three channel pictures, and directly output the relative pose between two adjacent pictures end-to-end. The main idea is to use FlowNetSimple network to extract the geometric relationship features of two adjacent pictures in the picture sequence, then input the feature sequence to LSTM network for feature learning on the time series, and finally reduce the dimension of the features to realize the regression of the relative pose between consecutive multi frame pictures.

Drawbacks & Future

1. From the comparison of trajectories between KITTI and nuScenes, we can find that the predicted results on nuScenes dataset are much better than those on KITTI. This is because routes in nuScenes have been divided by 850 scenes and each scene is just a nearly straight line. There are many turnings on KITTI dataset. In result, we can use token to concatenate multiple scenes to form a complete road for training in the future.
2. Design a new network structure. FlowNetSimple has many parameters and the training process is time-consuming. Therefore, we can consider using a reduced version of the network to train the model quickly in the future, such as SqueezeNet and MobileNet. For DeepVO, the GRU with fewer parameters can be used to replace the LSTM, or the attention mechanism can be referenced in the RNN to achieve higher accuracy attitude estimation.