RoRD : Rotation Robust Descriptors and Orthographic Views for Local Feature Matching

Udit Singh Parihar

​

Accepted IROS 2021

​

Problem Statement

• Developing local descriptors invariant to high viewpoints
• Use in relative pose estimation
• Front-end pipelines in SLAM system
• Image Retrievals

Introduction

Feature Correspondences

RORD (ours)

SIFT

✔️

x

Contributions

• Proposed deep learnt descriptors which works under extreme viewpoints
• Used self-supervised training to generate huge training data with simple homographic transformations
• Achieves good generalisation across datasets and perform state of art in variety of tasks
• Proposed dataset comprising of images from high change in camera viewpoint
• Dual headed architecture, where one head is rotation invariant and other is illumination invariant

Correspondences from RoRD

• Results from extreme viewpoints
• Works in both Indoor and outdoor

Improvements

• MMA in HPatches Dataset
• Pose estimation on DiverseView Dataset
• Use of rotation robust descriptors in Visual Place Recognition Task

Supervised Training

• Generates correspondences from monocular images by applying SfM
• Data labelling doesn’t require human labelling
• Inherit biases from SfM data pipeline which uses classical methods

SfM Data

Self Supervised Training

• Training on primitive synthetic shapes and deploying on real world scenarios

Homographic Transformations

Orthographic Views

• Intermediate representation of scene as orthographic views
• Orthographic views improves image retrieval, feature matching and planning
• Ability to calculates orthographic views during 6 DoF camera motion

Pipeline

• Orthographic view rectifies the perspectivity
• Planar patches extraction and surface normal calculation

Orthographic view

Pipeline

Ensemble Architecture

• Dual headed architecture for illumination and rotation invariance
• D2Net model is supervised using 3D depth and pose information
• RoRD head is trained using self-supervised rotational homographies

Orthographic View Generation

• Depth information along with desired ROI for 3D scene representation

​

​

• Virtual Camera anti parallel to surface normal

Ortographic to Perspective Matching

• Inverse homography to project correspondences from orthographic view to perspective view

IPM for Autonomous Driving

• Challenge of matching images from front and rear camera
• Discriminative road patches

VPR for Autonomous Driving

• Image pairs with maximum inliers are considered a match

Network Architecture

• VGG-16 common backbone architecture for better efficency
• Last layer is fine-tune for D2-Net and RoRd
• D2-Net head is train using SfM data while RoRD is trained using Homography data
• Feature correspondences are calculated independently for each head
• RANSAC based geometric verification for geometric verification

Loss Function

• Related via rotation homography
• Triplet margin loss, with margin
• p(c) is euclidean distance between descriptors
• Negative distance is calculated via hardest negative

Training Data

• Training data from phototourim and Oxford Robot Car Dataset

Homographic Transformations

​

Results

Datasets and Tasks

• MMA on Hpatches Dataset
• VPR on Oxford RobotCar Dataset
• Pose Estimation Error on DiverseView Dataset

HPatches Dataset

• Extended Hpatches Dataset to scenes having high rotation
• Evaluated for Mean matching accuracy, by using ground truth homography

• D2 Net

• RoRD

Qualitative Results (MMA)

• Comparison on Standard and extended HPatches Dataset

Oxford RobotCar Dataset

• D2-Net with Orthographic View

• RoRD with Orthographic View

VPR Results with Front and Rear Camera

• Testing VPR results on the sequence not seen during training

Video

Recall for VPR

• Obtain twice the recall compared to second best on challenging Oxford RobotCar Dataset

DiverseView Dataset

• Orthographic views in tandem with RoRD gives best correspondences

Pose Estimation Results

Indoor

Video

Pose Estimation Results

Outdoor

Video

Ablation Study for Pose Estimation

• Rotation and Translation error

Opposite View Loop Closures in SLAM

• Global feature matching for VPR and local feature matching for transformations

Transformation Estimation using Rotation Invariant Descriptors

Video Results of Loop Closure in Lab Dataset

Pose Graph Optimization on Lab Dataset

• Extended RTABMAP to work with scenes where robot revisits the scene

Code and Dataset

• Code and Dataset are publicly available
• https://github.com/UditSinghParihar/RoRD

Topological Mapping for Manhattan-like Repetitive Environments

Sai Shubodh Puligilla ^*, Satyajit Tourani ^*, Tushar Vaidya ^*,

Udit Singh Parihar ^*, Ravi Kiran Sarvadevabhatla and K. Madhava Krishna

​

​

*Denotes authors with equal contribution

Accepted to International Conference on Robotics and Automation 2020

​

​

Problem Formulation

1. Loop closure constraints are proposed by MLP and computed using ICP.
2. Manhattan constraints are obtained from orthogonal relations between nodes in Manhattan graph.

Pose Graph Optimization Pipeline

RESULTS

1. Nodes of the pose graph are labelled based on the topological labels by CNN to create a topological graph

1. A highly distorted unoptimized pose graph with labelled topologies is recovered using an intermediate Manhattan Graph.
2. A loop detection event between two Manhattan Nodes in the Manhattan Graph is shown in bold color. This results in both Manhattan and/or Loop constraints.

RECOVERED TRAJECTORIES

Top row shows unoptimized trajectories, middle row shows trajectories recovered using our pipeline and last row shows ground truth trajectories

Improving RTABMAP with Topological Mapping

​

Benchmarking RTABMAP

• Evaluation of RTABMAP on warehouse dataset leads to detection of many False Positive loop closure constraints, due to repeating corridors.
• Incorporation of Topological constraints leads to better ATE.
• Our topological comparator utilizes geometric structure of the topological representation.

References

• RoRD - Rotation Robust Descriptors and Orthographic Views for Local Feature Matching, IEEE IROS 2021
• Early Bird : Loop Closures from Opposing Viewpoints for Perceptually-Aliased Indoor Environments, VISAPP 2020
• Topological Mapping for Manhattan-like Repetitive Environments, IEEE ICRA 2020

​

Thanks