1 of 12

Exploring mmWave Radar and Camera Fusion for High-Resolution�and Long-Range Depth Imaging

Diana Zhang*

Akarsh Prabhakara*

Anthony Rowe

Chao Li

Sirajum Munir

Aswin C.

Sankaranarayanan

Swarun Kumar

2 of 12

3 of 12

150 m

75 m

CAR

4 of 12

150 m

75 m

CAR

5 of 12

150 m

75 m

CAR

Single vantage point high-res long-range depth imaging

6 of 12

Monocular Depth Estimation

​

10s of meters depth error at long ranges

Stand alone lidar

​

Options Today

7 of 12

​

Monocular Camera

Radar

Long-range Depth Estimation Accuracy

Poor

Good

Case for Radar

Cruise Control Radar already used for

100s of meters depth estimation

Angular Resolution

Good

Poor

8 of 12

Exploring mmWave Radar and Camera Fusion for High-Resolution�and Long-Range Depth Imaging

Fusion Dance From Dragon Ball

METAMORAN

Camera

mmWave Radar

9 of 12

Azimuth

Elevation

Strong clutter sources

Wrong detection

Azimuth

Elevation

Challenges

Correct detection

10 of 12

Object

detected

Monocular

Estimate

Image

Segmentation

3D object shape

Extract an

elevation slice

Contour

Synthesized Radar

Contour template

Correlate

Robust Object Detection in Clutter

​

More details in paper

​

1. Clutter suppression�2. Creating a higher res depth image

​

11 of 12

Results

Evaluation showing

- Error of 28 cm at long ranges

- 13x better than naïve radar-camera fusion and 23x better than monocular alone

​

12 of 12

Conclusion

Resources: https://witechlab.com/metamoran

Fuse mmWave radar and camera for single vantage point long-range depth imaging

​

​

System design to help radar leverage information from camera

  • Detect true objects in radar at long ranges
  • Suppress clutter
  • Create high resolution depth images

​

​

Evaluation at long ranges of 100-300 meters

Camera

mmWave Radar