1 of 14

Izzeddin Teeti

Visual Artificial Intelligence Laboratory, Oxford Brookes University

June 11, 2022

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

2 of 14

Content

  • Intro
  • Problem Formulation.
  • Challenges
  • Datasets
  • Metrices
  • Research Trends
  • Conclusion

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

2

June 11, 2022

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

3 of 14

Introduction

  • AVs industry worth USD 1.45 billion in 2020[1]
  • Develped by research labs, companies, startups
  • Saftey is a concern
  • Prediction is a saftey measure
  • Contribuations: 1. Formulation of a generic AV prediction pipeline� 2. A guide for researchers to select the correct ML technique� 3. A comparison of the relevant datasets� 4. A summary of the latest methodological trends and future directions

​

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

3

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

[1] Autonomous Cars Market Size, Share, Growth & Forecast [2028]. (2021). Fortune Business Insights. https://www.fortunebusinessinsights.com/industry-reports/autonomous-cars-market-100141

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

4 of 14

Problem Formulation

  •  

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

4

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

[1] Girase, H., Gang, H., Malla, S., Li, J., Kanehara, A., Mangalam, K., & Choi, C. (2021). LOKI: Long Term and Key Intentions for Trajectory Prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision, 9803–9812.

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

5 of 14

Challenges

  1. Dynamic
  2. Multi-agent
  3. Stochastic
  4. Partially observable
  5. Real-time requirement

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

5

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

6 of 14

Datasets

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

6

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

7 of 14

Evaluation Metrices

  •  

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

7

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

8 of 14

Research Trends

1. Ego-Motion and Dynamics:�[Cai et al., 2020; Neumann and Vedaldi, 2021;Salzmann et al., 2020]

​

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

8

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

9 of 14

Research Trends

2. Ego-Camera vs Bird’s Eye View:�

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

9

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

[Rasouli et al., 2019; Rasouli et al., 2021; Neumann and Vedaldi, 2021; Bhattacharyya et al., 2021]

[Salzmann et al., 2020; Varadarajan et al., 2021]

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

10 of 14

Research Trends

3. Temporal Encoding:

  • 3D-CNN [Fang et al., 2021; Kotseruba et al., 2021]
  • HMM [Cai et al., 2020]
  • Transformers [Yuan et al., 2021]
  • RNNs
    • RNN [Girase et al., 2021]
    • LSTM [Bhattacharyya et al., 2021; Fang et al., 2021; Kosaraju et al., 2019; Rasouli et al., 2021; Salzmann et al., 2020; Weng et al., 2021;Yuan et al., 2021]
    • GRU [Gilleset al., 2021]

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

10

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

11 of 14

Research Trends

4. Social Encoding:

  • GNN
    • Fully connected [Gilles et al., 2021; Kosaraju et al., 2019]
    • Sparse
      • Bidirectional [Fang et al., 2021;Girase et al., 2021; Salzmann et al., 2020; Weng et al., 2021]
      • Unidirectional [Kosaraju et al., 2019; Yuan et al., 2021]
    • Heterogeneous [Girase et al., 2021]
  • Semantic Segmentation with Attention [Rasouli et al., 2021; Mangalam et al., 2021]
  • Social pooling with Attention [Pang et al., 2021]
  • Conditioned latent space [Bhattacharyya et al., 2021]
  • Masked Transformer [Yuan et al., 2021]

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

11

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

12 of 14

Research Trends

5. Goal-Conditioning:

  • Inspired from neuroscience.
  • Epistemic and aleatoric
  • [Girase et al., 2021;Mangalam et al., 2021; Pang et al., 2021;Gilleset al., 2021]
  • Achived SOTA

​

​

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

12

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

13 of 14

Research Trends

6. Multimodality:

  • Uncertainity
  • CVAEs [Girase et al., 2021; Weng et al., 2021;Yuan et al., 2021]
  • GAN [Kosaraju et al., 2019]

​

​

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

13

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

14 of 14

Conclusion

  • ‘real-time’ challenge is still to be addressed.
  • the ideal loss function for generative models is still to be identified
  • Explore multimodal intention prediction
  • self-supervised learning.

​

​

THE VISUAL ARTIFICIAL INTELLIGENCE LABORATORY, IZZEDDIN TEETI

14

June 11, 2022

Intro

Problem

Challenges

Datasets

Metrices

Research Trends

Conclusion

​

iteeti@brookes.ac.uk

Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey