Behavior Prediction

This repository is a paper digest of multi-agent behavior (motion / trajectory) prediction (forecasting / generation), especially in driving scenes. Papers are listed in alphabetical order of the first character. All links to the materials are freely accessible.

:star2:Recommendation

Helpful Learning Resource:thumbsup::thumbsup::thumbsup:

• (Survey) Social Interactions for Autonomous Driving: A Review and Perspectives [paper]
• (Lecture) Pedestrian Trajectory Prediction [video]
• (Presentation) Trajectory Forecasting in the Modern Robotic Autonomy Stack [video]
• (Keynote) The 7 Foundational Principles behind Autonomous Mobility [video], Behavior Models for Autonomous Driving [video]
• (Workshop) CVPR 2023 Workshop on Autonomous Driving [video], CVPR 2022 Workshop on Autonomous Driving [video], CVPR 2021 Workshop on Autonomous Driving [video]

:bookmark:Benchmarks

ICCV 2021

• WOSAC (The Waymo Open Sim Agents Challenge) [paper] [code] [challenge]
  • Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nick Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, Brandyn White, Dragomir Anguelov
• WOMD (Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset) [paper] [code] [challenge]
  • Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, Dragomir Anguelov

:bookmark:Approaches

Selected Preprint

• CU-aware (Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting) [paper] [code] [demo]
  • Bohan Tang, Yiqi Zhong, Chenxin Xu, Wei-Tao Wu, Ulrich Neumann, Yanfeng Wang, Ya Zhang, Siheng Chen
• DCMS (DCMS: Motion Forecasting with Dual Consistency and Multi-Pseudo-Target Supervision) [paper] [code] [demo]
  • Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tengfei Wang, Tongyi Cao, Qifeng Chen
• DyGroupNet (Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoning) [paper] [code] [demo]
  • Chenxin Xu, Yuxi Wei, Bohan Tang, Sheng Yin, Ya Zhang, Siheng Chen
• HDGT (HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding) [paper] [code] [demo]
  • Xiaosong Jia, Penghao Wu, Li Chen, Hongyang Li, Yu Liu, Junchi Yan
• MTR++ (MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying) [paper] [code] [demo]
  • Shaoshuai Shi, Li Jiang, Dengxin Dai, Bernt Schiele
• QCNeXt (QCNeXt: A Next-Generation Framework For Joint Multi-Agent Trajectory Prediction) [paper] [code] [demo]
  • Zikang Zhou, Zihao Wen, Jianping Wang, Yung-Hui Li, Yu-Kai Huang

CVPR 2023

• EqMotion (EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning) [paper] [code] [demo]
  • Chenxin Xu, Robby T. Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, Yanfeng Wang
• FJMP (FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs) [paper] [code] [demo]
  • Luke Rowe, Martin Ethier, Eli-Henry Dykhne, Krzysztof Czarnecki
• MotionDiffuser (MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion) [paper] [code] [demo]
  • Chiyu Max Jiang, Andre Cornman, Cheolho Park, Ben Sapp, Yin Zhou, Dragomir Anguelov
• QCNet (Query-Centric Trajectory Prediction) [paper] [code] [demo]
  • Zikang Zhou, Jianping Wang, Yung-Hui Li, Yu-Kai Huang

CoRL 2023

• JFP (JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving) [paper&review] [code] [demo]
  • Wenjie Luo, Cheolho Park, Andre Cornman, Benjamin Sapp, Dragomir Anguelov

CVPR 2022

• GroupNet (GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning) [paper] [code] [demo]
  • Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, Siheng Chen
• HiVT (HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction) [paper] [code] [demo]
  • Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, Kejie Lu
• M2I (M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction) [paper] [code] [demo]
  • Qiao Sun, Xin Huang, Junru Gu, Brian C. Williams, Hang Zhao

NeurIPS 2022

• IMMA (Interaction Modeling with Multiplex Attention) [paper&review] [code] [demo]
  • Fan-Yun Sun, Isaac Kauvar, Ruohan Zhang, Jiachen Li, Mykel Kochenderfer, Jiajun Wu, Nick Haber
• MTR (Motion Transformer with Global Intention Localization and Local Movement Refinement) [paper&review] [code] [demo]
  • Shaoshuai Shi, Li Jiang, Dengxin Dai, Bernt Schiele

ICLR 2022

• Scene Transformer (Scene Transformer: A Unified Architecture for Predicting Multiple Agent Trajectories) [paper&review] [code] [demo]
  • Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, Jonathon Shlens
• THOMAS (THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling) [paper&review] [code] [demo]
  • Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, Fabien Moutarde

ICRA 2022

• MultiPath++ (MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction) [paper] [code] [demo]
  • Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, Benjamin Sapp
• PredictionNet (PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation) [paper] [code] [demo]
  • Alexey Kamenev, Lirui Wang, Ollin Boer Bohan, Ishwar Kulkarni, Bilal Kartal, Artem Molchanov, Stan Birchfield, David Nistér, Nikolai Smolyanskiy

CVPR 2021

• TPCN (TPCN: Temporal Point Cloud Networks for Motion Forecasting) [paper] [code] [demo]
  • Maosheng Ye, Tongyi Cao, Qifeng Chen
• TrafficSim (TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors) [paper] [code] [demo]
  • Simon Suo, Sebastian Regalado, Sergio Casas, Raquel Urtasun

NeurIPS 2021

• CU-based (Collaborative Uncertainty in Multi-Agent Trajectory Forecasting) [paper&review] [code] [demo]
  • Bohan Tang, Yiqi Zhong, Ulrich Neumann, Gang Wang, Ya Zhang, Siheng Chen

ICCV 2021

• AgentFormer (AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting) [paper] [code] [demo]
  • Ye Yuan, Xinshuo Weng, Yanglan Ou, Kris Kitani
• DenseTNT (DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets) [paper] [code] [demo]
  • Junru Gu, Chen Sun, Hang Zhao

CoRL 2021

• CEAV (Multi-Agent Trajectory Prediction by Combining Egocentric and Allocentric Views) [paper&review] [code] [demo]
  • Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, Wei Zhan

WACV 2021

• GraphTCN (GraphTCN: Spatio-Temporal Interaction Modeling for Human Trajectory Prediction) [paper] [code] [demo]
  • Chengxin Wang, Shaofeng Cai, Gary Tan

CVPR 2020

• RSBG (Recursive Social Behavior Graph for Trajectory Prediction) [paper] [code] [demo]
  • Jianhua Sun, Qinhong Jiang, Cewu Lu
• VectorNet (VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation) [paper] [code] [demo]
  • Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, Cordelia Schmid

NeurIPS 2020

• EvolveGraph (EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning) [paper&review] [code] [demo]
  • Jiachen Li, Fan Yang, Masayoshi Tomizuka, Chiho Choi

ECCV 2020

• DSDNet (DSDNet: Deep Structured Self-Driving Network) [paper] [code] [demo]
  • Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, Raquel Urtasun
• LaneGCN (Learning Lane Graph Representations for Motion Forecasting) [paper] [code] [demo]
  • Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, Raquel Urtasun
• STAR (Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction) [paper] [code] [demo]
  • Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, Shuai Yi
• Trajectron++ (Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data) [paper] [code] [demo]
  • Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, Marco Pavone

CoRL 2020

• DROGON (DROGON: A Trajectory Prediction Model based on Intention-Conditioned Behavior Reasoning) [paper] [code] [demo]
  • Chiho Choi, Srikanth Malla, Abhishek Patil, Joon Hee Choi
• MATS (MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control) [paper] [code] [demo]
  • Boris Ivanovic, Amine Elhafsi, Guy Rosman, Adrien Gaidon, Marco Pavone
• TNT (TNT: Target-Driven Trajectory Prediction) [paper] [code] [demo]
  • Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, Dragomir Anguelov

ICCV 2019

• Trajectron (The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs) [paper] [code] [demo]
  • Boris Ivanovic, Marco Pavone

CoRL 2019

• MultiPath (MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction) [paper] [code] [demo]
  • Yuning Chai, Benjamin Sapp, Mayank Bansal, Dragomir Anguelov