Awesome Interaction-aware Behavior and Trajectory Prediction

This is a checklist of state-of-the-art research materials (datasets, blogs, papers and public codes) related to trajectory prediction. Wish it could be helpful for both academia and industry. (Still updating)

Maintainers: Jiachen Li, Hengbo Ma, Jinning Li (University of California, Berkeley)

Emails: {jiachen_li, hengbo_ma, jinning_li}@berkeley.edu

Please feel free to pull request to add new resources or send emails to us for questions, discussion and collaborations.

Note: Here is also a collection of materials for reinforcement learning, decision making and motion planning.

Please consider citing our work if you found this repo useful:

@inproceedings{li2020evolvegraph,
  title={EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning},
  author={Li, Jiachen and Yang, Fan and Tomizuka, Masayoshi and Choi, Chiho},
  booktitle={2020 Advances in Neural Information Processing Systems (NeurIPS)},
  year={2020}
}

@inproceedings{li2019conditional,
  title={Conditional Generative Neural System for Probabilistic Trajectory Prediction},
  author={Li, Jiachen and Ma, Hengbo and Tomizuka, Masayoshi},
  booktitle={2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={6150--6156},
  year={2019},
  organization={IEEE}
}

Table of Contents

• Datasets
  • Vehicles and Traffic
  • Pedestrians
  • Sport Players
• Literature and Codes
  • Survey Papers
  • Physics Systems with Interaction
  • Intelligent Vehicles & Traffic
  • Pedestrians
  • Mobile Robots
  • Sport Players
  • Benchmark and Evaluation Metrics
  • Others

Datasets

Vehicles and Traffic

Pedestrians

Sport Players

Literature and Codes

Survey Papers

• Modeling and Prediction of Human Driver Behavior: A Survey, 2020. [paper]
• Human Motion Trajectory Prediction: A Survey, 2019. [paper]
• A literature review on the prediction of pedestrian behavior in urban scenarios, ITSC 2018. [paper]
• Survey on Vision-Based Path Prediction. [paper]
• Autonomous vehicles that interact with pedestrians: A survey of theory and practice. [paper]
• Trajectory data mining: an overview. [paper]
• A survey on motion prediction and risk assessment for intelligent vehicles. [paper]

Physics Systems with Interaction

• EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning, NeurIPS 2020. [paper]
• Interaction Templates for Multi-Robot Systems, IROS 2019. [paper]
• Factorised Neural Relational Inference for Multi-Interaction Systems, ICML workshop 2019. [paper] [code]
• Physics-as-Inverse-Graphics: Joint Unsupervised Learning of Objects and Physics from Video, 2019. [paper]
• Neural Relational Inference for Interacting Systems, ICML 2018. [paper] [code]
• Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks, UAI 2018. [paper]
• Relational inductive biases, deep learning, and graph networks, 2018. [paper]
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions, ICLR 2018. [paper]
• Graph networks as learnable physics engines for inference and control, ICML 2018. [paper]
• Flexible Neural Representation for Physics Prediction, 2018. [paper]
• A simple neural network module for relational reasoning, 2017. [paper]
• VAIN: Attentional Multi-agent Predictive Modeling, NIPS 2017. [paper]
• Visual Interaction Networks, 2017. [paper]
• A Compositional Object-Based Approach to Learning Physical Dynamics, ICLR 2017. [paper]
• Interaction Networks for Learning about Objects, Relations and Physics, 2016. [paper][code]

Intelligent Vehicles & Traffic

• EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning, NeurIPS 2020. [paper]
• V2VNet- Vehicle-to-Vehicle Communication for Joint Perception and Prediction, ECCV 2020. [paper]
• SMART- Simultaneous Multi-Agent Recurrent Trajectory Prediction, ECCV 2020. [paper]
• SimAug- Learning Robust Representations from Simulation for Trajectory Prediction, ECCV 2020. [paper]
• Learning Lane Graph Representations for Motion Forecasting, ECCV 2020. [paper]
• Implicit Latent Variable Model for Scene-Consistent Motion Forecasting, ECCV 2020. [paper]
• Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding, ECCV 2020. [paper]
• Kernel Trajectory Maps for Multi-Modal Probabilistic Motion Prediction, CoRL 2019. [paper] [code]
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network, 2020. [paper]
• Forecasting Trajectory and Behavior of Road-Agents Using Spectral Clustering in Graph-LSTMs, 2019. [paper] [code]
• Joint Prediction for Kinematic Trajectories in Vehicle-Pedestrian-Mixed Scenes, ICCV 2019. [paper]
• Analyzing the Variety Loss in the Context of Probabilistic Trajectory Prediction, ICCV 2019. [paper]
• Looking to Relations for Future Trajectory Forecast, ICCV 2019. [paper]
• Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles, IROS 2019. [paper]
• Sharing Is Caring: Socially-Compliant Autonomous Intersection Negotiation, IROS 2019. [paper]
• INFER: INtermediate Representations for FuturE PRediction, IROS 2019. [paper] [code]
• Deep Predictive Autonomous Driving Using Multi-Agent Joint Trajectory Prediction and Traffic Rules, IROS 2019. [paper]
• NeuroTrajectory: A Neuroevolutionary Approach to Local State Trajectory Learning for Autonomous Vehicles, IROS 2019. [paper]
• Urban Street Trajectory Prediction with Multi-Class LSTM Networks, IROS 2019. [N/A]
• Spatiotemporal Learning of Directional Uncertainty in Urban Environments with Kernel Recurrent Mixture Density Networks, IROS 2019. [paper]
• Conditional generative neural system for probabilistic trajectory prediction, IROS 2019. [paper]
• Interaction-aware multi-agent tracking and probabilistic behavior prediction via adversarial learning, ICRA 2019. [paper]
• Generic Tracking and Probabilistic Prediction Framework and Its Application in Autonomous Driving, IEEE Trans. Intell. Transport. Systems, 2019. [paper]
• Coordination and trajectory prediction for vehicle interactions via bayesian generative modeling, IV 2019. [paper]
• Wasserstein generative learning with kinematic constraints for probabilistic interactive driving behavior prediction, IV 2019. [paper]
• GRIP: Graph-based Interaction-aware Trajectory Prediction, ITSC 2019. [paper]
• AGen: Adaptable Generative Prediction Networks for Autonomous Driving, IV 2019. [paper]
• TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions, CVPR 2019. [paper], [code]
• Multi-Step Prediction of Occupancy Grid Maps with Recurrent Neural Networks, CVPR 2019. [paper]
• Argoverse: 3D Tracking and Forecasting With Rich Maps, CVPR 2019 [paper]
• Robust Aleatoric Modeling for Future Vehicle Localization, CVPR 2019. [paper]
• Pedestrian occupancy prediction for autonomous vehicles, IRC 2019. [paper]
• Context-based path prediction for targets with switching dynamics, 2019.[paper]
• Deep Imitative Models for Flexible Inference, Planning, and Control, 2019. [paper]
• Infer: Intermediate representations for future prediction, 2019. [paper][code]
• Multi-agent tensor fusion for contextual trajectory prediction, 2019. [paper]
• Context-Aware Pedestrian Motion Prediction In Urban Intersections, 2018. [paper]
• Generic probabilistic interactive situation recognition and prediction: From virtual to real, ITSC 2018. [paper]
• Generic vehicle tracking framework capable of handling occlusions based on modified mixture particle filter, IV 2018. [paper]
• Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs, 2018. [paper]
• Sequence-to-sequence prediction of vehicle trajectory via lstm encoder-decoder architecture, 2018. [paper]
• R2P2: A ReparameteRized Pushforward Policy for diverse, precise generative path forecasting, ECCV 2018. [paper]
• Predicting trajectories of vehicles using large-scale motion priors, IV 2018. [paper]
• Vehicle trajectory prediction by integrating physics-and maneuver based approaches using interactive multiple models, 2018. [paper]
• Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks, 2018. [paper]
• Generative multi-agent behavioral cloning, 2018. [paper]
• Deep Sequence Learning with Auxiliary Information for Traffic Prediction, KDD 2018. [paper], [code]
• Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction, 2017. [paper]
• Probabilistic long-term prediction for autonomous vehicles, IV 2017. [paper]
• Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network, ITSC 2017. [paper]
• Desire: Distant future prediction in dynamic scenes with interacting agents, CVPR 2017. [paper][code]
• Imitating driver behavior with generative adversarial networks, 2017. [paper][code]
• Infogail: Interpretable imitation learning from visual demonstrations, 2017. [paper][code]
• Long-term planning by short-term prediction, 2017. [paper]
• Long-term path prediction in urban scenarios using circular distributions, 2017. [paper]
• Deep learning driven visual path prediction from a single image, 2016. [paper]
• Understanding interactions between traffic participants based on learned behaviors, 2016. [paper]
• Visual path prediction in complex scenes with crowded moving objects, CVPR 2016. [paper]
• A game-theoretic approach to replanning-aware interactive scene prediction and planning, 2016. [paper]
• Intention-aware online pomdp planning for autonomous driving in a crowd, ICRA 2015. [paper]
• Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression, 2014. [paper]
• Patch to the future: Unsupervised visual prediction, CVPR 2014. [paper]
• Mobile agent trajectory prediction using bayesian nonparametric reachability trees, 2011. [paper]

Pedestrians

• Learning Structured Representations of Spatial and Interactive Dynamics for Trajectory Prediction in Crowded Scenes, IEEE Robotics and Automation Letters 2021 [paper], [code]
• Social NCE: Contrastive Learning of Socially-aware Motion Representations. [paper], [code]
• EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning, NeurIPS 2020. [paper]
• Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction, ECCV 2020. [paper]
• It is not the Journey but the Destination- Endpoint Conditioned Trajectory Prediction, ECCV 2020. [paper]
• How Can I See My Future? FvTraj: Using First-person View for Pedestrian Trajectory Prediction, ECCV 2020. [paper]
• Dynamic and Static Context-aware LSTM for Multi-agent Motion Prediction, ECCV 2020. [paper]
• Human Trajectory Forecasting in Crowds: A Deep Learning Perspective, 2020. [paper], [code]
• SimAug: Learning Robust Representations from 3D Simulation for Pedestrian Trajectory Prediction in Unseen Cameras, ECCV 2020. [paper], [code]
• DAG-Net: Double Attentive Graph Neural Network for Trajectory Forecasting, ICPR 2020. [paper] [code]
• Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision, WACV 2020. [paper]
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network, 2020. [paper]
• Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction, CVPR 2020. [Paper], [Code]
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction, CVPR 2020. [paper], [code/dataset]
• Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision, WACV 2020. [paper]
• The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs, ICCV 2019. [paper] [code]
• STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction, ICCV 2019. [paper] [code]
• Instance-Level Future Motion Estimation in a Single Image Based on Ordinal Regression, ICCV 2019. [paper]
• Social and Scene-Aware Trajectory Prediction in Crowded Spaces, ICCV workshop 2019. [paper] [code]
• Stochastic Sampling Simulation for Pedestrian Trajectory Prediction, IROS 2019. [paper]
• Long-Term Prediction of Motion Trajectories Using Path Homology Clusters, IROS 2019. [paper]
• StarNet: Pedestrian Trajectory Prediction Using Deep Neural Network in Star Topology, IROS 2019. [paper]
• Learning Generative Socially-Aware Models of Pedestrian Motion, IROS 2019. [paper]
• Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model, CVWW 2019. [paper]
• Path predictions using object attributes and semantic environment, VISIGRAPP 2019. [paper]
• Probabilistic Path Planning using Obstacle Trajectory Prediction, CoDS-COMAD 2019. [paper]
• Human Trajectory Prediction using Adversarial Loss, hEART 2019. [paper], [code]
• Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs, CVPR 2019. [Precognition Workshop], [paper], [code]
• Peeking into the Future: Predicting Future Person Activities and Locations in Videos, CVPR 2019. [paper], [code]
• Learning to Infer Relations for Future Trajectory Forecast, CVPR 2019. [paper]
• TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions, CVPR 2019. [paper]
• Which Way Are You Going? Imitative Decision Learning for Path Forecasting in Dynamic Scenes, CVPR 2019. [paper]
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction, CVPR 2019. [paper][code]
• Sophie: An attentive gan for predicting paths compliant to social and physical constraints, CVPR 2019. [paper][code]
• Pedestrian path, pose, and intention prediction through gaussian process dynamical models and pedestrian activity recognition, 2019. [paper]
• Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing, 2019. [paper]
• The simpler the better: Constant velocity for pedestrian motion prediction, 2019. [paper]
• Pedestrian trajectory prediction in extremely crowded scenarios, 2019. [paper]
• Srlstm: State refinement for lstm towards pedestrian trajectory prediction, 2019. [paper]
• Location-velocity attention for pedestrian trajectory prediction, WACV 2019. [paper]
• Pedestrian Trajectory Prediction in Extremely Crowded Scenarios, Sensors, 2019. [paper]
• A data-driven model for interaction-aware pedestrian motion prediction in object cluttered environments, ICRA 2018. [paper]
• Move, Attend and Predict: An attention-based neural model for people’s movement prediction, Pattern Recognition Letters 2018. [paper]
• GD-GAN: Generative Adversarial Networks for Trajectory Prediction and Group Detection in Crowds, ACCV 2018, [paper], [demo]
• Ss-lstm: a hierarchical lstm model for pedestrian trajectory prediction, WACV 2018. [paper]
• Social Attention: Modeling Attention in Human Crowds, ICRA 2018. [paper][code]
• Pedestrian prediction by planning using deep neural networks, ICRA 2018. [paper]
• Joint long-term prediction of human motion using a planning-based social force approach, ICRA 2018. [paper]
• Human motion prediction under social grouping constraints, IROS 2018. [paper]
• Future Person Localization in First-Person Videos, CVPR 2018. [paper]
• Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks, CVPR 2018. [paper][code]
• Group LSTM: Group Trajectory Prediction in Crowded Scenarios, ECCV 2018. [paper]
• Mx-lstm: mixing tracklets and vislets to jointly forecast trajectories and head poses, CVPR 2018. [paper]
• Intent prediction of pedestrians via motion trajectories using stacked recurrent neural networks, 2018. [paper]
• Transferable pedestrian motion prediction models at intersections, 2018. [paper]
• Probabilistic map-based pedestrian motion prediction taking traffic participants into consideration, 2018. [paper]
• A Computationally Efficient Model for Pedestrian Motion Prediction, ECC 2018. [paper]
• Context-aware trajectory prediction, ICPR 2018. [paper]
• Set-based prediction of pedestrians in urban environments considering formalized traffic rules, ITSC 2018. [paper]
• Building prior knowledge: A markov based pedestrian prediction model using urban environmental data, ICARCV 2018. [paper]
• Depth Information Guided Crowd Counting for Complex Crowd Scenes, 2018. [paper]
• Tracking by Prediction: A Deep Generative Model for Mutli-Person Localisation and Tracking, WACV 2018. [paper]
• “Seeing is Believing”: Pedestrian Trajectory Forecasting Using Visual Frustum of Attention, WACV 2018. [paper]
• Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty, CVPR 2018. [paper], [code+data]
• Encoding Crowd Interaction with Deep Neural Network for Pedestrian Trajectory Prediction, CVPR 2018. [paper], [code]
• Walking Ahead: The Headed Social Force Model, 2017. [paper]
• Real-time certified probabilistic pedestrian forecasting, 2017. [paper]
• A multiple-predictor approach to human motion prediction, ICRA 2017. [paper]
• Forecasting interactive dynamics of pedestrians with fictitious play, CVPR 2017. [paper]
• Forecast the plausible paths in crowd scenes, IJCAI 2017. [paper]
• Bi-prediction: pedestrian trajectory prediction based on bidirectional lstm classification, DICTA 2017. [paper]
• Aggressive, Tense or Shy? Identifying Personality Traits from Crowd Videos, IJCAI 2017. [paper]
• Natural vision based method for predicting pedestrian behaviour in urban environments, ITSC 2017. [paper]
• Human Trajectory Prediction using Spatially aware Deep Attention Models, 2017. [paper]
• Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection, 2017. [paper]
• Forecasting Interactive Dynamics of Pedestrians with Fictitious Play, CVPR 2017. [paper]
• Social LSTM: Human trajectory prediction in crowded spaces, CVPR 2016. [paper][code]
• Comparison and evaluation of pedestrian motion models for vehicle safety systems, ITSC 2016. [paper]
• Age and Group-driven Pedestrian Behaviour: from Observations to Simulations, 2016. [paper]
• Structural-RNN: Deep learning on spatio-temporal graphs, CVPR 2016. [paper][code]
• Intent-aware long-term prediction of pedestrian motion, ICRA 2016. [paper]
• Context-based detection of pedestrian crossing intention for autonomous driving in urban environments, IROS 2016. [paper]
• Novel planning-based algorithms for human motion prediction, ICRA 2016. [paper]
• Learning social etiquette: Human trajectory understanding in crowded scenes, ECCV 2016. [paper][code]
• GLMP-realtime pedestrian path prediction using global and local movement patterns, ICRA 2016. [paper]
• Knowledge transfer for scene-specific motion prediction, ECCV 2016. [paper]
• STF-RNN: Space Time Features-based Recurrent Neural Network for predicting People Next Location, SSCI 2016. [code]
• Goal-directed pedestrian prediction, ICCV 2015. [paper]
• Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations, 2015. [paper]
• Predicting and recognizing human interactions in public spaces, 2015. [paper]
• Learning collective crowd behaviors with dynamic pedestrian-agents, 2015. [paper]
• Modeling spatial-temporal dynamics of human movements for predicting future trajectories, AAAI 2015. [paper]
• Unsupervised robot learning to predict person motion, ICRA 2015. [paper]
• A controlled interactive multiple model filter for combined pedestrian intention recognition and path prediction, ITSC 2015. [paper]
• Real-Time Predictive Modeling and Robust Avoidance of Pedestrians with Uncertain, Changing Intentions, 2014. [paper]
• Behavior estimation for a complete framework for human motion prediction in crowded environments, ICRA 2014. [paper]
• Pedestrian’s trajectory forecast in public traffic with artificial neural network, ICPR 2014. [paper]
• Will the pedestrian cross? A study on pedestrian path prediction, 2014. [paper]
• BRVO: Predicting pedestrian trajectories using velocity-space reasoning, 2014. [paper]
• Context-based pedestrian path prediction, ECCV 2014. [paper]
• Pedestrian path prediction using body language traits, 2014. [paper]
• Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression, 2014. [paper]
• Learning intentions for improved human motion prediction, 2013. [paper]

Mobile Robots

• Social NCE: Contrastive Learning of Socially-aware Motion Representations. [paper], [code]
• Multimodal probabilistic model-based planning for human-robot interaction, ICRA 2018. [paper][code]
• Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning, ICRA 2017. [paper]
• Augmented dictionary learning for motion prediction, ICRA 2016. [paper]
• Predicting future agent motions for dynamic environments, ICMLA 2016. [paper]
• Bayesian intention inference for trajectory prediction with an unknown goal destination, IROS 2015. [paper]
• Learning to predict trajectories of cooperatively navigating agents, ICRA 2014. [paper]

Sport Players

• EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning, NeurIPS 2020. [paper]
• Imitative Non-Autoregressive Modeling for Trajectory Forecasting and Imputation, CVPR 2020. [paper]
• DAG-Net: Double Attentive Graph Neural Network for Trajectory Forecasting, ICPR 2020. [paper] [code]
• Diverse Generation for Multi-Agent Sports Games, CVPR 2019. [paper]
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations, ICLR 2019. [paper]
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision, ICLR 2019. [paper]
• Generative Multi-Agent Behavioral Cloning, ICML 2018. [paper]
• Where Will They Go? Predicting Fine-Grained Adversarial Multi-Agent Motion using Conditional Variational Autoencoders, ECCV 2018. [paper]
• Coordinated Multi-Agent Imitation Learning, ICML 2017. [paper]
• Generating long-term trajectories using deep hierarchical networks, 2017. [paper]
• Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction, ICDM 2014. [paper]
• Generative Modeling of Multimodal Multi-Human Behavior, 2018. [paper]
• What will Happen Next? Forecasting Player Moves in Sports Videos, ICCV 2017, [paper]

Benchmark and Evaluation Metrics

• OpenTraj: Assessing Prediction Complexity in Human Trajectories Datasets, ACCV 2020. [paper] [code]
• Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction, ECCV 2020. [paper]
• PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction, ICCV 2019. [paper]
• Towards a fatality-aware benchmark of probabilistic reaction prediction in highly interactive driving scenarios, ITSC 2018. [paper]
• How good is my prediction? Finding a similarity measure for trajectory prediction evaluation, ITSC 2017. [paper]
• Trajnet: Towards a benchmark for human trajectory prediction. [website]

Others

• Cyclist trajectory prediction using bidirectional recurrent neural networks, AI 2018. [paper]
• Road infrastructure indicators for trajectory prediction, 2018. [paper]
• Using road topology to improve cyclist path prediction, 2017. [paper]
• Trajectory prediction of cyclists using a physical model and an artificial neural network, 2016. [paper]