Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction (2021-02-19T00:00:00.000000Z)

TL;DR

Latent Variable Sequential Set Transformers are encoder-decoder architectures that generate scene-consistent multi-agent trajectories that achieve top results on the global nuScenes vehicle motion prediction leaderboard, and produces strong results onThe Argoverse vehicle prediction challenge.

Abstract

Robust multi-agent trajectory prediction is essential for the safe control of robotic systems. A major challenge is to efficiently learn a representation that approximates the true joint distribution of contextual, social, and temporal information to enable planning. We propose Latent Variable Sequential Set Transformers which are encoder-decoder architectures that generate scene-consistent multi-agent trajectories. We refer to these architectures as"AutoBots". The encoder is a stack of interleaved temporal and social multi-head self-attention (MHSA) modules which alternately perform equivariant processing across the temporal and social dimensions. The decoder employs learnable seed parameters in combination with temporal and social MHSA modules allowing it to perform inference over the entire future scene in a single forward pass efficiently. AutoBots can produce either the trajectory of one ego-agent or a distribution over the future trajectories for all agents in the scene. For the single-agent prediction case, our model achieves top results on the global nuScenes vehicle motion prediction leaderboard, and produces strong results on the Argoverse vehicle prediction challenge. In the multi-agent setting, we evaluate on the synthetic partition of TrajNet++ dataset to showcase the model's socially-consistent predictions. We also demonstrate our model on general sequences of sets and provide illustrative experiments modelling the sequential structure of the multiple strokes that make up symbols in the Omniglot data. A distinguishing feature of AutoBots is that all models are trainable on a single desktop GPU (1080 Ti) in under 48h.

Authors

References48 items

GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

Multimodal Motion Prediction with Stacked Transformers

TPCN: Temporal Point Cloud Networks for Motion Forecasting

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption

PRANK: motion Prediction based on RANKing

What-If Motion Prediction for Autonomous Driving

TNT: Target-driveN Trajectory Prediction

End-to-end Contextual Perception and Prediction with Interaction Transformer

Learning Lane Graph Representations for Motion Forecasting

Implicit Latent Variable Model for Scene-Consistent Motion Forecasting

Spatially Aware Multimodal Transformers for TextVQA

Human Trajectory Forecasting in Crowds: A Deep Learning Perspective

Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction

VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation

Multi-Head Attention with Joint Agent-Map Representation for Trajectory Prediction in Autonomous Driving

EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning

Variational Transformers for Diverse Response Generation

Transformer Networks for Trajectory Forecasting

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Trajectron++: Multi-Agent Generative Trajectory Forecasting With Heterogeneous Data for Control

Sound Event Detection of Weakly Labelled Data With CNN-Transformer and Automatic Threshold Optimization

CoverNet: Multimodal Behavior Prediction Using Trajectory Sets

Multiple Futures Prediction

T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion

Argoverse: 3D Tracking and Forecasting With Rich Maps

nuScenes: A Multimodal Dataset for Autonomous Driving

RED: A Simple but Effective Baseline Predictor for the TrajNet Benchmark

Convolutional Social Pooling for Vehicle Trajectory Prediction

Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension

Attention is All you Need

DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

Deep Sets

Layer Normalization

Social LSTM: Human Trajectory Prediction in Crowded Spaces

Human-level concept learning through probabilistic program induction

Generating Sentences from a Continuous Space

Multi-view Convolutional Neural Networks for 3D Shape Recognition

MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

Pattern Recognition and Machine Learning

Scene Transformer: A unified multi-task model for behavior prediction and planning

Multi-head attention for joint multi-modal vehicle motion forecasting

Set transformer: A framework for attention-based permutation-invariant neural networks

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Multi-head Attention Block Decoders

Deep Learning for Predicting Human Strategic Behavior