Scene Transformer: A unified architecture for predicting multiple agent trajectories (2021-06-15T00:00:00.000000Z)

TL;DR

A model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents is formulated, and it is shown that the model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction.

Abstract

Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g. vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions can make it challenging to represent the future interaction possibilities between different agents, leading to sub-optimal planning. In this work, we formulate a model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model, enabling one to invoke a single model to predict agent behavior in many ways, such as potentially conditioned on the goal or full future trajectory of the autonomous vehicle or the behavior of other agents in the environment. Our model architecture employs attention to combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction, and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy, we show that our model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction.

Authors

References76 items

MotionCNN: A Strong Baseline for Motion Prediction in Autonomous Driving

DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets

Rethinking Trajectory Forecasting Evaluation

HOME: Heatmap Output for future Motion Estimation

Toward Causal Representation Learning

Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

Identifying Driver Interactions via Conditional Behavior Prediction

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Multimodal Motion Prediction with Stacked Transformers

TPCN: Temporal Point Cloud Networks for Motion Forecasting

LambdaNetworks: Modeling Long-Range Interactions Without Attention

Bottleneck Transformers for Visual Recognition

Deep Structured Reactive Planning

TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors

Trajformer: Trajectory Prediction with Local Self-Attentive Contexts for Autonomous Driving

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

PRANK: motion Prediction based on RANKing

What-If Motion Prediction for Autonomous Driving

TNT: Target-driveN Trajectory Prediction

SoDA: Multi-Object Tracking with Soft Data Association

End-to-end Contextual Perception and Prediction with Interaction Transformer

Learning Lane Graph Representations for Motion Forecasting

Implicit Latent Variable Model for Scene-Consistent Motion Forecasting

MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction

VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation

Transformer Networks for Trajectory Forecasting

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving

Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data

Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Plans

Multiple Futures Prediction

Learning an Uncertainty-Aware Object Detector for Autonomous Driving

SpAGNN: Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data

MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction

INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps

Axial Attention in Multidimensional Transformers

Invariant Risk Minimization

Stand-Alone Self-Attention in Vision Models

Argoverse: 3D Tracking and Forecasting With Rich Maps

Rules of the Road: Predicting Driving Behavior With a Convolutional Model of Semantic Interactions

End-To-End Interpretable Neural Motion Planner

Diverse Generation for Multi-Agent Sports Games

PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings

Attention Augmented Convolutional Networks

Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

nuScenes: A Multimodal Dataset for Autonomous Driving

TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions

Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks

Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

Attention is All you Need

In-datacenter performance analysis of a tensor processing unit

DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Social LSTM: Human Trajectory Prediction in Crowded Spaces

Rethinking the Inception Architecture for Computer Vision

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Trajectory planning for Bertha — A local, continuous method

Understanding the difficulty of training deep feedforward neural networks

You'll never walk alone: Modeling social behavior for multi-target tracking

Junior: The Stanford entry in the Urban Challenge

A Personal Account of the Development of Stanley, the Robot That Won the DARPA Grand Challenge

Multi-modal interactive agent trajectory prediction using heterogeneous edge-enhanced graph attention network

Multi-head attention for joint multi-modal vehicle motion forecasting

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Lyft level 5 perception dataset 2020

TensorFlow: Large-scale machine learning on heterogeneous systems

Figure 8: Pseudo-code in TensorFlow

The DARPA urban challenge: autonomous vehicles in city traffic, volume

Waymo open dataset challenge 2021 winners

Recoat: A deep learning framework with attention mechanism for multi-modal motion prediction