TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022-05-31T00:00:00.000000Z)

TL;DR

This work proposes TransFuser, a mechanism to integrate image and LiDAR representations using self-attention, which outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin.

Abstract

How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g., object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.

Authors

Andreas Geiger

18 papers

Zehao Yu

2 papers

Kashyap Chitta

4 papers

TL;DR

Abstract

Authors

References134 items

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

Cross-view Transformers for real-time Map-view Semantic Segmentation

KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients

M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation

CAT-Det: Contrastively Augmented Transformer for Multimodal 3D Object Detection

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Interactive Multi-scale Fusion of 2D and 3D Features for Multi-object Tracking

Learning from All Vehicles

TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers

FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction

gDNA: Towards Generative Detailed Neural Avatars

A ConvNet for the 2020s

Driver attention prediction based on convolution and transformers

EPNet++: Cascade Bi-Directional Fusion for Multi-Modal 3D Object Detection

Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior

GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving

Translating Images into Maps

SafetyNet: Safe Planning for Real-World Self-Driving Vehicles Using Machine-Learned Policies

NEAT: Neural Attention Fields for End-to-End Autonomous Driving

A Review of 3D Object Detection for Autonomous Driving of Electric Vehicles

Improving 3D Object Detection with Channel-wise Transformer

End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

Scene Transformer: A unified architecture for predicting multiple agent trajectories

Keyframe-Focused Visual Imitation Learning

Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation

Learning to drive from a world on rails

MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

Affordance-based Reinforcement Learning for Urban Driving

Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models

Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving

Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

Fighting Copycat Agents in Behavioral Cloning from Observation Histories

A Software Architecture for Autonomous Vehicles: Team LRM-B Entry in the First CARLA Autonomous Driving Challenge

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

RoIFusion: 3D Object Detection From LiDAR and Vision

Depth Completion via Inductive Fusion of Planar LIDAR and Monocular Camera

CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

End-to-end Contextual Perception and Prediction with Interaction Transformer

Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations

Driving among Flatmobiles: Bird-Eye-View occupancy grids from a monocular camera for holistic trajectory planning

Generative Pretraining From Pixels

Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?

MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views

MultiXNet: Multiclass Multistage Multimodal Motion Prediction

Learning Situational Driving

PnPNet: End-to-End Perception and Prediction With Tracking in the Loop

Towards Streaming Perception

Label Efficient Visual Abstractions for Autonomous Driving

Multi-Modal Sensor Fusion-Based Deep Neural Network for End-to-End Autonomous Driving With Scene Understanding

STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Prediction

Scalable Active Learning for Object Detection

Designing Network Design Spaces

Probabilistic Future Prediction for Video Scene Understanding

LaserFlow: Efficient and Probabilistic Object Detection and Motion Forecasting

3DSSD: Point-Based 3D Single Stage Object Detector

ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

Learning by Cheating

SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning

End-to-End Model-Free Reinforcement Learning for Urban Driving Using Implicit Affordances

PointPainting: Sequential Fusion for 3D Object Detection

Attention-based Hierarchical Deep Reinforcement Learning for Lane Change Behaviors in Autonomous Driving

SpAGNN: Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data

End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction

Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks

Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving

Multimodal End-to-End Autonomous Driving