VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023-03-21T00:00:00.000000Z)

TL;DR

VAD, an end-to-end vectorized paradigm for autonomous driving, which models the driving scene as a fully vectorized representation, achieves state-of-the-art end-to-end planning performance on the nuScenes dataset, outperforming the previous best method by a large margin.

Abstract

Autonomous driving requires a comprehensive understanding of the surrounding environment for reliable trajectory planning. Previous works rely on dense rasterized scene representation (e.g., agent occupancy and semantic map) to perform planning, which is computationally intensive and misses the instance-level structure information. In this paper, we propose VAD, an end-to-end vectorized paradigm for autonomous driving, which models the driving scene as a fully vectorized representation. The proposed vectorized paradigm has two significant advantages. On one hand, VAD exploits the vectorized agent motion and map elements as explicit instance-level planning constraints which effectively improves planning safety. On the other hand, VAD runs much faster than previous end-to-end planning methods by getting rid of computation-intensive rasterized representation and hand-designed post-processing steps. VAD achieves state-of-the-art end-to-end planning performance on the nuScenes dataset, outperforming the previous best method by a large margin. Our base model, VAD-Base, greatly reduces the average collision rate by 29.0% and runs 2.5× faster. Besides, a lightweight variant, VAD-Tiny, greatly improves the inference speed (up to 9.3×) while achieving comparable planning performance. We believe the excellent performance and the high efficiency of VAD are critical for the real-world deployment of an autonomous driving system. Code and models are available at https://github.com/hustvl/VAD for facilitating future research.

Authors

References52 items

Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes

Lane Graph as Path: Continuity-preserving Path-wise Modeling for Online Lane Graph Construction

Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction

PlanT: Explainable Planning Transformers via Object-Level Representations

Differentiable Raycasting for Self-supervised Occupancy Forecasting

MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction

ViP3D: End-to-End Visual Trajectory Prediction via 3D Agent Queries

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation

HOPE: Hierarchical Spatial-temporal Network for Occupancy Flow Prediction

VectorMapNet: End-to-end Vectorized HD Map Learning

Efficient and Robust 2D-to-BEV Representation Learning via Geometry-guided Kernel Transformer

BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving

Cross-view Transformers for real-time Map-view Semantic Segmentation

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

PETR: Position Embedding Transformation for Multi-View 3D Object Detection

GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

HDMapNet: An Online HD Map Construction and Evaluation Framework

Scene Transformer: A unified architecture for predicting multiple agent trajectories

Safe Local Motion Planning with Self-Supervised Freespace Forecasting

Autonomous Vehicle Motion Planning via Recurrent Spline Optimization

What data do we need for training an AV motion planner?

Learning to drive from a world on rails

FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Multimodal Motion Prediction with Stacked Transformers

MP3: A Unified Model to Map, Perceive, Predict and Plan

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations

Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

Learning Lane Graph Representations for Motion Forecasting

VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation

Learning by Cheating

End-to-End Model-Free Reinforcement Learning for Urban Driving Using Implicit Affordances

CoverNet: Multimodal Behavior Prediction Using Trajectory Sets

MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction

TL;DR

Abstract

Authors

References52 items

Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes

Lane Graph as Path: Continuity-preserving Path-wise Modeling for Online Lane Graph Construction

Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction

PlanT: Explainable Planning Transformers via Object-Level Representations

Differentiable Raycasting for Self-supervised Occupancy Forecasting

MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction

ViP3D: End-to-End Visual Trajectory Prediction via 3D Agent Queries

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation

HOPE: Hierarchical Spatial-temporal Network for Occupancy Flow Prediction

VectorMapNet: End-to-end Vectorized HD Map Learning

Efficient and Robust 2D-to-BEV Representation Learning via Geometry-guided Kernel Transformer

BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving

Cross-view Transformers for real-time Map-view Semantic Segmentation

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

PETR: Position Embedding Transformation for Multi-View 3D Object Detection

GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

HDMapNet: An Online HD Map Construction and Evaluation Framework

Scene Transformer: A unified architecture for predicting multiple agent trajectories

Safe Local Motion Planning with Self-Supervised Freespace Forecasting

Autonomous Vehicle Motion Planning via Recurrent Spline Optimization

What data do we need for training an AV motion planner?

Learning to drive from a world on rails

FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Multimodal Motion Prediction with Stacked Transformers

MP3: A Unified Model to Map, Perceive, Predict and Plan

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations

Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

Learning Lane Graph Representations for Motion Forecasting

VectorNet: Encoding HD Maps and Agent Dynamics From Vectorized Representation

Learning by Cheating

End-to-End Model-Free Reinforcement Learning for Urban Driving Using Implicit Affordances

CoverNet: Multimodal Behavior Prediction Using Trajectory Sets

MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction

End-To-End Interpretable Neural Motion Planner

Exploring the Limitations of Behavior Cloning for Autonomous Driving

nuScenes: A Multimodal Dataset for Autonomous Driving

Learning to Drive in a Day

Decoupled Weight Decay Regularization

CARLA: An Open Urban Driving Simulator

Focal Loss for Dense Object Detection

Attention is All you Need

SGDR: Stochastic Gradient Descent with Warm Restarts

A Review of Motion Planning Techniques for Automated Vehicles

Deep Residual Learning for Image Recognition

Goal-oriented Autonomous Driving

ALVINN, an autonomous land vehicle in a neural network

8314 applicable license agreement with IEEE. Restrictions apply

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names