A Survey of End-to-End Driving: Architectures and Training Methods (2020-03-13T00:00:00.000000Z)

TL;DR

A deeper look on the so-called end-to-end approaches for autonomous driving, where the entire driving pipeline is replaced with a single neural network, and concludes with an architecture that combines the most promising elements of the end- to-end autonomous driving systems.

Abstract

Autonomous driving is of great interest to industry and academia alike. The use of machine learning approaches for autonomous driving has long been studied, but mostly in the context of perception. In this article, we take a deeper look on the so-called end-to-end approaches for autonomous driving, where the entire driving pipeline is replaced with a single neural network. We review the learning methods, input and output modalities, network architectures, and evaluation schemes in end-to-end driving literature. Interpretability and safety are discussed separately, as they remain challenging for this approach. Beyond providing a comprehensive overview of existing methods, we conclude the review with an architecture that combines the most promising elements of the end-to-end autonomous driving systems.

Authors

D. Fishman

2 papers

Ardi Tampuu

2 papers

Tambet Matiisen

2 papers

TL;DR

Abstract

Authors

References152 items

Generative Adversarial Networks

Introduction to Reinforcement Learning

Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?

Instance Segmentation of LiDAR Point Clouds

Learning by Cheating

Dota 2 with Large Scale Deep Reinforcement Learning

Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Efficient Black-box Assessment of Autonomous Vehicle Safety

Urban Driving with Conditional Imitation Learning

Simulation-Based Reinforcement Learning for Real-World Autonomous Driving

Grandmaster level in StarCraft II using multi-agent reinforcement learning

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

A survey on Image Data Augmentation for Deep Learning

A Survey of Autonomous Driving: Common Practices and Emerging Technologies

Multimodal End-to-End Autonomous Driving

End-To-End Interpretable Neural Motion Planner

Grounding Human-To-Vehicle Advice for Self-Driving Vehicles

Causal Confusion in Imitation Learning

Does computer vision matter for action?

Early Failure Detection of Deep End-to-End Control Policy by Reinforcement Learning

Exploring the Limitations of Behavior Cloning for Autonomous Driving

nuScenes: A Multimodal Dataset for Autonomous Driving

Learning Accurate, Comfortable and Human-like Driving

The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

PointPillars: Fast Encoders for Object Detection From Point Clouds

A Commute in Data: The comma2k19 Dataset

PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud

Learning to Drive from Simulation without Real World Labels

ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst

Variational End-to-End Navigation and Localization

IntentNet: Learning to Predict Intention from Raw Sensor Data

End-To-End Multi-Modal Sensors Fusion System For Urban Automated Driving

SECOND: Sparsely Embedded Convolutional Detection

A Modern Take on the Bias-Variance Tradeoff in Neural Networks

VisualBackProp: Efficient Visualization of CNNs for Autonomous Driving

On Offline Evaluation of Vision-based Driving Models

Textual Explanations for Self-Driving Vehicles

CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving

Learning to Drive in a Day

Conditional Affordance Learning for Driving in Urban Environments

LiDAR-Video Driving Dataset: Learning Driving Policies Effectively

Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning

PIXOR: Real-time 3D Object Detection from Point Clouds

Robust Physical-World Attacks on Deep Learning Visual Classification

Planning and Decision-Making for Autonomous Vehicles

BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling

Failure Prediction for Autonomous Driving

Driving Policy Transfer via Modularity and Abstraction

End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners

Complex-YOLO: An Euler-Region-Proposal for Real-Time 3D Object Detection on Point Clouds

OIL: Observational Imitation Learning

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control

Fairness in Machine Learning: Lessons from Political Philosophy

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation

VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection

CARLA: An Open Urban Driving Simulator

End-to-End Driving Via Conditional Imitation Learning

Agile Autonomous Driving using End-to-End Deep Imitation Learning

MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving

DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars

Proximal Policy Optimization Algorithms

Aggressive Deep Driving: Model Predictive Control with a CNN Cost Model

Attention is All you Need

Deep learning algorithm for autonomous driving using GoogLeNet

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

DDD17: End-To-End DAVIS Driving Dataset

Learning how to drive in a real world simulation with deep Q-Networks

Real Time Image Saliency for Black Box Classifiers

DeepXplore: Automated Whitebox Testing of Deep Learning Systems

Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car

Virtual to Real Reinforcement Learning for Autonomous Driving

Deep Reinforcement Learning framework for Autonomous Driving

Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention