Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022-06-16T00:00:00.000000Z)

TL;DR

An integrated approach that has two branches for trajectory planning and direct control, respectively, and involves a novel multi-step prediction scheme such that the relationship between current actions and future states can be reasoned.

Abstract

Current end-to-end autonomous driving methods either run a controller based on a planned trajectory or perform control prediction directly, which have spanned two separately studied lines of research. Seeing their potential mutual benefits to each other, this paper takes the initiative to explore the combination of these two well-developed worlds. Specifically, our integrated approach has two branches for trajectory planning and direct control, respectively. The trajectory branch predicts the future trajectory, while the control branch involves a novel multi-step prediction scheme such that the relationship between current actions and future states can be reasoned. The two branches are connected so that the control branch receives corresponding guidance from the trajectory branch at each time step. The outputs from two branches are then fused to achieve complementary advantages. Our results are evaluated in the closed-loop urban driving setting with challenging scenarios using the CARLA simulator. Even with a monocular camera input, the proposed approach ranks first on the official CARLA Leaderboard, outperforming other complex candidates with multiple sensors or fusion mechanisms by a large margin. The source code is publicly available at https://github.com/OpenPerceptionX/TCP

Authors

Junchi Yan

17 papers

Li Chen

3 papers

Yu Qiao

2 papers

TL;DR

Abstract

Authors

References72 items

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

Learning Mixture of Domain-Specific Experts via Disentangled Factors for Autonomous Driving

Learning from All Vehicles

PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmark

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Multi-Task Conditional Imitation Learning for Autonomous Navigation at Crowded Intersections

CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Driving

GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving

NEAT: Neural Attention Fields for End-to-End Autonomous Driving

YOLOP: You Only Look Once for Panoptic Driving Perception

End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

Learning by Watching

Learning to drive from a world on rails

Multi-task Learning with Attention for End-to-end Autonomous Driving

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Enhancing Object Detection for Autonomous Driving by Optimizing Anchor Generation and Addressing Class Imbalance

OmniDet: Surround View Cameras Based Multi-Task Visual Perception Network for Autonomous Driving

Multi-Task End-to-End Self-Driving Architecture for CAV Platoons

MP3: A Unified Model to Map, Perceive, Predict and Plan

Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges

The NVIDIA PilotNet Experiments

Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations

Eigen-CAM: Class Activation Map using Principal Components

Learning Situational Driving

Multi-model ensemble with rich spatial information for object detection

Learning by Cheating

SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning

Urban Driving with Conditional Imitation Learning

End-to-End Model-Free Reinforcement Learning for Urban Driving Using Implicit Affordances

Multimodal End-to-End Autonomous Driving

End-To-End Interpretable Neural Motion Planner

Multi-Task Multi-Sensor Fusion for 3D Object Detection

Deep Local Trajectory Replanning and Control for Robot Navigation

Exploring the Limitations of Behavior Cloning for Autonomous Driving

MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning

Model predictive path following control for autonomous cars considering a measurable disturbance: Implementation, testing, and verification

Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks

Deep Imitative Models for Flexible Inference, Planning, and Control

Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability

Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers

CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving

Learning to Drive in a Day

End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptions

CARLA: An Open Urban Driving Simulator

End-to-End Driving Via Conditional Imitation Learning

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

End-to-End Learning of Driving Models from Large-Scale Video Datasets

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

End to End Learning for Self-Driving Cars

Deep Residual Learning for Image Recognition

Learning to propose objects

Adam: A Method for Stochastic Optimization

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

ImageNet: A large-scale hierarchical image database

Model Predictive Control

Off-Road Obstacle Avoidance through End-to-End Learning

Multiple Classifier Systems

Multitask Learning

Adaptive Mixtures of Local Experts

Multi-task Learning with Future States for Vision-Based Autonomous Driving

Supplementary Material for Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving

Multi-Task Mutual Learning for Vehicle Re-Identification

ALVINN, an autonomous land vehicle in a neural network

Книги издательства Springer Science & Business Media

autonomous driving leaderboard

Did you mention the license of the assets? [Yes] See the Supplementary

c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes

e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content? [No]

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

If you are using existing assets

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Field of Study

Journal Information

Name

Volume