MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

Published in

(2016)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper presents an approach to joint classification, detection and semantic segmentation using a unified architecture where the encoder is shared amongst the three tasks, and performs extremely well in the challenging KITTI dataset.

Abstract

While most approaches to semantic reasoning have focused on improving performance, in this paper we argue that computational times are very important in order to enable real time applications such as autonomous driving. Towards this goal, we present an approach to joint classification, detection and semantic segmentation using a unified architecture where the encoder is shared amongst the three tasks. Our approach is very simple, can be trained end-to-end and performs extremely well in the challenging KITTI dataset. Our approach is also very efficient, allowing us to perform inference at more then 23 frames per second. Training scripts and trained weights to reproduce our results can be found here: https://github.com/MarvinTeichmann/MultiNet

Authors

R. Urtasun

12 papers

R. Cipolla

17 papers

Johann Marius Zöllner

3 papers

References68 items

Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

Mask R-CNN

Fast LIDAR-based road detection using fully convolutional neural networks

Fast LIDAR-based Road Detection Using Convolutional Neural Networks

MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

Published in

(2016)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

R. Urtasun

12 papers

R. Cipolla

17 papers

Johann Marius Zöllner

3 papers

References68 items

Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

Mask R-CNN

Fast LIDAR-based road detection using fully convolutional neural networks

Fast LIDAR-based Road Detection Using Convolutional Neural Networks

Marvin Teichmann

1 papers

Michael Weber

1 papers

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Monocular 3D Object Detection for Autonomous Driving

Find your way by observing the sun and other semantic cues

Map-supervised road detection

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

End-to-End Instance Segmentation and Counting with Recurrent Attention

Exploiting fully convolutional neural networks for fast road detection

Learning to Refine Object Segments

A guide to convolution arithmetic for deep learning

Generating images with recurrent adversarial networks

Instance-Aware Semantic Segmentation via Multi-task Network Cascades

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

3D Object Proposals for Accurate Object Class Detection

Multi-Scale Context Aggregation by Dilated Convolutions

Convolutional Models for Joint Object Categorization and Pose Estimation

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

End-to-End People Detection in Crowded Scenes

You Only Look Once: Unified, Real-Time Object Detection

Rotating your face using multi-task deep neural network

Learning Multiple Tasks with Deep Relationship Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Learning Deconvolution Network for Semantic Segmentation

Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Fully Connected Deep Structured Networks

What Makes for Effective Detection Proposals?

Conditional Random Fields as Recurrent Neural Networks

Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Adam: A Method for Stochastic Optimization

Highly Efficient Forward and Backward Propagation of Convolutional Neural Networks for Pixelwise Classification

Fully convolutional networks for semantic segmentation

Deep Deconvolutional Networks for Scene Parsing

Facial Landmark Detection by Deep Multi-task Learning

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Simultaneous Detection and Segmentation

How good are detection proposals, really?

OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks

Scalable Object Detection Using Deep Neural Networks

Visualizing and Understanding Convolutional Networks

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

A new performance measure and evaluation benchmark for road detection algorithms

Vision meets robotics: The KITTI dataset

Fast image scanning with deep max-pooling convolutional neural networks

ImageNet classification with deep convolutional neural networks

Are we ready for autonomous driving? The KITTI vision benchmark suite

Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation

Joint segmentation and classification of human actions in video

Deconvolutional networks

ImageNet: A large-scale hierarchical image database

Beyond sliding windows: Object localization by efficient subwindow search

Single Shot Multibox Detector를 적용한 실시간 선박 검출 및 분류 시스템 개발

HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition

Towards Road Type Classiﬁcation with Occupancy Grids

Efficient deep methods for monocular road segmentation

Kitti road public benchmark

The PASCAL Visual Object Classes Challenge

The Pascal Visual Object Classes Challenge 2006 ( VOC 2006 ) Results

Field of Study

Computer Science

Journal Information

Name

2018 IEEE Intelligent Vehicles Symposium (IV)

Page

1013-1020

TL;DR

Abstract

Authors

References68 items

Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

Mask R-CNN

Fast LIDAR-based road detection using fully convolutional neural networks

Fast LIDAR-based Road Detection Using Convolutional Neural Networks

TL;DR

Abstract

Authors

References68 items

Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

Mask R-CNN

Fast LIDAR-based road detection using fully convolutional neural networks

Fast LIDAR-based Road Detection Using Convolutional Neural Networks

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Monocular 3D Object Detection for Autonomous Driving

Find your way by observing the sun and other semantic cues

Map-supervised road detection

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

End-to-End Instance Segmentation and Counting with Recurrent Attention

Exploiting fully convolutional neural networks for fast road detection

Learning to Refine Object Segments

A guide to convolution arithmetic for deep learning

Generating images with recurrent adversarial networks

Instance-Aware Semantic Segmentation via Multi-task Network Cascades

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

3D Object Proposals for Accurate Object Class Detection

Multi-Scale Context Aggregation by Dilated Convolutions

Convolutional Models for Joint Object Categorization and Pose Estimation

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

End-to-End People Detection in Crowded Scenes

You Only Look Once: Unified, Real-Time Object Detection

Rotating your face using multi-task deep neural network

Learning Multiple Tasks with Deep Relationship Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Learning Deconvolution Network for Semantic Segmentation

Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Fast R-CNN

Fully Connected Deep Structured Networks

What Makes for Effective Detection Proposals?

Conditional Random Fields as Recurrent Neural Networks

Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Adam: A Method for Stochastic Optimization

Highly Efficient Forward and Backward Propagation of Convolutional Neural Networks for Pixelwise Classification

Fully convolutional networks for semantic segmentation

Deep Deconvolutional Networks for Scene Parsing

Facial Landmark Detection by Deep Multi-task Learning

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Simultaneous Detection and Segmentation

How good are detection proposals, really?

OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks

Scalable Object Detection Using Deep Neural Networks

Visualizing and Understanding Convolutional Networks

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

A new performance measure and evaluation benchmark for road detection algorithms

Vision meets robotics: The KITTI dataset

Fast image scanning with deep max-pooling convolutional neural networks

ImageNet classification with deep convolutional neural networks

Are we ready for autonomous driving? The KITTI vision benchmark suite

Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation

Joint segmentation and classification of human actions in video

Deconvolutional networks

ImageNet: A large-scale hierarchical image database

Beyond sliding windows: Object localization by efficient subwindow search

Single Shot Multibox Detector를 적용한 실시간 선박 검출 및 분류 시스템 개발

HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition

Towards Road Type Classiﬁcation with Occupancy Grids

Efficient deep methods for monocular road segmentation

Kitti road public benchmark

The PASCAL Visual Object Classes Challenge