RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Published in

European Conference on Computer Vision(2020)

External Links:

Generate Graph

TL;DR

Recurrent All-Pairs Field Transforms achieves state-of-the-art performance on the KITTI and Sintel datasets and has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count.

Abstract

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance on the KITTI and Sintel datasets. In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count.

Authors

Jia Deng

20 papers

Zachary Teed

2 papers

References70 items

MaskFlownet: Asymmetric Feature Matching With Learnable Occlusion Mask

ScopeFlow: Dynamic Scene Scoping for Optical Flow

Improving Optical Flow on a Pyramid Level

The Five Elements of Flow

PyTorch: An Imperative Style, High-Performance Deep Learning Library

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Published in

European Conference on Computer Vision(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Jia Deng

20 papers

Zachary Teed

2 papers

References70 items

MaskFlownet: Asymmetric Feature Matching With Learnable Occlusion Mask

ScopeFlow: Dynamic Scene Scoping for Optical Flow

Improving Optical Flow on a Pyramid Level

The Five Elements of Flow

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Differentiable Convex Optimization Layers

Deep Equilibrium Models

Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation

Hierarchical Deep Stereo Matching on High-Resolution Images

DeepView: View Synthesis With Learned Gradient Descent

SelFlow: Self-Supervised Learning of Optical Flow

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

A Lightweight Optical Flow CNN —Revisiting Data Fidelity and Regularization

Taking a Deeper Look at the Inverse Compositional Algorithm

Hierarchical Discrete Distribution Decomposition for Match Density Estimation

DeepV2D: Video to Depth with Differentiable Structure from Motion

Trellis Networks for Sequence Modeling

Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

DeepTAM: Deep Tracking and Mapping

Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining

BA-Net: Dense Bundle Adjustment Network

FlowFields++: Accurate Optical Flow Correspondences Meet Robust Interpolation

End-to-End Learning of Motion Representation for Video Understanding

Devon: Deformable Volume Network for Learning Optical Flow

Learning for Disparity Estimation Through Feature Constancy

Decoupled Weight Decay Regularization

Automatic differentiation in PyTorch

Variational Networks: Connecting Variational Methods and Deep Learning

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

Learned Primal-Dual Reconstruction

Optical Flow in Mostly Rigid Scenes

Accurate Optical Flow via Direct Cost Volume Processing

Solving ill-posed inverse problems using iterative deep neural networks

The 2017 DAVIS Challenge on Video Object Segmentation

OptNet: Differentiable Optimization as a Layer in Neural Networks

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Optical Flow Estimation Using a Spatial Pyramid Network

The HCI Benchmark Suite: Stereo and Flow Ground Truth with Uncertainties for Urban Autonomous Driving

Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids

A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

Discrete Optimization for Optical Flow

Object scene flow for autonomous vehicles

FlowNet: Learning Optical Flow with Convolutional Networks

Non-local Total Generalized Variation for Optical Flow Estimation

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Vision meets robotics: The KITTI dataset

A Naturalistic Open Source Movie for Optical Flow Evaluation

Slow and fast scales for superprocess limits of age-structured populations

Large displacement optical flow

A Duality Based Approach for Realtime TV-L1 Optical Flow

A symmetry problem

A framework for the robust estimation of optical flow

Determining Optical Flow

Volumetric Correspondence Networks for Optical Flow

DeepView: High-quality view synthesis by learned gradient descent

Supplementary Material for LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation

Author manuscript, published in "IEEE Intenational Conference on Computer Vision (ICCV), Sydney: Australie (2013)" DeepFlow: Large displacement optical flow with deep matching

A rst-order primal-dual algorithm for convex problems with applications to imaging

Stereo Processing by Semiglobal Matching and Mutual Information

The state of the Journal of Mathematical Imaging and Vision

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Puerto Rico June pages Ego Motion Estimation Using Optical Flow Fields Observed from Multiple Cameras

In Advances in Neural Information Processing Systems

In 1993 (4th) International Conference on Computer Vision

Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Stereo Processing by Semi-global Matching and Mutual Information

A Network Architecture Conv7x7 (64)

Recurrent All-Pairs

Unit (32) Res. Unit (32) Res. Unit (64) Res. Unit (64) Res. Unit (96) Res. Unit(96)

Field of Study

Computer ScienceMaterials Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

European Conference on Computer Vision

Type

conference

URL

https://link.springer.com/conference/eccv

Alternate Names

ECCV
Eur Conf Comput Vis

TL;DR

Abstract

Authors

References70 items

MaskFlownet: Asymmetric Feature Matching With Learnable Occlusion Mask

ScopeFlow: Dynamic Scene Scoping for Optical Flow

Improving Optical Flow on a Pyramid Level

The Five Elements of Flow

PyTorch: An Imperative Style, High-Performance Deep Learning Library

TL;DR

Abstract

Authors

References70 items

MaskFlownet: Asymmetric Feature Matching With Learnable Occlusion Mask

ScopeFlow: Dynamic Scene Scoping for Optical Flow

Improving Optical Flow on a Pyramid Level

The Five Elements of Flow

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Differentiable Convex Optimization Layers

Deep Equilibrium Models

Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation

Hierarchical Deep Stereo Matching on High-Resolution Images

DeepView: View Synthesis With Learned Gradient Descent

SelFlow: Self-Supervised Learning of Optical Flow

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

A Lightweight Optical Flow CNN —Revisiting Data Fidelity and Regularization

Taking a Deeper Look at the Inverse Compositional Algorithm

Hierarchical Discrete Distribution Decomposition for Match Density Estimation

DeepV2D: Video to Depth with Differentiable Structure from Motion

Trellis Networks for Sequence Modeling

Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

DeepTAM: Deep Tracking and Mapping

Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining

BA-Net: Dense Bundle Adjustment Network

FlowFields++: Accurate Optical Flow Correspondences Meet Robust Interpolation

End-to-End Learning of Motion Representation for Video Understanding

Devon: Deformable Volume Network for Learning Optical Flow

Learning for Disparity Estimation Through Feature Constancy

Decoupled Weight Decay Regularization

Automatic differentiation in PyTorch

Variational Networks: Connecting Variational Methods and Deep Learning

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

Learned Primal-Dual Reconstruction

Optical Flow in Mostly Rigid Scenes

Accurate Optical Flow via Direct Cost Volume Processing

Solving ill-posed inverse problems using iterative deep neural networks

The 2017 DAVIS Challenge on Video Object Segmentation

OptNet: Differentiable Optimization as a Layer in Neural Networks

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Optical Flow Estimation Using a Spatial Pyramid Network

The HCI Benchmark Suite: Stereo and Flow Ground Truth with Uncertainties for Urban Autonomous Driving

Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids

A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

Discrete Optimization for Optical Flow

Object scene flow for autonomous vehicles

FlowNet: Learning Optical Flow with Convolutional Networks

Non-local Total Generalized Variation for Optical Flow Estimation

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Vision meets robotics: The KITTI dataset

A Naturalistic Open Source Movie for Optical Flow Evaluation

Slow and fast scales for superprocess limits of age-structured populations

Large displacement optical flow

Et al

A Duality Based Approach for Realtime TV-L1 Optical Flow

A symmetry problem

A framework for the robust estimation of optical flow

Determining Optical Flow

Volumetric Correspondence Networks for Optical Flow

DeepView: High-quality view synthesis by learned gradient descent

Supplementary Material for LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation

pages 439–454

Author manuscript, published in "IEEE Intenational Conference on Computer Vision (ICCV), Sydney: Australie (2013)" DeepFlow: Large displacement optical flow with deep matching

A rst-order primal-dual algorithm for convex problems with applications to imaging

Stereo Processing by Semiglobal Matching and Mutual Information

The state of the Journal of Mathematical Imaging and Vision

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Puerto Rico June pages Ego Motion Estimation Using Optical Flow Fields Observed from Multiple Cameras

In Advances in Neural Information Processing Systems

In 1993 (4th) International Conference on Computer Vision

volume 281

Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Stereo Processing by Semi-global Matching and Mutual Information