Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks (2020-03-14T00:00:00.000000Z)

TL;DR

Spike-FlowNet is presented, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs without sacrificing the performance.

Abstract

Event-based cameras display great potential for a variety of tasks such as high-speed motion detection and navigation in low-light environments where conventional frame-based cameras suffer critically. This is attributed to their high temporal resolution, high dynamic range, and low-power consumption. However, conventional computer vision methods as well as deep Analog Neural Networks (ANNs) are not suited to work well with the asynchronous and discrete nature of event camera outputs. Spiking Neural Networks (SNNs) serve as ideal paradigms to handle event camera outputs, but deep SNNs suffer in terms of performance due to the spike vanishing phenomenon. To overcome these issues, we present Spike-FlowNet, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs without sacrificing the performance. The network is end-to-end trained with self-supervised learning on Multi-Vehicle Stereo Event Camera (MVSEC) dataset. Spike-FlowNet outperforms its corresponding ANN-based method in terms of the optical flow prediction capability while providing significant computational efficiency.

Authors

K. Roy

2 papers

Kostas Daniilidis

11 papers

A. Z. Zhu

3 papers

TL;DR

Abstract

Authors

References39 items

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Towards Scalable, Efficient and Accurate Deep Spiking Neural Networks with Backward Residual Connections, Stochastic Softmax and Hybridization

Live Demonstration: Unsupervised Event-Based Learning of Optical Flow, Depth and Egomotion

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Event-based Plane-fitting Optical Flow for Dynamic Vision Sensors in FPGA

ABMOF: A Novel Optical Flow Algorithm for Dynamic Vision Sensors

A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation

EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

The Multivehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception

Loihi: A Neuromorphic Manycore Processor with On-Chip Learning

Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

Spiking Optical Flow for Event-Based Sensors Using IBM's TrueNorth Neurosynaptic System

Event-based feature tracking with probabilistic data association

Training Deep Spiking Neural Networks Using Backpropagation

Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness

Unsupervised learning of digit recognition using spike-timing-dependent plasticity

Bio-inspired Motion Estimation with Event-Driven Sensors

U-Net: Convolutional Networks for Biomedical Image Segmentation

On event-based optical flow detection

Adam: A Method for Stochastic Optimization

Theoretical Neuroscience

A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor

A million spiking-neuron integrated circuit with a scalable communication network and interface

1.1 Computing's energy problem (and what we can do about it)

Event-Based Visual Flow

A spiking neural network architecture for visual motion estimation

A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them

Asynchronous frameless event-based optical flow

Fly motion vision.

A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor

A review of the integrate-and-fire neuron model: II. Inhomogeneous synaptic input and network properties

A Review of the Integrate-and-fire Neuron Model: I. Homogeneous Synaptic Input

Backpropagation Through Time: What It Does and How to Do It

An Iterative Image Registration Technique with an Application to Stereo Vision

Semi-Supervised Learning for Optical Flow with Generative Adversarial Networks

> Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names