Event-based Temporally Dense Optical Flow Estimation with Sequential Learning (2022-10-03T00:00:00.000000Z)

TL;DR

This work shows that a temporally dense flow estimation at 100 Hz can be achieved by treating the flow estimation as a sequential problem using two different variants of recurrent networks – Long-short term memory (LSTM) and spiking neural network (SNN).

Abstract

Event cameras provide an advantage over traditional frame-based cameras when capturing fast-moving objects without a motion blur. They achieve this by recording changes in light intensity (known as events), thus allowing them to operate at a much higher frequency and making them suitable for capturing motions in a highly dynamic scene. Many recent studies have proposed methods to train neural networks (NNs) for predicting optical flow from events. However, they often rely on a spatio-temporal representation constructed from events over a fixed interval, such as 10 Hz used in training on the DSEC dataset. This limitation restricts the flow prediction to the same interval (10 Hz) whereas the fast speed of event cameras, which can operate up to 3 kHz, has not been effectively utilized. In this work, we show that a temporally dense flow estimation at 100 Hz can be achieved by treating the flow estimation as a sequential problem using two different variants of recurrent networks – Long-short term memory (LSTM) and spiking neural network (SNN). First, We utilize the NN model constructed similar to the popular EV-FlowNet but with LSTM layers to demonstrate the efficiency of our training method. The model not only produces 10× more frequent optical flow than the existing ones, but the estimated flows also have 13% lower errors than predictions from the baseline EV-FlowNet. Second, we construct an EV-FlowNet SNN but with leaky integrate and fire neurons to efficiently capture the temporal dynamics. We found that simple inherent recurrent dynamics of SNN lead to significant parameter reduction compared to the LSTM model. In addition, because of its event-driven computation, the spiking model is estimated to consume only 1.5% energy of the LSTM model, highlighting the efficiency of SNN in processing events and the potential for achieving temporally dense flow.

References37 items

Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamics

Efficient Neuromorphic Signal Processing with Loihi 2

Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation

E-RAFT: Dense Optical Flow from Event Cameras

CarSNN: An Efficient Spiking Neural Network for Event-Based Autonomous Cars on the Loihi Neuromorphic Research Processor

Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures

DSEC: A Stereo Event Camera Dataset for Driving Scenarios

Holistic LSTM for Pedestrian Trajectory Prediction

Unsupervised Learning of Dense Optical Flow, Depth and Egomotion with Event-Based Sensors

Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks

EVDodgeNet: Deep Dynamic Obstacle Dodging with Event Cameras

TrueNorth: Accelerating From Zero to 64 Million Neurons in 10 Years

Event-Based Vision: A Survey

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Long short-term memory and Learning-to-learn in networks of spiking neurons

EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

Loihi: A Neuromorphic Manycore Processor with On-Chip Learning

Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

Optic flow-based collision-free strategies: From insects to robots.

SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks

Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art

Optical Flow Estimation Using a Spatial Pyramid Network

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

FlowNet: Learning Optical Flow with Convolutional Networks

Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition

Sequence to Sequence Learning with Neural Networks

1.1 Computing's energy problem (and what we can do about it)

Speech recognition with deep recurrent neural networks

Long Short-Term Memory

The computation of optical flow

Determining Optical Flow

Optic flow computation : a unified perspective

9801 applicable license agreement with IEEE. Restrictions apply

Garrick

Unsupervised event-based learning of 9802 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions