NeRV: Neural Representations for Videos (2021-10-26T00:00:00.000000Z)

TL;DR

A novel neural representation for videos (NeRV) which encodes videos in neural networks taking frame index as input, which can be used as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches.

Abstract

We propose a novel neural representation for videos (NeRV) which encodes videos in neural networks. Unlike conventional representations that treat videos as frame sequences, we represent videos as neural networks taking frame index as input. Given a frame index, NeRV outputs the corresponding RGB image. Video encoding in NeRV is simply fitting a neural network to video frames and decoding process is a simple feedforward operation. As an image-wise implicit representation, NeRV output the whole image and shows great efficiency compared to pixel-wise implicit representation, improving the encoding speed by 25x to 70x, the decoding speed by 38x to 132x, while achieving better video quality. With such a representation, we can treat videos as neural networks, simplifying several video-related tasks. For example, conventional video compression methods are restricted by a long and complex pipeline, specifically designed for the task. In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H.264, HEVC \etc). Besides compression, we demonstrate the generalization of NeRV for video denoising. The source code and pre-trained model can be found at https://github.com/haochen-rye/NeRV.git.

Authors

Abhinav Shrivastava

9 papers

Ser-Nam Lim

4 papers

Bo He

3 papers

TL;DR

Abstract

Authors

References63 items

COIN: COmpression with Implicit Neural representations

Adaptive Gradient Quantization for Data-Parallel SGD

Hierarchical Autoregressive Modeling for Neural Video Compression

Conditional Entropy Coding for Efficient Video Compression

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Implicit Neural Representations with Periodic Activation Functions

Scale-Space Flow for End-to-End Optimized Video Compression

UVG dataset: 50/120fps 4K sequences for video codec analysis and development

Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction

Local Implicit Grid Representations for 3D Scenes

Convolutional Occupancy Networks

Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

Deep Structured Implicit Functions

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Learning Image and Video Compression Through Spatial-Temporal Energy Compaction

Texture Fields: Learning Texture Representations in Function Space

Learning Shape Templates With Structured Implicit Functions

Training Deep Neural Networks with 8-bit Floating Point Numbers

DVC: An End-To-End Deep Video Compression Framework

Learned Video Compression

On the Spectral Bias of Neural Networks

Quantizing deep convolutional networks for efficient inference: A whitepaper

Scalable Methods for 8-bit Training of Neural Networks

Video Compression through Image Interpolation

Model compression via distillation and quantization

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Learning Efficient Object Detection Models with Knowledge Distillation

Deep Image Prior

Attention is All you Need

Doubly Convolutional Neural Networks

MCL-JCV: A JND-based H.264/AVC video quality assessment dataset

SGDR: Stochastic Gradient Descent with Warm Restarts

Learning Structured Sparsity in Deep Neural Networks

Gaussian Error Linear Units (GELUs)

Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

A guide to convolution arithmetic for deep learning

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Group Equivariant Convolutional Networks

Exploiting Cyclic Symmetry in Convolutional Neural Networks

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Distilling the Knowledge in a Neural Network

Deep Learning with Limited Numerical Precision

Adam: A Method for Stochastic Optimization

Speeding up Convolutional Neural Networks with Low Rank Expansions

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Do Deep Nets Really Need to be Deep?

Learning Separable Filters

Overview of the High Efficiency Video Coding (HEVC) Standard

On discrete cosine transform

Converting video formats with FFmpeg

Multiscale structural similarity for image quality assessment

Overview of the H.264/AVC video coding standard

The JPEG 2000 still image compression standard

Image coding using wavelet transform

The JPEG still picture compression standard

Multilayer feedforward networks are universal approximators

A method for the construction of minimum-redundancy codes

Nerf: Representing scenes as neural radiance fields for view synthesis

Improving the speed of neural networks on CPUs

Review : Multimedia Encoding Techniques Review : MPEG : A Video Compression Standard for Multimedia Applications

ffmpeg − i FILE / f%05d . png −c : v h264 − p r e s e t medium \ −bf 0 − c r f CRF FILE

Field of Study

Venue Information

Name

Type

URL

Alternate Names