The Pose Knows: Video Forecasting by Generating Pose Futures

Published in

IEEE International Conference on Computer Vision(2017)

External Links:

Generate Graph DownloadPDF

TL;DR

This work exploits human pose detectors as a free source of supervision and breaks the video forecasting problem into two discrete steps, and uses the structured space of pose as an intermediate representation to sidestep the problems that GANs have in generating video pixels directly.

Abstract

Current approaches to video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, since these approaches try to model all the structure and scene dynamics at once, in unconstrained settings they often generate uninterpretable results. Our insight is that forecasting needs to be done first at a higher level of abstraction. Specifically, we exploit human pose detectors as a free source of supervision and break the video forecasting problem into two discrete steps. First we explicitly model the high level structure of active objects in the scene (humans) and use a VAE to model the possible future movements of humans in the pose space. We then use the future poses generated as conditional information to a GAN to predict the future frames of the video in pixel space. By using the structured space of pose as an intermediate representation, we sidestep the problems that GANs have in generating video pixels directly. We show through quantitative and qualitative evaluation that our method outperforms state-of-the-art methods for video prediction.

Authors

A. Gupta

21 papers

M. Hebert

6 papers

Jacob Walker

2 papers

References57 items

Déjà Vu: - Motion Prediction in Static Images

Multi-context Attention for Human Pose Estimation

First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Online Semantic Activity Forecasting with DARKO

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

The Pose Knows: Video Forecasting by Generating Pose Futures

Published in

IEEE International Conference on Computer Vision(2017)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

A. Gupta

21 papers

M. Hebert

6 papers

Jacob Walker

2 papers

References57 items

Déjà Vu: - Motion Prediction in Static Images

Multi-context Attention for Human Pose Estimation

First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Online Semantic Activity Forecasting with DARKO

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

Kenneth Marino

4 papers

Image-to-Image Translation with Conditional Adversarial Networks

Learning What and Where to Draw

Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes

Video Pixel Networks

Generating Videos with Scene Dynamics

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Learning Activity Progression in LSTMs for Activity Detection and Early Detection

An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Dynamic Filter Networks

Unsupervised Learning for Physical Interaction through Video Prediction

Context Encoders: Feature Learning by Inpainting

Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding

Stacked Hourglass Networks for Human Pose Estimation

Generative Image Modeling Using Style and Structure Adversarial Networks

Convolutional Pose Machines

Pixel Recurrent Neural Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Spatio-temporal video autoencoder with differentiable memory

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Deep multi-scale video prediction beyond mean square error

Recurrent Network Models for Human Dynamics

Towards Good Practices for Very Deep Two-Stream ConvNets

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

ActivityNet: A large-scale video benchmark for human activity understanding

Training generative neural networks via Maximum Mean Discrepancy optimization

Dense Optical Flow Prediction from a Static Image

Anticipating the future by watching unlabeled video

Unsupervised Learning of Video Representations using LSTMs

Generative Moment Matching Networks

Video (language) modeling: a baseline for generative models of natural videos

Going deeper with convolutions

A Hierarchical Representation for Future Action Prediction

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Patch to the Future: Unsupervised Visual Prediction

Large-Scale Video Classification with Convolutional Neural Networks

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

From Actemes to Action: A Strongly-Supervised Representation for Detailed Action Understanding

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

ImageNet classification with deep convolutional neural networks

Max-Margin Early Event Detectors

A Kernel Two-Sample Test

HMDB: A large video database for human motion recognition

A Data-Driven Approach for Event Prediction

GENERATIVE ADVERSARIAL NETS

Supervision Beyond Manual Annotations for Learning Visual Representations

THUMOS challenge: Action recognition with a large number of classes

Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

Research Showcase @ CMU

Field of Study

Computer Science

Journal Information

Name

2017 IEEE International Conference on Computer Vision (ICCV)

Page

843-852

Venue Information

Name

IEEE International Conference on Computer Vision

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome/1000149/all-proceedings

Alternate Names

ICCV
IEEE Int Conf Comput Vis
ICCV Workshops
ICCV Work