Reconstructing Animatable Categories from Videos

Published in

Computer Vision and Pattern Recognition(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

RAC, a method to build category-level 3D models from monocular videos, disentangling variations over instances and motion over time is presented, using 3D background models to disentangle objects from the background.

Abstract

Building animatable 3D models is challenging due to the need for 3D scans, laborious registration, and rigging. Recently, differentiable rendering provides a pathway to obtain high-quality 3D models from monocular videos, but these are limited to rigid categories or single instances. We present RAC, a method to build category-level 3D models from monocular videos, disentangling variations over instances and motion over time. Three key ideas are introduced to solve this problem: (1) specializing a category-level skeleton to instances, (2) a method for latent space regularization that encourages shared structure across a category while maintaining instance details, and (3) using 3D background models to disentangle objects from the background. We build 3D models for humans, cats and dogs given monocular videos. Project page: https://gengshan-y.github.io/rac-www/.

Authors

Deva Ramanan

21 papers

Chaoyang Wang

3 papers

Dinesh Reddy Narapureddy

3 papers

References82 items

SMPL: A Skinned Multi-Person Linear Model

MoDA: Modeling Deformable 3D Objects from Casual Videos

MagicPony: Learning Articulated 3D Animals in the Wild

CASA: Category-agnostic Skeletal Animal Reconstruction

TAVA: Template-free Animatable Volumetric Actors

Reconstructing Animatable Categories from Videos

Published in

Computer Vision and Pattern Recognition(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Deva Ramanan

21 papers

Chaoyang Wang

3 papers

Dinesh Reddy Narapureddy

3 papers

References82 items

SMPL: A Skinned Multi-Person Linear Model

MoDA: Modeling Deformable 3D Objects from Casual Videos

MagicPony: Learning Articulated 3D Animals in the Wild

CASA: Category-agnostic Skeletal Animal Reconstruction

TAVA: Template-free Animatable Volumetric Actors

Gengshan Yang

4 papers

WALT: Watch And Learn 2D amodal representation from Time-lapse imagery

CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism

BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video

BANMo: Building Animatable 3D Neural Models from Many Casual Videos

Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects

ICON: Implicit Clothed humans Obtained from Normals

Efficient Geometry-aware 3D Generative Adversarial Networks

Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

CodeNeRF: Disentangled Neural Radiance Fields for Object Categories

DOVE: Learning Deformable 3D Objects by Watching Videos

To The Point: Correspondence-driven monocular 3D category reconstruction

Discovering Relationships between Object Categories via Universal Canonical Maps

HuMoR: 3D Human Motion Model for Robust Pose Estimation

LASR: Learning Articulated Shape Reconstruction from a Monocular Video

FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling

Neural Articulated Radiance Field

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks

NeRF--: Neural Radiance Fields Without Known Camera Parameters

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

Shelf-Supervised Mesh Prediction in the Wild

Online Adaptation for Consistent Mesh Reconstruction in the Wild

Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

D-NeRF: Neural Radiance Fields for Dynamic Scenes

Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

Nerfies: Deformable Neural Radiance Fields

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Continuous Surface Embeddings

NeRF++: Analyzing and Improving Neural Radiance Fields

MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video

Volumetric human teleportation

3D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single View

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild

Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop

Implicit Mesh Reconstruction from Unannotated Image Collections

PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

Self-supervised Single-view 3D Reconstruction via Semantic Consistency

PointRend: Image Segmentation As Rendering

VIBE: Video Inference for Human Body Pose and Shape Estimation

Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

What Do Single-View 3D Reconstruction Networks Learn?

Tex2Shape: Detailed Full Human Body Geometry From a Single Image

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

From Variational to Deterministic Autoencoders

Occupancy Networks: Learning 3D Reconstruction in Function Space

Monocular Total Capture: Posing Face, Body, and Hands in the Wild

Creatures great and SMAL: Recovering the shape and motion of animals from video

Interpolating between Optimal Transport and MMD using Sinkhorn Divergences

Lions and Tigers and Bears: Capturing Non-rigid, 3D, Articulated Shape from Images

Learning Category-Specific Mesh Reconstruction from Image Collections

The 2017 DAVIS Challenge on Video Object Segmentation

3D Menagerie: Modeling the 3D Shape and Pose of Animals

Density estimation using Real NVP

Robust and accurate skeletal rigging from mesh sequences

FAUST: Dataset and Evaluation for 3D Mesh Registration

Stretchable and Twistable Bones for Skeletal Shape Deformation

Background Subtraction for Freely Moving Cameras

Articulated mesh animation from multi-view silhouettes

Skinning with dual quaternions

SCAPE: shape completion and animation of people

Marching cubes: A high resolution 3D surface construction algorithm

On the Analysis of Accumulative Difference Pictures from Image Sequences of Real World Scenes

ICON: Implicit Clothed humans Obtained 17004 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. from Normals

NeRF-: Neural Radiance Fields Without Known Camera Parameters

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

Nerf: Representing scenes as neural radiance ﬁelds for view synthesis

Volumetric Correspondence Networks for Optical Flow

Shape and viewpoints without keypoints

Field of Study

Computer Science

Journal Information

Name

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Page

10323-10333

Venue Information

Name

Computer Vision and Pattern Recognition

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000147

Alternate Names

CVPR
Comput Vis Pattern Recognit

TL;DR

Abstract

Authors

References82 items

SMPL: A Skinned Multi-Person Linear Model

MoDA: Modeling Deformable 3D Objects from Casual Videos

MagicPony: Learning Articulated 3D Animals in the Wild

CASA: Category-agnostic Skeletal Animal Reconstruction

TAVA: Template-free Animatable Volumetric Actors

TL;DR

Abstract

Authors

References82 items

SMPL: A Skinned Multi-Person Linear Model

MoDA: Modeling Deformable 3D Objects from Casual Videos

MagicPony: Learning Articulated 3D Animals in the Wild

CASA: Category-agnostic Skeletal Animal Reconstruction

TAVA: Template-free Animatable Volumetric Actors

WALT: Watch And Learn 2D amodal representation from Time-lapse imagery

CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism

BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video

BANMo: Building Animatable 3D Neural Models from Many Casual Videos

Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects

ICON: Implicit Clothed humans Obtained from Normals

Efficient Geometry-aware 3D Generative Adversarial Networks

Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

CodeNeRF: Disentangled Neural Radiance Fields for Object Categories

DOVE: Learning Deformable 3D Objects by Watching Videos

HyperNeRF

To The Point: Correspondence-driven monocular 3D category reconstruction

Discovering Relationships between Object Categories via Universal Canonical Maps

HuMoR: 3D Human Motion Model for Robust Pose Estimation

LASR: Learning Articulated Shape Reconstruction from a Monocular Video

FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling

Neural Articulated Radiance Field

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks

NeRF--: Neural Radiance Fields Without Known Camera Parameters

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

Shelf-Supervised Mesh Prediction in the Wild

Online Adaptation for Consistent Mesh Reconstruction in the Wild

Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

D-NeRF: Neural Radiance Fields for Dynamic Scenes

Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

Nerfies: Deformable Neural Radiance Fields

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Continuous Surface Embeddings

NeRF++: Analyzing and Improving Neural Radiance Fields

MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video

Volumetric human teleportation

3D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single View

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild

Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop

Implicit Mesh Reconstruction from Unannotated Image Collections

PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

RigNet

PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

Self-supervised Single-view 3D Reconstruction via Semantic Consistency

PointRend: Image Segmentation As Rendering

VIBE: Video Inference for Human Body Pose and Shape Estimation

Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

What Do Single-View 3D Reconstruction Networks Learn?

Tex2Shape: Detailed Full Human Body Geometry From a Single Image

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

From Variational to Deterministic Autoencoders

Occupancy Networks: Learning 3D Reconstruction in Function Space

Monocular Total Capture: Posing Face, Body, and Hands in the Wild

Creatures great and SMAL: Recovering the shape and motion of animals from video

Interpolating between Optimal Transport and MMD using Sinkhorn Divergences

Lions and Tigers and Bears: Capturing Non-rigid, 3D, Articulated Shape from Images

Learning Category-Specific Mesh Reconstruction from Image Collections

MonoPerfCap

The 2017 DAVIS Challenge on Video Object Segmentation

3D Menagerie: Modeling the 3D Shape and Pose of Animals

Density estimation using Real NVP