Sapiens: Foundation for Human Vision Models (2024-08-22T00:00:00.000000Z)

TL;DR

Sapiens is a family of models for four fundamental human-centric vision tasks -- 2D pose estimation, body-part segmentation, depth estimation, depth estimation, and surface normal prediction that consistently surpasses existing baselines across various human-centric benchmarks.

Abstract

We present Sapiens, a family of models for four fundamental human-centric vision tasks -- 2D pose estimation, body-part segmentation, depth estimation, and surface normal prediction. Our models natively support 1K high-resolution inference and are extremely easy to adapt for individual tasks by simply fine-tuning models pretrained on over 300 million in-the-wild human images. We observe that, given the same computational budget, self-supervised pretraining on a curated dataset of human images significantly boosts the performance for a diverse set of human-centric tasks. The resulting models exhibit remarkable generalization to in-the-wild data, even when labeled data is scarce or entirely synthetic. Our simple model design also brings scalability -- model performance across tasks improves as we scale the number of parameters from 0.3 to 2 billion. Sapiens consistently surpasses existing baselines across various human-centric benchmarks. We achieve significant improvements over the prior state-of-the-art on Humans-5K (pose) by 7.6 mAP, Humans-2K (part-seg) by 17.1 mIoU, Hi4D (depth) by 22.4% relative RMSE, and THuman2 (normal) by 53.5% relative angular error. Project page: https://about.meta.com/realitylabs/codecavatars/sapiens.

Authors

Timur M. Bagautdinov

2 papers

Rawal Khirodkar

2 papers

Julieta Martinez

1 papers

TL;DR

Abstract

Authors

References121 items

Project Starline: A high-fidelity telepresence system

Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

Scalable Pre-training of Large Autoregressive Image Models

Sequential Modeling Enables Scalable Learning for Large Vision Models

HUGS: Human Gaussian Splats

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks

Mistral 7B

Effective Whole-body Pose Estimation with Two-stages Distillation

MiDaS v3.1 - A Model Zoo for Robust Monocular Relative Depth Estimation

Llama 2: Open Foundation and Fine-Tuned Chat Models

Towards Zero-Shot Scale-Aware Monocular Depth Estimation

Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model

DINOv2: Learning Robust Visual Features without Supervision

Segment Anything

Hi4D: 4D Instance Segmentation of Close Human Interaction

The effectiveness of MAE pre-pretraining for billion-scale pretraining

EVA-02: A Visual Representation for Neon Genesis

GPT-4 Technical Report

RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose

LLaMA: Open and Efficient Foundation Language Models

ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

Adding Conditional Control to Text-to-Image Diffusion Models

Scaling Vision Transformers to 22 Billion Parameters

Scalable Diffusion Models with Transformers

ECON: Explicit Clothed humans Optimized via Normal integration

EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

LAION-5B: An open large-scale dataset for training next generation image-text models

Metaverse through the prism of power and addiction: what will happen when the virtual world becomes more attractive than reality?

PaLI: A Jointly-Scaled Multilingual Language-Image Model

ZoomNAS: Searching for Whole-Body Human Pose Estimation in the Wild

MegaPortraits: One-shot Megapixel Neural Head Avatars

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation

Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing

BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation

Training Compute-Optimal Large Language Models

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video

High-Resolution Image Synthesis with Latent Diffusion Models

Are Large-scale Datasets Necessary for Self-Supervised Pre-training?

ICON: Implicit Clothed humans Obtained from Normals

Masked-attention Mask Transformer for Universal Image Segmentation

Swin Transformer V2: Scaling Up Capacity and Resolution

iBOT: Image BERT Pre-Training with Online Tokenizer

Masked Autoencoders Are Scalable Vision Learners

Exploring the Limits of Large Scale Pre-training

Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

BEiT: BERT Pre-Training of Image Transformers

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

Emerging Properties in Self-Supervised Vision Transformers

Pixel Codec Avatars

How to decay your learning rate

Mixture of volumetric primitives for efficient neural rendering

Self-supervised Pretraining of Visual Features in the Wild

Learning Transferable Visual Models From Natural Language Supervision

Multi-Instance Pose Networks: Rethinking Top-Down Pose Estimation

AdaBins: Depth Estimation Using Adaptive Bins

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Whole-Body Human Pose Estimation in the Wild

Language Models are Few-Shot Learners

PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

A Simple Framework for Contrastive Learning of Visual Representations

Momentum Contrast for Unsupervised Visual Representation Learning

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

XLNet: Generalized Autoregressive Pretraining for Language Understanding

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Deep High-Resolution Representation Learning for Human Pose Estimation

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Improving Multi-Person Pose Estimation using Label Correction

Everybody Dance Now

Instance-level Human Parsing via Part Grouping Network

Macro-Micro Adversarial Network for Human Parsing