EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling (2023-12-31T00:00:00.000000Z)

TL;DR

Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results.

Abstract

We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEAT2 combines a MoShed SMPL-X body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results. Our code and dataset are available. 1

Authors

Haiyang Liu

2 papers

You Zhou

2 papers

Zihao Zhu

1 papers

TL;DR

Abstract

Authors

References81 items

Emotional Speech-Driven 3D Body Animation via Disentangled Latent Diffusion

Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-Speech Gesture Generation

LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation

Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer

BlendFace: Re-designing Identity Encoders for Face-Swapping

Emotional Speech-Driven Animation with Content-Emotion Disentanglement

EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation

QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation

DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

Makeup Extraction of 3D Representation via Illumination‐Aware Image Decomposition

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Generating Holistic 3D Human Motion from Speech

MotionBERT: A Unified Perspective on Learning Human Motion Representations

DisCo: Disentangled Implicit Content and Rhythm Learning for Diverse Co-Speech Gestures Synthesis

BareSkinNet: De‐makeup and De‐lighting via 3D Face Reconstruction

ZeroEGGS: Zero‐shot Example‐based Gesture Generation from Speech

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

Audio-driven Neural Gesture Reenactment with Video Motion Graphs

Differentiable Dynamics for Articulated 3d Human Motion Reconstruction

Vision Transformer Adapter for Dense Predictions

Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation

BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Masked Autoencoders Are Scalable Vision Learners

Probabilistic Human-like Gesture Synthesis from Speech using GRU-based WGAN

Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates

Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders

Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanism

Double-DCCCAE: Estimation of Body Gestures From Speech Waveform

ExpressGesture: Expressive gesture generation from speech through database matching

MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

AST: Audio Spectrogram Transformer

Learning Speech-driven 3D Conversational Gestures from Video

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

Modeling the Conditional Distribution of Co-Speech Upper Body Gesture Jointly Using Conditional-GAN and Unrolled-GAN

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Speech gesture generation from the trimodal context of text, audio, and speaker identity

Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach

Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics

Moving Fast and Slow: Analysis of Representations and Post-Processing in Speech-Driven Automatic Gesture Generation

Reinforcement Learning based Neural Architecture Search for Audio Tagging

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Adversarial gesture generation with realistic gesture phasing

Skeleton-aware networks for deep motion retargeting

Style‐Controllable Speech‐Driven Gesture Synthesis Using Normalising Flows

EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding

Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Learning Individual Styles of Conversational Gesture

Capture, Learning, and Synthesis of 3D Speaking Styles

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

AMASS: Archive of Motion Capture As Surface Shapes

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Investigating the use of recurrent motion modelling for speech gesture generation

Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Learning a model of facial shape and expression from 4D scans

Neural Discrete Representation Learning

Enriching Word Vectors with Subword Information

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Direct Iterative Closest Point for real-time visual odometry

A 3-D Audio-Visual Corpus of Affective Communication

Pix4Point: Image Pretrained Transformers for 3D Point Cloud Understanding

ARK-itscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Spatio-temporal priors in 3d human motion

Vicon study of dynamic object tracking accuracy