VirtualConductor: Music-driven Conducting Video Generation System
Published in
arXiv.org(2021)External Links:
TL
TL;DR
This work combines 3D animation rendering and a pose transfer model to synthesize conducting video from a single given user's image and proposes Audio Motion Correspondence Network and adversarial-perceptual learning to learn the cross-modal relationship and generate diverse, plausible, music-synchronized motion.
Abstract
In this demo, we present VirtualConductor, a system that can generate conducting video from any given music and a single user's image. First, a large-scale conductor motion dataset is collected and constructed. Then, we propose Audio Motion Correspondence Network (AMCNet) and adversarial-perceptual learning to learn the cross-modal relationship and generate diverse, plausible, music-synchronized motion. Finally, we combine 3D animation rendering and a pose transfer model to synthesize conducting video from a single given user's image. Therefore, any user can become a virtual conductor through the system.
References4 items
1
Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis
2
Deep Collaborative Embedding for Social Image Understanding
3
RMPE: Regional Multi-person Pose Estimation
4