ObamaNet: Photo-realistic lip-sync from text
Published in
arXiv.org(2017)External Links:
TL
TL;DR
The first architecture that generates both audio and synchronized photo-realistic lip-sync videos from any new text is presented, and it is claimed that this architecture is the first to be composed of fully trainable neural modules.
Abstract
We present ObamaNet, the first architecture that generates both audio and synchronized photo-realistic lip-sync videos from any new text. Contrary to other published lip-sync approaches, ours is only composed of fully trainable neural modules and does not rely on any traditional computer graphics methods. More precisely, we use three main modules: a text-to-speech network based on Char2Wav, a time-delayed LSTM to generate mouth-keypoints synced to the audio, and a network based on Pix2Pix to generate the video frames conditioned on the keypoints.
References6 items
1
Audio-driven facial animation by joint end-to-end learning of pose and emotion
2
Char2Wav: End-to-End Speech Synthesis
3
Image-to-Image Translation with Conditional Adversarial Networks
4
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
5