You Said That?: Synthesising Talking Faces from Audio (2019-02-13T00:00:00.000000Z)

TL;DR

An encoder–decoder convolutional neural network model is developed that uses a joint embedding of the face and audio to generate synthesised talking face video frames and proposed methods to re-dub videos by visually blending the generated face into the source video frame using a multi-stream CNN model.

Abstract

We describe a method for generating a video of a talking face. The method takes still images of the target face and an audio speech segment as inputs, and generates a video of the target face lip synched with the audio. The method runs in real time and is applicable to faces and audio not seen at training time. To achieve this we develop an encoder–decoder convolutional neural network (CNN) model that uses a joint embedding of the face and audio to generate synthesised talking face video frames. The model is trained on unlabelled videos using cross-modal self-supervision. We also propose methods to re-dub videos by visually blending the generated face into the source video frame using a multi-stream CNN model.

Authors

Andrew Zisserman

62 papers

Joon Son Chung

11 papers

A. Jamaludin

1 papers

TL;DR

Abstract

Authors

References52 items

Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation

VoxCeleb2: Deep Speaker Recognition

Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching

Photographic Image Synthesis with Cascaded Refinement Networks

Audio-driven facial animation by joint end-to-end learning of pose and emotion

A deep learning approach for generalized speech animation

Synthesizing Obama

Unsupervised Learning of Disentangled Representations from Video

Look, Listen and Learn

You said that?

Image-to-Image Translation with Conditional Adversarial Networks

Out of Time: Automated Lip Sync in the Wild

Lip Reading Sentences in the Wild

SoundNet: Learning Sound Representations from Unlabeled Video

Virtual Immortality: Reanimating Characters from TV Shows

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Image Style Transfer Using Convolutional Neural Networks

Conditional Image Generation with PixelCNN Decoders

Generative Adversarial Text to Image Synthesis

Context Encoders: Feature Learning by Inpainting

Colorful Image Colorization

Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification

Pixel Recurrent Neural Networks

Visually Indicated Sounds

Learning visual groups from co-occurrences in space and time

Spatio-temporal video autoencoder with differentiable memory

Accurate Image Super-Resolution Using Very Deep Convolutional Networks

Unsupervised Visual Representation Learning by Context Prediction

U-Net: Convolutional Networks for Biomedical Image Segmentation

Unsupervised Learning of Visual Representations Using Videos

VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track

Photo-real talking head with deep bidirectional LSTM

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

MatConvNet: Convolutional Neural Networks for MATLAB

Deep visual-semantic alignments for generating image descriptions

Show and tell: A neural image caption generator

One millisecond face alignment with an ensemble of regression trees

Generative Adversarial Nets

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Dlib-ml: A Machine Learning Toolkit

Poisson image editing

Reliable Transition Detection in Videos: A Survey and Practitioner's Guide

Visual Speech Synthesis by Morphing Visemes

An Iterative Image Registration Technique with an Application to Stereo Vision

GENERATIVE ADVERSARIAL NETS

Accurate image superresolution using very deep convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition

Deep Face Recognition

Features and methods for improving large scale face recognition

Phoneme-to-viseme Mapping for Visual Speech Recognition

This PDF file includes: Materials and Methods

Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

Self-Supervised Video Representation Learning with Odd-One-Out Networks

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names