We propose an image-based, facial reenactment system that replaces the face of an actor in an existing target video with the face of a user from a source video, while preserving the original target performance. Our system is fully automatic and does not require a database of source expressions. Instead, it is able to produce convincing reenactment results from a short source video captured with an off-the-shelf camera, such as a webcam, where the user performs arbitrary facial gestures. Our reenactment pipeline is conceived as part image retrieval and part face transfer: The image retrieval is based on temporal clustering of target frames and a novel image matching metric that combines appearance and motion to select candidate frames from the source video, while the face transfer uses a 2D warping strategy that preserves the user's identity. Our system excels in simplicity as it does not rely on a 3D face model, it is robust under head motion and does not require the source and target performance to be similar. We show convincing reenactment results for videos that we recorded ourselves and for low-quality footage taken from the Internet.
@cite_6: The authors present a model-based algorithm that estimates 3D motion and facial expressions from 2D image sequences showing head and shoulder scenes typical of video telephone and teleconferencing applications.
@cite_24: We present a complete integrated system for live facial puppetry that enables high-resolution real-time facial expression tracking with transfer to another person's face. The system utilizes a real-time structured light scanner that provides dense 3D data and texture. A generic template mesh, fitted to a rigid reconstruction of the actor's face, is tracked offline in a training stage through a set of expression sequences. These sequences are used to build a person-specific linear face model that is subsequently used for online face tracking and expression transfer. Even with just a single rigid pose of the target face, convincing real-time facial animations are achievable. The actor becomes a puppeteer with complete and accurate control over a digital face.
@cite_23: 
@cite_5: This paper presents a system for performance-based character animation that enables any user to control the facial expressions of a digital avatar in realtime. The user is recorded in a natural environment using a non-intrusive, commercially available 3D sensor. The simplicity of this acquisition device comes at the cost of high noise levels in the acquired data. To effectively map low-quality 2D images and 3D depth maps to realistic facial expressions, we introduce a novel face tracking algorithm that combines geometry and texture registration with pre-recorded animation priors in a single optimization. Formulated as a maximum a posteriori estimation in a reduced parameter space, our method implicitly exploits temporal coherence to stabilize the tracking. We demonstrate that compelling 3D facial dynamics can be reconstructed in realtime without the use of face markers, intrusive lighting, or complex scanning hardware. This makes our system easy to deploy and facilitates a range of new applications, e.g. in digital gameplay or social interactions.
@cite_12: Face Transfer is a method for mapping videorecorded performances of one individual to facial animations of another. It extracts visemes (speech-related mouth articulations), expressions, and three-dimensional (3D) pose from monocular video or film footage. These parameters are then used to generate and drive a detailed 3D textured face mesh for a target identity, which can be seamlessly rendered back into target footage. The underlying face model automatically adjusts for how the target performs facial expressions and visemes. The performance data can be easily edited to change the visemes, expressions, pose, or even the identity of the target---the attributes are separably controllable. This supports a wide variety of video rewrite and puppetry applications.Face Transfer is based on a multilinear model of 3D face meshes that separably parameterizes the space of geometric variations due to different attributes (e.g., identity, expression, and viseme). Separability means that each of these attributes can be independently varied. A multilinear model can be estimated from a Cartesian product of examples (identities × expressions × visemes) with techniques from statistical analysis, but only after careful preprocessing of the geometric data set to secure one-to-one correspondence, to minimize cross-coupling artifacts, and to fill in any missing examples. Face Transfer offers new solutions to these problems and links the estimated model with a face-tracking algorithm to extract pose, expression, and viseme parameters.
@cite_20: This course describes how high-resolution face scanning, advanced character rigging, and performance-driven facial animation were combined to create Digital Emily, a believably photorealistic digital actor. Actress Emily O'Brien was scanned in the USC ICT light stage in 35 different facial poses using a new high-resolution face-scanning process capable of capturing geometry and textures down to the level of skin pores and fine wrinkles. These scans were assembled into a rigged digital character, which could then be driven by Image Metrics video-based facial animation technology. The real Emily was captured speaking on a small set, and her movements were used to drive a complete digital face replacement of her character, including its diffuse, specular, and animated displacement maps. HDRI lighting reconstruction techniques were used to reproduce the lighting on her original performance. The most recent results show new real-time animation and rendering research for the Digital Emily character.
@cite_17: We present a method for replacing facial performances in video. Our approach accounts for differences in identity, visual appearance, speech, and timing between source and target videos. Unlike prior work, it does not require substantial manual operation or complex acquisition hardware, only single-camera video. We use a 3D multilinear model to track the facial performance in both videos. Using the corresponding 3D geometry, we warp the source to the target face and retime the source to match the target performance. We then compute an optimal seam through the video volume that maintains temporal consistency in the final composite. We showcase the use of our method on a variety of examples and present the result of a user study that suggests our results are difficult to distinguish from real video footage.
A first type of methods tracks a morphable 3D model of the face that parameterizes identity, facial expressions and other nuances. Such systems can produce accurate 3D textured meshes and can establish a one-to-one expression mapping between source user and target actor, thereby simplifying and speeding up expression transfer. The generation of such a model, however, can be time consuming and is either done by learning a detailed 3D multilinear model from example data spanning a large variety of identities and expressions @cite_12 @cite_17 , or by purposely building a person-specific blend shape model from scans of a specific actor using specialized hardware @cite_6 @cite_23 @cite_20 @cite_24 @cite_5 . Moreover, the difficulty of stably tracking a 3D model over time generally necessitates a fair amount of manual interaction.