This Real-Time Face Reenactment Can Make Youtube Videos Say Anything

While making characters talk artificially has been around for years now, doing it in real-time is something very novel. Doing that with just an off-the-shelf webcam is quite remarkable.

Face2Face is a project by Matthias Nießner and others, that can take common Youtube video interviews and change what the famous celebrity, politician or other talking head is saying.

The system works by filming an input subject making facial expressions, the system then creates a 3D transfer overlay from that input. The final magic happens when that overlay is applied to an interview video. Moving the outside of the face is fine, but without teeth and a tongue this wouldn’t convince anyone, so the team has added convincing reconstruction of the mouth interior to their bag of tricks too. Have a look:

Here’s a bit more detail from Matthias Nießner at Stanford:

We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.

Screen Shot 2016-03-23 at 9.19.27 am

Screen Shot 2016-03-23 at 9.21.07 am

Share this Story