GeezerVison is showing confies.github.io — CoNFies: Controllable Neural Face Avatars
CoNFies: Controllable Neural Face Avatars
(FG 2023)
Abstract
Neural Radiance Fields (NeRF) are compelling tech- niques for modeling dynamic 3D scenes from 2D image collections.
These volumetric representations would be well suited for synthesizing novel facial expressions but for two problems.
First, deformable NeRFs are object agnostic and model holistic movement of the scene: they can replay how the motion changes
over time, but they cannot alter it in an interpretable way. Second, control- lable volumetric representations typically require
either time-consuming manual annotations or 3D supervision to provide semantic meaning to the scene. We propose a controllable
neural representation for face self-portraits (CoNFies), that solves both of these problems within a common framework, and it
can rely on automated processing. We use automated facial action recognition (AFAR) to characterize facial expressions as a
combi- nation of action units (AU) and their intensities. AUs provide both the semantic locations and control labels for the
system. CoNFies outperformed competing methods for novel view and expression synthesis in terms of visual and anatomic fidelity
of expressions.
Localized Control Results
Control eye opening (Left: CoNeRF. Right: ours.)
Control mouth opening (Left: CoNeRF. Right: ours.)
Citation
@article{yu2022confies,
title={CoNFies: Controllable Neural Face Avatars},
author={Yu, Heng and Niinuma, Koichiro and Jeni, Laszlo A},
journal={arXiv preprint arXiv:2211.08610},
year={2022}
}
Acknowledgements
This research was supported by Fujitsu. We thank Joel Julin from University of Pittsburgh for helping with data collection
and comments that greatly improved the manuscript. We thank Xuxin Cheng from Carnegie Mellon University who provided the data
acquisition equipment. We would also like to show our gratitude to Nian-Hsuan Tsai from Carnegie Mellon University who helped
build the OpenFace system.
