AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild":

Published in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020)

Public Dataset: RealFaceDB

AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs:

Published in IEEE Transaction on Pattern Analysis and Machine Intelligence (TPAMI 2021)

Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Baris Gecer,
Abhijeet Ghosh, Stefanos Zafeiriou
Imperial College London

Overview

AvatarMe++ is a new method for reconstructing renderable photorealistic 3D faces from a single ‘in-the-wild”. Extending on AvatarMe and our public dataset RealFaceDB, we define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method.

Method

Summary of the AvatarMe++ method: We acquire a 3DMM fitting and upscale its texture using a state-of-the-art super resolution network, trained on synthetic data rendered in the texture’s domain. A image-translation network is then used to transform the upscaled texture and normals to reflectance maps, namely the Diffuse Albedo, Specular Albedo, Diffuse Normals and Specular Normals.

We create a rendering module, with subsurface-scattering and self-occlusion approximation. During training, it is used to create synthetic data pairs, by rendering the captured data in the target’s environment and random ones, which are used for photorealistic rendering loss. The complete high resolution (up to 6k×4k) BRDF maps can be used for photorealistic rendering, while the specular normals can be used to enhance the original geometry with fine details.

Results

Our high resolution mesh and reflectance textures (Diffuse Albedo, Specular Albedo, Diffuse Normals, Specular Normals) can be used with most open-source or commercial renderers to render the reconstructed subjects in various illumination environments.

Thanks to the photorealistic differentiable renderer and the stochastic rendering loss, in contrast to AvatarMe, AvatarMe++ can be generalize for most examples with frontal facing illumination. As shown above, AvatarMe++ can be effectively used in reconstructions with (Chen et al., 2019), (Gecer et al., 2021) and datasets such as Facescape, Superface and subjects captured with a 3dMD device.

Public Dataset

RealfaceDB can now be obtained by accredited researchers and used for training/testing methods similar to AvatarMe and AvatarMe++.

The dataset contains patches of facial reflectance as described in the paper, namely the diffuse albedo, diffuse normals, specular albedo, specular normals, as well as the shape in UV space. For the shape, reconstructed meshes have been registered to a common topology and the XYZ values of the points have been mapped to the RGB in UV coordinates and interpolated to complete the UV map. From the complete UV maps of 6144x4096 pixels, patches of 512x512 pixels have been sampled. The dataset contains 7500 such patches (1500 of each datatype) that are anonymized, randomized and sampled so that they do not contain identifiable features.

The AvatarMe++ paper contains a section on photorealistically rendering such data, using Pytorch3D. For any further questions, please contact us.

To obtain access to the dataset, please complete and sign our licence agreement, and forward it to the authors Alexandros Lattas ([email protected]) and Stylianos Moschoglou ([email protected]). The licence should be signed by a full-time academic staff member (not a student) and sent through your academic email. Please also include your name, position and academic page. We will verify your request and contact you regarding how to download the dataset. Note that the agreement requires that:

• The data must be used for non-commercial research and education purposes only.
• You agree not to copy, sell, trade, or exploit the model for any commercial purposes.
• You must destroy the data after 2 years since the first download.
• If you will be publishing any work using this dataset, please cite the following paper.

For any queries regarding commercial licenses for the dataset, please contacnt James Chan ([email protected]).

Please note that the above apply to the public dataset. The code and trained models have been commercialized and are not publicly available.

Citation

If you find this work useful in your research, please cite our papers AvatarMe, AvatarMe++ using:

@InProceedings{lattas2020avatarme,
  author = {Lattas, Alexandros and Moschoglou, Stylianos and Gecer, Baris and Ploumpis, Stylianos and Triantafyllou, Vasileios and Ghosh, Abhijeet and Zafeiriou, Stefanos},
  title = {AvatarMe: Realistically Renderable 3D Facial Reconstruction "In-the-Wild"},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2020}
}

@article{lattas2021avatarme++,
  title={AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs},
  author={Lattas, Alexandros and Moschoglou, Stylianos and Ploumpis, Stylianos and Gecer, Baris and Ghosh, Abhijeet and Zafeiriou, Stefanos P},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  publisher={IEEE}
}