Georgeretsi Smirk Save
Official Pytorch Implementation of SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024)
SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis
This repository is the official implementation of the CVPR 2024 paper 3D Facial Expressions through Analysis-by-Neural Synthesis.
SMIRK reconstructs 3D faces from monocular images with facial geometry that faithfully recover extreme, asymmetric, and subtle expressions.
Installation
You need to have a working version of PyTorch and Pytorch3D installed. We provide a requirements.txt file that can be used to install the necessary dependencies for a Python 3.9 setup with CUDA 11.7:
conda create -n smirk python=3.9
pip install -r requirements.txt
# install pytorch3d now
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py39_cu117_pyt201/download.html
Then, in order to download the required models, run:
bash quick_install.sh
The above installation includes downloading the FLAME model. This requires registration. If you do not have an account you can register at https://flame.is.tue.mpg.de/
Demo
We provide two demos. One that can be used to test the model on a single image,
python demo.py --input_path samples/test_image2.png --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt --crop
and one that can be used to test the model on a video,
python demo_video.py --input_path samples/dafoe.mp4 --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt --crop --render_orig
Training
Dataset Preparation
SMIRK was trained on a combination of the following datasets: LRS3, MEAD, CelebA, and FFHQ.
-
§§Download the LRS3 dataset from here.We are aware that currently this dataset has been removed from the website. It can be replaced with any other similar dataset, e.g. LRS2. -
Download the MEAD dataset from here.
-
Download the CelebA dataset from here. You can download directly the aligned images
img_align_celeba.zip. -
Download the FFHQ256 dataset from here.
After downloading the datasets we need to extract the landmarks using mediapipe and FAN. We provide the scripts for preprocessing in datasets/preprocess_scripts. Example usage:
python datasets/preprocess_scripts/apply_mediapipe_to_dataset.py --input_dir PATH_TO_FFHQ256/images --output_dir PATH_TO_FFHQ256/mediapipe_landmarks
and for FAN:
python datasets/preprocess_scripts/apply_fan_to_dataset.py --input_dir PATH_TO_FFHQ256/images --output_dir PATH_TO_FFHQ256/fan_landmarks
Note that for obtaining the FAN landmarks we use the implementation in https://github.com/hhj1897/face_alignment.
Next, make sure to update the config files in configs with the correct paths to the datasets and their landmarks.
Pretraining
At the pretraining stage, we train all 3 encoders (pose, shape, and expression) using only the extracted landmarks and the output of MICA.
python train.py configs/config_pretrain.yaml train.log_path="logs/pretrain"
Training
After pretraining, at the core stage of SMIRK, we freeze the shape and pose encoders and train the expression encoder with the full SMIRK framework (reconstruction path and cycle path).
python train.py configs/config_train.yaml resume=logs/pretrain/first_stage_pretrained_encoder.pt train.loss_weights.emotion_loss=1.0
Citation
If you find this work useful, please consider citing:
@inproceedings{SMIRK:CVPR:2024,
title = {3D Facial Expressions through Analysis-by-Neural-Synthesis},
author = {Retsinas, George and Filntisis, Panagiotis P., and Danecek, Radek and Abrevaya, Victoria F. and Roussos, Anastasios and Bolkart, Timo and Maragos, Petros},
booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2024}
}
Acknowledgements
We acknowledge the following repositories and papers that were used in this work:
Open Source Agenda Badge
