Preprocessing to evaluate 3D face reconstruction from 2D image
The project's aim is to compare the results obtained with the pretrained Extreme 3D Faces CNN (https://github.com/anhttran/extreme_3d_faces) [1] and a ground truth using a well-known dataset (http://kunzhou.net/zjugaps/facewarehouse/) [2] to measure the similarity between the two models and, hence, estimate the performance of the net. The aligment phase is realized with cpd rigid algorithm [3].
To build and run the files, you will need the following libraries:
To run the whole project, you are required to follow some steps. The parameters of each function will be explained in the next subsection.
.ply
format (in this case, you will also
need to adapt the next steps)..ply
files of the network in the filePlyRete
folder.faceWarehouse
folder.plyTotxt()
function in plyConverter.py
choosing the right values of
the parameters. This will convert the .ply
files
in a bunch of .txt
(stored in the data
folder) files in order to make the
next steps easier.MatTotxt()
function in the matConverter.py
file. This will convert
the dataset's .mat
3D models in a .txt
format and store them in a groudtruth
folder.distancesTest()
in test.py
to find out how much the two models are
similar. If you want an overview of the accuracy of the different poses, you can
also run posesPrecision()
.open3Dheatmap()
function
in the heatmap.py
file.plyTotxt()
: directory
stands for the directory where you can find the .ply
files. If you follow the steps above, this should be set to 'filePlyRete/'
;
compressionLevel
stands for the sampling parameter K
of the KNNSearch algorithm:
since a single model has over 170.000 points, you will probably need to sample
the model (depending on your hardware). If you have a single-node cluster, we
recommend you to leave this parameter to its default value; radius
stands for
the maximum distance (from the nose tip) of the useful points: the CNN we've
used reconstructs the ears, while the ground truth models have not also those
points so, to achieve a better alignment, we've focused just on the face setting this
parameter properly.MatTotxt()
: same as plyTotxt()
, but with ground truth Matlab files.distancesTest()
: the FaceWarehouse dataset is structured this way: there are 150
candidates and 20 poses for each candidate. This means we can choose a subset of the
candidates and also a subset of poses to run the test. testers
and poses
are
two lists in which you just need to put the number of the testers and the poses you
want to test.posesPrecision()
: same as distancesTest()
open3Dheatmap()
: target
stands for the ground truth .txt
model file path,
while source
stands for the CNN .txt
model file path.[1] A. Tran, T. Hassner, I. Masi, E. Paz, Y. Nirkin, G. Medioni, Extreme 3D Face Reconstruction: Seeing Through Occlusions, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[2] Cao Chen, Yanlin Weng, Shun Zhou, Yiying Tong, Kun Zhou: "FaceWarehouse: a 3D Facial Expression Database for Visual Computing", IEEE Transactions on Visualization and Computer Graphics, 20(3): 413-425, 2014.
[3] Andriy Myronenko and Xubo Song, Point Set Registration: Coherent Point Drift, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 32, issue 12, pp. 2262-2275, 2009.