Estimate absolute 3D human poses from RGB images.
This repository contains code for the following paper:
MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation
by István Sárándi, Timm Linder, Kai O. Arras, Bastian Leibe
IEEE Transactions on Biometrics, Behavior, and Identity Science (T-BIOM), Selected Best Works From
Automated Face and Gesture Recognition 2020.
The repo has been updated to an improved version employed in the following paper:
Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats
by István Sárándi, Alexander Hermans, Bastian Leibe
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
We release standalone TensorFlow models (SavedModel) to allow easy application in downstream research. After loading the model, you can run inference in a single line of Python without having this codebase as a dependency. Try it in action in Google Colab.
import tensorflow as tf
import tensorflow_hub as tfhub
model = tfhub.load('https://bit.ly/metrabs_l')
image = tf.image.decode_jpeg(tf.io.read_file('img/test_image_3dpw.jpg'))
pred = model.detect_poses(image)
pred['boxes'], pred['poses2d'], pred['poses3d']
See also the demos folder for more examples.
NOTE: The models can only be used for non-commercial purposes due to the licensing of the used training datasets.
Alternatively, you can try the experimental PyTorch version:
wget -O - https://bit.ly/metrabs_l_pt | tar -xzvf -
python -m metrabs_pytorch.scripts.demo_image --model-dir metrabs_eff2l_384px_800k_28ds_pytorch --image img/test_image_3dpw.jpg
./demo.py
to auto-download the model, predict on a sample image and display the
result with Matplotlib or PoseViz (if installed)../demo_video.py
filepath-or-url-to-video.mp4
to run inference on a video.skeleton
(e.g. COCO,
SMPL, H36M)num_aug
sets the number of TTA crops per detection)See the docs directory.
If you find this work useful in your research, please cite it as:
@article{sarandi2021metrabs,
title={{MeTRAbs:} Metric-Scale Truncation-Robust Heatmaps for Absolute 3{D} Human Pose Estimation},
author={S\'ar\'andi, Istv\'an and Linder, Timm and Arras, Kai O. and Leibe, Bastian},
journal={IEEE Transactions on Biometrics, Behavior, and Identity Science},
year={2021},
volume={3},
number={1},
pages={16-30},
doi={10.1109/TBIOM.2020.3037257}
}
The above paper is an extended journal version of the FG'2020 conference paper:
@inproceedings{Sarandi20FG,
title={Metric-Scale Truncation-Robust Heatmaps for 3{D} Human Pose Estimation},
author={S\'ar\'andi, Istv\'an and Linder, Timm and Arras, Kai O. and Leibe, Bastian},
booktitle={IEEE International Conference on Automatic Face and Gesture Recognition},
pages={677-684},
year={2020}
}
The newer large-scale models correspond to the WACV'23 paper:
@inproceedings{Sarandi2023dozens,
author = {S\'ar\'andi, Istv\'an and Hermans, Alexander and Leibe, Bastian},
title = {Learning {3D} Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats},
booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year = {2023}
}
Code in this repository was written by István Sárándi (RWTH Aachen University) unless indicated otherwise.
Got any questions or feedback? Drop a mail to [email protected]!