SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos (ECCV 2022)

This repo is the official implementation of "SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos". [Paper] [Project]

Update

• Support SmoothNet in MMPose Release v0.25.0 and MMHuman3D as a smoothing strategy!

• Clean version is released!

• To further improve SmoothNet as a near online smoothing strategy, we reduce the original window size 64 to 32 frames by default!

• We also provide the pretrained models with the window size 8, 16, 32 and 64 frames here.

It currently includes code, data, log and models for the following tasks:

• 2D human pose estimation
• 3D human pose estimation
• Body recovery via a SMPL model

Major Features

• Model training and evaluation for 2D pose, 3D pose, and SMPL body representation
• Supporting 6 popular datasets (AIST++, Human3.6M, Sub-JHMDB, MPI-INF-3DHP, MuPoTS-3D, 3DPW) and providing cleaned estimation results of 13 popular pose estimation backbones(SPIN, TCMR, VIBE, CPN, FCN, Hourglass, HRNet, RLE, VideoPose3D, TposeNet, EFT, PARE, SimplePose)

Description

When analyzing human motion videos, the output jitters from existing pose estimators are highly-unbalanced with varied estimation errors across frames. Most frames in a video are relatively easy to estimate and only suffer from slight jitters. In contrast, for rarely seen or occluded actions, the estimated positions of multiple joints largely deviate from the ground truth values for a consecutive sequence of frames, rendering significant jitters on them.

To tackle this problem, we propose to attach a dedicated temporal-only refinement network to existing pose estimators for jitter mitigation, named SmoothNet. Unlike existing learning-based solutions that employ spatio-temporal models to co-optimize per-frame precision and temporal smoothness at all the joints, SmoothNet models the natural smoothness characteristics in body movements by learning the long-range temporal relations of every joint without considering the noisy correlations among joints. With a simple yet effective motion-aware fully-connected network, SmoothNet improves the temporal smoothness of existing pose estimators significantly and enhances the estimation accuracy of those challenging frames as a side-effect. Moreover, as a temporal-only model, a unique advantage of SmoothNet is its strong transferability across various types of estimators and datasets. Comprehensive experiments on five datasets with eleven popular backbone networks across 2D and 3D pose estimation and body recovery tasks demonstrate the efficacy of the proposed solution. Our code and datasets are provided in the supplementary materials.

Results

SmoothNet is a plug-and-play post-processing network to smooth any outputs of existing pose estimators. To fit well across datasets, backbones, and modalities with lower MPJPE and PA-MPJPE, we provide THREE pre-trained models (Train on AIST-VIBE-3D, 3DPW-SPIN-3D, and H36M-FCN-3D) to handle all existing issues.

Please refer to our supplementary materials to check the cross-model validation in detail. Noted that all models can obtain lower and similar Accels than the compared backbone estimators. The differences are in MPJPEs and PA-MPJPEs.

Due to the temporal-only network without spatial modelings, SmoothNet is trained on 3D position representations only, and can be tested on 2D, 3D, and 6D representations, respectively.

3D Keypoint Results

2D Keypoint Results

SMPL Results

• * means the used pose estimators are using temporal information.
• The usage of SmoothNet for better performance: a SOTA single-frame estimator (e.g., PARE) + SmoothNet
• Since TCMR uses a sliding window method to smooth the poses, which causes over-smoothness issue, SmoothNet will be hard to further decrease the MPJPE, PA-MPJPE.

Getting Started

Environment Requirement

SmoothNet has been implemented and tested on Pytorch 1.10.1 with python >= 3.6. It supports both GPU and CPU inference.

Clone the repo:

git clone https://github.com/cure-lab/SmoothNet.git

We recommend you prepare the environment using conda:

# conda
source scripts/install_conda.sh

Prepare Data

All the data used in our experiment can be downloaded here.

Google Drive

Baidu Netdisk

The sructure of the repository should look like this:

|-- configs
    |-- aist_vibe_3D.yaml
    |-- ...
|-- data
    |-- checkpoints         # pretrained checkpoints
    |-- poses               # cleaned detected poses and groundtruth poses
    |-- smpl                # SMPL parameters
|-- lib
    |-- core
        |-- ...
    |-- dataset
        |-- ...
    |-- models
        |-- ...
    |-- utils
        |-- ...
|-- results                 # folders including log files, checkpoints, running config and tensorboard logs
|-- scripts
    |-- install_conda.sh
|-- eval_smoothnet.py       # SmoothNet evaluation
|-- train_smoothnet.py      # SmoothNet training
|-- README.md
|-- LICENSE
|-- requirements.txt

If you want to add your own dataset, please follow these steps (noted that this is also how the provided data is organized):

1. Organize your data into corresponding type according to the body representation. The file structure is shown as follows:

   |-- [your dataset]\_[estimator]\_[2D/3D/smpl]
       |-- detected
           |-- [your dataset]\_[estimator]\_[2D/3D/smpl]_test.npz
           |-- [your dataset]\_[estimator]\_[2D/3D/smpl]_train.npz
       |-- groundtruth
           |-- [your dataset]\_gt\_[2D/3D/smpl]_test.npz
           |-- [your dataset]\_gt\_[2D/3D/smpl]_train.npz
   

   It is fine if you only have training or testing data. The content in each .npz file is consist of "imgname" and "human poses", which is related to the body representation you use.

   • 3D keypoints:

     • imgname: Strings containing the image and sequence name with format [sequence_name]/[image_name](string "" if the sequence_name and image_name not available).
     • keypoints_3d: 3D joint position. The shape of each sequence is corresponding_sequence_length*(keypoints_number*3). The order of it is the same with imgname

   • 2D keypoints

     • imgname: Strings containing the image and sequence name with format [sequence_name]/[image_name](string "" if the sequence_name and image_name not available).
     • keypoints_2d: 2D joint position. The shape of each sequence is corresponding_sequence_length*(keypoints_number*2). The order of it is the same with imgname

   • SMPL

     • imgname: Strings containing the image and sequence name with format [sequence_name]/[image_name](string "" if the sequence_name and image_name not available).
     • pose: pose parameters. The shape of each sequence is corresponding_sequence_length*72. The order of it is the same with imgname
     • shape: shape parameters. The shape of each sequence is corresponding_sequence_length*10. The order of it is the same with imgname

2. If you use 3D keypoints as the body representation, add the root of all keypoints cfg.DATASET.ROOT_[your dataset]_[estimator]_3D in evaluate_config.py, train_config.py or visualize_config.py according to your purpose(test, train or visualize).

3. Construct your own dataset following the existing dataset files. You might need to modify the detailed implementation depending on your data characteristics.

Training

Run the commands below to start training:

python train_smoothnet.py --cfg [config file] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size]

For example, you can train on 3D representation of Human3.6M using backbone estimator FCN with silde window size 8 by:

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name h36m --estimator fcn --body_representation 3D --slide_window_size 8

You can easily train on multiple datasets using "," to split multiple datasets / estimator / body representation. For example, you can train on AIST++ - VIBE - 3D and 3DPW - SPIN - 3D with silde window size 8 by:

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name aist,pw3d --estimator vibe,spin --body_representation 3D,3D  --slide_window_size 8

Note that the training and testing datasets should be downloaded and prepared before training.

Evaluation

Run the commands below to start evaluation:

python eval_smoothnet.py --cfg [config file] --checkpoint [pretrained checkpoint] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size] --tradition [savgol/oneeuro/gaus1d]

For example, you can evaluate MPI-INF-3DHP - TCMR - 3D and MPI-INF-3DHP - VIBE - 3D using SmoothNet trained on 3DPW - SPIN - 3D with silde window size 8, and compare the results with traditional filters oneeuro by:

python eval_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoint_8.pth.tar --dataset_name mpiinf3dhp,mpiinf3dhp --estimator tcmr,vibe --body_representation 3D,3D --slide_window_size 8 --tradition oneeuro

Note that the pretrained checkpoints and testing datasets should be downloaded and prepared before evaluation.

The data and checkpoints used in our experiment can be downloaded here.

Google Drive

Baidu Netdisk

Visualization

Here, we only provide demo visualization based on offline processed detected poses of specific datasets(e.g. AIST++, Human3.6M, and 3DPW). To visualize on arbitrary given video, please refer to the inference/demo of MMHuman3D.

un the commands below to start evaluation:

python visualize_smoothnet.py --cfg [config file] --checkpoint [pretrained checkpoint] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size] --visualize_video_id [visualize sequence id] --output_video_path [visualization output video path]

For example, you can visualize the second sequence of 3DPW - SPIN - 3D using SmoothNet trained on 3DPW - SPIN - 3D with silde window size 32, and output the video to ./visualize by:

python visualize_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoint_8.pth.tar --dataset_name pw3d --estimator spin --body_representation 3D --slide_window_size 32 --visualize_video_id 2 --output_video_path ./visualize

Citing SmoothNet

If you find this repository useful for your work, please consider citing it as follows:

@inproceedings{zeng2022smoothnet,
      title={SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos},
      author={Zeng, Ailing and Yang, Lei and Ju, Xuan and Li, Jiefeng and Wang, Jianyi and Xu, Qiang},
      booktitle={European Conference on Computer Vision},
      year={2022},
      organization={Springer}
}

Please remember to cite all the datasets and backbone estimators if you use them in your experiments.

License

This code is available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using this code you agree to the terms in the LICENSE. Third-party datasets and software are subject to their respective licenses.