中文文档 ｜ Tutorial Video (coming soon)

❓ What is UniMoCap?

UniMoCap is a community implementation to unify the text-motion mocap datasets. In this repository, we unify the AMASS-based text-motion datasets (HumanML3D, BABEL, and KIT-ML). We support to process the AMASS data to both :

• body-only H3D-format (263-dim, 24 joints)
• whole-body SMPL-X-format (322-dim SMPL-X parameters).

We believe this repository will be useful for training models on larger mocap text-motion data. We will support more T-M mocap datasets in near feature.

We make the data processing as simple as possible. For those who are not familiar with the datasets, we will provide a video tutorial to tell you how to do it in the following weeks. This is a community implementation to support text-motion datasets. For the Chinese community, we provide a Chinese document (中文文档) for users.

🏃🏼 TODO List

• Support SMPL-X motion representation (including position, velocity, and rotations for body and hands) calculation (expected in 1-2 week).
• Support seg only and seq only BABEL unifier.
• Provide a tutorial video.
• Support more language documents.
• Support more T-M datasets (e.g.: FLAG3D, STDM). Welcome to support your own dataset here!
• Provide trained models based on UniMoCap.

🛠️ Installation

pip install -r requirements.txt

🚀 How to use?

1. Data Preparing

./body_models
├── dmpls
│   ├── female
│   │   └── model.npz
│   ├── male
│   │   └── model.npz
│   └── neutral
│       └── model.npz
├── smplh
│   ├── female
│   │   └── model.npz
│   ├── info.txt
│   ├── male
│   │   └── model.npz
│   └── neutral
│       └── model.npz
├── smplx
│   ├── female
│   │   ├── model.npz
│   │   └── model.pkl
│   ├── male
│   │   ├── model.npz
│   │   └── model.pkl
│   └── neutral
│       ├── model.npz
└───────└── model.pkl

./datasets/amass_data/
├── ACCAD
├── BioMotionLab_NTroje
├── BMLhandball
├── BMLmovi
├── CMU
├── DanceDB
├── DFaust_67
├── EKUT
├── Eyes_Japan_Dataset
├── GRAB
├── HUMAN4D
├── HumanEva
├── KIT
├── MPI_HDM05
├── MPI_Limits
├── MPI_mosh
├── SFU
├── SOMA
├── SSM_synced
├── TCD_handMocap
├── TotalCapture
└── Transitions_mocap

mkdir datasets
cd datasets
git clone https://github.com/EricGuo5513/HumanML3D/tree/main
mkdir pose_data
unzip HumanML3D/pose_data/humanact12.zip -d pose_data/
mv pose_data/humanact12/humanact12/*.npy pose_data/humanact12/
rm -rf pose_data/humanact12/humanact12
cd HumanML3D/HumanML3D
unzip texts.zip
cd ../../..

2. Generate mapping files and text files

In this step, we will get mapping files (.csv) and text files (./{dataset}_new_text).

python h3d_to_h3d.py

python babel.py

python kitml.py

3. Extract and Process Data

In this step, we follow the method in HumanML3D to extract motion feature.

Now, you are at the root position of ./UniMoCap. To generated body-only motion feature, we create a folder and copy some tools provided by HumanML3D first:

mkdir body-only-unimocap
cp -r ./datasets/HumanML3D/common ./
cp -r ./datasets/HumanML3D/human_body_prior ./
cp -r ./datasets/HumanML3D/paramUtil.py ./

If you would like to get body-only motions, please refer to Body-only data preprocessing.

If you would like to get whole-body motions, please refer to Whole-body motion processing.

💖 Opening for Community Contributions

We sincerely wish the community to support more text-motion mocap datasets.

🌹 Acknowledgement

Our code is modified on the basis of TMR, AMASS-Annotation-Unifier, and HumanML3D, thanks to all contributors!

Star History

🤝🏼 Citation

If you use this repository for research, you need to cite:

@article{chen2023unimocap,
  title={UniMocap: Unifier for BABEL, HumanML3D, and KIT},
  author={Chen, Ling-Hao and UniMocap, Contributors},
  journal={https://github.com/LinghaoChan/UniMoCap},
  year={2023}
}

As some components of UniMoCap are borrowed from AMASS-Annotation-Unifier and HumanML3D. You need to cite them accordingly.

@inproceedings{petrovich23tmr,
    title     = {{TMR}: Text-to-Motion Retrieval Using Contrastive {3D} Human Motion Synthesis},
    author    = {Petrovich, Mathis and Black, Michael J. and Varol, G{\"u}l},
    booktitle = {International Conference on Computer Vision ({ICCV})},
    year      = {2023}
}

@InProceedings{Guo_2022_CVPR,
    author    = {Guo, Chuan and Zou, Shihao and Zuo, Xinxin and Wang, Sen and Ji, Wei and Li, Xingyu and Cheng, Li},
    title     = {Generating Diverse and Natural 3D Human Motions From Text},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {5152-5161}
}

If you use the dataset, you need to cite subset KIT-ML and AMASS.

@article{Plappert2016,
    author = {Matthias Plappert and Christian Mandery and Tamim Asfour},
    title = {The {KIT} Motion-Language Dataset},
    journal = {Big Data}
    publisher = {Mary Ann Liebert Inc},
    year = {2016},
    month = {dec},
    volume = {4},
    number = {4},
    pages = {236--252}
}

@conference{AMASS2019,
  title = {AMASS: Archive of Motion Capture as Surface Shapes},
  author = {Mahmood, Naureen and Ghorbani, Nima and Troje, Nikolaus F. and Pons-Moll, Gerard and Black, Michael J.},
  booktitle = {International Conference on Computer Vision},
  pages = {5442--5451},
  month = oct,
  year = {2019},
  month_numeric = {10}
}

If you use the Motion-X dataset, please cite it accordingly.

@article{lin2023motionx,
  title={Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset},
  author={Lin, Jing and Zeng, Ailing and Lu, Shunlin and Cai, Yuanhao and Zhang, Ruimao and Wang, Haoqian and Zhang, Lei},
  journal={Advances in Neural Information Processing Systems},
  year={2023}
}

If you have any question, please contact Ling-Hao CHEN (thu [DOT] lhchen [AT] gmail [DOT] com).