SAGA: Stochastic Whole-Body Grasping with Contact

SAGA: Stochastic Whole-Body Grasping with Contact
ECCV 2022
Yan Wu*, Jiahao Wang*, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, Siyu Tang

This repository is the official implementation for the ECCV 2022 paper: SAGA: Stochastic Whole-Body Grasping with Contact.

[Project Page | Paper]

Introduction

Given an object in 3D space and a human initial pose, we aim to generate diverse human motion sequences to approach and grasp the given object. We propose a two-stage pipeline to address this problem by generating grasping ending pose first and then infilling the in-between motion.

Input	First-stage result	Second-stage result

Contents

• Installation
• Dataset Preparation
• Pretrained models
• Train
• Grasping poses and motions generation for given object (object position and orientation can be customized)
• Visualization

Installation

• Packages

  • python>=3.8
  • pytorch==1.12.1
  • human-body-prior
  • SMPLX
  • Chamfer Distance
  • Open3D

• Body Models
  Download SMPL-X body model and vposer v1.0 model and put them under /body_utils/body_models folder as below:

SAGA
│
└───body_utils
    │
    └───body_models 
        │
        └───smplx
        │   └───SMPLX_FEMALE.npz
        │   └───...
        │   
        └───vposer_v1_0
        │   └───snapshots
        │       └───TR00_E096.pt
        │   └───...
        │
        └───VPoser
        │   └───vposerDecoderWeights.npz
        │   └───vposerEnccoderWeights.npz
        │   └───vposerMeanPose.npz
    │
    └───...
│
└───...

Dataset

Download GRAB object mesh

Download dataset for the first stage (GraspPose) from [Google Drive]

Download dataset for the second stage (GraspMotion) from [Google Drive]

Put them under /dataset as below,

SAGA
│
└───dataset 
    │
    └───GraspPose
    │   └───train
    │       └───s1
    │       └───...
    │   └───eval
    │       └───s1
    │       └───...
    │   └───test
    │       └───s1
    │       └───...
    │   
    └───GraspMotion
    │   └───Processed_Traj
    │   └───s1
    │   └───...
    │   
    └───contact_meshes
    │   └───airplane.ply
    │   └───...
│
└───... 

Pretrained models

Download pretrained models from [Google Drive], and the pretrained models include:

• Stage 1: pretrained WholeGrasp-VAE for male and female respectively
• Stage 2: pretrained TrajFill-VAE and LocalMotionFill-VAE (end to end)

Train

First Stage: WholeGrasp-VAE training

python train_grasppose.py --data_path ./dataset/GraspPose --gender male --exp_name male

Second Stage: MotionFill-VAE training

Can train TrajFill-VAE and LocalMotionFill-VAE separately first (download separately trained models from [Google Drive]), and then train them end-to-end:

python train_graspmotion.py --pretrained_path_traj $PRETRAINED_MODEL_PATH/TrajFill_model_separate_trained.pkl --pretrained_path_motion $PRETRAINED_MODEL_PATH/LocalMotionFill_model_separate_trained.pkl

Inference

First Stage: WholeGrasp-VAE sampling + GraspPose-Opt

At the first stage, we generate grasping poses for the given object.
The example command below generates 10 male pose samples to grasp camera, where the object's height and orientation are randomly set within a reasonable range. You can also easily customize your own setting accordingly.

python opt_grasppose.py --exp_name pretrained_male --gender male --pose_ckpt_path $PRETRAINED_MODEL_PATH/male_grasppose_model.pt --object camera --n_object_samples 10

Second Stage: MotionFill-VAE sampling + GraspMotion-Opt

At the second stage, with generated ending pose from the first stage and a customizable human initial pose, we generate in-between motions.
The example command below generates male motion samples to grasp camera, where the human initial pose and the initial distance away from the given object are randomly set within a reasonable range. You can also easily customize your own setting accordingly.

python opt_graspmotion.py --GraspPose_exp_name pretrained_male --object camera --gender male --traj_ckpt_path $PRETRAINED_MODEL_PATH/TrajFill_model.pkl --motion_ckpt_path $PRETRAINED_MODEL_PATH/LocalMotionFill_model.pkl

Visualization

We provide visualization script to visualize the generated grasping ending pose results which is saved at (by default) /results/$EXP_NAME/GraspPose/$OBJECT/fitting_results.npz.

cd visualization
python vis_pose.py --exp_name pretrained_male --gender male --object camera

We provide visualization script to visualize the generated grasping motion result which is saved at (by default) /results/$EXP_NAME/GraspMotion/$OBJECT/fitting_results.npy, from 3 view points, the first-person view, third-person view and the bird-eye view.

cd visualization
python vis_motion.py --GraspPose_exp_name pretrained_male --gender male --object camera

Contact

If you have any questions, feel free to contact us:

• Yan Wu: [email protected]
• Jiahao Wang: [email protected]

Citation

@inproceedings{wu2022saga,
  title = {SAGA: Stochastic Whole-Body Grasping with Contact},
  author = {Wu, Yan and Wang, Jiahao and Zhang, Yan and Zhang, Siwei and Hilliges, Otmar and Yu, Fisher and Tang, Siyu},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year = {2022}
}