VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024)
This repository is the official implementation of VMC.
[CVPR 2024] VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
Hyeonho Jeong*,
Geon Yeong Park*,
Jong Chul Ye,
Given an input video with any type of motion patterns, our framework, VMC fine-tunes only the Keyframe Generation Module within hierarchical Video Diffusion Models for motion-customized video generation.
pip install -r requirements.txt
The following command will run "train & inference" at the same time:
accelerate launch train_inference.py --config configs/car_forest.yml
*Additional scripts of 'train_only' and 'inference_with_pretrained' will be uploaded too.
Input Videos | Output Videos |
Reversed Videos | Output Videos |
If you find our work interesting, please cite our paper.
@article{jeong2023vmc,
title={VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models},
author={Jeong, Hyeonho and Park, Geon Yeong and Ye, Jong Chul},
journal={arXiv preprint arXiv:2312.00845},
year={2023}
}
Thanks all for open-sourcing!