[CVPR'22 Oral] TTVSR: Learning Trajectory-Aware Transformer for Video Super-Resolution
This is the official PyTorch implementation of the paper Learning Trajectory-Aware Transformer for Video Super-Resolution.
We proposed an approach named TTVSR to study video super-resolution by leveraging long-range frame dependencies. TTVSR introduces Transformer architectures in video super-resolution tasks and formulates video frames into pre-aligned trajectories of visual tokens to calculate attention along trajectories.
We propose a novel trajectory-aware Transformer, which is one of the first works to introduce Transformer into video super-resolution tasks. TTVSR reduces computational costs and enables long-range modeling in videos. TTVSR can outperform existing SOTA methods in four widely-used VSR benchmarks.
Pre-trained models can be downloaded from onedrive, google drive, and baidu cloud(nbgc).
The output results on REDS4, Vid4 and UMD10 can be downloaded from onedrive, google drive, and baidu cloud(nbgc).
Training set
├────REDS
├────train
├────train_sharp
├────000
├────...
├────269
├────train_sharp_bicubic
├────X4
├────000
├────...
├────269
sep_trainlist.txt
file listing the training samples in the download zip file.
├────vimeo_septuplet
├────sequences
├────00001
├────...
├────00096
├────sequences_BD
├────00001
├────...
├────00096
├────sep_trainlist.txt
├────sep_testlist.txt
Testing set
sep_testlist.txt
file listing the testing samples in the download zip file.├────VID4
├────BD
├────calendar
├────...
├────HR
├────calendar
├────...
├────UDM10
├────BD
├────archpeople
├────...
├────HR
├────archpeople
├────...
git clone https://github.com/researchmm/TTVSR.git
cd TTVSR
./checkpoint
configs/TTVSR_reds4.py
and configs/TTVSR_vimeo90k.py
# REDS model
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./tools/dist_test.sh configs/TTVSR_reds4.py checkpoint/TTVSR_REDS.pth 8 [--save-path 'save_path']
# Vimeo model
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./tools/dist_test.sh configs/TTVSR_vimeo90k.py checkpoint/TTVSR_Vimeo90K.pth 8 [--save-path 'save_path']
save_path
.git clone https://github.com/researchmm/TTVSR.git
cd TTVSR
configs/TTVSR_reds4.py
and configs/TTVSR_vimeo90k.py
# REDS
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./tools/dist_train.sh configs/TTVSR_reds4.py 8
# Vimeo
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./tools/dist_train.sh configs/TTVSR_vimeo90k.py 8
./ttvsr_reds4
and ./ttvsr_vimeo90k
(also can be set by modifying "work_dir" in configs/TTVSR_reds4.py
and configs/TTVSR_vimeo90k.py
)We also sincerely recommend some other excellent works related to us. :sparkles:
If you find the code and pre-trained models useful for your research, please consider citing our paper. :blush:
@InProceedings{liu2022learning,
author = {Liu, Chengxu and Yang, Huan and Fu, Jianlong and Qian, Xueming},
title = {Learning Trajectory-Aware Transformer for Video Super-Resolution},
booktitle = {CVPR},
year = {2022},
month = {June}
}
This code is built on mmediting. We thank the authors of BasicVSR for sharing their code.
If you meet any problems, please describe them in issues or contact: