Winning solution of AIM2020 VTSR Challenge (video interpolation). EQVI-Enhanced Quadratic Video Interpolation.
Authors: Yihao Liu*, Liangbin Xie*, Li Siyao, Wenxiu Sun, Yu Qiao, Chao Dong [paper]
*equal contribution
If you find our work is useful, please kindly cite it.
@InProceedings{liu2020enhanced,
author = {Yihao Liu and Liangbin Xie and Li Siyao and Wenxiu Sun and Yu Qiao and Chao Dong},
title = {Enhanced quadratic video interpolation},
booktitle = {European Conference on Computer Vision Workshops},
year = {2020},
}
synthesize_video.py
to help synthesize video from consecutive frames. We slightly modified the file models/scopeflow_models/IRR_PWC_V2.py
. If you can run the testing codes successfully before, you can ignore the changes. We update this file to avoid some latent problems when testing.:construction_worker: The list gose on and on...
So many things to do, let me have a break... :see_no_evil:
In our implementation, we use ScopeFlow as a pretrained flow estimation module.
Please follow the instructions to install the required correlation package:
cd models/scopeflow_models/correlation_package
python setup.py install
Note:
if you use CUDA>=9.0, just execute the above commands straightforward;
if you use CUDA==8.0, you need to change the folder name correlation_package_init
into correlation_package
, and then execute the above commands.
Please refer to ScopeFlow and irr for more information.
:zap: Currently we only provide EQVI models trained on REDS_VTSR dataset.
:zap: We empirically find that the training datasets have significant influence on the performance. That is to say, there exists a large dataset bias. When the distribution of training and testing data mismatch, the model performance could dramatically drop. Thus, the generalizability of video interpolation methods is worth investigating.
The pretrained models can be downloaded at Google Drive or Baidu Drive (token: s4sv).
Unzip the downloaded zip file.
unzip checkpoints.zip
There should be five models in the checkpoints
folder:
checkpoints/scopeflow/Sintel_ft/checkpoint_best.ckpt
checkpoints/Stage3_RCSN_RQFP/Stage3_checkpoint.ckpt
checkpoints/Stage4_MSFuion/Stage4_checkpoint.ckpt
checkpoints/Stage123_scratch/Stage123_scratch_checkpoint.ckpt
checkpoints/Stage123_scratch_vgg/Stage123_scratch_vgg_checkpoint.ckpt
Model | baseline | RCSN | RQFP | MS-Fusion | REDS_VTSR val (30 clips)* | REDS_VTSR5 (5 clips) ** |
---|---|---|---|---|---|---|
Stage3 RCSN+RQFP | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | 24.0354 | 24.9633/0.7268 |
Stage4 MS-Fusion | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | 24.0562 | 24.9706/0.7263 |
Stage123 scratch | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | 24.0962 | 25.0699/0.7296 |
Stage123 scratch vgg | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | 24.0069 | 24.9684/0.7237 |
* The performance is evaluated by x2 interpolation (interpolate 1 frame between two given frames).
** Poposed in our [EQVI paper]. Clip 002, 005, 010, 017 and 025 of REDS_VTSR validation set.
Clarification:
checkpoints/Stage123_scratch/Stage123_scratch_checkpoint.ckpt
), since it achieves the best quantitative performance on REDS_VTSR validation set.checkpoints/Stage123_scratch_vgg/Stage123_scratch_vgg_checkpoint.ckpt
). It can produce more visually pleasing results with sharper and clearer edges and textures.For convenient comparison, we now provide two predicted results produced by EQVI (Stage123_scratch_checkpoint.ckpt) and EQVI-P (Stage123_scratch_vgg_checkpoint) on REDS_VTSR validation set. You can download them at Google Drive.
The REDS_VTSR training and validation dataset can be found here.
More datasets and models will be included soon.
configs/config_xxx.py
, including:testset_root
test_size
test_crop_size
inter_frames
preserve_input
store_path
interpolate_REDS_VTSR.py
to produce the interpolated frames in the same naming manner. For example, given the input frames 00000000.png and 00000008.png, if we choose to interpolate 3 frames (inter_frames=3
), then the output frames are automatically named as 00000002.png, 00000004.png and 00000006.png.CUDA_VISIBLE_DEVICES=0 python interpolate_REDS_VTSR.py configs/config_xxx.py
Note: interpolate_REDS_VTSR.py
is specially coded with REDS_VTSR dataset.
:zap: Now we support testing for arbitrary dataset with a generic inference script interpolate_EQVI.py
.
inter_frames=3
), then the output frames will be named as 001_0.png, 001_1.png, 001_2.png.CUDA_VISIBLE_DEVICES=0 python interpolate_EQVI.py configs/config_xxx.py
The output results will be stored in the specified $store_path$
.
configs/config_train_EQVI_VTSR.py
CUDA_VISIBLE_DEVICES=0,1,2,3 python train_EQVI_lap_l1.py --config configs/config_train_EQVI_VTSR.py