Deep-SloMo

Official PyTorch implementation of "Deep Slow Motion Video Reconstruction with Hybrid Imaging System" Project | Paper
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) and ICCP 2020.

This project is an appliaction of Computational Photography to enhance the slow motion video capturing capabilities of cameras by leveraging dual camera setups.

Example: If a dual camera smartphone is capable of capturing 240fps at 480p and 30fps at 1080p, we can generate 240fps 1080p videos by simultaneously utilizing both cameras.

Prerequisites

This codebase was developed and tested with PyTorch 1.3.1, CUDA 0.4.2 and Python 3.7.5.

Installation

conda install pytorch==1.3.1 torchvision==0.4.2 cudatoolkit=10.1 numpy -c pytorch
pip install tqdm

Download repository

https://github.com/avinashpaliwal/Deep-SloMo.git

Build correlation package for PWCNet

$ cd PWCNet/correlation_package_pytorch1_0
$ ./build.sh

Test

Pretrained model

The pretrained model checkpoints are in the Checkpoints folder.

Scenes

You can download the synthetic (Juggler and Horse), real digital camera (Basketball, Fountain and Veggies) and smartphone (Jump, Dribble and Car) scenes here.

Synthetic scenes

python test.py --scene_root path/to/juggler --interpolation_sequence 12

Digital camera scenes

python test.py --scene_root path/to/basketball --interpolation_sequence 13 12 12

Smartphone scenes

python test.py --scene_root path/to/jump --interpolation_sequence 3

The --interpolation_sequence depends on the low and high frame rates. Example: For 30 and 400 fps, since they are not divisible, the sequence is 13 12 12.
The --scene_root is path to the scene folder containing low-res LR and high-res HR folders.

Video

Our real camera rigs

References

@article{paliwal2020slowmotion,
  author={A. {Paliwal} and N. {Khademi Kalantari}}, 
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Deep Slow Motion Video Reconstruction With Hybrid Imaging System}, 
  year={2020},
  volume={42},
  number={7},
  pages={1557-1569},}