High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction

This Project is a Tensorflow implementation of

• "High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction" IEEE Transactions on Pattern Analysis and Machine Intelligence, Nan Meng, Hayden K-H. So, Xing Sun, Edmund Y. Lam, 2019. [Paper]
• "High-order Residual Network for Light Field Super-Resolution" The 34th AAAI Conference on Artificial Intelligence, Nan Meng, Xiaofei Wu, Jianzhuang Liu, Edmund Y. Lam, 2020. [Paper]

Requirements

• Python2==2.7
• Python3>=3.5
• Tensorflow r1.*>=r1.8
• tqmd
• opencv
• unrar

Installation

• Download the project

git clone https://github.com/monaen/LightFieldReconstruction.git --branch master --single-branch

Training

Training models for spatial or angular or both super-resolution tasks

• Training the model for spatial super-resolution (e.g. Sx4). You need to specify the gamma_S for different spatial SR tasks.

python train_SpatialSR.py --datadir data/train/Spatial/5x5
                          --gamma_S 4
                          --verbose
                          --perceptual_loss

• Training the model for angular super-resolution (e.g. Ax4). You need to specify the gamma_A for different angular SR tasks.

python train_ViewSysthesis.py --datadir data/train/Angular/9x9
                              --gamma_A 4
                              --verbose

• Training the model for spatio-angular super-resolution (e.g. Sx2Ax2). You need to specify both the gamma_S and gamma_A for different spatio-angular SR tasks.

python train_SpatialAngularSR.py --datadir data/train/SpatialAngular/5x5
                                 --gamma_S 2
                                 --gamma_A 2
                                 --verbose
                                 --perceptual_loss

Training arguments

usage:train_SpatialSR.py [-h] [--datadir DATADIR] [--lr_start LR_START] [--lr_beta1 LR_BETA1] [--batchSize BATCHSIZE]
                         [--imageSize IMAGESIZE] [--viewSize VIEWSIZE] [--channels CHANNELS] [--verbose VERBOSE]
                         [--num_epoch NUM_EPOCH] [--start_epoch START_EPOCH] [--gamma_S {1,2,3,4}]
                         [--gamma_A {0,1,2,3,4}] [--num_GRL_HRB NUM_GRL_HRB] [--num_SRe_HRB NUM_SRE_HRB]
                         [--resume RESUME] [--select_gpu SELECT_GPU] [--perceptual_loss PERCEPTUAL_LOSS]
                         [--vgg_model VGG_MODEL] [--save_folder SAVE_FOLDER]

optional arguments:
  -h, --help                Show this help message and exit
  --datadir                 The training and testing data path
  --lr_start                The start learning rate
  --lr_beta1                The exponential decay rate for the 1st moment estimates
  --batchSize               The batchsize of the input data
  --imageSize               Spatial size of the input light fields
  --viewSize                Angular size of the input light fields
  --channels                Channels=1 means only the luma channel; Channels=3 means RGB channels (not supported)
  --verbose                 Whether print the network structure or not
  --num_epoch               The total number of training epoch
  --start_epoch             The start epoch counting number
  --gamma_S {1,2,3,4}       Spatial downscaling factor
  --gamma_A {0,1,2,3,4}     Angular downscaling factor, '0' represents 3x3->7x7
  --num_GRL_HRB             The number of HRB in GRLNet (only for AAAI model)
  --num_SRe_HRB             The number of HRB in SReNet (only for AAAI model)
  --resume                  Need to resume the pretrained model or not
  --select_gpu              Select the gpu for training or evaluation
  --perceptual_loss         Need to use perceptual loss or not, if true, one also have to set the vgg_model item
  --vgg_model               Pretrained VGG model path
  --save_folder             Model save path

Training data samples

We provide the training samples to facilitate users to generate custom training set.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_trainingsamples.sh

Evaluation

• Spatial SR evaluation (Sx4, Sx3, Sx2)

python evaluation_SpatialSR.py --datapath $PATH_TO_THE_EVALUATION_LF_DATA \
                               --gamma_S $Upscaling_Factor \
                               --pretrained_model $PATH_TO_THE_PRETRAINED_MODEL \
                               --select_gpu 0 --verbose

Here, we provide an example to evaluate the performance on Sx4 task as guidance.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_evaluation_data.sh
cd ../pretrained_models
bash download_pretrained_models_HDDRNet_Sx4.sh
cd ..

python evaluation_SpatialSR.py --datapath data/evaluation/buddha.mat \
                               --gamma_S 4 \
                               --pretrained_model pretrained_models/HDDRNet/Sx4/HDDRNet \
                               --select_gpu 0 --verbose

• Angular SR evaluation (Ax4, Ax3, Ax2, A3x3_7x7)

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_occlusions20.sh
cd ../pretrained_models
bash download_pretrained_models_HDDRNet_Ax4.sh
cd ..

python evaluation_ViewSynthesis.py --datapath data/testset/occlusions20/occlusions_48.mat \
                                   --gamma_A 4 \
                                   --pretrained_model pretrained_models/HDDRNet/Ax4/HDDRNet \
                                   --select_gpu 0 --verbose


python evaluation_ViewSynthesis_Patchwise.py --datapath data/testset/occlusions20/occlusions_48.mat \
                                             --gamma_A 4 \
                                             --pretrained_model pretrained_models/HDDRNet/Ax4/HDDRNet \
                                             --select_gpu 0 --verbose

• Spatial and Angular SR evaluation (Sx2Ax2, Sx3Ax2)

Note: We provide the pretrained model for Sx2Ax2, but the model is not well-trained. We also provide the code for Sx3Ax2 task. Here, we provide an example to evaluate the performance on Sx2Ax2 task as guidance.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_evaluation_data.sh
cd ../pretrained_models
bash download_pretrained_models_Sx2Ax2.sh
cd ..

python evaluation_SpatialAngularSR.py --datapath data/evaluation/buddha.mat \
                                      --gamma_S 2 \
                                      --gamma_A 2 \
                                      --pretrained_model pretrained_models/Others/Sx2Ax2/weights \
                                      --select_gpu 0 --verbose

• Evaluate the entire dataset

Note: All the reported results are tested using Matlab, and here we provide an example to show how to evaluate the results on reflective20 dataset.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_reflective20.sh
cd ../pretrained_models
bash download_pretrained_models_M-HDDRNet_Ax4.sh
cd ..

python evaluation_on_dataset.py --datafolder data/testset/reflective20 \
                                --gamma_S 1 \
                                --gamma_A 4 \
                                --pretrained_model pretrained_models/M-HDDRNet/Ax4/M-HDDRNet \
                                --result_folder results \
                                --select_gpu 0 --verbose


# After the execution of "evaluation_on_dataset.py", run the m-file "evaluation.m"

matlab -nodisplay -nosplash -nodesktop -r "run evaluation.m; exit;"

Using the pretrained model

To download the pretrained mdoels, please change the directory into the folder pretrained_models and run the corresponding bash files. For example, to download the HDDRNet_Sx4 pretrained model,

# path = Path to LightFieldReconstruction
cd pretrained_models
bash download_pretrained_models_HDDRNet_Sx4.sh

We provide a detailed instruction on how to download the pretrained models for differnet SR models and tasks.

Reference

Paper

@article{Meng2019High,
  title        = {High-dimensional dense residual convolutional neural network for light field reconstruction},
  author       = {Meng, Nan and So, Hayden Kwok-Hay and Sun, Xing and Lam, Edmund},
  journal      = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year         = {2019}
}
@article{Meng2020High,
  title        = {High-order residual network for light field super-resolution},
  author       = {Meng, Nan and Wu, Xiaofei and Liu, Jianzhuang and Lam, Edmund},
  journal      = {Association for the Advancement of Artificial Intelligence},
  volume       = {34},
  number       = {7},
  pages        = {11757-11764},
  month        = {February},
  year         = {2020}
}

Frequently Asked Questions

TODO List

• T-PAMI

• Spatial super-resolution 2x framework.
• Spatial super-resolution 3x framework.
• Spatial super-resolution 4x framework.
• Angular super-resolution 2x (5x5->9x9) framework.
• Angular super-resolution 2.3x (3x3->7x7) framework.
• Angular super-resolution 3x (3x3->9x9) framework.
• Angular super-resolution 4x (2x2->8x8) framework.
• Spatial super-resolution 2x Angular super-resolution 2x framework.
• The pretrained models for spatial Sx2, Sx3, Sx4.
• The pretrained models for angular Ax2, Ax2.3, Ax3, Ax4.
• The pretrained models for spatio-angular Sx2Ax2.
• Training samples.
• Testset.
• Evaluation code.
• Results and analysis.

• AAAI

• Spatial super-resolution 2x framework.
• Spatial super-resolution 3x framework.
• Spatial super-resolution 4x framework.
• Angular super-resolution 2x (5x5->9x9).
• Angular super-resolution 2.3x (3x3->7x7).
• Angular super-resolution 3x (3x3->9x9).
• Angular super-resolution 4x (2x2->8x8).
• Other materials.

• Matlab

• Matlab evaluation code.

For some reasons, we are not able to provide the pre-trained models for the AAAI work.

Acknowledgement

Our code is released under MIT License. We would like to appreciate the GPU support from Lee Ultrasound Imaging Group of Prof.Wei-Ning Lee