Defocus Deblurring Using Dual-Pixel Data

Abdullah Abuolaim¹ and Michael S. Brown^1,2

¹York University, Toronto, Canada      ²Samsung AI Center, Toronto, Canada

Reference github repository for the paper Defocus Deblurring Using Dual-Pixel Data. Abdullah Abuolaim and Michael S. Brown, Proceedings of the European Conference on Computer Vision (ECCV) 2020 (YouTube presentation). If you use our dataset or code, please cite our paper:

@inproceedings{abuolaim2020defocus,
  title={Defocus deblurring using dual-pixel data},
  author={Abuolaim, Abdullah and Brown, Michael S},
  booktitle={European Conference on Computer Vision},
  pages={111--126},
  year={2020},
  organization={Springer}
}

Dataset

Dual-Pixel Defocus Deblurring (DPDD) dataset contains 500 carefully captured scenes. This dataset consists of 2000 images i.e., 500 DoF blurred images with their 1000 dual-pixel (DP) sub-aperture views and 500 corresponding all-in-focus images all at full-frame resolution (i.e., 6720x4480 pixels).

• Images captured by Canon EOS 5D Mark IV DSLR camera

  • Indoor CR2 RAW images

  • Outdoor CR2 RAW images

  • All images used for training/testing (processed to an sRGB and encoded with a lossless 16-bit depth)

• Images captured by Google Pixel 4 smartphone camera

  • Pixel 4 images used for testing in our experiments
  • We refer redears to this GitHub repository created by Google research in order to extract DP views for Pixel smartphone
  • Recall, the smartphone camera currently has limitations that make it challenging to train our DPDNet with
    • The Google Pixel smartphone cameras do not have adjustable apertures, so we are unable to capture corresponding sharp images using a small aperture
    • The data currently available from the Pixel smartphones are not full-frame, but are limited to only one of the Green channels in the raw-Bayer frame

• Extracting DP left and right views from the RAW CR2

  • Download Digital Photo Professional CANON software
  • Save the DP views in high-quality 16-bit TIF images as follows

• Training, validation, and testing sets

  • The dataset is divided randomly into:
    • 70% training, 15% validation, and 15% testing
  • Each set has a balanced number of indoor/outdoor scenes
  • The image names for each set can be found in ./DPDNet/file_names/ directory

• The dataset is organized based on the following directory structure

$dir_name$_c: directory of the final output combined images
$dir_name$_l: directory of the corresponding DP left view images
$dir_name$_r: directory of the corresponding DP right view images
source: images exhibiting defocus blur
target: the corresponding all-in-focus images

Code

Prerequisites

• The code tested with:
  • Python 3.6.9
  • TensorFlow 1.9.0
  • Keras 2.2.4
  • Numpy 1.17.2
  • Scikit-image 0.16.2
  • OpenCV 3.4.2

Despite not tested, the code may work with library versions other than the specified

Installation

Clone with HTTPS this project to your local machine

git clone https://github.com/Abdullah-Abuolaim/defocus-deblurring-dual-pixel.git
cd defocus-deblurring-dual-pixel

Evaluation Metrics

Results are reported on traditional signal processing metrics i.e., MSE, PSNR, SSIM, and MAE. Implementation can be found in ./DPDNet/metrics.py. We also incorporate the recent learned perceptual image patch similarity (LPIPS) proposed by R. Zhang et al. [1]. Implementation can be found in this GitHub repository

Testing

• Download pretrained model defocus_deblurring_dp_l5_s512_f0.7_d0.4.hdf5 inside ./DPDNet/ModelCheckpoints/

• Canon dataset

  • Download dataset dd_dp_dataset_canon.zip and unzip it inside ./DPDNet/

  • Recall that we test with 16-bit images (bit_depth=16 in ./DPDNet/config.py)

  • Run:

    ython ./DPDNet/main.py
    

• Pixel 4 test images

  • Download dataset dd_dp_dataset_pixel.zip and unzip it inside ./DPDNet/

  • Change the dataset name variable inside ./DPDNet/config.py to be dataset_name='_pixel'

  • Recall that we test with 16-bit images (bit_depth=16 in ./DPDNet/config.py)

  • Run:

    ython ./DPDNet/main.py
    

Training

• Pre-processe data to extract patches for training

  • As we mentioned in the main paper, we discard 30% of the patches that have the lowest sharpness energy

  • We found it faster to prepare training patches beforehand instead of extract patches during the runtime

  • Download dataset dd_dp_dataset_canon.zip and unzip it inside ./DPDNet/

  • The patch size, overlapping ratio, and sharpness energy filtering ratio are set as described in the main paper

  • Run:

    ython ./DPDNet/image_to_patch_filter.py
    

  • Running above creates a new directory dd_dp_dataset_canon_patch that has the training/validation patches

• Start training based on the procedure and hyper-parameters described in the main paper

  • ./DPDNet/config.py module contains all the configurations and hyper-parameters

  • Change op_phase='test' to op_phase='train' in ./DPDNet/config.py

  • Recall that we train with 16-bit images (bit_depth=16 in ./DPDNet/config.py)

  • Run:

    ython ./DPDNet/main.py
    

Contact

Should you have any question/suggestion, please feel free to reach out:

Abdullah Abuolaim ([email protected])

Related Links

• ECCV'18 paper: Revisiting Autofocus for Smartphone Cameras   [project page]
• WACV'20 paper: Online Lens Motion Smoothing for Video Autofocus   [project page]   [presentation]
• ICCP'20 paper: Modeling Defocus-Disparity in Dual-Pixel Sensors   [github]   [presentation]
• ECCV'20 paper: Defocus Deblurring Using Dual-Pixel Data   [project page]   [presentation]
• ICCV'21 paper: Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data   [github]   [presentation]
• CVPRW'21 paper: NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results   [pdf]   [presentation]
• WACV'22 paper: Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning   [github]   [presentation]
• WACVW'22 paper: Multi-View Motion Synthesis via Applying Rotated Dual-Pixel Blur Kernels   [pdf]   [presentation]

Reference

[1] R. Zhang et al. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.