Beyond Joint Demosaicking and Denoising (BJDD)

This is the official implementation of paper title "Beyond Joint Demosaicking and Denoising: An Image Processing Pipeline for a Pixel-bin Image Sensor" from CVPRW21. To download full paper [Click Here].

Please consider to cite this paper as follows:

@inproceedings{a2021beyond,
  title={Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor},
  author={Sharif, SMA and Naqvi, Rizwan Ali and Biswas, Mithun},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={233--242},
  year={2021}
}

Follow-up study on pixel-bin sensor has published in BMVC'21

Paper title: SAGAN: Adversarial Spatial-asymmetric Attention for Noisy Nona-Bayer Reconstruction

[Paper Link] | [Supplemental] | [Code]

Please do consider citing SAGAN as follows:

@inproceedings{a2021beyond,
  title={SAGAN: Adversarial Spatial-asymmetric Attention for Noisy Nona-Bayer Reconstruction},
  author={Sharif, SMA and Naqvi, Rizwan Ali and Biswas, Mithun},
  booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
  pages={0-0},
  year={2021}
}

Overview

BJDD tackle the challenges of joint demosaicing and denoising (JDD) of artefact prone non-Bayer Quad-Bayer CFA and Bayer CFA pattern. Notably, such CFA pair is substantially leveraging in the recent smartphone cameras (denoted as Pixel-bin image sensor).

Figure: Commonly used CFA patterns of pixel-bin image 086 sensors. Left : Quad-Bayer CFA. Right: Bayer-CFA.

Figure: Overview of the proposed method, including network architecture and submodules.

Comparison with state-of-the-art deep JDD methods

Figure: Quantitative comparison between proposed method and existing JDD methods for Quad-Bayer reconstruction.

Prerequisites

Python 3.8
CUDA 10.1 + CuDNN
pip
Virtual environment (optional)

Installation

Please consider using a virtual environment to continue the installation process.

git clone https://github.com/sharif-apu/BJDD_CVPR21.git
cd BJDD_CVPR21
pip install -r requirement.txt

Testing

** [Click Here]** to download pretrained weights and save it to weights/ directory for inferencing with Quad-bayer CFA python main.py -i

A few testing images are provided in a sub-directory under testingImages (i.e., testingImages/sampleImages/) In such occasion, reconstructed image(s) will be available in modelOutput/sampleImages/.

To inference with custom setting execute the following command: python main.py -i -s path/to/inputImages -d path/to/outputImages -ns=sigma(s) Here,-ns specifies the standard deviation of a Gaussian distribution (i.e., -ns=5, 10, 15),-s specifies the root directory of the source images (i.e., testingImages/), and -d specifies the destination root (i.e., modelOutput/).

Training

To start training we need to sampling the images according to the CFA pattern and have to pair with coresponding ground-truth images. To sample images for pair training please execute the following command:

python main.py -ds -s /path/to/GTimages/ -d /path/to/saveSamples/ -g 2 -n 10000 Here -s flag defines your root directory of GT images, -d flag defines the directory where sampled images should be saved, and -g flag defines the binnig factr (i.e., 1 for bayer CFA, 2 for Quad-bayer), -n defines the number of images have to sample (optional)

After extracting samples, please execute the following commands to start training:

python main.py -ts -e X -b Y To specify your trining images path, go to mainModule/config.json and update "gtPath" and "targetPath" entity. You can specify the number of epoch with -e flag (i.e., -e 5) and number of images per batch with -b flag (i.e., -b 12).

For transfer learning execute: python main.py -tr -e -b

Bayer Testing

We also trained our model with Bayer CFA. To download pretrained Bayer weights [Click Here]. In such occasion, please update binning factor entity in mainModule/config.json file.

Others

Check model configuration: python main.py -ms Create new configuration file: python main.py -c Update configuration file: python main.py -u Overfitting testing python main.py -to

Contact

For any further query, feel free to contact us through the following emails: [email protected], [email protected], or [email protected]