Official implementation of "Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision", CVPR Workshops 2020.
Official implementation (PyTorch) of the paper:
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision, CVPR Workshops 2020 [arXiv] [project].
Fredrik K. Gustafsson, Martin Danelljan, Thomas B. Schön.
We propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. It is specifically designed to test the robustness required in real-world computer vision applications. We also apply our proposed framework to provide the first properly extensive and conclusive comparison of the two current state-of-the-art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates.
Youtube video with qualitative results:
If you find this work useful, please consider citing:
@inproceedings{gustafsson2020evaluating,
title={Evaluating scalable Bayesian deep learning methods for robust computer vision},
author={Gustafsson, Fredrik K and Danelljan, Martin and Sch{\"o}n, Thomas B},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
year={2020}
}
The code has been tested on Ubuntu 16.04. Docker images are provided (see below).
#!/bin/bash
# DEFAULT VALUES
GPUIDS="0"
NAME="toyProblems_depthCompletion_GPU"
NV_GPU="$GPUIDS" nvidia-docker run -it --rm --shm-size 12G \
-p 5700:5700\
--name "$NAME""0" \
-v /home/fregu482:/root/ \
fregu856/evaluating_bdl:pytorch_pytorch_0.4_cuda9_cudnn7_evaluating_bdl bash
(Inside the image, /root/ will now be mapped to /home/fregu482, i.e., $ cd -- takes you to the regular home folder)
(To create more containers, change the lines GPUIDS="0", --name "$NAME""0" and -p 5700:5700)
General Docker usage:
Download the KITTI depth completion dataset (data_depth_annotated.zip, data_depth_selection.zip and data_depth_velodyne.zip) and place it in /root/data/kitti_depth (/root/data/kitti_depth should contain the folders train, val and depth_selection).
Create /root/data/kitti_raw and download the KITTI raw dataset using download_kitti_raw.py.
Create /root/data/kitti_rgb. For each folder in /root/data/kitti_depth/train (e.g. 2011_09_26_drive_0001_sync), copy the corresponding folder in /root/data/kitti_raw and place it in /root/data/kitti_rgb/train.
Download the virtual KITTI dataset (vkitti_1.3.1_depthgt.tar and vkitti_1.3.1_rgb.tar) and place in /root/data/virtualkitti (/root/data/virtualkitti should contain the folders vkitti_1.3.1_depthgt and vkitti_1.3.1_rgb).
Example usage:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/depthCompletion/ensembling_train_virtual.py
#!/bin/bash
# DEFAULT VALUES
GPUIDS="0,1"
NAME="segmentation_GPU"
NV_GPU="$GPUIDS" nvidia-docker run -it --rm --shm-size 12G \
-p 5900:5900 \
--name "$NAME""01" \
-v /home/fregu482:/home/ \
fregu856/evaluating_bdl:rainbowsecret_pytorch04_20180905_evaluating_bdl bash
(Inside the image, /home/ will now be mapped to /home/fregu482, i.e., $ cd home takes you to the regular home folder)
(To create more containers, change the lines GPUIDS="0,1", --name "$NAME""01" and -p 5900:5900)
General Docker usage:
Download resnet101-imagenet.pth from here and place it in segmentation.
Download the Cityscapes dataset and place it in /home/data/cityscapes (/home/data/cityscapes should contain the folders leftImg8bit and gtFine).
Download the Synscapes dataset and place it in /home/data/synscapes (/home/data/synscapes should contain the folder img, which in turn should contain the folders rgb-2k and class).
Run segmentation/utils/preprocess_synscapes.py (This will, among other things, create /home/data/synscapes_meta/train_img_ids.pkl and /home/data/synscapes_meta/val_img_ids.pkl by randomly selecting subsets of examples. The ones used in the paper are found in segmentation/lists/synscapes).
Example usage:
$ sudo sh start_docker_image_segmentation.sh
$ cd home
$ /root/miniconda3/bin/python evaluating_bdl/segmentation/ensembling_train_syn.py
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyRegression/Ensemble-Adam/train.py
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyClassification/Ensemble-Adam/train.py
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/depthCompletion/ensembling_train_virtual.py
criterion.py: Definitions of losses and metrics.
datasets.py: Definitions of datasets, for KITTI depth completion (KITTI) and virtualKITTI.
model.py: Definition of the CNN.
model_mcdropout.py: Definition of the CNN, with inserted dropout layers.
%%%%%
ensembling_train.py: Code for training M model.py models, on KITTI train.
ensembling_train_virtual.py: As above, but on virtualKITTI train.
ensembling_eval.py: Computes the loss and RMSE for a trained ensemble, on KITTI val. Also creates visualization images of the input data, ground truth, prediction and the estimated uncertainty.
ensembling_eval_virtual.py: As above, but on virtualKITTI val.
ensembling_eval_auce.py: Computes the AUCE (mean +- std) for M = [1, 2, 4, 8, 16, 32] on KITTI val, based on a total of 33 trained ensemble members. Also creates calibration plots.
ensembling_eval_auce_virtual.py: As above, but on virtualKITTI val.
ensembling_eval_ause.py: Computes the AUSE (mean +- std) for M = [1, 2, 4, 8, 16, 32] on KITTI val, based on a total of 33 trained ensemble members. Also creates sparsification plots and sparsification error curves.
ensembling_eval_ause_virtual.py: As above, but on virtualKITTI val.
ensembling_eval_seq.py: Creates visualization videos (input data, ground truth, prediction and the estimated uncertainty) for a trained ensemble, on all sequences in KITTI val.
ensembling_eval_seq_virtual.py: As above, but on all sequences in virtualKITTI val.
%%%%%
mcdropout_train.py: Code for training M model_mcdropout.py models, on KITTI train.
mcdropout_train_virtual.py: As above, but on virtualKITTI train.
mcdropout_eval.py: Computes the loss and RMSE for a trained MC-dropout model with M forward passes, on KITTI val. Also creates visualization images of the input data, ground truth, prediction and the estimated uncertainty.
mcdropout_eval_virtual.py: As above, but on virtualKITTI val.
mcdropout_eval_auce.py: Computes the AUCE (mean +- std) for M = [1, 2, 4, 8, 16, 32] forward passes on KITTI val, based on a total of 16 trained MC-dropout models. Also creates calibration plots.
mcdropout_eval_auce_virtual.py: As above, but on virtualKITTI val.
mcdropout_eval_ause.py: Computes the AUSE (mean +- std) for M = [1, 2, 4, 8, 16, 32] forward passes on KITTI val, based on a total of 16 trained MC-dropout models. Also creates sparsification plots and sparsification error curves.
mcdropout_eval_ause_virtual.py: As above, but on virtualKITTI val.
mcdropout_eval_seq.py: Creates visualization videos (input data, ground truth, prediction and the estimated uncertainty) for a trained MC-dropout model with M forward passes, on all sequences in KITTI val.
mcdropout_eval_seq_virtual.py: As above, but on all sequences in virtualKITTI val.
$ sudo sh start_docker_image_segmentation.sh
$ cd home
$ /root/miniconda3/bin/python evaluating_bdl/segmentation/ensembling_train_syn.py
models:
utils:
datasets.py: Definitions of datasets, for Cityscapes and Synscapes.
%%%%%
ensembling_train.py: Code for training M model.py models, on Cityscapes train.
ensembling_train_syn.py: As above, but on Synscapes train.
ensembling_eval.py: Computes the mIoU for a trained ensemble, on Cityscapes val. Also creates visualization images of the input image, ground truth, prediction and the estimated uncertainty.
ensembling_eval_syn.py: As above, but on Synscapes val.
ensembling_eval_ause_ece.py: Computes the AUSE (mean +- std) and ECE (mean +- std) for M = [1, 2, 4, 8, 16] on Cityscapes val, based on a total of 26 trained ensemble members. Also creates sparsification plots, sparsification error curves and reliability diagrams.
ensembling_eval_ause_ece_syn.py: As above, but on Synscapes val.
ensembling_eval_seq.py: Creates visualization videos (input image, prediction and the estimated uncertainty) for a trained ensemble, on the three demo sequences in Cityscapes.
ensembling_eval_seq_syn.py: Creates a visualization video (input image, ground truth, prediction and the estimated uncertainty) for a trained ensemble, showing the 30 first images in Synscapes val.
%%%%%
mcdropout_train.py: Code for training M model_mcdropout.py models, on Cityscapes train.
mcdropout_train_syn.py: As above, but on Synscapes train.
mcdropout_eval.py: Computes the mIoU for a trained MC-dropout model with M forward passes, on Cityscapes val. Also creates visualization images of the input image, ground truth, prediction and the estimated uncertainty.
mcdropout_eval_syn.py: As above, but on Synscapes val.
mcdropout_eval_ause_ece.py: Computes the AUSE (mean +- std) and ECE (mean +- std) for M = [1, 2, 4, 8, 16] forward passes on Cityscapes val, based on a total of 8 trained MC-dropout models. Also creates sparsification plots, sparsification error curves and reliability diagrams.
mcdropout_eval_ause_ece_syn.py: As above, but on Synscapes val.
mcdropout_eval_seq.py: Creates visualization videos (input image, prediction and the estimated uncertainty) for a trained MC-dropout model with M forward passes, on the three demo sequences in Cityscapes.
mcdropout_eval_seq_syn.py: Creates a visualization video (input image, ground truth, prediction and the estimated uncertainty) for a trained MC-dropout model with M forward passes, showing the 30 first images in Synscapes val.
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyRegression/Ensemble-Adam/train.py
Ensemble-Adam:
Ensemble-MAP-Adam:
Ensemble-MAP-Adam-Fixed:
Ensemble-MAP-SGD:
Ensemble-MAP-SGDMOM:
MC-Dropout-MAP-02-Adam:
MC-Dropout-MAP-02-SGD
MC-Dropout-MAP-02-SGDMOM:
SGLD-256:
SGLD-64:
SGHMC-256:
SGHMC-64:
HMC:
Deterministic:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyClassification/Ensemble-Adam/train.py
Ensemble-Adam:
Ensemble-Adam-Fixed:
Ensemble-MAP-Adam:
Ensemble-MAP-SGD:
Ensemble-MAP-SGDMOM:
MC-Dropout-MAP-01-Adam:
MC-Dropout-MAP-02-SGD
MC-Dropout-MAP-02-SGDMOM:
SGLD-256:
SGLD-64:
SGHMC-256:
SGHMC-64:
HMC:
depthCompletion:
segmentation: