FourierHeatmap (latest release: v0.2.0)

This is an unofficial pytorch implementation of Fourier Heat Map which is proposed in the paper, A Fourier Perspective on Model Robustness in Computer Vision [Yin+, NeurIPS2019].

Fourier Heat Map allows to investigate the sensitivity of CNNs to high and low frequency corruptions via a perturbation analysis in the Fourier domain.

News

• We release v0.2.0. API is renewed and some useful libraries (e.g. hydra) are added.

• Previous version is still available as v0.1.0.

• Docker is supported. Now, you can evaluate Fourier Heat Map on the Docker container.

Requirements

This library requires following as a pre-requisite.

• python 3.9+
• poetry

Note that I run the code with Ubuntu 20, Pytorch 1.8.1, CUDA 11.0.

Installation

This repo uses poetry as a package manager. The following code will install all necessary libraries under .venv/.

$ git clone [email protected]:gatheluck/FourierHeatmap.git
$ cd FourierHeatmap
$ pip install poetry  # If you haven't installed poetry yet.
$ poetry install

Setup

Dataset

This codes expect datasets exist under data/. For example, if you want to evaluate Fourier Heat Map for ImageNet, please set up like follows:

FourierHeatmap
├── data
│	└── imagenet
│		├── train/
│		└── val/

Usage

Visualizing Fourier basis

The script fhmap/fourier/basis.py generates Fourier base functions. For example:

$ poetry run python fhmap/fourier/basis.py

will generate 31x31 2D Fourier basis and save as an image under outputs/basis.png. The generated image should be like follows.

Evaluating Fourier Heat Map

The script fhmap/apps/eval_fhmap.py eveluate Fourier Heat Map for a model. For example:

$ poetry run python fhmap/apps/eval_fhmap.py dataset=cifar10 arch=resnet56 weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0

will generate 31x31 Fourier Heat Map for ResNet56 on CIFAR-10 dataset and save as an image under outputs/eval_fhmap/. The generated image should be like follows.

Note that the L2 norm size (=eps) of Fourier basis use in original paper is following:

Evaluating custom dataset and model

Evaluating your custom dataset

If you want to evaluate Fourier Heat Map on your custom dataset, please refer follwing instraction.

• Implement YourCustomDatasetStats class:

  • This class holds basic dataset information.
  • YourCustomDatasetStats class should inherit from original DatasetStats class in factory/dataset module and also shoud be placed in factory/dataset module.
  • For details, please refer to the Cifar10Stats class in factory/dataset module.

• Implement YourCustomDataModule class:

  • This class is responsible for preprocess, transform (includes adding Fourier Noise to image) and create test dataset.
  • YourCustomDataModule class should inherit from BaseDataModule class in factory/dataset module and also shoud be placed in factory/dataset module.
  • For details, please refer to the Cifar10DataModule class in factory/dataset module.

• Implement YourCustomDatasetConfig class:

  • This class is needed for applying hydra's dynamic object instantiation to dataset class.
  • YourCustomDatasetConfig class should inherit from DatasetConfig class in schema/dataset module and also shoud be placed in schema/dataset module. Please add YourCustomDatasetConfig to schema/__init__.
  • For details, please refer to the Cifar10Config class in schema/dataset module.

• Add option for your custom dataset:

  • Lastly, please add the config of your custom dataset to ConfigStore class by adding a follwing line to apps/eval_fhmap.

  s.store(group="dataset", name="yourcustomdataset", node=schema.YourCustomDatasetConfig)
  

Now, you will be able to call your custom dataset like following.

$ poetry run python fhmap/apps/eval_fhmap.py dataset=yourcustomdataset arch=resnet50 weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0

Evaluating your custom architecture (model)

If you want to evaluate Fourier Heat Map on your custom architecture (model), please refer follwing instraction.

• Implement YourCustomArch class:

  • Please implement class or function which return your custom architecture. The custom architecture have to subclass of torch.nn.module.
  • For details, please refer to the factory/archs/resnet module.

• Implement YourCustomArchConfig class:

  • This class is needed for applying hydra's dynamic object instantiation to architecture class.
  • YourCustomArchConfig class should inherit from ArchConfig class in schema/arch module and also shoud be placed in schema/arch module. Please add YourCustomArchConfig to schema/__init__.
  • For details, please refer to the Resnet56Config class in schema/arch module.
  • If you want to use architectures which is provided by other libs like pytorch or timm, please refere to the Resnet50Config class in schema/arch module.

• Add option for your custom architecture:

  • Lastly, please add the config of your custom architecture to ConfigStore class by adding a follwing line to apps/eval_fhmap.

  s.store(group="arch", name="yourcustomarch", node=schema.YourCustomArchConfig)
  

Now, you will be able to call your custom arch like following.

$ poetry run python fhmap/apps/eval_fhmap.py dataset=cifar10 arch=yourcustomarch weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0

Evaluating Fourier Heat Map through Docker

In order to use FourierHeatmap throgh docker, please install Docker with NVIDIA Container Toolkit beforehand. For detail, please refere official installation guide.

If nvidia-smi is able to run through docker like following, it is successfully installed.

$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

Tue Apr 27 06:46:09 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
| N/A   56C    P0    42W /  N/A |   1809MiB /  8114MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

We use environmental variables to specify the arguments. The variables that can be specified and their meanings are as follows:

For example:

$ export HOST_DATADIR=[DATASET_DIRECTORY_PATH]
$ export HOST_OUTPUTSDIR=[OUTPUTS_DIRECTORY_PATH]
$ export HOST_WEIGHTDIR=[WEIGHT_DIRECTORY_PATH]
$ export WEIGHTFILE=[PYTORCH_MODEL_FILE]
$ cd provision/docker
$ sudo -E docker-compose up  # -E option is needed to inherit environment variables.

will generate 31x31 Fourier Heat Map for ResNet56 on CIFAR-10 dataset and save as an image under OUTPUTS_DIRECTORY_PATH/eval_fhmap/.

References

• Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin Gilmer. "A Fourier Perspective on Model Robustness in Computer Vision.", in NeurIPS, 2019.
• Justin Gilmer and Dan Hendrycks. "Adversarial Example Researchers Need to Expand What is Meant by ‘Robustness’.", in Distill, 2019.