VisualizingNDF

Background: Neural decision forest (NDF) [3] combines the representation power of deep neural networks (DNNs) and the divide-and-conquer idea of traditional decision trees. It conducts inference by making decisions based on image features extracted by DNNs. This decision-making process can be traced and visualized with Decision Saliency Maps (DSMs) [1], which highlight important regions of the input that influence the decision process more.

Contents: This repository contains official Pytorch code for training and visualizing NDF. Pre-processed data and pre-trained models are also released. Both classification and regression problems are considered and specific tasks include:

1. Image classification for MNIST, CIFAR-10 and Nexperia semiconductor. The last is a Kaggle competition organized by MATH 6380O (Advanced Topics in Deep Learning) in HKUST.
2. Facial age estimation on the large-scale Cross-Age Celebrity Dataset (CACD). Pre-processed data and a new model (RNDF) [2] is released. RNDF achieves state-of-the-art accuracy while comsumes less memory.

Example: decision-making for image classification

The left-most column shows the input images. Each row visualizes one path from the root node towards the leaf node in a soft decision tree. Each image in the row represents the DSM [1] for one splitting node, where (Na, Pb) means the input arrives at node a with probability b. For example, the input arrives at the root node with probability 1 is indicated by (N1, P1.0). Each DSM highlights the spatial region that has larger influence on the corresponding splitting node. For example, the foreground object is more important for NDF when making decisions.

Example: decision-making for facial age estimation

Note how the irrelevant texture (e.g. hair) is ignored by NDF during its decision making process.

Performance on Cross-Age Celebrity Dataset (CACD)

Dependency

• Python 3.6 (not tested for other versions)
• PyTorch >= 1.0
• Matplotlib
• Numpy
• CUDA (CPU mode is not implemented)

Pre-trained models

You can download the pre-trained models here and place them in the "pre-trained" folder.

Usage: visualizing pre-trained NDF for image classification

After downloading the pre-trained models, go to /classification and run

python ndf_vis.py 

for CIFAR-10.

For MNIST, run

python ndf_vis.py -dataset 'mnist'

Usage: visualizing pre-trained RNDF for facial age estimation

To visualize NDF for CACD dataset:

1. Download the pre-processed images here and decompress it into the "/data" folder.
2. Download the metadata folder here and place it under "/data".
3. Go to /regression and run

python ndf_vis.py 

Please refer to the classification counterpart for detailed comments. Future updates will introduce more comments for regression.

Usage: training NDF for image classification

To train a deep neural decision forest for CIFAR-10, go to /classification and run

python main.py

For MNIST, run

python main.py -dataset 'mnist' -epochs 50

Usage: training NDF for facial age estimation

To train a RNDF (R stands for residual) for CACD dataset, follow the same step 1 and 2 as in visualization. Finally, go to /regression and run

python main.py -train True

To test the pre-trained model on CACD, go to /regression and run

python main.py -evaluate True -test_model_path "YourPATH/CACD_MAE_4.59.pth"

The released model should give a MAE of 4.59

License

MIT

Citation

Please consider citing the related papers in your publications if they help your research:

@inproceedings{li2019visualizing,
  title={Visualizing the Decision-making Process in Deep Neural Decision Forest},
  author={Li, Shichao and Cheng, Kwang-Ting},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  pages={114--117},
  year={2019}
}

@article{li2019facial,
  title={Facial age estimation by deep residual decision making},
  author={Li, Shichao and Cheng, Kwang-Ting},
  journal={arXiv preprint arXiv:1908.10737},
  year={2019}
}

@inproceedings{kontschieder2015deep,
  title={Deep neural decision forests},
  author={Kontschieder, Peter and Fiterau, Madalina and Criminisi, Antonio and Rota Bulo, Samuel},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={1467--1475},
  year={2015}
}

Links to the papers:

1. Visualizing the decision-making process in deep neural decision forest
2. Facial age estimation by deep residual decision making
3. Deep neural decision forests