Global-Local Capsule Network (GLCapsNet) is a capsule-based architecture able to provide context-based eye fixation prediction for several autonomous driving scenarios, while offering interpretability both globally and locally.
Code for the paper entitled Interpretable Global-Local Dynamics for the Prediction of Eye Fixations in Autonomous Driving Scenarios, publicly available in IEEE Access. Supplementary material as videos and images are provided along with the paper in the IEEE Access site.
Global-Local Capsule Network (GLCapsNet) block diagram. It predicts eye fixations based on several contextual conditions of the scene, which are represented as combinations of several spatio-temporal features (RGB, Optical Flow and Semantic Segmentation). Its hierarchical multi-task approach routes Feature Capsules to Condition Capsules both globally and locally, which allows for the interpretation of visual attention in autonomous driving scenarios.
/path_output_in_config/[all,rgb,of,segmentation_probabilities]/conv_block/caps_block/experiment_id/config_train.py
/path_output_in_config/[all,rgb,of,segmentation_probabilities]/conv_block/caps_block/experiment_id/checkpoints/weights.h5
/path_output_in_config/[all,rgb,of,segmentation_probabilities]/conv_block/caps_block/experiment_id/logs/tensorboard-logs
/path_output_in_config/[all,rgb,of,segmentation_probabilities]/conv_block/caps_block/experiment_id/logs/log.csv
/path_output_in_config/[all,rgb,of,segmentation_probabilities]/conv_block/caps_block/experiment_id/logs/trace_sampling.npy
/path_output_in_config/[all,rgb,of,segmentation_probabilities]/conv_block/caps_block/experiment_id/predictions/[test_id,prediction_id]/[resulting_files]
python3.6 execute.py -m train -f rgb --conv_block cnn_generic_branch
python3.6 execute.py -m train -f of --conv_block cnn_generic_branch
python3.6 execute.py -m train -f segmentation_probabilities --conv_block cnn_generic_branch
python3.6 execute.py -m train -f all --conv_block cnn_generic_fusion
python3.6 execute.py -m train -f all --conv_block cnn_generic_fusion
python3.6 execute.py -m train -f all --conv_block cnn_generic_branch --caps_block ns_sc
python3.6 execute.py -m train -f all --conv_block cnn_generic_branch --caps_block ns_sc
python3.6 execute.py -m train -f all --conv_block cnn_generic_branch --caps_block triple_ns_sc
python3.6 execute.py -m train -f all --conv_block cnn_generic_branch --caps_block mask_triple_ns_sc
python3.6 execute.py -m train -f all --conv_block cnn_generic_branch --caps_block glcapsnet
python3.6 execute.py -m train -f all --conv_block cnn_generic_branch --caps_block glcapsnet
Model function names are required to be unique per conv_block or caps_block, as the code manage the executions via that names.
If you use portions of this code or ideas from the paper, please cite our work:
@article{martinez2020glcapsnet,
title={Interpretable Global-Local Dynamics for the Prediction of Eye Fixations in Autonomous Driving Scenarios},
author={J. {Martínez-Cebrián} and M. {Fernández-Torres} and F. {Díaz-de-María}},
journal={IEEE Access},
volume={8},
pages={217068-217085},
year={2020},
publisher={IEEE},
doi={10.1109/ACCESS.2020.3041606}
}
Plese, any question or comment email me at [email protected]. I will be happy to discuss anything related to the topic of the paper.