Always sparse. Never dense. But never say never. A Sparse Training repository for the Adaptive Sparse Connectivity concept and its algorithmic instantiation, i.e. Sparse Evolutionary Training, to boost Deep Learning scalability on various aspects (e.g. memory and computational time efficiency, representation and generalization power).
Proof of concept implementations of various sparse artificial neural network models with adaptive sparse connectivity trained with the Sparse Evolutionary Training (SET) algorithm - https://arxiv.org/abs/1707.04780, 15 July 2017
SET was the first algorithm which demonstrated that sparse neural networks can be trained from scratch to outperform dense neural networks within the framework of gradient descent and introduced the idea of optimizing the sparse connections between neurons together with the weights during training.
On short, SET laid the ground for what is today known as sparse training with dynamic sparsity (also referred to in some papers as dynamic sparse training, pruning and growth strategies, and so on).
The following implementations are distributed in the hope that they may be useful, but without any warranties; Their use is entirely at the user's own risk.
An improved version of this Implementation can be found here https://github.com/SelimaC/Tutorial-SCADS-Summer-School-2020-Scalable-Deep-Learning
Proof of concept implementation of Sparse Evolutionary Training (SET) for Multi Layer Perceptron (MLP) on lung dataset using Python, SciPy sparse data structures, and (optionally) Cython.
This implementation was developed just in the last stages of the reviewing process, and we are briefly discussing about it in the "Peer Review File" which can be downloaded from Reference 1 website.
This implementation can be used to create SET-MLP with hundred of thousands of neurons on a standard laptop. It was made starting from the vanilla fully connected MLP implementation of Ritchie Vink (https://www.ritchievink.com/) and we would like to acknowledge his work and thank him. Also, we would like to thank Thomas Hagebols for analyzing the performance of SciPy sparse matrix operations. We thank also to Amarsagar Reddy Ramapuram Matavalam from Iowa State University ([email protected]), who provided us a faster implementation of the "weightsEvolution" method, after the initial release of this code.
If you would like to try large SET-MLP models, below are the expected running times measured on my laptop (16 GB RAM) using the original implementation of the "weightsEvolution" method. I have used exactly the model and the dataset from the file "set_mlp_sparse_data_structures.py" and I just changed the number of hidden neurons per layer:
If you would like to try out SET-MLP with various activation functions, optimization methods and so on (in the detriment of scalability) please use Implementation 1.
For an easy understanding of these implementations please read the following articles. Also, if you use parts of this code in your work, please cite the corresponding ones:
@article{Mocanu2018SET, author = {Mocanu, Decebal Constantin and Mocanu, Elena and Stone, Peter and Nguyen, Phuong H. and Gibescu, Madeleine and Liotta, Antonio}, journal = {Nature Communications}, title = {Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science}, year = {2018}, doi = {10.1038/s41467-018-04316-3}, url = {https://www.nature.com/articles/s41467-018-04316-3 }}
@article{Mocanu2016XBM, author={Mocanu, Decebal Constantin and Mocanu, Elena and Nguyen, Phuong H. and Gibescu, Madeleine and Liotta, Antonio}, title={A topological insight into restricted Boltzmann machines}, journal={Machine Learning}, year={2016}, volume={104}, number={2}, pages={243--270}, doi={10.1007/s10994-016-5570-z}, url={https://doi.org/10.1007/s10994-016-5570-z }}
@phdthesis{Mocanu2017PhDthesis, title = {Network computations in artificial intelligence}, author = {Mocanu, Decebal Constantin}, year = {2017}, isbn = {978-90-386-4305-2}, publisher = {Eindhoven University of Technology}, url={https://pure.tue.nl/ws/files/69949254/20170629_CO_Mocanu.pdf } }
@article{Liu2019onemillion, author = {Liu, Shiwei and Mocanu, Decebal Constantin and Mocanu and Ramapuram Matavalam, Amarsagar Reddy and Pei, Yulong Pei and Pechenizkiy, Mykola}, journal = {arXiv:1901.09181}, title = {Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware}, year = {2019}, url={https://arxiv.org/abs/1901.09181 } }
SET shows that large sparse neural networks can be built if topological sparsity is created from the design phase, before training. There are many algorithmic and implementation improvements which can be made. If you find this work interesting, please share the links to this Github page and to Reference 1. For any question, suggestion, feedback please feel free to contact me by email.
Some time ago, I had a very pleasant unexpected surprise when I found out that Michael Klear released "Synapses". This library implements SET layers in PyTorch and as Michael says it is "truly sparse". For more details please read his article:
https://towardsdatascience.com/the-sparse-future-of-deep-learning-bce05e8e094a
And try out "Synapses" yourself:
https://github.com/AlliedToasters/synapses
Many things can be improved in "Synapses". If interested, please contact and help Michael in developing further the project.
Our paper "Topological insights into sparse neural networks" https://arxiv.org/pdf/2006.14085.pdf has been accepted at ECMLPKDD 2020. It proposes Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. The code is here https://github.com/Shiweiliuiiiiiii/Sparse_Topology_Distance. Also, it shows in a principled manner that sparse training easily unveils a plenitude of sparse sub-networks with very different topologies which outperform the dense networks.
For an interesting quick read about sparse training, please have a look on this blog https://numenta.com/blog/2020/10/30/case-for-sparsity-in-neural-networks-part-2-dynamic-sparsity
To see how sparse training can be used for feature selection please check our latest paper, titled "Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders", here: https://arxiv.org/abs/2012.00560
and the corresponding truly sparse implementation here: https://github.com/zahraatashgahi/QuickSelection
Many thanks,
Decebal