๐ PILOT: A Pre-trained Model-Based Continual Learning Toolbox
๐Introduction โข
๐Methods Reproduced โข
๐Reproduced Results
โ๏ธHow to Use โข
๐จโ๐ซAcknowledgments โข
๐คContact
Welcome to PILOT, a pre-trained model-based continual learning toolbox [Paper]. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT also fits typical class-incremental learning algorithms (e.g., FOSTER, and MEMO) within the context of pre-trained models to evaluate their effectiveness.
If you use any content of this repo for your work, please cite the following bib entry:
@article{sun2023pilot,
title={PILOT: A Pre-Trained Model-Based Continual Learning Toolbox},
author={Sun, Hai-Long and Zhou, Da-Wei and Ye, Han-Jia and Zhan, De-Chuan},
journal={arXiv preprint arXiv:2309.07117},
year={2023}
}
@article{zhou2024continual,
title={Continual Learning with Pre-Trained Models: A Survey},
author={Zhou, Da-Wei and Sun, Hai-Long and Ning, Jingyi and Ye, Han-Jia and Zhan, De-Chuan},
journal={arXiv preprint arXiv:2401.16386},
year={2024}
}
@article{zhou2023class,
author = {Zhou, Da-Wei and Wang, Qi-Wei and Qi, Zhi-Hong and Ye, Han-Jia and Zhan, De-Chuan and Liu, Ziwei},
title = {Deep Class-Incremental Learning: A Survey},
journal = {arXiv preprint arXiv:2302.03648},
year = {2023}
}
FineTune
: Baseline method which simply updates parameters on new tasks.iCaRL
: iCaRL: Incremental Classifier and Representation Learning. CVPR 2017 [paper]Coil
: Co-Transport for Class-Incremental Learning. ACMMM 2021 [paper]DER
: DER: Dynamically Expandable Representation for Class Incremental Learning. CVPR 2021 [paper]FOSTER
: Feature Boosting and Compression for Class-incremental Learning. ECCV 2022 [paper]MEMO
: A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. ICLR 2023 Spotlight [paper]SimpleCIL
: Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need. arXiv 2023 [paper]L2P
: Learning to Prompt for Continual Learning. CVPR 2022 [paper]DualPrompt
: DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning. ECCV 2022 [paper]CODA-Prompt
: CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning. CVPR 2023 [paper]ADAM
: Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need. arXiv 2023 [paper]RanPAC
: RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023 [paper]Ease
: Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CVPR 2024 [paper]Intended authors are welcome to contact us to reproduce your methods in our repo. Feel free to merge your algorithm into PILOT if you are using our codebase!
For exemplar parameters, Coil, DER, iCaRL, MEMO, and FOSTER set the
fixed_memory
option to false and retain thememory_size
of 2000 for CIFAR100, while settingfixed_memory
option to true and retaining thememory_per_class
of 20 for ImageNet-R. On the contrary, other models are exemplar-free.
Clone this GitHub repository:
git clone https://github.com/sun-hailong/LAMDA-PILOT
cd LAMDA-PILOT
Edit the [MODEL NAME].json
file for global settings and hyperparameters.
Run:
python main.py --config=./exps/[MODEL NAME].json
hyper-parameters
When using PILOT, you can edit the global parameters and algorithm-specific hyper-parameter in the corresponding json file.
These parameters include:
finetune
, icarl
, coil
, der
, foster
, memo
, simplecil
, l2p
, dualprompt
, coda-prompt
and adam
.fixed_memory
is set to false, assuming there are $K$ classes at the current stage, the model will preserve $\left[\frac{{memory-size}}{K}\right]$ exemplars for each class. L2P, DualPrompt, SimpleCIL, ADAM, and CODA-Prompt do not require exemplars. Therefore, parameters related to the exemplar are not utilized.fixed memory
is set to true, the model will preserve a fixed number of memory_per_class
exemplars for each class.We have implemented the pre-processing datasets as follows:
These subsets are sampled from the original datasets. Please note that I do not have the right to distribute these datasets. If the distribution violates the license, I shall provide the filenames instead.
When training not on CIFAR100
, you should specify the folder of your dataset in utils/data.py
.
def download_data(self):
assert 0,"You should specify the folder of your dataset"
train_dir = '[DATA-PATH]/train/'
test_dir = '[DATA-PATH]/val/'
We thank the following repos providing helpful components/functions in our work.
If there are any questions, please feel free to propose new features by opening an issue or contact with the author: Hai-Long Sun([email protected]) and Da-Wei Zhou([email protected]). Enjoy the code.