A PyTorch Lightning extension that accelerates and enhances foundation model experimentation with flexible fine-tuning schedules.
A PyTorch Lightning extension that enhances model experimentation with flexible fine-tuning schedules.
FinetuningScheduler is simple to use yet powerful, offering a number of features that facilitate model research and exploration:
pip install finetuning-scheduler
pip install finetuning-scheduler['examples']
pip install finetuning-scheduler['all']
git clone https://github.com/speediedan/finetuning-scheduler.git
cd finetuning-scheduler
python -m pip install -e ".[all]" -r requirements/docs.txt
pytorch-lighting
package:export FTS_VERSION=2.0.0
export PACKAGE_NAME=pytorch
git clone -b v${FTS_VERSION} https://github.com/speediedan/finetuning-scheduler
cd finetuning-scheduler
python -m pip install -e ".[all]" -r requirements/docs.txt
Note, publishing of new finetuning-scheduler
version-specific docker images was paused after the 2.0.2
patch release. If new version-specific images are required, please raise an issue.
import lightning as L
from finetuning_scheduler import FinetuningScheduler
trainer = L.Trainer(callbacks=[FinetuningScheduler()])
Get started by following the Fine-Tuning Scheduler introduction which includes a CLI-based example or by following the notebook-based Fine-Tuning Scheduler tutorial.
pytorch-lightning
Packageapplicable to versions >= 2.0.0
Now that the core Lightning package is lightning
rather than pytorch-lightning
, Fine-Tuning Scheduler (FTS) by default depends upon the lightning
package rather than the standalone pytorch-lightning
. If you would like to continue to use FTS with the standalone pytorch-lightning
package instead, you can still do so as follows:
Install a given FTS release (for example v2.0.0) using standalone pytorch-lightning
:
export FTS_VERSION=2.0.0
export PACKAGE_NAME=pytorch
wget https://github.com/speediedan/finetuning-scheduler/releases/download/v${FTS_VERSION}/finetuning-scheduler-${FTS_VERSION}.tar.gz
pip install finetuning-scheduler-${FTS_VERSION}.tar.gz
Fine-Tuning Scheduler is rigorously tested across multiple CPUs, GPUs and against major Python and PyTorch versions. Each Fine-Tuning Scheduler minor release (major.minor.patch) is paired with a Lightning minor release (e.g. Fine-Tuning Scheduler 2.0 depends upon Lightning 2.0).
To ensure maximum stability, the latest Lightning patch release fully tested with Fine-Tuning Scheduler is set as a maximum dependency in Fine-Tuning Scheduler's requirements.txt (e.g. <= 1.7.1). If you'd like to test a specific Lightning patch version greater than that currently in Fine-Tuning Scheduler's requirements.txt, it will likely work but you should install Fine-Tuning Scheduler from source and update the requirements.txt as desired.
System / (PyTorch/Python ver) | 2.0.1/3.8 | 2.3.0/3.8, 2.3.0/3.11 |
---|---|---|
Linux [GPUs**] | - | |
Linux (Ubuntu 22.04) | ||
OSX (11) | ||
Windows (2022) |
Fine-Tuning Scheduler is developed and maintained by the community in close communication with the Lightning team. Thanks to everyone in the community for their tireless effort building and improving the immensely useful core Lightning project.
PR's welcome! Please see the contributing guidelines (which are essentially the same as Lightning's).
Please cite:
@misc{Dan_Dale_2022_6463952,
author = {Dan Dale},
title = {{Fine-Tuning Scheduler}},
month = Feb,
year = 2022,
doi = {10.5281/zenodo.6463952},
publisher = {Zenodo},
url = {https://zenodo.org/record/6463952}
}
Feel free to star the repo as well if you find it useful or interesting. Thanks 😊!