optimizer & lr scheduler & loss function collections in PyTorch
The major version is updated! (v2.12.0
-> v3.0.0
) (#164)
Many optimizers, learning rate schedulers, and objective functions are in pytorch-optimizer
.
Currently, pytorch-optimizer
supports 67 optimizers (+ bitsandbytes
), 11 lr schedulers, and 13 loss functions, and reached about 4 ~ 50K downloads / month (peak is 75K downloads / month)!
The reason for updating the major version from v2
to v3
is that I think it's a good time to ship the recent implementations (the last update was about 7 months ago) and plan to pivot to new concepts like training utilities while maintaining the original features (e.g. optimizers).
Also, rich test cases, benchmarks, and examples are on the list!
Finally, thanks for using the pytorch-optimizer
, and feel free to make any requests :)
REX
lr scheduler. (#217, #222)
Aida
optimizer. (#220, #221)
WSAM
optimizer. (#213, #216)
GaLore
optimizer. (#224, #228)
Adalite
optimizer. (#225, #229)bSAM
optimizer. (#212, #233)
Schedule-Free
optimizer. (#230, #233)
EMCMC
. (#231, #233)
Python 3.7
support officially. (#221)
bitsandbytes
to 0.43.0
. (#228)Ranger21 optimizer
document. (#214, #215)WSAM
optimizer paper link. (#219)thanks to @sdbds, @i404788
bitsandbytes
optimizer. (#211)
pip3 install pytorch-optimizer[bitsandbytes]
bnb_adagrad8bit
, bnb_adam8bit
, bnb_adamw8bit
, bnb_lion8bit
, bnb_lamb8bit
, bnb_lars8bit
, bnb_rmsprop8bit
, bnb_sgd8bit
.mkdocs
with material
theme. (#204, #206)
Optimizer
.rectify
to False
in AdaBelief
optimizer (#203)DynamicLossScaler
test casethanks to @georg-wolflein
BaseOptimizer
(#176, #177)
Gaussian
, Rademacher
).AdamD
feature for AdaHessian optimizer (#177)thanks to @i404788
amsbound
, amsgrad
terms into ams_bound
, #149pytorch_optimizer.experimental
under pytorch_optimizer.*.experimental
update
in Lion optimizer, #135momentum_buffer
in SGDP optimizer, #139