Fast and Accurate ML in 3 Lines of Code
We're happy to announce the AutoGluon 0.6 release. 0.6 contains major enhancements to Tabular, Multimodal, and Time Series modules, along with many quality of life improvements and fixes.
As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 263 commits from 25 contributors!
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.5.2...v0.6.0
Special thanks to @cheungdaven, @suzhoum, @BingzhaoZhu, @liangfu, @Harry-zzh, @gidler, @yongxinw, @martinschaef, @giswqs, @Jalagarto, @geoalgo, @lujiaying and @leloykun who were first time contributors to AutoGluon this release!
Full Contributor List (ordered by # of commits):
@shchur, @yinweisu, @zhiqiangdon, @Innixma, @FANGAreNotGnu, @canerturkmen, @sxjscience, @gradientsky, @cheungdaven, @bryanyzhu, @suzhoum, @BingzhaoZhu, @yongxinw, @tonyhoo, @liangfu, @Harry-zzh, @Raldir, @gidler, @martinschaef, @giswqs, @Jalagarto, @geoalgo, @lujiaying, @leloykun, @yiqings
This version supports Python versions 3.7 to 3.9. This is the last release that will support Python 3.7.
AutoGluon Multimodal (a.k.a AutoMM) supports three new features: 1) object detection, 2) named entity recognition, and 3) multimodal matching. In addition, the HPO backend of AutoGluon Multimodal has been upgraded to ray 2.0. It also supports fine-tuning billion-scale FLAN-T5-XL model on a single AWS g4.2x-large instance with improved parameter-efficient finetuning. Starting from 0.6, we recommend using autogluon.multimodal rather than autogluon.text or autogluon.vision and added deprecation warnings.
Object Detection
"object_detection"
.Named Entity Recognition
"ner"
.Multimodal Matching
"text_similarity"
, "image_similarity"
, "image_text_similarity"
.Miscellaneous minor fixes. @cheungdaven @FANGAreNotGnu @geoalgo @zhiqiangdon (#2402, #2409, #2026, #2401, #2418)
New experimental model FT_TRANSFORMER
. @bingzhaozhu, @innixma (#2085, #2379, #2389, #2410)
FT_TRANSFORMER
key
in the hyperparameters
dictionary or via presets="experimental_best_quality"
.New experimental model compilation support via predictor.compile_models()
. @liangfu, @innixma (#2225, #2260, #2300)
pip install autogluon.tabular[all,skl2onnx]
..compile_models
is called only at the very end.Added predictor.clone(...)
method to allow perfectly cloning a predictor object to a new directory.
This is useful to preserve the state of a predictor prior to altering it
(such as prior to calling .save_space
, .distill
, .compile_models
, or .refit_full
. @innixma (#2071)
Added simplified num_gpus
and num_cpus
arguments to predictor.fit
to control total resources.
@yinweisu, @innixma (#2263)
Improved stability and effectiveness of HPO functionality via various refactors regarding our usage of ray. @yinweisu, @innixma (#1974, #1990, #2094, #2121, #2133, #2195, #2253, #2263, #2330)
Upgraded dependency versions: XGBoost 1.7, CatBoost 1.1, Scikit-learn 1.1, Pandas 1.5, Scipy 1.9, Numpy 1.23. @innixma (#2373)
Added python version compatibility check when loading a fitted TabularPredictor. Will now error if python versions are incompatible. @innixma (#2054)
Added fit_weighted_ensemble
argument to predictor.fit
. This allows the user to disable the weighted ensemble.
@innixma (#2145)
Added cascade ensemble foundation logic. @innixma (#1929)
infer_limit
. @innixma (#2014)Scorer
classes to be easier to use, plus added comprehensive unit tests for all metrics. @innixma (#2242)TimeSeriesPredictor
now supports static features (a.k.a. time series metadata, static covariates) and **
time-varying covariates** (a.k.a. dynamic features or related time series). @shchur @canerturkmen (#1986, #2238,
#2276, #2287)DeepAR
and SimpleFeedForward
), removing the dependency
on MXNet. @canerturkmen (#2074, #2205, #2279)AutoGluonTabular
relies on XGBoost, LightGBM and CatBoost under the hood via the autogluon.tabular
module. Naive
and SeasonalNaive
forecasters are simple methods that provide strong baselines with no increase in
training time. TemporalFusionTransformerMXNet
brings the TFT transformer architecture to AutoGluon. @shchur (#2106,
#2188, #2258, #2266)ETS
, ARIMA
and Theta
, as well as WeightedEnsemble
. @shchur @canerturkmen (#2001, #2033, #2040, #2067, #2072, #2073, #2180,
#2293, #2305)TimeSeriesPredictor
now handles irregularly sampled time series with ignore_index
. @canerturkmen, @shchur (#1993,
#2322)TimeSeriesEvaluator
@shchur (#2147, #2150)Improved documentation and new tutorials:
@shchur (#2120, #2127, #2146, #2174, #2187, #2354)
@shchur
@canerturkmen
v0.5.2 is a security hotfix release.
This release is non-breaking when upgrading from v0.5.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.5.1...v0.5.2
This version supports Python versions 3.7 to 3.9.
v0.4.3 is a security hotfix release.
This release is non-breaking when upgrading from v0.4.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.4.2...v0.4.3
This version supports Python versions 3.7 to 3.9.
We're happy to announce the AutoGluon 0.5 release. This release contains major optimizations and bug fixes to autogluon.multimodal and autogluon.timeseries modules, as well as inference speed improvements to autogluon.tabular.
This release is non-breaking when upgrading from v0.5.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 58 commits from 14 contributors!
Full Contributor List (ordered by # of commits):
This version supports Python versions 3.7 to 3.9.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.5.0...v0.5.1
Changed to a new namespace autogluon.multimodal
(AutoMM), which is a deep learning "model zoo" of model zoos. On one hand, AutoMM can automatically train deep models for unimodal (image-only, text-only or tabular-only) problems. On the other hand, AutoMM can automatically solve multimodal (any combinations of image, text, and tabular) problems by fusing multiple deep learning models. In addition, AutoMM can be used as a base model in AutoGluon Tabular and participate in the model ensemble.
Supported zero-shot learning with CLIP (#1922) @zhiqiangdon
Improved efficient finetuning
Added more data augmentation techniques
Enhanced teacher-student model distillation
Beginner tutorials of applying AutoMM to image, text, or multimodal (including tabular) data. (#1861, #1908, #1858, #1869) @bryanyzhu @sxjscience @zhiqiangdon
A zero-shot image classification tutorial with the CLIP model. (#1942) @bryanyzhu
A tutorial of using CLIP model to extract embeddings for image-text retrieval. (#1957) @bryanyzhu
A tutorial to introduce comprehensive AutoMM configurations (#1861). @zhiqiangdon
AutoMM for tabular data examples (#1752, #1893, #1903). @yiqings
AutoMM distillation example (#1846). @FANGAreNotGnu
A Kaggle notebook about how to use AutoMM to predict pet adoption: https://www.kaggle.com/code/linuxdex/use-autogluon-to-predict-pet-adoption. The model achieves the score equivalent to top 1% (20th/3537) in this kernel-only competition (test data is only available in the kernel without internet access) (#1796, #1847, #1894, #1943). @Linuxdex
We're happy to announce the AutoGluon 0.5 release. This release contains major new modules autogluon.timeseries
and autogluon.multimodal
. In collaboration with the Yu Group of Statistics and EECS from UC Berkeley, we have added interpretable models (imodels) to autogluon.tabular
.
This release is non-breaking when upgrading from v0.4.2. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 91 commits from 13 contributors!
Full Contributor List (ordered by # of commits):
The imodels integration is based on the following work,
Singh, C., Nasseri, K., Tan, Y.S., Tang, T. and Yu, B., 2021. imodels: a python package for fitting interpretable models. Journal of Open Source Software, 6(61), p.3192.
This version supports Python versions 3.7 to 3.9.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.4.1...v0.5.0
Full release notes will be available shortly.
v0.4.2 is a hotfix release to fix breaking change in protobuf.
This release is non-breaking when upgrading from v0.4.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.4.1...v0.4.2
This version supports Python versions 3.7 to 3.9.
We're happy to announce the AutoGluon 0.4.1 release. 0.4.1 contains minor enhancements to Tabular, Text, Image, and Multimodal modules, along with many quality of life improvements and fixes.
This release is non-breaking when upgrading from v0.4.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 55 commits from 10 contributors!
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.4.0...v0.4.1
Special thanks to @yiqings, @leandroimail, @huibinshen who were first time contributors to AutoGluon this release!
Full Contributor List (ordered by # of commits):
This version supports Python versions 3.7 to 3.9.
Added optimization.efficient_finetune
flag to support multiple efficient finetuning algorithms. (#1666) @sxjscience
bit_fit
: "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models"
norm_fit
: An extension of the algorithm in "Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs" and BitFit. We finetune both the parameters in the norm layers as long as the biases.Enabled knowledge distillation for AutoMM (#1670) @zhiqiangdon
AutoMMPredictor
reuses the .fit()
function:from autogluon.text.automm import AutoMMPredictor
teacher_predictor = AutoMMPredictor(label="label_column").fit(train_data)
student_predictor = AutoMMPredictor(label="label_column").fit(
train_data,
hyperparameters=student_and_distiller_hparams,
teacher_predictor=teacher_predictor,
)
Option to turn on returning feature column information (#1711) @zhiqiangdon
requires_column_info
flag in data processors and a utility function to turn this flag on or off.FT-Transformer implementation for tabular data in AutoMM (#1646) @yiqings
Make CLIP support multiple images per sample (#1606) @zhiqiangdon
Avoid using eos
as the sep token for CLIP. (#1710) @zhiqiangdon
Update fusion transformer in AutoMM (#1712) @yiqings
polynomial_decay
scheduler.[CLS]
token in numerical/categorical transformer.Added more image augmentations: verticalflip
, colorjitter
, randomaffine
(#1719) @Linuxdex, @sxjscience
Added prompts for the percentage of missing images during image column detection. (#1623) @zhiqiangdon
Support average_precision
in AutoMM (#1697) @sxjscience
Convert roc_auc
/ average_precision
to log_loss
for torchmetrics (#1715) @zhiqiangdon
torchmetrics.AUROC
requires both positive and negative examples are available in a mini-batch. When training a large model, the per gpu batch size is probably small, leading to an incorrect roc_auc
score. Conversion from roc_auc
to log_loss
improves training stablility.Added pytorch-lightning
1.6 support (#1716) @sxjscience
Updated the names of top-k checkpoint average methods and support customizing model names for terminal input (#1668) @zhiqiangdon
union_soup
-> uniform_soup
and best_soup
-> best
.customize_config_names
-> customize_model_names
and verify_config_names
-> verify_model_names
) to make it easier to understand them.Implemented the GreedySoup algorithm proposed in paper. Added union_soup
, greedy_soup
, best_soup
flags and changed the default value correspondingly. (#1613) @sxjscience
Updated the standalone
flag in automm.predictor.save()
to save the pertained model for offline deployment (#1575) @yiqings
Simplified checkpoint template (#1636) @zhiqiangdon
AutoMMPredictor
's final model checkpoint.ckpt_path
argument to pytorch lightning's trainer only when resume=True
.Unified AutoMM's model output format and support customizing model names (#1643) @zhiqiangdon
timm_image
, hf_text
, clip
, numerical_mlp
, categorical_mlp
, and fusion_mlp
) as prefixes. This is helpful when users want to simultaneously use two models of the same type, e.g., hf_text
. They can just use names hf_text_0
and hf_text_1
.Support standalone
feature in TextPredictor
(#1651) @yiqings
Fixed saving and loading tokenizers and text processors (#1656) @zhiqiangdon
0.4.0
.Change load from a classmethod to staticmethod to avoid incorrect usage. (#1697) @sxjscience
Added AutoMMModelCheckpoint
to avoid evaluating the models to obtain the scores (#1716) @sxjscience
Extract column features from AutoMM's model outputs (#1718) @zhiqiangdon
timm_image
, hf_text
, and clip
.Make AutoMM dataloader return feature column information (#1710) @zhiqiangdon
Fixed calling save_pretrained_configs
in AutoMMPrediction.save(standalone=True)
when no fusion model exists (here) (#1651) @yiqings
Fixed error raising for setting key that does not exist in the configuration (#1613) @sxjscience
Fixed warning message about bf16. (#1625) @sxjscience
Fixed the corner case of calculating the gradient accumulation step (#1633) @sxjscience
Fixes for top-k averaging in the multi-gpu setting (#1707) @zhiqiangdon
Limited RF max_leaf_nodes
to 15000 (previously uncapped) (#1717) @Innixma
high_quality
preset.Limit KNN to 32 CPUs to avoid OpenBLAS error (#1722) @Innixma
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
Segmentation fault: 11
This error occurred when the machine had many CPU cores (>64 vCPUs) due to too many threads being created at once. By limiting to 32 cores used, the error is avoided.
Improved memory warning thresholds (#1626) @Innixma
Added get_results
and model_base_kwargs
(#1618) @Innixma
get_results
to searchers, useful for debugging and for future extensions to HPO functionality.
Added new way to init a BaggedEnsembleModel
that avoids having to init the base model prior to initing the bagged ensemble model.Update resource logic in models (#1689) @Innixma
auto
for resources, fixed in this PR.get_minimum_resources
to explicitly define minimum resource requirements within a method.Updated feature importance default subsample_size
1000 -> 5000, num_shuffle_sets 3
-> 5 (#1708) @Innixma
Added notice to ensure serializable custom metrics (#1705) @Innixma
Fixed evaluate
when weight_evaluation=True
(#1612) @Innixma
predictor.evaluate(...)
or predictor.evaluate_predictions(...)
when self.weight_evaluation==True
.Fixed RuntimeError: dictionary changed size during iteration (#1684, #1685) @leandroimail
Fixed CatBoost custom metric & F1 support (#1690) @Innixma
Fixed HPO not working for bagged models if the bagged model is loaded from disk (#1702) @Innixma
Fixed Feature importance erroring if self.model_best
is None
(can happen if no Weighted Ensemble is fit) (#1702) @Innixma
updated the text tutorial of cutomizing hyperparameters (#1620) @zhiqiangdon
Improved implementations and docstrings of save_pretrained_models
and convert_checkpoint_name
. (#1656) @zhiqiangdon
Added cheat sheet to website (#1605) @yinweisu
Doc fix to use correct predictor when calling leaderboard (#1652) @Innixma
[security] updated pillow
to 9.0.1
+ (#1615) @gradientsky
[security] updated ray
to 1.10.0
+ (#1616) @yinweisu
Tabular regression tests improvements (#1555) @willsmithorg
TabularPredictor
on both regression and classification tasks, multiple presets etc.Disabled image/text predictor when gpu is not available in TabularPredictor
(#1676) @yinweisu
Use class property to set keys in model classes. In this way, if we customize the prefix key, other keys are automatically updated. (#1669) @zhiqiangdon
We're happy to announce the AutoGluon 0.4 release. 0.4 contains major enhancements to Tabular and Text modules, along with many quality of life improvements and fixes.
This release is non-breaking when upgrading from v0.3.1. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 151 commits from 14 contributors!
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.3.1...v0.4.0
Special thanks to @zhiqiangdon, @willsmithorg, @DolanTheMFWizard, @truebluejason, @killerSwitch, and @Xilorole who were first time contributors to AutoGluon this release!
Full Contributor List (ordered by # of commits):
This version supports Python versions 3.7 to 3.9.
pip install autogluon.text
will error on import if installed standalone due to missing autogluon.features
as a dependency. To fix: pip install autogluon.features
. This will be resolved in v0.4.1 release.AutoGluon-Text is refactored with PyTorch Lightning. It now supports backbones in huggingface/transformers. The new version has better performance, faster training time, and faster inference speed. In addition, AutoGluon-Text now supports solving multilingual problems and a new AutoMMPredictor
has been implemented for automatically building multimodal DL models.
presets="high_quality"
, the win-rate increased to 77.8% thanks to the DeBERTa-v3 backbone.presets="multilingual"
. You can now train a model on the English dataset and directly apply the model on datasets in other languages such as German, Japanese, Italian, etc..fit()
again after a previous trained model has been loaded.Thanks to @zhiqiangdon and @sxjscience for contributing the AutoGluon-Text refactors! (#1537, #1547, #1557, #1565, #1571, #1574, #1578, #1579, #1581, #1585, #1586)
AutoGluon-Tabular has been majorly enhanced by numerous optimizations in 0.4. In summation, these improvements have led to a:
Specific updates:
infer_limit
and infer_limit_batch_size
as new fit-time constraints (Tutorial). This allows users to specify
the desired end-to-end inference latency of the final model and AutoGluon will automatically train models
to satisfy the constraint. This is extremely useful for online-inference scenarios where you need to satisfy an
end-to-end latency constraint (for example 50ms). @Innixma (#1541, #1584)TabularPredictor.fit_pseudolabel(...)
! @DolanTheMFWizard (#1323, #1382)TabularPredictor.fit(..., feature_prune_kwargs={})
! @truebluejason (#1274, #1305)calibrate
fit argument. @DolanTheMFWizard (#1336, #1374, #1502)refit_full
logic to majorly simplify user model contributions and improve multimodal support with advanced presets. @Innixma (#1567)As part of the migration from MXNet to Torch, we have created a Torch based counterpart to the prior MXNet tabular neural network model. This model has several major advantages, such as:
This model has replaced the MXNet tabular neural network model in the default hyperparameters configuration, and is enabled by default.
Thanks to @jwmueller and @Innixma for contributing TabularNeuralNetTorchModel to AutoGluon! (#1489)
VowpalWabbit has been added as a new model in AutoGluon. VowpalWabbit is not installed by default, and must be installed separately.
VowpalWabbit is used in the hyperparameters='multimodal'
preset, and the model is a great option to use for datasets containing text features.
To install VowpalWabbit, specify it via pip install autogluon.tabular[all, vowpalwabbit]
or pip install "vowpalwabbit>=8.10,<8.11"
Thanks to @killerSwitch for contributing VowpalWabbitModel to AutoGluon! (#1422)
Linear models have been accelerated by 20x in training and 20x in inference thanks to a variety of optimizations.
To get the accelerated training speeds, please install scikit-learn-intelex via pip install "scikit-learn-intelex>=2021.5,<2021.6"
Note that currently LinearModel is not enabled by default in AutoGluon,
and must be specified in hyperparameters
via the key 'LR'
.
Further testing is planned to incorporate LinearModel as a default model in future releases.
Thanks to the scikit-learn-intelex
team and @Innixma for the LinearModel optimizations! (#1378)
autogluon.common
to simplify dependency management for future submodules. @Innixma (#1386)autogluon.mxnet
and autogluon.extra
submodules as part of code cleanup. @Innixma (#1397, #1411, #1414)v0.3.1 is a hotfix release which fixes several major bugs as well as including several model quality improvements.
This release is non-breaking when upgrading from v0.3.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 9 commits from 4 contributors.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.3.0...v0.3.1
Thanks to the 4 contributors that contributed to the v0.3.1 release!
Special thanks to @yinweisu who is a first time contributor to AutoGluon and fixed a major bug in ImagePredictor HPO!
Full Contributor List (ordered by # of commits):
@Innixma, @gradientsky, @yinweisu, @sackoh
best_quality
preset.-1
as n_jobs
value instead of using os.cpu_count()
. @sackoh (#1289)v0.3.0 introduces multi-modal image, text, tabular support to AutoGluon. In just a few lines of code, you can train a multi-layer stack ensemble using text, image, and tabular data! To our knowledge this is the first publicly available implementation of a model that handles all 3 modalities at once. Check it out in our brand new multimodal tutorial! v0.3.0 also features a major model quality improvement for Tabular, with a 57.6% winrate vs v0.2.0 on the AutoMLBenchmark, along with an up to 10x online inference speedup due to low level numpy and pandas optimizations throughout the codebase! This inference optimization enables AutoGluon to have sub 30 millisecond end-to-end latency for real-time deployment scenarios when paired with model distillation. Finally, AutoGluon can now train PyTorch image models via integration with TIMM. Specify any TIMM model to ImagePredictor
or TabularPredictor
to train them with AutoGluon!
This release is non-breaking when upgrading from v0.2.0. As always, only load previously trained models using the same version of AutoGluon that they were originally trained on. Loading models trained in different versions of AutoGluon is not supported.
This release contains 70 commits from 10 contributors.
See the full commit change-log here: https://github.com/awslabs/autogluon/compare/v0.2.0...v0.3.0
Thanks to the 10 contributors that contributed to the v0.3.0 release!
Special thanks to the 3 first-time contributors! @rxjx, @sallypannn, @sarahyurick
Special thanks to @talhaanwarch who opened 21 GitHub issues (!) and participated in numerous discussions during v0.3.0 development. His feedback was incredibly valuable when diagnosing issues and improving the user experience throughout AutoGluon!
Full Contributor List (ordered by # of commits):
@Innixma, @zhreshold, @jwmueller, @gradientsky, @sxjscience, @ValerioPerrone, @taesup-aws, @sallypannn, @rxjx, @sarahyurick
tree_method='hist'
for improved performance. @Innixma (#1239)groups
parameter. Now users can specify the exact split indices in a groups
column when performing model bagging. This solution leverages sklearn's LeaveOneGroupOut cross-validator. @Innixma (#1224)use_bag_holdout
argument. @Innixma (#1105)predictor.features()
to get the original feature names used during training. @Innixma (#1257)problem_type
support to ImagePredictor. @sallypannn (#1165)predict_proba
. @Innixma (#1206)eval_metric='average_precision'
. @rxjx (#1092)__version__
. @Innixma (#1122)