Low-code framework for building custom LLMs, neural networks, and other AI models
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.7...v0.7.1
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.6.3...v0.6.4
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.6.2...v0.6.3
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.6.1...v0.6.2
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.6...v0.6.1
Ludwig 0.6 introduces several exciting features focused on modeling, deployment, and testing that make it more flexible, reliable, and easy to use in production.
Additional improvements include a new global configuration section, time-based dataset splitting and more flexible hyperparameter optimization configurations. Read more about each specific feature below.
If you are learning about Ludwig for the first time, or if these new features are relevant and exciting to your research or application, we'd love to hear from you. Join our Ludwig Slack Community here.
Historically, Ludwig has been built around a single, flexible neural network architecture called ECD (for Encoder-Combiner-Decoder). With the release of 0.6 we are, adding support for a different model architecture: gradient-boosted tree models (GBM).
This is motivated by the fact that tree models still outperform neural networks on some tabular datasets, and the fact that tree models are generally less compute-intensive, making them a better choice for some applications. In Ludwig, users can now experiment with both neural and tree-based architectures within the same framework, taking advantage of all of the additional functionalities and conveniences that Ludwig offers like: preprocessing, hyperparameter optimization, integration with different backends (local
, ray
, horovod
), and interoperability with different data sources (pandas
, dask
, modin
).
Install the tree
extra package with pip install ludwig[tree]
. After the installation, you can use the new gbm
model type in the configuration. Ludwig will default to using the ECD architecture, which can be overridden as follows to use GBM:
In some initial benchmarking we found that GBMs are particularly performant on smaller tabular datasets and can sometimes deal better with class imbalance compared to neural networks. Stay tuned for a more in-depth blogpost on the topic. Like the ECD neural networks, GBMs can be sensitive to hyperparameter values, and hyperparameter tuning is important to get a well-performing model.
Under the hood, Ludwig uses LightGBM for training gradient-boosted tree models, and the LightGBM trainer
parameters can be configured in the trainer section of the configuration. For serving, the LightGBM model is converted to a PyTorch graph using Hummingbird for efficient evaluation and inference.
Ludwig's initial support for GBM is limited to tabular data (binary, categorical and numeric features) with a single output feature target.
Suppose your model outputs a class probability of 90%. Is there a 90% chance that the model prediction is correct? Do the probabilities given by your model match the true likelihood of the data? With deep neural networks, they often don't.
Drawing on the methods described in On Calibration of Modern Neural Networks (Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger), Ludwig now supports temperature scaling for binary and category output features. Temperature scaling brings a model's output probabilities closer to the true likelihood while preserving the same accuracy and top k predictions.
To enable calibration, add calibration: true
to any binary or category output feature configuration:
With calibration enabled, Ludwig will find a scale factor (temperature) which will bring the class probabilities closer to their true likelihoods in the validation set. The calibration scale factor is determined in a short phase after training is complete. If no validation split is provided, the training set is used instead.
To visualize the effects of calibration in Ludwig, you can use Calibration Plots, which bin the data based on model probability and plot the model probability (X) versus observed (Y) for each bin (see code examples).
In a perfectly calibrated model, the observed probability equals the predicted probability, and all predictions will land on the dotted line y=x
. In this example using the forest cover dataset, the uncalibrated model in blue gives over-confident predictions near the left and right edges close to probability values of 0 or 1. Temperature scaling learns a scale factor of 0.51
which improves the calibration curve in orange, moving it closer to y=x
.
Calibration is currently limited to models with binary
and category
output features.
Ludwig configurations are flexible by design, as they internally map to Python function signatures. This allows configurations for expressive configurations with many parameters for the users to play with, but we have found that users would too easily have typos in their configs like incorrect value types or other syntactical inconsistencies that were not easy to catch.
We have now formalized the Ludwig config with a strongly typed schema, serving as a centralized source of truth for parameter documentation and config validation. Ludwig validation now explicitly restricts each parameter's values to valid ones, decreasing the chance of syntactical and logical errors and signaling immediately to the user where the issues lie, before processing data or starting training. Schemas also provide many future benefits including autocompletion.
We have also restructured the way that encoders and decoders are configured to now use a nested structure, consistent with other modules in Ludwig such as combiners and loss.
As these changes impact what constitutes a valid Ludwig config, we also introduced a mechanism for ensuring backward compatibility that invisibly and automatically upgrades older configs to the current config structure.
We hope with the new Ludwig schema and the improved encoder/decoder nesting structure, that you find using Ludwig to be a much more robust and user friendly experience!
In Ludwig 0.5, users could specify global preprocessing parameters on a per-feature-type basis through the preprocessing section in Ludwig configs. This is useful if users know they always want to apply certain transformations to their data for every feature of the same type. However, there was no equivalent mechanism for global encoder, decoder or loss related parameters.
For example, say we have a mammography dataset to predict breast cancer that contains many categorical features. In Ludwig 0.5, we might define our input features with encoder parameters in the following way:
Here, the problem is that we have to redefine the same encoder parameters (type
, dropout
, and embedding_size
) for each of the input features if we want to override the default value across all categorical features.
In Ludwig 0.6, we are introducing a new defaults section within the Ludwig config to define feature-type defaults for preprocessing
, encoders
, decoders
, and loss
. Default preprocessing and encoder configurations will be applied to all input_features
of that feature type, while decoder
and loss
configurations will be applied to all output_features
of that feature type.
Note that you can still specify feature specific parameters as usual, and these will override any default parameter values that come from the global defaults
section.
The same mammography config above could be defined in the following, much more concise way in Ludwig 0.6:
Here, the encoder defaults for type
, dropout
and embedding_size
are applied to all three categorical features. The he_normal
embedding initializer is only applied to tumor_size
and inv_nodes
since we didn't specify this parameter in their feature definitions, but breast_quadrant
will use the glorot_normal
initializer since it will override the value from the defaults
section.
Additionally, in Ludwig 0.6, we have moved all global feature-type preprocessing within this new defaults section from the preprocessing section.
The defaults section enables the same fine-grained control with the benefit of making your config easier to define and read.
The defaults section has also been added to hyperopt, so that users can define feature-type level parameters for individual trials. This makes the definition of the hyperopt search space more convenient, without the need to define individual parameters for each of the features in instances where the dataset has a large number of input or output features.
For example, if you want to hyperopt over different encoders for all text features for each of the trials, one can do so by defining a parameter this way:
This will sample one of the three encoders for text features and apply it to all the text features for that particular trial.
We have extended the range of hyperopt parameters to support parameter choices that consist of partial or complete blocks of nested Ludwig config sections. This allows users to search over a set of Ludwig configs, as opposed to needing to specify config params individually and search over all combinations.
To provide a parameter that represents a full top-level Ludwig config, the .
key name can be used.
For example, we can define a hyperopt search space where we sample partial Ludwig configs in the following way would create hyperopt samples that look like the following:
In Ludwig v0.6, we improved the TorchScript model export functionality, making it easier than ever to train and deploy models for high performance inference.
At the core of our implementation is a pipeline-based approach to exporting models. After training a Ludwig model, users can run the export_torchscript
command in the CLI, or call LudwigModel.save_torchscript
. If model training was performed on a GPU device, doing so produces three new TorchScript artifacts:
These artifacts represent a single LudwigModel as three modules, each separated by stage: preprocessing, prediction, and postprocessing. These artifacts can be pipelined together using the InferenceModule
class method InferenceModule.from_directory
, or with some tools such as NVIDIA Triton.
One of the most significant benefits is that TorchScripted models are backend and environment independent and different parts can run on different hardware to maximize throughput. They can be loaded up in either a C++ or Python backend, and in either, minimal dependencies are required to run model inference. Such characteristics ensure that the model itself is both highly portable and backward compatible.
In Ludwig v0.6, we have added the ability to split based on a date column such that the data is ordered by date (ascending) and then split into train-validation-test along the time dimension. To make this possible, we have reworked the way splitting is handled in the Ludwig configuration to support a dedicated split
section:
In this example, by setting probabilities: [0.7, 0.1, 0.2]
, the earliest 70% of the data will be used for training, the middle 10% used for validation, and the last 20% used for testing.
This feature is important to support backtesting strategies where the user needs to know if a model trained on historical data would have performed well on unseen future data. If we were to use a uniformly random split strategy in these cases, then the model performance may not reflect the model's ability to generalize well if the data distribution is subject to change over time. For example, imagine a model that is predicting housing prices. If we both train and test on data from around the same time, we may fool ourselves into believing our model has learned something fundamental about housing valuations when in reality it might just be basing its predictions on recent trends in the market (trends that will likely change once the model is put into production). Splitting the training from the test data along the time dimension is one way to avoid this false sense of confidence, by showing how well the model should do on unseen data from the future.
Prior to Ludwig v0.6, the preprocessing configuration supported splitting based on a split column, split probabilities (train-val-test), or stratified splitting based on a category, all of which were flattened into the top-level of the preprocessing section:
This approach was limiting in that every new split type required reconciling all of the above params and determining how they should interact with the new type. To resolve this complexity, all of the existing split types have been similarly reworked to follow the new structure supported for datetime splitting.
Splitting by row at random (default):
Splitting based on a fixed column.
Stratified splits using a chosen stratification category column.
Be on the lookout as we continue to add additional split strategies in the future to support advanced usage such as bucketed backtesting. If you are interested in these kinds of scenarios, please reach out!
A significant step was taken in this release to improve the code quality of Ludwig components, e.g., encoders
, combiners
, and decoders
. Deep neural networks have many layers composed of a large number of parameters that must be updated to converge to a solution. Depending on the particular algorithm, the code for updating parameters during training can be quite complex. As a result, it is near impossible for a developer to reason through an analysis that confirms model parameters are updated.
To address this difficulty, we implemented a reusable utility to perform a quick sanity check to ensure parameters, such as tensor weights and biases, are updated during one cycle of a forward-pass / backward-pass / optimizer step. This work was inspired by these earlier blog postings: How to unit test machine learning code and Testing Your PyTorch Models with Torcheck.
This utility was added to unit tests for existing Ludwig components. With this addition, unit tests for Ludwig now ensure the following:
The above is an example of a unit test. First, it sets the random number seed to ensure repeatability. Next, the test instantiates the Ludwig component and processes synthetic data to ensure the component does not raise an error and that the output has the expected shape. Finally, the unit test checks if the parameters are updated under the different combinations of configuration settings.
In addition to the new parameter update check utility, Ludwig's Developer Guide contains instructions for using the utility. This allows an advanced user or a contributor, who is developing custom encoders, combiners, or decoders, to ensure the quality of their custom component.
Ludwig thriving open source community gathers on Slack, join it to get involved!
If you are interested in adopting Ludwig in the enterprise, check out Predibase, the declarative ML platform that connects with your data, manages the training, iteration, and deployment of your models, and makes them available for querying, reducing time to value of machine learning projects.
InferenceModule
as a pipelined module with separate preprocessor, predictor, and postprocessor modules by @brightsparc in https://github.com/ludwig-ai/ludwig/pull/2105
FloatOrAuto
and IntegerOrAuto
schema fields, and use them. by @justinxzhao in https://github.com/ludwig-ai/ludwig/pull/2219
inference_utils.py
by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2213
BaseTrainerConfig
an abstract class by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2273
--device
argument to export_torchscript
CLI command by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2275
ParameterMetadata
to JSON by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2348
OneOfField
that accepts other fields as arguments. by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2285
cell_type
field schema by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2428
export_torchscript
by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2459
batch_size: auto
for CPU-only training by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2455
Congratulations to our new contributors!
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.5.3...v0.6
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.6.beta...v0.6rc1
InferenceModule
as a pipelined module with separate preprocessor, predictor, and postprocessor modules by @brightsparc in https://github.com/ludwig-ai/ludwig/pull/2105
FloatOrAuto
and IntegerOrAuto
schema fields, and use them. by @justinxzhao in https://github.com/ludwig-ai/ludwig/pull/2219
inference_utils.py
by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2213
BaseTrainerConfig
an abstract class by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2273
--device
argument to export_torchscript
CLI command by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2275
ParameterMetadata
to JSON by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2348
OneOfField
that accepts other fields as arguments. by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2285
cell_type
field schema by @ksbrar in https://github.com/ludwig-ai/ludwig/pull/2428
export_torchscript
by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2459
batch_size: auto
for CPU-only training by @geoffreyangus in https://github.com/ludwig-ai/ludwig/pull/2455
Full Changelog: https://github.com/ludwig-ai/ludwig/compare/v0.5.3...v0.6.beta