Build, test, deploy, iterate - Dev and prod tool for data science pipelines
Prodmodel is a build system for data science pipelines. Users, testers, contributors are welcome!
A build system is a DAG of rules
(transformations), inputs
and targets
.
In Prodmodel inputs
can be
A rule
is transforming any of the above to an output (which can in turn be depended on by other rules). Therefore rules need to be
re-executed (and their outputs re-created) if any of their dependencies change. Prodmodel keeps track all of these dependencies.
The outputs of the rules
are targets
. Every target
corresponds to an output (e.g. a model or a dataset). These outputs
are cached and version controlled.
Prodmodel therefore ensures
Every rule is a statically typed function, where the inputs are targets, data, or configs. The execution of a rule outputs some data (e.g. a different feature set or a model), which can be used in other rules.
In order to use Prodmodel your code has to be structured as functions which the rules can call into.
Targets are created by rule functions. Targets can be executed to generate output files. IterableDataTarget
is a special target
which can be used as an iterable of dicts
to make iterating over datasets easier. Regular DataTargets
can represent any
Python object.
Prodmodel requires at least Python3.6. Use pip to install prodmodel.
pip install prodmodel --user
Create a build.py
file in your data science folder. The build file contains references to your inputs and the build rules you can execute.
from prodmodel.rules import rules
csv_data = rules.data_source(file='data.csv', type='csv', dtypes={...})
my_model = rules.transform(objects={'data': csv_data}, file='kmeans.py', fn='compute_kmeans')
Now you can build your model by running prodmodel my_model
from the directory of build.py
,
or prodmodel <path_to_my_directory>:my_model
from any directory.
Prodmodel creates a .prodmodel
directory under the home directory of the user to store log and config files.
Check out a complete example project for more examples.
The complete list of build rules can be found here.
Prodmodel searches for a config file under <user home dir>/.prodmodel/config
. The config file can be created manually
based on this template.
--force_external
: Some data sources are remote (e.g. an SQL server), therefore tracking changes is not always feasible.
This argument gives the user manual control over when to reload these data sources.--cache_data
: Cache local data files if changed. This can be useful for debugging / reproducibility by making sure every
data source used for a specific build is saved.--output_format
: One of none
, str
, bytes
and log
. The output format of the data produced by the build target
written to stdout.prodmodel ls <path_to_build>
to list targets in a build file where <path_to_build>
to the build file or its directory.prodmodel clean <target> --cutoff_date=<cutoff datetime>
to delete output cache files of a target created before
the cutoff datetime, which has to be in %Y-%m-%dT%H:%M%S
(YYYY-mm-ddTHH:MM:SS
) format.Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Feel free to email me at [email protected] if you have any question, need help or would like to contribute to the code.