:rocket: Build and manage real-life ML, AI, and data science projects with ease!
docs coming!
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.11...2.11.0
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.10...2.10.11
@pypi
dependenciesThis release adds support for git repositories as dependencies for the @pypi
decorator.
You can pin the version of the dependency with "@branch"
, "@commit"
, "@tag"
or the head of the main branch with ""
.
Under the hood @pypi
will resolve the dependency to a commit at the moment of creating the environment.
Currently conda and pypi environments are created in advance before deploying on remote platforms, this includes gathering all the dependencies and bundling them up. The dependencies for pypi have been limited to binaries only in order to support cross-platform deployments.
For the git sources, we build a wheel from the source and check whether it is compatible with the target platform. This limits using sources where the output wheel is a platform specific binary, for example if building on an ARM platform but trying to deploy on x86.
@pypi
by @savingoyal in https://github.com/Netflix/metaflow/pull/1681
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.9...2.10.10
@batch
This release adds support for mounting existing EFS volumes with @batch
You can specify one or multiple volumes to be mounted with the batch decorators efs_volumes=
attribute. The volumes will be mounted under /mnt/volume-name
by default, but you can also specify a custom mount point separated by a semicolon with the volume.
Examples
@batch(efs_volumes="fs-001")
@step
...
@batch(efs_volumes=["fs-002", "fs-003"])
@step
...
@batch(efs_volumes="fs-003:/mnt/custom-mountpoint"])
@step
@batch
This release also adds support for custom mount points for host volumes with @batch
.
Example
@batch(host_volumes="/home:/mnt/host-homedir")
@step
@conda
and @pypi
changes@pypi
can now handle dependencies with special characters in versions
Fixes a @conda
bug where environments were not appending to PATH
during runtime if the decorator was applied implicitly to the step. This bug affected dependencies that install a binary, causing them to not be found during runtime due to the environment missing from the PATH
.
Previously the lifecycle methods task_pre_step
and task_decorate
were being called consecutively on a per-decorator basis. This release changes the lifecycle so that task_pre_step
is called on all decorators before any task_decorate
is called.
This release improves the way that datastore dependencies are checked, in order to avoid an issue where previously run metadata could be published to the metadata service, but the execution fails to start due to missing dependencies for pushing data to the datastore.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.8...2.10.9
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.7...2.10.8
This release fixes support for pip environment variables that specify a custom location for the config file (PIP_CONFIG_FILE
or PIP_CONFIG
).
The release also adds support for defining a custom index-url through the pip supported environment variable PIP_INDEX_URL
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.6...2.10.7
pypi
decoratorThe pypi
decorator had a bug that caused it to be treated as disabled
unless specifically passing disabled=False
as an attribute to it.
This release fixes the default case so that pypi
environments activate correctly.
This release adds a METAFLOW_DEBUG_TRACING
environment variable to toggle more verbose output for tracing related issues.
By default any errors related to missing tracing dependencies are now silenced completely, in order to not affect platforms that might want tracing environment variables present for all deployments, whether they have the required dependencies or not.
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.5...2.10.6
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.4...2.10.5
With this release it is possible to gather telemetry data using an opentelemetry endpoint.
Specifying an endpoint in one of the environment variables
METAFLOW_OTEL_ENDPOINT
METAFLOW_ZIPKIN_ENDPOINT
will enable the corresponding tracing provider.
Some additional dependencies are required for the tracing functionality in the execution environment. These can be installed in the base Docker image, or supplied through a conda environment. The relevant packages are
opentelemetry-sdk, opentelemetry-api, opentelemetry-instrumentation, opentelemetry-instrumentation-requests
and depending on your endpoint, either opentelemetry-exporter-otlp
or opentelemetry-exporter-zipkin
pypi
decoratorThe pypi
decorator now supports using a custom index in the users Pip configuration under global.index-url
.
This enables using private indices, even ones that require authentication.
For example the following would set up one authenticated and two extra non-authenticated indices for package resolution
pip config set global.index-url "https://user:[email protected]"
pip config set global.extra-index-url "https://extra.example.com https://extra2.example.com"
resources
decoratorIt is now possible to specify the ephemeral storage size for Kubernetes jobs when using the resources
decorator with the disk=
attribute.
argo-workflows status
commandAdds a command for easily checking the current status of a workflow on Argo workflows.
python flow.py argo-workflows status [run-id]
There was an issue where relying solely on the Kubernetes apiserver for generating random pod names was resulting in significant collisions with sufficiently large number of executions.
This release adds more randomness to the pod names besides what is generated by Kubernetes.
resources
decorator in combination with step functionsThis release fixes an issue where deploying flows on AWS Step Functions was failing in the following cases
@resources(shared_memory=)
with any value@resources
and @batch(use_tmpfs=True)
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.3...2.10.4
pandas.DataFrame
indexes for default card by @amerberg in https://github.com/Netflix/metaflow/pull/1574
ArgoEvent.publish
by @savingoyal in https://github.com/Netflix/metaflow/pull/1587
Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.2...2.10.3