Metaflow Versions Save

:rocket: Build and manage real-life ML, AI, and data science projects with ease!

2.10.7

5 months ago

Improvements

pypi decorator enhancements

This release fixes support for pip environment variables that specify a custom location for the config file (PIP_CONFIG_FILE or PIP_CONFIG).

The release also adds support for defining a custom index-url through the pip supported environment variable PIP_INDEX_URL

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.6...2.10.7

2.10.6

6 months ago

Improvements

Fix environment activation issue with pypi decorator

The pypi decorator had a bug that caused it to be treated as disabled unless specifically passing disabled=False as an attribute to it. This release fixes the default case so that pypi environments activate correctly.

Add debug flag to tracing

This release adds a METAFLOW_DEBUG_TRACING environment variable to toggle more verbose output for tracing related issues.

By default any errors related to missing tracing dependencies are now silenced completely, in order to not affect platforms that might want tracing environment variables present for all deployments, whether they have the required dependencies or not.

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.5...2.10.6

2.10.5

6 months ago

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.4...2.10.5

2.10.4

6 months ago

Features

Support for tracing

With this release it is possible to gather telemetry data using an opentelemetry endpoint.

Specifying an endpoint in one of the environment variables

  • METAFLOW_OTEL_ENDPOINT
  • METAFLOW_ZIPKIN_ENDPOINT

will enable the corresponding tracing provider.

Some additional dependencies are required for the tracing functionality in the execution environment. These can be installed in the base Docker image, or supplied through a conda environment. The relevant packages are

opentelemetry-sdk, opentelemetry-api, opentelemetry-instrumentation, opentelemetry-instrumentation-requests

and depending on your endpoint, either opentelemetry-exporter-otlp or opentelemetry-exporter-zipkin

Custom index support for the pypi decorator

The pypi decorator now supports using a custom index in the users Pip configuration under global.index-url. This enables using private indices, even ones that require authentication.

For example the following would set up one authenticated and two extra non-authenticated indices for package resolution

pip config set global.index-url "https://user:[email protected]"
pip config set global.extra-index-url "https://extra.example.com https://extra2.example.com"

Specify Kubernetes job ephemeral storage size through resources decorator

It is now possible to specify the ephemeral storage size for Kubernetes jobs when using the resources decorator with the disk= attribute.

Introduce argo-workflows status command

Adds a command for easily checking the current status of a workflow on Argo workflows.

python flow.py argo-workflows status [run-id]

Improvements

Add more randomness to Kubernetes pod names to avoid collisions

There was an issue where relying solely on the Kubernetes apiserver for generating random pod names was resulting in significant collisions with sufficiently large number of executions.

This release adds more randomness to the pod names besides what is generated by Kubernetes.

Fix issues with resources decorator in combination with step functions

This release fixes an issue where deploying flows on AWS Step Functions was failing in the following cases

  • @resources(shared_memory=) with any value
  • combining @resources and @batch(use_tmpfs=True)

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.3...2.10.4

2.10.3

6 months ago

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.2...2.10.3

2.10.2

7 months ago

2.10.2

Features

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.0...2.10.2

2.10.0

7 months ago

Coming soon!

Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.15...2.10.0

2.9.15

7 months ago

Improvements

Improve the performance of parallel_map

We now check for processes in the order in which they complete not in the order in which they are launched. This also increases the likelihood of failing fast.

Fix issues with the environment escape mechanism

Deadlocks and errors could occur when using the environment escape mechanism in two cases: (a) GC would occur at an inopportune moment or (b) subprocesses were involved. Both issues were fixed.

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.14...2.9.15

2.9.14

7 months ago

Improvements

Fixes merging of log lines

This release fixes an issue with merging broken log lines.

Fix issue with using LD_LIBRARY_PATH with Conda environments

In a Conda environment, it is sometimes necessary to set LD_LIBRARY_PATH to first include the Conda's environment libraries before anything else. Prior to this release, this used to cause issues with the escape hatch.

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.13...2.9.14

2.9.13

8 months ago

Bug fix

Revert annotations changes to fix a regression

The recent annotations feature introduced an issue where project, flow_name or user annotations are not being populated for Kubernetes. This release reverts the changes.

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.9.12...2.9.13