Metaflow Versions Save

:rocket: Build and manage real-life ML, AI, and data science projects with ease!

2.11.5

2 weeks ago

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.11.4...2.11.5

2.11.4

1 month ago

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.11.3...2.11.4

2.11.3

1 month ago

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.11.2...2.11.3

2.11.2

1 month ago

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.11.1...2.11.2

2.11.1

1 month ago

Features

Auto generated stubs package

You can now install metaflow[stubs] to install an additional metaflow-stubs package that provides type hints for Metaflow decorators (and other objects). Some hints were previously included inline but this package expands their use.

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.11.0...2.11.1

2.11.0

2 months ago

What's Changed

docs coming!

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.11...2.11.0

2.10.11

2 months ago

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.10...2.10.11

2.10.10

2 months ago

Features

Support git repositories as @pypi dependencies

This release adds support for git repositories as dependencies for the @pypi decorator.

Usage

You can pin the version of the dependency with "@branch", "@commit", "@tag" or the head of the main branch with "". Under the hood @pypi will resolve the dependency to a commit at the moment of creating the environment.

Limitations

Currently conda and pypi environments are created in advance before deploying on remote platforms, this includes gathering all the dependencies and bundling them up. The dependencies for pypi have been limited to binaries only in order to support cross-platform deployments.

For the git sources, we build a wheel from the source and check whether it is compatible with the target platform. This limits using sources where the output wheel is a platform specific binary, for example if building on an ARM platform but trying to deploy on x86.

What's Changed

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.9...2.10.10

2.10.9

2 months ago

Features

EFS Volume support for @batch

This release adds support for mounting existing EFS volumes with @batch

Usage

You can specify one or multiple volumes to be mounted with the batch decorators efs_volumes= attribute. The volumes will be mounted under /mnt/volume-name by default, but you can also specify a custom mount point separated by a semicolon with the volume.

Examples

@batch(efs_volumes="fs-001")
@step
...
@batch(efs_volumes=["fs-002", "fs-003"])
@step
...
@batch(efs_volumes="fs-003:/mnt/custom-mountpoint"])
@step

Support custom mount point for host volumes with @batch

This release also adds support for custom mount points for host volumes with @batch. Example

@batch(host_volumes="/home:/mnt/host-homedir")
@step

Improvements

@conda and @pypi changes

@pypi can now handle dependencies with special characters in versions

Fixes a @conda bug where environments were not appending to PATH during runtime if the decorator was applied implicitly to the step. This bug affected dependencies that install a binary, causing them to not be found during runtime due to the environment missing from the PATH.

Decorator lifecycle improvement

Previously the lifecycle methods task_pre_step and task_decorate were being called consecutively on a per-decorator basis. This release changes the lifecycle so that task_pre_step is called on all decorators before any task_decorate is called.

Better checking of datastore dependencies

This release improves the way that datastore dependencies are checked, in order to avoid an issue where previously run metadata could be published to the metadata service, but the execution fails to start due to missing dependencies for pushing data to the datastore.

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.8...2.10.9

2.10.8

3 months ago

Improvements

  • Warnings about regex patterns not being raw strings were fixed.
  • An issue with how the environment escape dealt with aliased modules was fixed.

What's Changed

New Contributors

Full Changelog: https://github.com/Netflix/metaflow/compare/2.10.7...2.10.8