Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
In this release we're announcing two experimental features, namely (1) ArrayNode map tasks, and (2) Execution Tags.
ArrayNodes are described more fully in RFC 3346, but the summary is that ArrayNode map tasks are a drop-in replacement for regular map tasks, the only difference being the submodule used to import the map_task function.
More explicitly, let's say you have this code:
from typing import List
from flytekit import map_task, task, workflow
@task
def t(a: int) -> int:
...
@workflow
def wf(xs: List[int]) -> List[int]:
return map_task(t)(a=xs)
In order to switch to using array node map tasks you should import map_task from the flytekit.experimental module like so:
from typing import List
from flytekit import task, workflow
from flytekit.experimental import map_task
@task
def t(a: int) -> int:
...
@workflow
def wf(xs: List[int]) -> List[int]:
return map_task(t)(a=xs)
Execution tags allow users to can discover their executions and other flyte entities more easily, by creating smarter groupings. The feature is described in this RFC.
As mentioned before, this feature is shipped in an experimental capacity, the idea being that we're going to incorporate the feedback of the community as we iterate. More work is expected to give prominence to the feature in flyteconsole, in the meanwhile, the feature is supported via Remote.
FLYTE_SDK_RICH_TRACEBACKS=0 is set by @fg91 in https://github.com/flyteorg/flytekit/pull/1760
https://github.com/flyteorg/flyteconsole/releases/tag/v1.8.6 https://github.com/flyteorg/flyteconsole/releases/tag/v1.8.7
Full Changelog: https://github.com/flyteorg/flyteconsole/compare/v1.8.5...v1.8.7
In this release we're announcing support for Flyte Agents, a new way of writing backend plugins, only now with a much more tightly integrated developer experience. Also lots of bug fixes all around in a buch of first-time contributors.
kubeflow.pytorch plugin instead of legacy pytorch plugin in https://github.com/flyteorg/flytekit/pull/1678 by @fg91In this release we're announcing support for writing backend plugins in python. This is in experimental state, feedback and bug reports are welcome!
This release contains a database migration that remediates an issue that we discovered with very old installations of Flyte. For more details, please read the Flyte [https://github.com/flyteorg/flyte/releases/tag/v1.5.1](1.5.1 release notes).
As mentioned in the previous release, we are working to improve performance investigations. In this release we're announcing:
Lots of features shipped in 1.6, including:
For a full changelog, go to https://github.com/flyteorg/flytekit/releases/tag/v1.6.0.
parent_id column into bigint only if necessary by @eapolinario in https://github.com/flyteorg/flyteadmin/pull/554
This is a patch release that only contains one change - https://github.com/flyteorg/flyteadmin/pull/560 which cherry-picks https://github.com/flyteorg/flyteadmin/pull/554.
PR #554 adds a migration that remediates an issue that we discovered with very old installations of Flyte. Basically one of the tables node_executions has a self-referencing foreign key. The main id column of the table is a bigint whereas the self-foreign-key parent_id was an int. This was a rooted in an early version of gorm and should not affect most users. Out of an abundance of caution however, we are adding a migration to patch this issue in a manner that minimizes any locking.
When you deploy this release of Flyte, you should make sure that you have more than one pod for Admin running. (If you are running the flyte-binary helm chart, this patch release does not apply to you at all. All those deployments should already have the correct column type.) When the two new migrations that #554 added runs, the first one may take an extended period of time (hours). However, this is entirely non-blocking as long as there is another Admin instance available to serve traffic.
The second migration is locking, but even on very large tables, this migration was over in ~5 seconds, so you should not see any noticeable downtime whatsoever.
The migration will also check to see that your database falls into this category before running (ie, the parent_id and the id columns in node_executions are mismatched). You can also do check this yourself using psql. If this migration is not needed, the migration will simply mark itself as complete and be a no-op otherwise.
parent_id column into bigint only if necessary by @eapolinario in https://github.com/flyteorg/flyteadmin/pull/554
Moved controller-runtime start out of webhook Run function by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/546
Fixing recovering of SKIPPED nodes by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/551
Remove resource injection on the node for container task by @ByronHsu in https://github.com/flyteorg/flytepropeller/pull/544
Infer GOOS and GOARCH from environment by @jeevb in https://github.com/flyteorg/flytepropeller/pull/552
fix makefile to read variables from environment and overrides by @jeevb in https://github.com/flyteorg/flytepropeller/pull/554
Remove BarrierTick by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/545
Check for TerminateExecution error and eat Precondition status by @EngHabu in https://github.com/flyteorg/flytepropeller/pull/553
Setting primaryContainerName by default on Pod plugin by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/555
Implement ability to specify additional/override annotations when using Vault Secret Manager by @pradithya in https://github.com/flyteorg/flytepropeller/pull/556
Maintaining Interruptible and OverwriteCache for reference launchplans by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/557
Added support for aborting task nodes reported as failures by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/541
Added support for EnvironmentVariables on ExecutionConfig by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/558
Fast fail if task resource requests exceed k8s resource limits by @hamersaw in https://github.com/flyteorg/flytepropeller/pull/488
@ByronHsu made their first contribution in https://github.com/flyteorg/flytepropeller/pull/544
We're laying the foundation for an improved experience to help performance investigations. Stay tuned for more details!
We can now submit Ray jobs to separate clusters (other than the one flytepropeller is running). Thanks to https://github.com/flyteorg/flyteplugins/pull/321.
Several bug fixes, including:
One of the improvements planned requires us to clean up our database migrations. We have done so in this release so you should see a series of new migrations.
These should have zero impact if you are otherwise up-to-date on migrations (which is why they are all labeled noop) but please be aware that it will add a minute or so to the
init container/command that runs the migrations in the default Helm charts. Notably, because these should be a no-op, they also do not come with any rollback commands.
If you experience any issues, please let us know.
Python 3.11 is now officially supported.
The data persistence layer was completely revamped. We now rely exclusively on fsspec to handle IO.
Most users will benefit from a more performant IO subsystem, in other words, no change is needed in user code.
The data persistence layer has undergone a thorough overhaul. We now exclusively utilize fsspec for managing input and output operations.
For the majority of users, the improved IO subsystem provides enhanced performance, meaning that no modifications are required in their existing code.
This change opened the door for flytekit to rely on fsspec streaming capabilities. For example, let's say we want to stream a file, now we're able to do:
@task
def copy_file(ff: FlyteFile) -> FlyteFile:
new_file = FlyteFile.new_remote_file(ff.remote_path)
with ff.open("r", cache_type="simplecache", cache_options={}) as r:
with new_file.open("w") as w:
w.write(r.read())
return new_file
This feature is marked as experimental. We'd love feedback on the API!
We can use functools.partial to "freeze" some task arguments. Let's take a look at an example where we partially fix the parameter for a task:
@task
def t1(a: int, b: str) -> str:
return f"{a} -> {b}"
t1_fixed_b = functools.partial(t1, b="hello")
@workflow
def wf(a: int) -> str:
return t1_fixed_b(a=a)
Notice how calls to t1_fixed_b do not need to specify the b parameter.
This also works for MapTasks in a limited capacity. For example:
from flytekit import task, workflow, partial, map_task
@task
def t1(x: int, y: float) -> float:
return x + y
@workflow
def wf(y: List[float]):
partial_t1 = partial(t1, x=5)
return map_task(partial_t1)(y=y)
We are currently seeking feedback on this feature, and as a result, it is labeled as experimental for now.
Also worth mentioning that fixing parameters of type list is not currently supported. For example, if we try to register this workflow:
from functools import partial
from typing import List
from flytekit import task, workflow, map_task
@task
def t(a: int, xs: List[int]) -> str:
return f"{a} {xs}"
@workflow
def wf():
partial_t = partial(t, xs=[1, 2, 3])
map_task(partial_t)(a=[1, 2])
We're going to see this error:
❯ pyflyte run workflows/example.py wf
Failed with Unknown Exception <class 'ValueError'> Reason: Map tasks do not support partial tasks with lists as inputs.
Map tasks do not support partial tasks with lists as inputs.
Multiple bug fixes around waiting for external inputs. Better support for dataclasses in the launch form.