Elyra extends JupyterLab with an AI centric approach.
Elyra installations are pre-configured by default to support pipeline execution in local environments (via JupyterLab) and remote environments (Kubeflow Pipelines and Apache Airflow). Beginning with this release you can customize Elyra to only expose runtimes you are planning to use.
Refer to the new Configuring the pipeline editor user guide topic for details.
Kubeflow Pipelines runtime configurations now support authentication using static bearer tokens. To leverage this authentication mechanism, select EXISTING_BEARER_TOKEN
and specify the token value. Elyra passes the specified token as-is to Kubeflow Pipelines.
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.14.3...v3.15.0
bootstrapper.py
by @cjackal in https://github.com/elyra-ai/elyra/pull/3106
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.14.2...v3.14.3
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.14.1...v3.14.2
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.14.0...v3.14.1
Pipeline parameters allow for customization of pipeline runs and pipeline exports, without to need to modify the pipeline. A pipeline parameter is a typed variable that can be applied to generic or custom nodes. The Kubeflow Pipelines pipeline editor includes a new PIPELINE PARAMETERS
tab, where pipeline parameters can be defined.
To make a parameter value available to a generic node, select it from the pipeline parameters list:
Jupyter notebooks and scripts can access the selected parameters and their associated values using environment variables.
To pass a parameter value to a custom node, choose parameter as input and select the desired parameter:
Note that for custom nodes the parameter list only includes parameters that are type-compatible.
You can customize parameter values in the pipeline editor during pipeline submission and pipeline export:
You can also customize parameter values in the pipelines UI in the Kubeflow Central Dashboard.
For generic components components you can now specify a custom GPU vendor. Note that the appropriate device plugin must be installed in Kubernetes or node execution will fail. The default vendor remains nvidia.com/gpu
.
You can now customize the filename when exporting pipelines. Custom filenames make it easier to retain multiple versions of the same pipeline.
You can now use markdown in comments.
Python 3.11 is now officially supported:
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.13.0...v3.14.0
The amount of shared memory that Kubernetes makes available to pods that execute pipeline nodes can now be customized for individual nodes or all nodes using pipeline defaults.
The data volume node property can now be optionally configured to mount a volume in read-only mode (preventing accidental overwrites) and to mount a sub path within the specified volume. Use these new property attributes to impose stricter access control on nodes that should not have unlimited access to mounted data volumes.
The Elyra Pipeline Editor and the elyra-pipeline export
CLI command can now export Kubeflow Pipelines Python DSL in addition to the already supported YAML format.
cos_object_prefix
pipeline property by @kiersten-stokes in https://github.com/elyra-ai/elyra/pull/2972
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.12.0...v3.13.0
This release requires pipeline migration.
Kubernetes labels can now be defined as pipeline defaults or assigned to individual nodes.
Use labels to assign identifying metadata to the Kubernetes pods that execute the node.
Some runtime environments support node output caching, reducing the need to re-execute nodes, which can improve performance and reduce resource usage. For the Kubeflow Pipelines runtime you can now disable node caching for custom nodes that produce output in a non-deterministic way. Node caching can be disabled for all nodes by specifying a pipeline default or for individual nodes.
Generic nodes are currently never cached, due to an Elyra limitation.
Pipeline default properties and node properties can now be configured using custom widgets:
Refer to the documentation for widget descriptions:
In previous releases string-based input properties for custom components could only be provided by entering text or selecting the output of an upstream node. The text input widget in the pipeline editor was extended to also allow for text input from a local file:
In the example above local file url.txt
contains a valid URL. The content of this file is not stored in the .pipeline
file, only its name and location. The file is only processed by Elyra when you submit or export a pipeline using the pipeline editor or the elyra-pipeline
CLI.
The elyra-pipeline describe
CLI command was extended to list input files like these as local file dependencies that must be distributed with the .pipeline
file:
$ elyra-pipeline describe download.pipeline
...
Pipeline name: download
Description: None specified
Pipeline type: KUBEFLOW_PIPELINES
Pipeline format version: 7.5
Pipeline runtime: Kubeflow Pipelines
Number of generic nodes: 0
Number of custom nodes: 1
Script dependencies: None specified
Notebook dependencies: None specified
Local file dependencies:
- url.txt
...
The Apache Airflow package component catalog connector, the Apache Airflow provider package component catalog connector, and the URL component catalog connector now support secured environments where SSL server authenticity can only be validated using certificates based on private public key infrastructure with root and optionally intermediate certificate authorities that are not publicly trusted. Refer to the connector documentation links for details.
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.11.0...v3.12.0
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.11.0...v3.11.1
The JupyterLab launcher now includes a What's new
tile in the Elyra category, which links to the release summary of the release you are using, e.g. https://github.com/elyra-ai/elyra/releases/tag/v3.11.0 for Elyra version 3.11. The release summary highlights new features and provides links to release specific resources.
This feature is currently experimental. The Elyra Python editor was extended to make it easier to use the JupyterLab debugger. Refer to the user guide for more information.
This feature is currently experimental. The Elyra family of editors for JupyterLab now includes a Scala code editor. This editor can also be installed as a stand-alone extension from PyPI.
Note that Scala files are not supported by the Visual Pipeline Editor.
The Visual Pipeline Editor now allows for optional input of Kubernetes pod tolerations. Tolerations can be defined as pipeline defaults (applying to all nodes) and for individual nodes and are supported for Kubeflow Pipelines and Apache Airflow.
The Visual Pipeline Editor now allows for optional input of Kubernetes [pod] annotations. Annotations can be defined as pipeline defaults (applying to all nodes) and for individual nodes and are supported for Kubeflow Pipelines and Apache Airflow.
In earlier releases the Visual Pipeline Editor capped CPU and RAM resource requests at 99. The caps have been removed.
The Apache Airflow package component catalog connector, the Apache Airflow provider package component catalog connector, and the URL component catalog connector, now support locally stored files as source.
Locally stored files can improve the performance of the Visual Pipeline Editor by eliminating the need to download them from remote locations.
Runtime configurations provide Elyra access to external resources, such as Kubeflow Pipelines or Apache Airflow for scalable pipeline execution. In prior releases runtime configurations only allowed for specification of a single URL for cloud object storage. This limitation causes issues in the Elyra UI when object storage deployment policies are configured to enforce separation of read-only and write operations.
To support those deployments it's now possible to optionally configure a read-only endpoint (Public Cloud Object Storage Endpoint
) in addition to the existing write endpoint (Cloud Object Storage Endpoint
). Refer to the documentation for details.
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.10.1...v3.11.0
Full Changelog: https://github.com/elyra-ai/elyra/compare/v3.10.0...v3.10.1