Aqueducthq Aqueduct Versions Save

Aqueduct is no longer being maintained. Aqueduct allows you to run LLM and ML workloads on any cloud infrastructure.

v0.3.6

11 months ago

0.3.6

Released on June 6, 2023.

Key Features

  • Enables users to selectively or completely disable Aqueduct's artifact snapshotting functionality. When defining a workflow, users can now decide to disable snapshotting for an individual artifact or for all artifacts in a workflow; if selected, these artifacts will only be generated during computation time and then garbage collected. No data will be persisted beyond any data that is explicitly saved. See our documentation for more details.

Enhancements

  • Enables on-demand Kubernetes clusters on Google Cloud to use GPU nodes in an autoscaling capacity.

Bugfixes

  • Fixes bug where resources could not be deleted even when unused.
  • Fixes bug where workflows that succeeded with a warning check were shown in a regular success state instead of a warning state.
  • Fixes page load jitter when a large number of workflows are present.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.3.5...v0.3.6

All commits

v0.3.5

11 months ago

Released on May 31, 2023.

Key Features

  • Adds support for on-demand Kubernetes clusters on Google Cloud; you can now select the Google Cloud option when asking Aqueduct to create a Kubernetes cluster, and Aqueduct will automatically use the Google Kubernetes Engine. See our documentation here.
  • Adds support for Google Artifact Registry as a conatiner registry in Aqueduct. You can now use custom Docker images that are pushed to GAR. See our documentation here.

Bugfixes

  • Fixes bug where editing certain resource configurations from the UI would cause the resource edit dialog to error out.
  • Fixes bug where updating a workflow that has been previously registered but not run would cause the workflow to be duplicated.
  • Fixes bug where certain resources could be registered without specifying a name for the resource.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.3.4...v0.3.5

All commits

v0.3.4

1 year ago

Released on May 24, 2023.

Enhancements

  • Improves consistency of icon sizing on resources page.
  • Merges flow_id and name into a single argument when retrieving workflows from the SDK.
  • Adds ability to parametrize save operators when using AWS S3.

Bugfixes

  • Fixes bug where race condition could occur when syncing Airflow workflows to Aqueduct.
  • Fixes bug where listing Snowflake data objects would return an error.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.3.3...v0.3.4

All commits

v0.3.3

1 year ago

Released on May 17, 2023.

Enhancements

  • Adds ability to use a parameter in save operators to dynamically name saves.
  • Updates the artifact storage card on the resources page to be consistent with other resource cards.
  • Shows how many workflows are using notification resources.
  • Improves form validation with react-hook-forms.
  • Replaces the icon that signifies a workflow that has been registered but not executed.
  • On the resouces page, displays the Conda integration as modifying the Aqueduct server execution layer rather than as a separate execution layer.
  • Improves BigQuery error checking to ensure that a dataset exists before accessing it.

Bugfixes

  • Fixes bug where documentation links on the home page were broken.
  • Fixes bug where the SDK's generated docstrings were broken.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.3.2...v0.3.3

All commits

v0.3.2

1 year ago

Released on May 10, 2023.

Enhancements

  • Improves error handling when the user attempts to take an action without connecting to the Aqueduct server.
  • When connecting an S3 resource to Aqueduct, if the specified subdirectory does not exist, it is now automatically created.

Bugfixes

  • Replaces outdated integration terminology on the UI.
  • Fixes bug where previously created resource names were not editable.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.3.1...v0.3.2

All commits

v0.3.1

1 year ago

Key Features

  • Introduces the aqueduct.llm_op API and the aqueduct-llm package. Aqueduct now has support for invoking LLMs with a single API call and comes with pre-built Docker images optimized for executing LLMs on Kubernetes. The llm_op API supports both ad hoc execution, as pictured below, as well as batch execution over a list of inputs or a Pandas Series. See our documentation for more details. 
    from aqueduct import Client, llm_op

client = Client() # initialize Aqueduct client so we can check if the engine name below is valid
vicuna = llm_op('vicuna_7b', engine='my_k8s_engine')
vicuna('What is the best LLM?')
  • Reorganizes integrations around the concept of resources. Resources are any external tool, system, or API that Aqueduct can connect to; existing data and compute integrations are automatically converted into resources. A container registry resource is added in this release, and future releases will introduce new resource types. The recommended SDK API for accessing resources is now client.resource, with client.integration slated to deprecated in a future release.
  • Allows users to specify a custom Docker image when running an Aqueduct operator on Kubernetes. The Docker image is required to have the Aqueduct executor scaffolding installed; for more details, please see our documentation here.

Enhancements

  • Improves logging and error handling when an operator fails because it's able to successfully generate a result, typically in the setup phase.
  • Enables connecting a Databricks cluster to Aqueduct via the Python SDK.

Bugfixes

  • Fixes bug where installing pre-requisites for using Aqueduct-managed Kubernetes clusters would fail on an M1 Mac with certain configurations.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.2.12...v0.3.1

All commits

v0.2.12

1 year ago

Released on April 25, 2023.

Enhancements

  • Improves the describe method on integration objects in the SDK; if there was an error connecting or validating the integration, the error will be shown.
  • Eagerly validates integration connections, so users will be notified immediately if they are using a mis-configured integration when, for example, attempting to access data.
  • Replaces placeholder values in integration connection forms with more realistic values.
  • Adds support for the most recent version of numpy.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.2.11...v0.2.12

All commits * Fix isort in our code and automated linter by @kenxu95 in https://github.com/aqueducthq/aqueduct/pull/1227 * Bump numpy version to 1.24.2 by @kenxu95 in https://github.com/aqueducthq/aqueduct/pull/1228 * Eng 2635 Add automated periodic testing coverage for MariaDB by @eunice-chan in https://github.com/aqueducthq/aqueduct/pull/1225 * Track the execution state of every integration connection on the backend by @kenxu95 in https://github.com/aqueducthq/aqueduct/pull/1220 * [SDK] Fail when using any integrations that aren't connected yet by @kenxu95 in https://github.com/aqueducthq/aqueduct/pull/1223 * Improve placeholder values for integrations by @hsubbaraj-spiral in https://github.com/aqueducthq/aqueduct/pull/1233 * Refactor Spark Python Executor by @hsubbaraj-spiral in https://github.com/aqueducthq/aqueduct/pull/1231 * Fix documentation bug for missing local file by @Fanjia-Yan in https://github.com/aqueducthq/aqueduct/pull/1230 * Eng 2735 Add order by and limit parameters for V2 workflow results by @eunice-chan in https://github.com/aqueducthq/aqueduct/pull/1240 * Rc v0212 by @likawind in https://github.com/aqueducthq/aqueduct/pull/1249

v0.2.11

1 year ago

Released on April 18, 2023.

Enhancements

  • Extends the Aqueduct-Databricks integration to optionally support long-standing instance pools. If specified, Aqueduct will automatically use nodes in the instance pool, and otherwise, Aqueduct will create a cluster with new instances.
  • Improves form validation during integration connection to ensure that valid and invalid form fields are correctly detected and visualized.

Bugfixes

  • Fixes bug where MySQL dependency installation could fail on an M-series Mac.
  • Fixes bug where integration connections dialogs could re-render repeatedly, causing slight performance degradation.
  • Fixes bug where loading a Tensorflow object from a file could fail due to library version mismatches.
  • Fixes concurrency bug when multiple processes attempt to simultaneously create the same Kubernetes secret.
  • Fixes bug where the release notification banner introduced in the last release was not picking up new Aqueduct versions from PyPI.
  • Fixes bug where CUDA versions specified via the SDK may not have been faithfully installed at execution time.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.2.10...v0.2.11

All commits

v0.2.10

1 year ago

Released on April 11, 2023.

Enhancements

  • Enables subdirectory support when configuring the AWS S3 integration. When a subdirectory is specified, everything outside of that directory will be ignored by Aqueduct.
  • Extends support for creating artifact from the local file system to all types supported by Aqueduct.
  • If executing code on the Aqueduct server without Conda, the server will now proactively ensure that the Python version from your environment matches the server's Python environment. If there's a mismatch, an error will be thrown.
  • Adds a dismissable banner to the Aqueduct UI notifying users of new releases.

Bugfixes

  • Fixes bug where errors occurring during the execution of a metric were not being properly surfaced.
  • Fixes bug where checks with a severity level of warning were being shown as errors in check history.
  • Fixes bug where creating a schedule with helper functions on the SDK would fail for monthly schedules.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.2.9...v0.2.10

All commits

v0.2.9

1 year ago

Released on April 5, 2023.

Key Features

  • [Beta] Aqueduct now has support for loading dataframe and image data from the local filesystem as parameters. See client.create_param() for details.

Enhancements

  • Improves the artifact storage management process. The integrations page now shows which system is being used for artifact storage, and if a migration between artifact stores fails, the error will be surfaced on the integration details page until the user triggers a new artifact migration.
  • Adds metadata to read and write operators' sidesheets. Both sidesheets now show which system is being used for the IO operation, and the save operator now shows to what location and (if relevant) in what format the data is being saved.

Bugfixes

  • Fixes typos in MongoDB connection dialog.
  • Fixes bug where workflow status at the top of the workflow details page would not update in sync with other parts of the page.
  • Fixes a bug where stopping the Aqueduct server immediately after starting it could potentially trigger and then kill a workflow run that would then stay in a pending state permanently.

All Changes

https://github.com/aqueducthq/aqueduct/compare/v0.2.8...v0.2.9

All commits