Marquez Versions Save

Collect, aggregate, and visualize a data ecosystem's metadata

0.46.0

2 months ago

Changed

  • Web: various revisions #2770 @phixMe
    Includes clean up of issues in the UI and removal of non-useful elements.

Fixed

  • Streaming API: fix behaviour for COMPLETE/FAIL events within streaming jobs #2768 @pawel-big-lebowski New job_version is not created for a streaming job terminal event with no dataset information and existing version is kept.

0.45.0

2 months ago

Added

Redesigned Web UI Featuring Column Lineage

Fixed

0.45.0-rc.1

3 months ago

Added

Fixed

0.44.0

3 months ago

Added

  • Web: add dataset tags tabs for adding/deleting of tags #2714 @davidsharp7
    Adds a dataset tags component so that datasets can have tags added/deleted.
  • API: Add endpoint to delete field-level tags #2705 @davidsharp7
    Adds delete endpoint to remove dataset field tags.

Fixed

  • Web: fix dataset tag reducers bug #2716 @davidsharp7
    Removes result from dataset tags reducer to fix a sidebar bug.

0.43.1

4 months ago

Fixed

0.43.0

5 months ago

Added

  • API: refactor the RunDao SQL query #2685 @sophiely
    Improves the performance of the SQL query used for listing all runs.
  • API: refactor dataset version query #2683 @sophiely
    Improves the performance of the SQL query used for the dataset version.
  • API: add support for a DatasetEvent #2641 #2654 @pawel-big-lebowski
    Adds a feature for saving into the Marquez model datasets sent via the DatasetEvent event type. Includes optimization of the lineage query.
  • API: add support for a JobEvent #2661 @pawel-big-lebowski
    Adds a feature for saving into the Marquez model jobs and datasets sent via the JobEvent event type.
  • API: add support for streaming jobs #2682 @pawel-big-lebowski
    Creates job version and reference rows at the beginning of the job instead of on complete. Updates the job version within the run if anything changes.
  • API/spec: implement upstream run-level lineage #2658 @julienledem
    Returns the version of each job and dataset a run is depending on.
  • API: add DELETE endpoint for dataset tags #2698 @davidsharp7
    Creates a new endpoint for removing the linkage between a dataset and a tag in datasets_tag_mapping to supply a way to delete a tag from a dataset via the API.
  • Web: add a dataset drawer #2672 @davidsharp7
    Adds a drawer to the dataset column view in the GUI.

Fixed:

  • Client/Java: change url path encoding to match jersey decoding #2693 @davidjgoss
    Swaps out the implementation of MarquezPathV1::encode to use the UrlEscapers path segment escaper, which does proper URI encoding.
  • Web: fix pagination in the Jobs route #2655 @merobi-hub
    Hides job pagination in the case of no jobs.
  • Web: fix empty search experience #2679 @phixMe
    Use of the previous search value was resulting in a bad request for the first character of a search.

Removed:

  • Client/Java: remove maven-archiver dependency from the Java client #2695 @davidjgoss
    Removes a dependency from build.gradle that was bringing some transitive vulnerabilities.

0.42.0

6 months ago

Added

  • Client: add Java client method for dataset/job lineage #2623 @davidjgoss
    To add a method for the dataset/job-level endpoint (GET /lineage) to the Java SDK, this adds a new method to the MarquezClient for the endpoint, along with tests, and the necessary new subclasses of NodeData for datasets and jobs.
  • Web: add IO tab #2613 @phixme
    Improves experience with large graphs by adding a new tab to move between graph elements without looking at the graph itself.
  • Web: add hover-over Tag tooltip to datasets #2630 @davidsharp7
    For parity with columns in the GUI, this adds a Tag tooltip to datasets.

Changed

  • Docker: upgrade to Docker Compose V2 #2644 @merobi-hub
    Docker Compose V1 has been at EOL since June, but docker/up.sh uses the V1 format. This upgrades the up command in up.sh to V2.

Removed

  • API: drop table job_contexts and usage #2621 @wslulciuc
    Removes usage of job_contexts, which has been replaced by OpenLineage facets, and adds a migration to drop the table.
  • API: remove usage of current_job_context_uuid column #2622 @wslulciuc
    Removes usage of job_context_uuid and current_job_context_uuid. Column to be removed in 0.43.0.

Fixed

  • Web: fix Unix epoch time display for null endedAt values #2647 @merobi-hub
    Fixes the issue of the GUI displaying Unix epoch time (midnight on January 1, 1970) in the case of running jobs/null endedAt values.

0.41.0

7 months ago

Added

  • API: add support for the following parameters in the SearchDao #2556 @tati @wslulciuc
    This PR updates the search endpoint to enforce YYYY-MM-DD for query params, use YYYY-MM-DD as LocalDate, and support the following query params:
    • namespace - matches jobs or datasets within the given namespace.
    • before - matches jobs or datasets before YYYY-MM-DD.
    • after - matches jobs or datasets after YYYY-MM-DD.
  • Web: add paging on jobs and datasets #2614 @phixme
    Adds paging to jobs and datasets just like we already have on the lineage events page.
  • Web: add tag descriptions to tooltips #2612 @davidsharp7
    Get the tag descriptions from the tags endpoint and when a column has a tag display the corresponding description on hover over. Context can be found here.
  • Web: add available column-level tags #2606 @davidsharp7
    Adds a new column called "tags" to the dataset column view along with the tags associated with the dataset column.
  • Web: add HTML Tool Tip #2601 @davidsharp7
    Adds a Tool Tip to display basic node details.

Fixed

  • Web: fix dataset saga for paging #2615 @phixme
    Updates the saga, changes the default page size.
  • API: perf/improve jobdao query #2609 @algorithmy1
    Optimizes the query to make use of Common Table Expressions to fetch the required data more efficiently and before the join, fixing a significant bottleneck.

Changed

  • Docker: Postgres 14 #2607 @wslulciuc
    Bumps the recommended version of Postgres to 14. When deploying locally, you might need to run ./docker/down.sh to clean existing volumes.

Removed

  • Client: tolerate null transformation attrs in field model #2600 @davidjgoss
    Removes the @NonNull annotation from the client class and the @NotNull from the model class.

0.40.0

9 months ago

Added

  • API: lineage events paging update #2577 @phixme
    Updates the API for lineage events and restyles the lineage events page to fix a number of bugs and code duplication.
  • Chart: do not use hardcoded Postgres image for init container #2579 @terrpan
    Adds a template in chart/templates/helpers to use the global.imageRegistry input value for the wait-for-db container to improve performance on private registries.
  • Web: add copy button for lineage ID #2578 @AmandaYao00
    Adds a copy button to the IDs on the Events page.

Fixed

  • API: add defaults for idFromValue() and idFromValueAndType() #2581 @wslulciuc
    Replaces the null values in these functions in EventTypeResolver with defaults.
  • Client: correct example syntax #2575 @davidjgoss
    Removes errant parens from the sample code's client instantiation.

0.39.0

9 months ago

Added

  • Web: add full graph toggle #2569 @jlukenoff
    Adds a toggle to the Lineage UI to let users switch between viewing the full graph and only the selected paths.
  • Web: add ARIA labels to input fields #2562 @merobi-hub
    Adds i18next-compliant ARIA labels to input fields for improved accessibility.

Changed

  • Web: upgrade React to version 18 #2563 @Xavier-Cliquennois
    Upgrades the Web client in order to utilize the latest version of Node.js and update all dependencies to their respective latest versions.

Fixed

  • Web: fix the stylesheet for the date selector #2573 @phixme
    Fixes margins and moves the label to be more inline with what the defaults are to fix issues caused by the recent Material-UI upgrade.
  • Web: update i18n for general search filter and runInfo facets search #2557 @merobi-hub
    Adds missing i18n support for runInfo and search.
  • Docker: update web proxy import #2571 @phixme
    Updates the import style for the http-proxy-middleware.