Dagster Versions Save

An orchestration platform for the development, production, and observation of data assets.

1.1.8

1 year ago

New

  • Asset backfills launched from the asset graph now respect partition mappings. For example, if partition N of asset2 depends on partition N-1 of asset1, and both of those partitions are included in a backfill, asset2’s partition N won’t be backfilled until asset1’s partition N-1 has been materialized.
  • Asset backfills launched from the asset graph will now only materialize each non-partitioned asset once - after all upstream partitions within the backfill have been materialized.
  • Executors can now be configured with a tag_concurrency_limits key that allows you to specify limits on the number of ops with certain tags that can be executing at once within a single run. See the docs for more information.
  • ExecuteInProcessResult, the type returned by materialize, materialize_to_memory, and execute_in_process, now has an asset_value method that allows you to fetch output values by asset key.
  • AssetIns can now accept Nothing for their dagster_type, which allows omitting the input from the parameters of the @asset- or @multi_asset- decorated function. This is useful when you want to specify a partition mapping or metadata for a non-managed input.
  • The start_offset and end_offset arguments of TimeWindowPartitionMapping now work across TimeWindowPartitionsDefinitions with different start dates and times.
  • If add_output_metadata is called multiple times within an op, asset, or IO manager handle_output, the values will now be merged, instead of later dictionaries overwriting earlier ones.
  • materialize and materialize_to_memory now both accept a tags argument.
  • Added SingleDimensionDependencyMapping, a PartitionMapping object that defines a correspondence between an upstream single-dimensional partitions definition and a downstream MultiPartitionsDefinition.
  • The RUN_DEQUEUED event has been removed from the event log, since it was duplicative with the RUN_STARTING event.
  • When an Exception is raised during the execution of an op or asset, Dagit will now include the original Exception that was raised, even if it was caught and another Exception was raised instead. Previously, Dagit would only show exception chains if the Exception was included using the raise Exception() from e syntax.
  • [dagit] The Asset Catalog table in Dagit is now a virtualized infinite-scroll table. It is searchable and filterable just as before, and you can now choose assets for bulk materialization without having to select across pages.
  • [dagit] Restored some metadata to the Code Locations table, including image, python file, and module name.
  • [dagit] Viewing a partition on the asset details page now shows both the latest materialization and also all observations about that materialization.
  • [dagit] Improved performance of the loading time for the backfills page
  • [dagit] Improved performance when materializing assets with very large partition sets
  • [dagit] Moving around asset and op graphs while selecting nodes is easier - drag gestures no longer clear your selection.
  • [dagster-k8s] The Dagster Helm chart now allows you to set an arbitrary kubernetes config dictionary to be included in the launched job and pod for each run, using the runK8sConfig key in the k8sRunLauncher section. See the docs for more information.
  • [dagster-k8s] securityContext can now be set in the k8sRunLauncher section of the Dagster Helm chart.
  • [dagster-aws] The EcsRunLauncher can now be configured with cpu and memory resources for each launched job. Previously, individual jobs needed to be tagged with CPU and memory resources. See the docs for more information.
  • [dagster-aws] The S3ComputeLogManager now takes in an argument upload_extra_args which are passed through as the ExtraArgs parameter to the file upload call.
  • [dagster-airflow] added make_dagster_definitions_from_airflow_dags_path and make_dagster_definitions_from_airflow_dag_bag which are passed through as the ExtraArgs parameter to the file upload call.

Bugfixes

  • Fixed a bug where ad-hoc materializations of assets were not correctly retrieving metadata of upstream assets.
  • Fixed a bug that caused ExperimentalWarnings related to LogicalVersions to appear even when version-based staleness was not in use.
  • Fixed a bug in the asset reconciliation sensor that caused multi-assets to be reconciled when some, but not all, of the assets they depended on, were reconciled.
  • Fixed a bug in the asset reconciliation sensor that caused it to only act on one materialization per asset per tick, even when multiple partitions of an asset were materialized.
  • Fixed a bug in the asset reconciliation sensor that caused it to never attempt to rematerialize assets which failed in their last execution. Now, it will launch the next materialization for a given asset at the same time that it would have if the original run had completed successfully.
  • The load_assets_from_modules and load_assets_from_package_module utilities now will also load cacheable assets from the specified modules.
  • The dequeue_num_workers config setting on QueuedRunCoordinatoris now respected.
  • [dagit] Fixed a bug that caused a “Maximum recursion depth exceeded” error when viewing partitioned assets with self-dependencies.
  • [dagit] Fixed a bug where “Definitions loaded” notifications would constantly show up in cases where there were multiple dagit hosts running.
  • [dagit] Assets that are partitioned no longer erroneously appear "Stale" in the asset graph.
  • [dagit] Assets with a freshness policy no longer appear stale when they are still meeting their freshness policy.
  • [dagit] Viewing Dagit in Firefox no longer results in erroneous truncation of labels in the left sidebar.
  • [dagit] Timestamps on the asset graph are smaller and have an appropriate click target.
  • [dagster-databricks] The databricks_pyspark_step_launcher will now cancel the relevant databricks job if the Dagster step execution is interrupted.
  • [dagster-databricks] Previously, the databricks_pyspark_step_launcher could exit with an unhelpful error after receiving an HTTPError from databricks with an empty message. This has been fixed.
  • [dagster-snowflake] Fixed a bug where calling execute_queries or execute_query on a snowflake_resource would raise an error unless the parameters argument was explicitly set.
  • [dagster-aws] Fixed a bug in the EcsRunLauncher when launching many runs in parallel. Previously, each run risked hitting a ClientError in AWS for registering too many concurrent changes to the same task definition family. Now, the EcsRunLauncher recovers gracefully from this error by retrying it with backoff.
  • [dagster-airflow] Added make_dagster_definitions_from_airflow_dags_path and make_dagster_definitions_from_airflow_dag_bag for creating Dagster definitions from a given airflow Dag file path or DagBag

Community Contributions

  • Fixed a metadata loading error in UPathIOManager, thanks @danielgafni!
  • [dagster-aws]FakeS3Session now includes additional functions and improvements to align with the boto3 S3 client API, thanks @asharov!
  • Typo fix from @vpicavet, thank you!
  • Repository license file year and company update, thanks @vwbusguy!

Experimental

  • Added experimental BranchingIOManager to model use case where you wish to read upstream assets from production environments and write them into a development environment.
  • Add create_repository_using_definitions_args to allow for the creation of named repositories.
  • Added the ability to use Python 3 typing to define and access op and asset config.
  • [dagster-dbt] Added DbtManifestAssetSelection, which allows you to define selections of assets loaded from a dbt manifest using dbt selection syntax (e.g. tag:foo,path:marts/finance).

Documentation

  • There’s now only one Dagster Cloud Getting Started guide, which includes instructions for both Hybrid and Serverless deployment setups.
  • Lots of updates throughout the docs to clean up remaining references to @repository, replacing them with Definitions.
  • Lots of updates to the dagster-airflow documentation, a tutorial for getting started with Dagster from an airflow background, a migration guide for going to Dagster from Airflow and a terminology/concept map for Airflow onto Dagster.

All Changes

https://github.com/dagster-io/dagster/compare/1.1.7...1.1.8

See All Contributors
  • 95950df - Adjust resources guide to be in a Definitions world by @schrockn
  • f245696 - add thread name prefix to run dequeue workers (#11155) by @alangenfeld
  • 45b4c26 - make schedules produced by build_schedule_from_partitioned_job more p… (#11147) by @sryza
  • 2d3c914 - [k8s launcher] security context (#9788) by @alangenfeld
  • b655ca6 - [run coordinator] fix threaded tests (#11139) by @alangenfeld
  • 1f7980e - [docs] - [definitions] Update Dagit + tutorial screenshots (#11089) by @erinkcochran87
  • c5c6eff - [docs] - [definitions] Update Partitions concept docs (#11030) by @erinkcochran87
  • 6ef79f0 - Port asset sensor guide to Definitions by @schrockn
  • 8cf4dba - Use buildkite_deps.txt to declare explicit buildkite deps (#11025) by @jmsanders
  • 52c726f - Trigger builds when .ini files change (#11161) by @jmsanders
  • 7480066 - Revert "keep track of max timestamps client side for code location up… (#11162) by @prha
  • 679d2e1 - [docs] - Fix links (#11163) by @erinkcochran87
  • b16f30c - Add create_repository_using_definitions_args by @schrockn
  • b4f8705 - Change Graph-backed asset guide code examples to be on Definitions by @schrockn
  • e71dacf - Delete repository unit testing using load_all_definitions in testing guide by @schrockn
  • 2ae2ed6 - 1.1.7 Changelog (#11165) by @jamiedemaria
  • c0d8a9a - Do a single pip install in tox suites [OSS] (#11164) by @gibsondan
  • a763ec9 - Update declarative scheduling guide to include Definitions by @schrockn
  • 4467bc7 - add thread name prefix to grpc server (#11158) by @alangenfeld
  • fb59fda - Automation: versioned docs for 1.1.7 by @elementl-devtools
  • 1518150 - Fixup rename of requirements.txt -> buildkite_deps.txt (#11160) by @jmsanders
  • 9a46201 - fix precedence ordering when merging dictionaries in container context (#11169) by @gibsondan
  • bf4222c - support in and len on PartitionsSubsets (#11172) by @sryza
  • d64b050 - Disable breaking azure tests in master (#11183) by @schrockn
  • 722c3b6 - [docs] Fix typo in title (#11156) by @vpicavet
  • 630d9c6 - Fix regression with passing in None to snowflake resource (#11182) by @gibsondan
  • b2bb257 - [dagit] Add tests for partition health data parsing / accessors (#11114) by @bengotow
  • 3922523 - Allow setting resources on EcsContainerContext and EcsRunLauncher (#11170) by @gibsondan
  • d8555aa - fix a small bug in UPathIOManager (#11110) by @danielgafni
  • 15b0f27 - [dagster-dbt] in tests, pin dbt rpc < 0.3.0 (#11196) by @OwenKephart
  • baadabe - Move core_tests/storage_tests to storage_tests (#11180) by @schrockn
  • bcd1229 - skip sqlite env var test on windows (#11195) by @gibsondan
  • 59faa75 - Move old_sqlalchemy_tests to only run on storage_tests (#11181) by @schrockn
  • fb3d4f9 - [dagster-airflow] airflow terminology mapping (#11015) by @Ramshackle-Jamathon
  • c4e398b - Move core_tests/definitions_tests to definitions_tests (#11184) by @schrockn
  • ab3498a - fix dequeue_num_workers setting (#11198) by @alangenfeld
  • 3db32c1 - Move core_tests/asset_defs_tests to asset_defs_tests (#11186) by @schrockn
  • 8ae492c - update timeout in sensor run tests (#11193) by @jamiedemaria
  • 80557d2 - Move core_tests/launcher_tests to launcher_tests (#11187) by @schrockn
  • 4961453 - add assets example to AssetSelection apidoc (#11194) by @sryza
  • b242ed7 - Move various logging tests into logging_tests (#11188) by @schrockn
  • ed912c3 - feat(dbt-cloud): add Dagster run id to dbt Cloud run (#11005) by @rexledesma
  • d521afa - [docs] snowflake reference page (#10985) by @jamiedemaria
  • 6acf983 - [declarative-scheduling] Fix bug with declarative scheduling where repeated calls to get_latest_materialization_record could return incorrect results (#11214) by @OwenKephart
  • 732ab85 - only reconcile multi-assets if all parents are reconciled (#11190) by @sryza
  • 1ed7d08 - [dagit] Fix dragging on the DAG clearing your selection / clicking links (#11202) by @bengotow
  • e446201 - [dagster-aws] Extend fake S3 resource (#11105) by @asharov
  • b99fd34 - [dagit] Optimizations to backfill UI for large partition key sets (#11201) by @bengotow
  • 4d79412 - do not compute projected logical versions of partitioned assets (#11204) by @sryza
  • ae3ed2d - factor out asset reconciliation graph traversal into util (#11206) by @sryza
  • 6ee425f - [dagit] Fresh + Stale should not show “Stale” on the Asset Graph (#11234) by @bengotow
  • c5cb42a - [dagit] Round middle truncate calculations for Firefox (#11203) by @bengotow
  • cbf21af - support tags argument on materialize and materialize_to_memory (#11225) by @sryza
  • 530e118 - docs: update license to include year and company (#11231) by @vwbusguy
  • 1895517 - [docs] - [definitions] Update OSS deployment overview (#11199) by @erinkcochran87
  • d94cff0 - split backfill table so partition status is fetched lazily (#11205) by @prha
  • 934f7c3 - fix tslint (#11242) by @alangenfeld
  • 76b00ba - [dagit] webpack-bundle-analyzer (#11230) by @hellendag
  • 8f72eb0 - [dagit] Virtualized Asset Catalog (#11168) by @hellendag
  • cd523d9 - Can remove hardcoded_resource from project_fully_featured because of Definitions (#11243) by @schrockn
  • ced6042 - [dagit] Replace moment-timezone (#11197) by @hellendag
  • 4cb9a82 - Remove unused backfillStatus, which loads individual run status (#11246) by @prha
  • e763c59 - Retry registering task definitions (#11192) by @jmsanders
  • 9e5df46 - Rerun dagster_tests with --snapshot-update (#11208) by @schrockn
  • 17f13ea - Add test case for binding assets before passing to Definitions (#11216) by @schrockn
  • 2c03dfd - [dagster-airflow] from airflow to dagster guide updates (#11218) by @Ramshackle-Jamathon
  • 86338fa - [graphql test] share schema instance (#11236) by @alangenfeld
  • 9460504 - Use FromSourceAsset instead of FromRootInputManager when loading assets with input managers (#11233) by @jamiedemaria
  • 8850c6a - Use bare objects for the hacker news resources (#11249) by @schrockn
  • 34a5f5a - Use bare I/O manager in fully featured (#11250) by @schrockn
  • 3f30430 - Delete unused fixed_s3_pickle_io_manager (#11251) by @schrockn
  • 5df60bf - Consolidate _resolve_bound_config and have logger and resource use the same one (#11209) by @schrockn
  • df13fc6 - Delete op version of _resolve_bound_config and call generic one (#11211) by @schrockn
  • 2ecec63 - Skip race condition tests (#11286) by @jmsanders
  • eca24df - Skip flaky test (#11292) by @jmsanders
  • 5d83868 - [dagit] Replace remaining moment usage (#11278) by @hellendag
  • a28f5d5 - Update op-retries.mdx - no solid (#11284) by @yuhan
  • 11c1640 - [docs] airbyte guide repository -> definitions (#11296) by @yuhan
  • 8be6dc3 - [docs] fivetran guide repository -> definitions (#11297) by @yuhan
  • 547a6cb - [docs] dbt guide repository -> definitions (#11298) by @yuhan
  • ed212ed - [docs] dbt cloud guide repository -> definitions (#11299) by @yuhan
  • 52336d7 - [declarative-scheduling] Rework constraint passing (#11229) by @OwenKephart
  • 58a31a0 - [dagster-airflow] basic airflow migration guide (#11012) by @Ramshackle-Jamathon
  • d46f969 - [dagstermill] add retries to flaky tests (#11291) by @jamiedemaria
  • c8f3a0c - Product tour component (#11227) by @salazarm
  • bd372fd - [dagit] Utility for timezone-aware date/time formatting (#11285) by @hellendag
  • 8ca865f - [dagit] Restore some metadata on Code Locations page (#11281) by @hellendag
  • d8b5c3b - [docs] - [definitions] Update Loggers Concept docs (#11171) by @erinkcochran87
  • b09e95a - [docs] - Update ECS deployment guide (#11289) by @erinkcochran87
  • d235619 - [docs] - [definitions] Update Dagster instance docs (#11241) by @erinkcochran87
  • be59c3e - Pass duckdb_path to __init__ rather than relying on context (#11300) by @schrockn
  • 40cd16a - [structured config] Base structured config implementation (#11268) by @benpankow
  • a0152b8 - [structured config] Add support for default values (#11272) by @benpankow
  • a21feb4 - Add gql pin (#11312) by @gibsondan
  • d1ae079 - [structured config] Add support for class, field descriptions (#11274) by @benpankow
  • ec47d7c - Move env var injection earlier in step command (#11239) by @gibsondan
  • 3e47f52 - accrete metadata with multiple calls to add_output_metadata (#9518) by @sryza
  • 534e38d - [docs] - [definitions] - Update Executors docs (#11247) by @erinkcochran87
  • 905ef7a - [docs] - [definitions] Update Helm guide (#11320) by @erinkcochran87
  • a3963b1 - Eliminating unnecessary output_context.resource_config check (#11313) by @schrockn
  • 49dd2c7 - [docs] - Consolidate sections in Resources concept doc (#11316) by @erinkcochran87
  • 0e670b6 - [docs] - [definitions] Update SDA Concept docs (#11018) by @erinkcochran87
  • b22c163 - [docs] - [definitions] Update Run launchers guide (#11200) by @erinkcochran87
  • 9632313 - [docs] - Re-do Cloud Getting Started guides (#10429) by @erinkcochran87
  • f099158 - [docs] - [definitions] Update Docker guide (#11295) by @erinkcochran87
  • 09c13a5 - backfill perf: swap backfill requested for num cancelable (#11304) by @prha
  • a3c0e71 - allow upload config to pass through s3 compute log manager (#11317) by @prha
  • b911f34 - [apidoc] define_asset_job repository -> Definitions (#11302) by @yuhan
  • dbca38a - Ignore stale timestamps from code location updates (#11173) by @prha
  • 43d4283 - [structured config] Fix usage with Assets (#11327) by @benpankow
  • e72a3f5 - Include parent exceptions in Dagster errors, even if they weren't explicitly raised (#11306) by @gibsondan
  • ec38705 - [docs] - [definitions] Update Dagster daemon docs (#11226) by @erinkcochran87
  • e8e3c57 - Docs release backfill 1.1.7 (#11328) by @yuhan
  • e92ce74 - Use dark mode logo on README.md (#11326) by @hellendag
  • a88a38e - AssetGraph.from_external_assets().get_required_multi_asset_keys() (#11318) by @sryza
  • 8f80959 - [declarative-scheduling] Update retry logic to attempt to retry failed materializations after some time has passed (#11294) by @OwenKephart
  • c90056e - Add instance property to InputContext (#11331) by @schrockn
  • dbfc06a - chore: add method to strip error stack trace (#11307) by @rexledesma
  • 661b4e4 - Hoist schema and database to DbIOManager constructor (#11301) by @schrockn
  • 3b0fe94 - Rename _resolve_bound_config to resolve_bound_config (#11287) by @schrockn
  • 7839ee3 - Add asset_materialization property to EventLogEntry (#11340) by @schrockn
  • 3a11f32 - store repository on external asset graph (#11332) by @sryza
  • e830ac6 - Revert "Use dark mode logo on README.md (#11326)" (#11345) by @hellendag
  • 6e8d060 - [dagster-io/eslint-config] v1.0.6 (#11324) by @hellendag
  • abfc7b5 - Add get_implicit_global_asset_job on Definitions and RepositoryDefinition (#11279) by @schrockn
  • 73e6a02 - feat: add retry number and url to integration api call failure (#11308) by @rexledesma
  • 538e5b3 - Fix dagster-graphql circular import (BK broken) (#11339) by @smackesey
  • ccf7468 - Skipping dbt rpc resource tests (#11357) by @schrockn
  • 7434f5e - skip test_threaded_ephemeral_instance (#11359) by @schrockn
  • 6eb8673 - Allow specifying resources in Op/Asset params list (#11322) by @benpankow
  • 945eb03 - change version placeholder 0+dev -> 1!0+dev (#11334) by @smackesey
  • 2975629 - [structured config] Structured-config-backed Resources (#11321) by @benpankow
  • a34a02d - [structured config] Add traditional resource wrapper base class (#11337) by @benpankow
  • a842b2b - [structured config] Fix test importing functools.cached_property on py37 tests (#11361) by @benpankow
  • b8a7b0c - [structured config] Structured-config-backed IO managers (#11343) by @benpankow
  • 62e1527 - fix some bugs in ExternalAssetGraph (#11350) by @sryza
  • 02ed6c2 - more refactors to reconciliation sensor (#11223) by @sryza
  • 9d25371 - [dagster-io/eslint-config] v1.0.7 (#11347) by @hellendag
  • 586f15b - Add Materialize Button hook (#11319) by @salazarm
  • 1a57141 - Allow setting raw k8s config at the run launcher / container context level (#11333) by @gibsondan
  • af2ddc6 - Make it more likely that we hit our lock (#11290) by @jmsanders
  • 860ffe6 - fix __contains__ of TimeWindowPartitionsSubset (#11380) by @sryza
  • 2394272 - [docs] - add note on pandas integration to redirect users interested in pandas w/out validation (#11342) by @slopp
  • 4dbf286 - fix asset reconciliation bug that ignores earlier partitions (#11336) by @sryza
  • 0263eab - fix typo on partitions concepts page (#11349) by @sryza
  • 9832545 - Clicking on top-level concept sections takes you to a page (#10140) by @sryza
  • 81b192d - Add single dimension -> multidimension partition mapping (#10910) by @clairelin135
  • d761f14 - [RFC] Update cached_status_data column 1/2 (#10821) by @clairelin135
  • b60b9a1 - Remove stray console.log (#11410) by @gibsondan
  • 1c5733d - ExecuteInProcessResult.asset_value (#11403) by @sryza
  • 9504c26 - Elminate PIPELINE_DEQUEUED event (#11393) by @gibsondan
  • 551a532 - [dagit] Begin adding new graphql-codegen (#11411) by @hellendag
  • b88880c - Docs for new ECS resource options (#11391) by @gibsondan
  • 26e9853 - Docs for new k8s configuration options (#11390) by @gibsondan
  • 7854e62 - Fix malformed dagster_cloud.yaml in code location docs (#11416) by @gibsondan
  • be844f7 - [dagit] Refactor mutations to new GraphQL codegen (#11414) by @hellendag
  • ffe89bf - [dagit] Refactor queries in src/workspace (#11418) by @hellendag
  • f29e001 - [dagit] dedupeFragments (#11426) by @hellendag
  • 2bc5146 - relax matching criteria for TimeWindowPartitionMappings w/offsets (#11422) by @sryza
  • 5ba8f4e - [dagster-databricks] handle empty responses (#11430) by @OwenKephart
  • b5fbfed - Export RunRecord in the public API (#11427) by @gibsondan
  • 67c7650 - [dagit] Refactor queries in src/runs (#11419) by @hellendag
  • 03fcf9b - [dagit] Refactor queries in Assets (#11423) by @hellendag
  • b7b7d06 - [dagit] Refactor queries in instance, instigation, launchpad (#11429) by @hellendag
  • 384fe6b - Branching I/O Manager (#11315) by @schrockn
  • a4ea49f - Fix issues with a SerializableErrorInfo being coerced to a GraphenePythonError (#11437) by @gibsondan
  • 186e2c7 - export product tour component (#11314) by @salazarm
  • f12798f - [dagit] Refactor queries in remaining src (#11435) by @hellendag
  • ba3a381 - Eliminate unused config_or_config_fn arg in copy_for_configured (#11433) by @schrockn
  • a6dcbfe - [dagit] Don’t show assets as stale if projectedLogicalVersion is null (#11224) by @bengotow
  • 9ca91c5 - [dagit] Show observations about the latest materialization on Asset > Partitions (#11288) by @bengotow
  • 157d901 - asset backfill core logic (#11377) by @sryza
  • d72e34c - Allow AssetIn(dagster_type=Nothing) (#11436) by @sryza
  • 5ab885d - Rename and refactor in structured_config.py for clarity (#11372) by @schrockn
  • f00cec7 - Consolidate all logic and trickery to support private cached properties in pydantic in a single class (#11373) by @schrockn
  • 5a82bdc - deprecate job-level memoization in docs (#11392) by @sryza
  • 09a66b9 - Add StructuredIOManagerAdapter (#11383) by @schrockn
  • eb5e973 - [dagit] Fix line height and size of timestamps on the asset graph (#11465) by @bengotow
  • afaeb13 - Default new-style config mappings to *Source equivalents rather than raw scalars (#11386) by @schrockn
  • f2915c5 - Unexperimentalize LogicalVersion and quell warning messages whenever assets are materialized (#11407) by @sryza
  • 7d69c44 - Passthrough pydantic.ModelField.required to dagster.Field.is_required (#11388) by @schrockn
  • 8c7b92f - Support pydantic aliases in config schema mapping (#11389) by @schrockn
  • cdcfedb - Make conversion to *Source types work on direct annotations of the config parameter (#11469) by @schrockn
  • dafbf49 - [dagster-databricks] handle execution interrupts (#11421) by @OwenKephart
  • d74dcf4 - Skip flaky race condition test (#11471) by @jmsanders
  • 08d0b0a - remove AssetStoreHandle (#11412) by @sryza
  • ebd9fb2 - refactor: sequester job backfill code (#11252) by @sryza
  • 8ca3874 - Be more resilient to user code errors when dequeuing runs (#11406) by @gibsondan
  • 0ff7c7f - Make all arguments on DagsterInstance.create_run keyword-only (#11446) by @schrockn
  • 9dd7579 - Elimiinate default value for asset_selection param on DagsterInstance.create_run (#11447) by @schrockn
  • 5f23b2f - Eliminate default values for solid_selection, external_pipeline_origin, and pipeline_code_origin on create_run (#11448) by @schrockn
  • 662077b - rename create_run to create_run_for_test on TestQueuedRunCoordinator to increase greppability of create_run (#11449) by @schrockn
  • 4d205f5 - [dagit] Remove old GraphQL Codegen (#11474) by @hellendag
  • fe1620e - [dagit] Delete apollo CLI dep (#11478) by @hellendag
  • 359e16f - Add tag_concurrency_limits config to executors (#11472) by @gibsondan
  • 8ca33ca - add run filter for updated before (#11481) by @prha
  • 8d50157 - docs(dagster-dbt): clarify that the integration supports arbitrary dbt profiles (#11351) by @rexledesma
  • 057b369 - [structured config] Add support for Permissive fields (#11275) by @benpankow
  • 7751e64 - handle pure asset backfills in backfill daemon (#11378) by @sryza
  • 5bdc212 - Fix missing callsite of tag_concurrency_limits (#11496) by @gibsondan
  • 936660e - fix(docs): format docs (#11501) by @rexledesma
  • d7e687e - [dagster-dbt] add DbtManifestAssetSelection (#11473) by @OwenKephart
  • 41b616b - Add parameter invariants around external_pipeline_origin and pipeline_code_origin arguments (#11450) by @schrockn
  • 6a6bb02 - supply solid_selection to fix submitting runs from pure asset backfills (#11502) by @sryza
  • 75b9084 - telemetry: add num_assets_in_repo in repo-level metadata (#11490) by @yuhan
  • 5144eab - fix(docs): run mdx-format again (#11504) by @rexledesma
  • 0e654f6 - Tighten solids_to_execute and solid_selection invariant (#11451) by @schrockn
  • 7d69dad - Bump json5 from 1.0.1 to 1.0.2 in /js_modules/dagit (#11485) by @dependabot[bot]
  • 39c0573 - Add invariant for asset_selection (#11452) by @schrockn
  • 72654df - Typehint pipeline_name, run_id, and mode (#11453) by @schrockn
  • 4b91684 - graphql for pure asset backfills (#11379) by @sryza
  • ba75a8f - docs for new tag_concurrency_limits feature on executor (#11499) by @gibsondan
  • 8e47dc2 - Add support for loading cacheable assets from module (#10389) by @benpankow
  • ea185ef - enable filtering by asset tag when asset tags table is not present (#11509) by @sryza
  • c578c55 - only count materializations within backfill (#11506) by @sryza
  • 862ca6d - telemetry: add num_assets_in_repo in log_repo_stats metadata (#11513) by @yuhan
  • e6a1a94 - telemetry: add location_name in repo-level metadata (#11514) by @yuhan
  • 8349be9 - [dagster-airflow] warn on airflow dataset (#11498) by @Ramshackle-Jamathon
  • 893ed12 - [dagster-airflow] add make_dagster_definitions_from_airflow_dags_pathandmake_dagster_definitions_from_airflow_dag_bag apis (#11441) by @Ramshackle-Jamathon
  • 07c5d7d - Changelog 1.1.8 (#11526) by @clairelin135
  • 88c01f1 - 1.1.8 by @elementl-devtools

1.1.7

1 year ago

New

  • Definitions is no longer marked as experimental and is the preferred API over @repository for new users of Dagster. Examples, tutorials, and documentation have largely ported to this new API. No migration is needed. Please see GitHub discussion for more details.
  • The “Workspace” section of Dagit has been removed. All definitions for your code locations can be accessed via the “Deployment” section of the app. Just as in the old Workspace summary page, each code location will show counts of its available jobs, assets, schedules, and sensors. Additionally, the code locations page is now available at /locations.
  • Lagged / rolling window partition mappings: TimeWindowPartitionMapping now accepts start_offset and end_offset arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets.
  • Asset partitions can now depend on earlier time partitions of the same asset. The asset reconciliation sensor will respect these dependencies when requesting runs.
  • dagit can now accept multiple arguments for the -m and -f flags. For each argument a new code location is loaded.
  • Schedules created by build_schedule_from_partitioned_job now execute more performantly - in constant time, rather than linear in the number of partitions.
  • The QueuedRunCoordinator now supports options dequeue_use_threads and dequeue_num_workers options to enable concurrent run dequeue operations for greater throughput.
  • [dagster-dbt] load_assets_from_dbt_project, load_assets_from_dbt_manifest, and load_assets_from_dbt_cloud_job now support applying freshness policies to loaded nodes. To do so, you can apply dagster_freshness_policy config directly in your dbt project, i.e. config(dagster_freshness_policy={"maximum_lag_minutes": 60}) would result in the corresponding asset being assigned a FreshnessPolicy(maximum_lag_minutes=60).
  • The DAGSTER_RUN_JOB_NAME environment variable is now set in containerized environments spun up by our run launchers and executor.
  • [dagster-airflow] make_dagster_repo_from_airflow_dags_path ,make_dagster_job_from_airflow_dag and make_dagster_repo_from_airflow_dag_bag have a new connections parameter which allows for configuring the airflow connections used by migrated dags.

Bugfixes

  • Fixed a bug where the log property was not available on the RunStatusSensorContext context object provided for run status sensors for sensor logging.

  • Fixed a bug where the re-execute button on runs of asset jobs would incorrectly show warning icon, indicating that the pipeline code may have changed since you last ran it.

  • Fixed an issue which would cause metadata supplied to graph-backed assets to not be viewable in the UI.

  • Fixed an issue where schedules often took up to 5 seconds to start after their tick time.

  • Fixed an issue where Dagster failed to load a dagster.yaml file that specified the folder to use for sqlite storage in the dagster.yaml file using an environment variable.

  • Fixed an issue which would cause the k8s/docker executors to unnecessarily reload CacheableAssetsDefinitions (such as those created when using load_assets_from_dbt_cloud_job) on each step execution.

  • [dagster-airbyte] Fixed an issue where Python-defined Airbyte sources and destinations were occasionally recreated unnecessarily.

  • Fixed an issue with build_asset_reconciliation_sensor that would cause it to ignore in-progress runs in some cases.

  • Fixed a bug where GQL errors would be thrown in the asset explorer when a previously materialized asset had its dependencies changed.

  • [dagster-airbyte] Fixed an error when generating assets for normalization table for connections with non-object streams.

  • [dagster-dbt] Fixed an error where dbt Cloud jobs with dbt run and dbt run-operation were incorrectly validated.

  • [dagster-airflow] use_ephemeral_airflow_db now works when running within a PEX deployment artifact.

Documentation

  • New documentation for Code locations and how to define one using Definitions
  • Lots of updates throughout the docs to reflect the recommended usage of Definitions. Any content not ported to Definitions in this release is in the process of being updated.
  • New documentation for dagster-airflow on how to start writing dagster code from an airflow background.

All Changes

https://github.com/dagster-io/dagster/compare/1.1.6...1.1.7

See All Contributors
  • 858b9d2 - Non isolated runs docs (#10860) by @johannkm
  • 24bff5f - [dagit] Fix Gantt chart rendering of per-step resource init log messages (#10943) by @bengotow
  • cde76d9 - [dagit] Fix the “Assets” label on large asset runs (#10932) by @bengotow
  • 6cd734a - [dagit] Fix “Job In” label regression in Chrome v109 (#10934) by @bengotow
  • 523edb0 - [dagit] Pass repository tag when loading runs for Partitions page (#10948) by @bengotow
  • 49f7c4f - [dagit] Updated asset DAG styles, added additional compute tags (#10931) by @bengotow
  • fb574c0 - [docs] - add a guide for scheduling assets (#10949) by @slopp
  • 66959b7 - cap packaging requirement at 22.0 (#10968) by @smackesey
  • 5a45dd1 - Execution result typing (#10919) by @smackesey
  • 7d18ed2 - solid -> node method renames (#10920) by @smackesey
  • f19c6ea - [dagster-airflow] re-enable airflow 2.5.0 tests (#10966) by @Ramshackle-Jamathon
  • 0d80225 - [dagster-airbyte][docs] Use dagster-airbyte CLI alias in docs (#10955) by @benpankow
  • 162e791 - [dagster-slack] create slack_on_freshness_policy_sensor (#10960) by @OwenKephart
  • faf4f30 - [docs ] - fix image dimensions in hello-dagster materialize (#10977) by @slopp
  • 4babd9d - 1.1.6 Changelog (#10978) by @OwenKephart
  • 577e3eb - Automation: versioned docs for 1.1.6 by @elementl-devtools
  • 0b59fa9 - [convert-environment-variables-and-secrets-guide-stack-2] Convert env vars and secrets guide from repository to Definitions by @schrockn
  • 352d077 - unexperimentalize PartitionMapping (#10980) by @sryza
  • 231ddfa - remove validation in AssetGraph.get_child_partition_keys_of_parent an… (#10981) by @sryza
  • 305f1a2 - [convert-development-to-production-1] Move repository to __init__.py by @schrockn
  • d1054f9 - [convert-development-to-production-2] Changing development to production to use snowflake_pandas_io_manager by @schrockn
  • 7d5fabc - [convert-deployment-to-production-3] Convert @repository to Definitions by @schrockn
  • 196c086 - [convert-development-to-production-4] Use base object instead of resource by @schrockn
  • 435ddcd - [convert-development-to-production-6] Use pyproject.toml instead of workspace.yaml by @schrockn
  • 888660d - [convert-development-to-production-7] Convert guide to use Definitions by @schrockn
  • ad41a9f - [dagit] New Code Locations table (#10975) by @hellendag
  • 3008420 - Pin graphene to <3.2 by @schrockn
  • 5373a74 - fix missing metadata in dagit on graph-backed assets (#10988) by @OwenKephart
  • 7bbcb8a - [dagit] Export a few Code Location components for Cloud (#11008) by @hellendag
  • 39c84bb - Temporarily disable some Azure test suites (#11007) by @jmsanders
  • 41101f7 - [graphql] fix for graphene 3.2 (#11011) by @alangenfeld
  • a0e5c3d - [code-location-selector-stack] Code location sensor tests 1/N. Rename workspace_load_target function to create_workspace_load_target by @schrockn
  • 160aba2 - [code-location-selector-stack] Code location sensor tests 2/N Make instance_with_multiple_repos_with_sensors workspace_load_target parameterizable by @schrockn
  • d6ad971 - [code-location-selector-stack] Code location sensor tests 3/N. Refactor instance_with_multiple_repos_with_sensors to handle multiple code locations by @schrockn
  • 6e305c1 - [code-location-selector-stack] Code location sensor tests 4/N Actually add test to test cross code location selector by @schrockn
  • 5fc415a - [code-location-selector-stack Add CodeLocationSelector; Have run_status_sensor accept it by @schrockn
  • 3c2367f - Fix test_persistent by @schrockn
  • bf83710 - chore: auto-assign dependabot pull requests (#10953) by @rexledesma
  • a91525b - fix(dbt-cloud): parse command string to find materialization commands (#10989) by @rexledesma
  • 3be1547 - [code-location-selector-stack] Change typehint on make_slack_on_run_failure_sensor to accept CodeLocationSelector by @schrockn
  • c156ab3 - [docs] re-org snowflake integration guide (#10984) by @jamiedemaria
  • 33affcf - Check scheduler ticks right after each minute boundary instead of once every 5 seconds (#10886) by @gibsondan
  • 3b93947 - [docs] - [definitions] Update Configured API concept doc (#11020) by @erinkcochran87
  • 3cfc612 - Fix bug with re-execution snapshot ids (#10967) by @OwenKephart
  • 3fb76d9 - [bugfix] UPathIOManger load_input type checking (#11022) by @danielgafni
  • 546ece8 - Pathspec typing fix (#11036) by @smackesey
  • c729760 - [dagit] /code-locations -> /locations (#11024) by @hellendag
  • f606250 - Replace partition ranges with subsets (#10909) by @clairelin135
  • cec9f43 - [dagit] With multiple assets selected, backfill “missing” should include partially materialized partitions (#11027) by @bengotow
  • ae946c0 - [dagit] Add empty value string for invalid tag input in Runs filter (#11044) by @hellendag
  • 57e5def - [docs] - [definitions] Update Repository page for Definitions (#10986) by @erinkcochran87
  • 9300fc7 - [definitions-accessors] Add get_job_def to Definitions. by @schrockn
  • 1014add - 1/ definitions in create new project: update dagster project CLI (#10829) by @yuhan
  • 550ec43 - 2/ definitions in create new project: update create-new-project docs (#10830) by @yuhan
  • 7eb520d - [docs] - [definitions] - Update dbt tutorial to use Definitions (#10842) by @erinkcochran87
  • 891dc89 - 2.1/ definitions in in create new project: remove unnecessary asset dir in scaffold + update docs (#10831) by @yuhan
  • ad1fc35 - update isolated run docs to say that it's enabled by default (#11039) by @gibsondan
  • a4cb91e - [docs] - remove repository from hello dagster as its not needed (#11048) by @slopp
  • 90b3e01 - [dagster-fivetran][ez] Add missing docstring entry for build_fivetran_assets (#11049) by @benpankow
  • b4e5a80 - Add start_offset and end_offset to TimeWindowPartitionMapping (#10979) by @sryza
  • ec1d974 - fix subsettable multi asset case (#10878) by @OwenKephart
  • e98e5fb - move ecs task tagging to run_task call (#11037) by @prha
  • c1d1224 - Add custom resource key to load_assets_from_dbt_project (#10827) by @dpeng817
  • cc1925f - [dagit] Highlight Assets in top nav for global asset graph (#11052) by @hellendag
  • fc1ab1f - [dagster-airbyte][docs][ez] Fix AirbyteConnection apidoc rendering (#11059) by @benpankow
  • 795ae3f - convert-examples 1.1/ assets_dbt_python defs + pyproject (#11060) by @yuhan
  • c4d6792 - convert-examples 2.1/ assets_modern_data_stack repo -> defs, pyproject (#11062) by @yuhan
  • 09ac5ed - convert-examples 5/ assets_smoke_test repository -> definitions (#11071) by @yuhan
  • 48c7ae3 - convert-examples 6.1/ feature_graph_backed_assets pyproject, @repository -> Definitions (#11072) by @yuhan
  • 8a12bf3 - convert-examples 7.1/ quickstart_etl workspace->pyproject, repo->defs (#11074) by @yuhan
  • d85d59f - convert-examples 8.1/ with_great_expectations workspace->pyproject, repo->defs (#11077) by @yuhan
  • e4099ec - convert-examples 1.2/ assets_dbt_python file renames (#11061) by @yuhan
  • 4d38088 - convert-examples 6.2/ feature_graph_backed_assets file rename and reorder (#11073) by @yuhan
  • 58e0d88 - convert-examples 2.2/ assets_modern_data_stack file renames (#11063) by @yuhan
  • 99a62a1 - Removes codecov commands from tox (#10635) by @dpeng817
  • 0a675c6 - [dagit] Disable Re-execute menu item in run action menu for perms (#11026) by @hellendag
  • db21352 - Support loading multiple modules in single command line invocation by @schrockn
  • e06b9ae - Enable multiple files in CLI tools by @schrockn
  • 4d4aecb - Add get_sensor_def and get_schedule_def to Definitions by @schrockn
  • ba87836 - convert-examples 10/ with_pyspark_emr repository.py -> defs.py, update [docs] (#11080) by @yuhan
  • 32c3742 - convert-examples 9/ with_pyspark repository.py -> defs.py, update [docs] (#11079) by @yuhan
  • 0234c90 - Add load_asset_value and get_asset_value_loader to Definitions by @schrockn
  • c3a962a - [dagster-airflow] reorg dagster airflow api docs (#11009) by @Ramshackle-Jamathon
  • f2a705c - convert-examples 3.1/ assets_pandas_pyspark definitons, pyproject (#11067) by @yuhan
  • e1a93e1 - convert-examples 8.2/ with_great_expectations file rename (#11078) by @yuhan
  • b0bcc14 - convert-examples 7.2/ quickstart_etl file rename + update README (#11075) by @yuhan
  • e9651fc - Support module_name in tool.dagster section of pyproject.toml by @schrockn
  • f0ed982 - Move all examples to use module_name in pyproject.toml by @schrockn
  • 0510adf - Pin tox on windows to < 4 (#11013) by @jmsanders
  • b023c47 - [docs] - [definitions] Code location concept page (#10843) by @erinkcochran87
  • 606ff79 - Add instance for test to top level API (#10709) by @dpeng817
  • 86a4fe1 - [example] recomment code in dagstermill tutorial (#11098) by @jamiedemaria
  • e43e653 - Make JobSelector work on cross code location by @schrockn
  • 3693ccf - [convert-branch-deployments-stack] Convert branch deployments guide to Definitions by @schrockn
  • a541020 - Eliminate package_name from pyproject.toml loading spec by @schrockn
  • e1f51b0 - [docs] - [definitions] Update Enriching with SDAs guide (#10849) by @erinkcochran87
  • c706db1 - [dagit] Don't show View link on code location toast if already there (#11095) by @hellendag
  • 559ebb6 - set DAGSTER_RUN_JOB_NAME env var (#10888) by @alangenfeld
  • 9e58100 - remove parameter tag from dagstermill notebook (#11109) by @jamiedemaria
  • 195f975 - sort task definition config secrets and env vars to ensure consistent ordering (#10982) by @gibsondan
  • b32e5cb - [dagster-airflow] from airflow to dagster guide (#10923) by @Ramshackle-Jamathon
  • aefd575 - [dagit] Improvements to partition range selection interactions (#11017) by @bengotow
  • 07b6d4e - add tests for legacy storage (#11047) by @prha
  • 8884177 - keep track of max timestamps client side for code location updates (#11100) by @prha
  • 5caeb9f - in default IO manager, handle PartitionMappings that return 0 partitions (#11065) by @sryza
  • 8552f8f - Fix bug with logical versions for assets with changed deps (#11083) by @smackesey
  • cbf36be - Adding apidocs for Definitions by @schrockn
  • 822f1e8 - convert-examples 3.2/ asset_pandas_pyspark re-arrange files + [docs] (#11068) by @yuhan
  • 5ebe03e - [dagit] Persist AssetPartitions state to URL, open Materialize panel with same selection (#11042) by @bengotow
  • ceb2050 - [dagit] Render assets with self-dependencies in Dagit (#11043) by @bengotow
  • ccb2b47 - run queue daemon refactor (#11090) by @alangenfeld
  • fa6b48b - Allow assets to depend on earlier partitions of themselves (#11066) by @sryza
  • 1811e25 - [docs] - [definitions] Update Schedules documentation (#11046) by @erinkcochran87
  • afc161c - Re-add section for customizing Docker images pre-fast-deploys (#11112) by @shalabhc
  • 8a5ade0 - Allow setting base_dir as a StringSource in sqlite storage (#11031) by @gibsondan
  • c1ca6ef - handle self-dependencies in reconciliation sensor (#11085) by @sryza
  • 621f9fb - [docs] - [definitions] Update Workspaces concept page (#10944) by @erinkcochran87
  • d5feb2b - Coerce IOManager objects into IOManagerDefinitions in Definitions by @schrockn
  • 1c7542a - Revert "[docs] - [definitions] Update Workspaces concept page (#10944)" by @alangenfeld
  • 1292df7 - Allow overriding the default max_concurrent for the default executor via env var (#11116) by @gibsondan
  • 2ec20a3 - convert-examples 4.1/ assets_pandas_type_metadata definitions, pyproject (#11069) by @yuhan
  • 1f9c37c - [docs] [definitions] convert snowflake guide to definitions (#11029) by @jamiedemaria
  • 132ce0d - [docs] - Fix workspace link (#11121) by @erinkcochran87
  • e64f0de - convert-examples 4.2/ assets_pandas_type_metadata repository.py -> __init__.py (new) (#11127) by @yuhan
  • d36a5e6 - Add executor and loggers to Definitions by @schrockn
  • ab743c4 - [docs] - [definitions] Update Sensors documentation (#11050) by @erinkcochran87
  • 4506c12 - threaded run queue daemon (#11113) by @alangenfeld
  • c87bf8d - convert-examples 11.1/ project_fully_featured defs, pyproject, no with_resources (#11081) by @yuhan
  • 9491833 - [dagster-airbyte] Fix normalization table generation, materializations for non-object Streams (#10899) by @benpankow
  • c221574 - update assets_modern_data_stack example readme (#11122) by @yuhan
  • f0ff38c - [CacheableAssets] Fix bug that would cause definitions to be recomputed when using a StepDelegatingExecutor (#11086) by @OwenKephart
  • bd0801b - convert-examples 11.2/ project_fully_featured repository.py -> __init__.py (#11082) by @yuhan
  • 4e72f19 - update assets_dbt_python example readme (#11126) by @yuhan
  • 7d92ffc - [dagster-dbt] Enable setting FreshnessPolicies on dbt assets (#11103) by @OwenKephart
  • c14d793 - Mark Definitions as not experimental by @schrockn
  • a0268ff - [declarative-scheduling] Fix issue where in-progress runs were not properly handled (#11118) by @OwenKephart
  • 2a66629 - [dagster-airbyte] Avoid unnecessarily recreating sources, destinations w/ managed ingestion (#11117) by @benpankow
  • 7ab1263 - [dagster-airflow] pex compatibility for ephemeral db (#11115) by @Ramshackle-Jamathon
  • 47e3c31 - [dagster-airflow] provide interface for passing connection models directly to dagster (#11006) by @Ramshackle-Jamathon
  • 372aae8 - [dagit] Left nav: show entire code location string for non-dunder repos (#11133) by @hellendag
  • 439a32a - move quickstart_aws back to mono repo (#11130) by @yuhan
  • 650f837 - move quickstart_gcp back to mono repo (#11131) by @yuhan
  • 30bd794 - move quickstart_snowflake back to mono repo (#11132) by @yuhan
  • 479ab1a - Update Definitions docs by @schrockn
  • a5a8c84 - Use wrap_resources_for_execution in Definitions by @schrockn
  • 8182da0 - Use i/o manager coercion in assets_pandas_pyspark example by @schrockn
  • 5fdecbd - convert-examples-repo-to-defs cloud nux quickstart_aws (#11135) by @yuhan
  • 2169bf8 - convert-examples-repo-to-defs cloud nux quickstart_gcp (#11136) by @yuhan
  • 96750d3 - convert-examples-repo-to-defs cloud nux quickstart_snowflake (#11137) by @yuhan
  • 841efdc - enable run status sensor logging (#11145) by @prha
  • defd785 - Move jobs concept guide to refer to Definitions by @schrockn
  • ef723a6 - Move I/O Manager Guide to be on Definitions by @schrockn
  • 2d9ff0b - add thread name prefix to run dequeue workers (#11155) by @alangenfeld
  • 1670918 - Adjust resources guide to be in a Definitions world by @schrockn
  • 61b43cb - make schedules produced by build_schedule_from_partitioned_job more p… (#11147) by @sryza
  • 4760e16 - [docs] - [definitions] Update Dagit + tutorial screenshots (#11089) by @erinkcochran87
  • 9df72f3 - [docs] - [definitions] Update Partitions concept docs (#11030) by @erinkcochran87
  • 799d333 - Port asset sensor guide to Definitions by @schrockn
  • 9290f54 - Revert "keep track of max timestamps client side for code location up… (#11162) by @prha
  • 468f39b - [docs] - Fix links (#11163) by @erinkcochran87
  • ecd4895 - Merge branch 'release-1.1.7' of github.com:dagster-io/dagster into release-1.1.7 by @jamiedemaria
  • de634ac - 1.1.7 Changelog (#11165) by @jamiedemaria
  • 9882250 - 1.1.7 by @elementl-devtools

1.1.6

1 year ago

New

  • [dagit] Throughout Dagit, when the default repository name __repository__ is used for a repo, only the code location name will be shown. This change also applies to URL paths.
  • [dagster-dbt] When attempting to generate software-defined assets from a dbt Cloud job, an error is now raised if none are created.
  • [dagster-dbt] Software-defined assets can now be generated for dbt Cloud jobs that execute multiple commands.

Bugfixes

  • Fixed a bug that caused load_asset_value to error with the default IO manager when a partition_key argument was provided.
  • Previously, trying to access context.partition_key or context.asset_partition_key_for_output when invoking an asset directly (e.g. in a unit test) would result in an error. This has been fixed.
  • Failure hooks now receive the original exception instead of RetryRequested when using a retry policy.
  • The LocationStateChange GraphQL subscription has been fixed (thanks @****roeij !)**
  • Fixed a bug where a sqlite3.ProgrammingError error was raised when creating an ephemeral DagsterInstance, most commonly when build_resources was called without passing in an instance parameter.
  • [dagstermill] Jupyter notebooks now correctly render in Dagit on Windows machines.
  • [dagster-duckdb-pyspark] New duckdb_pyspark_io_manager helper to automatically create a DuckDB I/O manager that can store and load PySpark DataFrames.
  • [dagster-mysql] Fixed a bug where versions of mysql < 8.0.31 would raise an error on some run queries.
  • [dagster-postgres] connection url param “options“ are no longer overwritten in dagit.
  • [dagit] Dagit now allows backfills to be launched for asset jobs that have partitions and required config.
  • [dagit] Dagit no longer renders the "Job in repo@location" label incorrectly in Chrome v109.
  • [dagit] Dagit's run list now shows improved labels on asset group runs of more than three assets
  • [dagit] Dagit's run gantt chart now renders per-step resource initialization markers correctly.
  • [dagit] In op and asset descriptions in Dagit, rendered markdown no longer includes extraneous escape slashes.
  • Assorted typos and omissions fixed in the docs — thanks @C0DK and @akan72!

Experimental

  • As an optional replacement of the workspace/repository concepts, a new Definitions entrypoint for tools and the UI has been added. A single Definitions object per code location may be instantiated, and accepts typed, named arguments, rather than the heterogenous list of definitions returned from an @repository-decorated function. To learn more about this feature, and provide feedback, please refer to the Github Discussion.
  • [dagster-slack] A new make_slack_on_freshness_policy_status_change_sensor allows you to create a sensor to alert you when an asset is out of date with respect to its freshness policy (and when it’s back on time!)

Documentation

1.1.5

1 year ago

Bugfixes

  • [dagit] Fixed an issue where the Partitions tab sometimes failed to load for asset jobs.

1.1.4

1 year ago

Community Contributions

  • Fixed a typo in GCSComputeLogManager docstring (thanks reidab)!
  • [dagster-airbyte] job cancellation on run termination is now optional. (Thanks adam-bloom)!
  • [dagster-snowflake] Can now specify snowflake role in config to snowflake io manager (Thanks binhnefits)!
  • [dagster-aws] A new AWS systems manager resource (thanks zyd14)!
  • [dagstermill] Retry policy can now be set on dagstermill assets (thanks nickvazz)!
  • Corrected typo in docs on metadata (thanks C0DK)!

New

  • Added a job_name parameter to InputContext.
  • Fixed inconsistent io manager behavior when using execute_in_process on a GraphDefinition (it would use the fs_io_manager instead of the in-memory io manager).
  • Compute logs will now load in Dagit even when websocket connections are not supported.
  • [dagit] A handful of changes have been made to our URLs:
    • The /instance URL path prefix has been removed. E.g. /instance/runs can now be found at /runs.
    • The /workspace URL path prefix has been changed to /locations. E.g. the URL for job my_job in repository foo@bar can now be found at /locations/foo@bar/jobs/my_job.
  • [dagit] The “Workspace” navigation item in the top nav has been moved to be a tab under the “Deployment” section of the app, and is renamed to “Definitions”.
  • [dagstermill] Dagster events can now be yielded from asset notebooks using dagstermill.yield_event.
  • [dagstermill] Failed notebooks can be saved for inspection and debugging using the new save_on_notebook_failure parameter.
  • [dagster-airflow] Added a new option use_ephemeral_airflow_db which will create a job run scoped airflow db for airflow dags running in dagster
  • [dagster-dbt] Materializing software-defined assets using dbt Cloud jobs now supports partitions.
  • [dagster-dbt] Materializing software-defined assets using dbt Cloud jobs now supports subsetting. Individual dbt Cloud models can be materialized, and the proper filters will be passed down to the dbt Cloud job.
  • [dagster-dbt] Software-defined assets from dbt Cloud jobs now support configurable group names.
  • [dagster-dbt] Software-defined assets from dbt Cloud jobs now support configurable AssetKeys.

Bugfixes

  • Fixed regression starting in 1.0.16 for some compute log managers where an exception in the compute log manager setup/teardown would cause runs to fail.
  • The S3 / GCS / Azure compute log managers now sanitize the optional prefix argument to prevent badly constructed paths.
  • [dagit] The run filter typeahead no longer surfaces key-value pairs when searching for tag:. This resolves an issue where retrieving the available tags could cause significant performance problems. Tags can still be searched with freeform text, and by adding them via click on individual run rows.
  • [dagit] Fixed an issue in the Runs tab for job snapshots, where the query would fail and no runs were shown.
  • [dagit] Schedules defined with cron unions displayed “Invalid cron string” in Dagit. This has been resolved, and human-readable versions of all members of the union will now be shown.

Breaking Changes

  • You can no longer set an output’s asset key by overriding get_output_asset_key on the IOManager handling the output. Previously, this was experimental and undocumented.

Experimental

  • Sensor and schedule evaluation contexts now have an experimental log property, which log events that can later be viewed in Dagit. To enable these log views in dagit, navigate to the user settings and enable the Experimental schedule/sensor logging view option. Log links will now be available for sensor/schedule ticks where logs were emitted. Note: this feature is not available for users using the NoOpComputeLogManager.

1.1.3

1 year ago

Bugfixes

  • Fixed a bug with the asset reconciliation sensor that caused duplicate runs to be submitted in situations where an asset has a different partitioning than its parents.
  • Fixed a bug with the asset reconciliation sensor that caused it to error on time-partitioned assets.
  • [dagster-snowflake] Fixed a bug when materializing partitions with the Snowflake I/O manager where sql BETWEEN was used to determine the section of the table to replace. BETWEEN included values from the next partition causing the I/O manager to erroneously delete those entries.
  • [dagster-duckdb] Fixed a bug when materializing partitions with the DuckDB I/O manager where sql BETWEEN was used to determine the section of the table to replace. BETWEEN included values from the next partition causing the I/O manager to erroneously delete those entries.

1.1.2

1 year ago

Bugfixes

  • In Dagit, assets that had been materialized prior to upgrading to 1.1.1 were showing as "Stale". This is now fixed.
  • Schedules that were constructed with a list of cron strings previously rendered with an error in Dagit. This is now fixed.
  • For users running dagit version >= 1.0.17 (or dagster-cloud) with dagster version < 1.0.17, errors could occur when hitting "Materialize All" and some other asset-related interactions. This has been fixed.

1.1.1

1 year ago

Major Changes since 1.0.0 (core) / 0.16.0 (libraries)

Core

  • You can now create multi-dimensional partitions definitions for software-defined assets, through the MultiPartitionsDefinition API. In Dagit, you can filter and materialize certain partitions by providing ranges per-dimension, and view your materializations by dimension.
  • The new asset reconciliation sensor automatically materializes assets that have never been materialized or whose upstream assets have changed since the last time they were materialized. It works with partitioned assets too. You can construct it using build_asset_reconciliation_sensor.
  • You can now add a FreshnessPolicy to any of your software-defined assets, to specify how up-to-date you expect that asset to be. You can view the freshness status of each asset in Dagit, alert when assets are missing their targets using the @freshness_policy_sensor, and use the build_asset_reconciliation_sensor to make a sensor that automatically kick off runs to materialize assets based on their freshness policies.
  • You can now version your asset ops and source assets to help you track which of your assets are stale. You can do this by assigning op_version s to software-defined assets or observation_fn s to SourceAssets. When a set of assets is versioned in this way, their “Upstream Changed” status will be based on whether upstream versions have changed, rather than on whether upstream assets have been re-materialized. You can launch runs that materialize only stale assets.
  • The new @multi_asset_sensor decorator enables defining custom sensors that trigger based on the materializations of multiple assets. The context object supplied to the decorated function has methods to fetch latest materializations by asset key, as well as built-in cursor management to mark specific materializations as “consumed”, so that they won’t be returned in future ticks. It can also fetch materializations by partition and mark individual partitions as consumed.
  • RepositoryDefinition now exposes a load_asset_value method, which accepts an asset key and invokes the asset’s I/O manager’s load_input function to load the asset as a Python object. This can be used in notebooks to do exploratory data analysis on assets.
  • With the new asset_selection parameter on @sensor and SensorDefinition, you can now define a sensor that directly targets a selection of assets, instead of targeting a job.
  • When running dagit or dagster-daemon locally, environment variables included in a .env file in the form KEY=value in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily use environment variables during local development.

Dagit

  • The Asset Graph has been redesigned to make better use of color to communicate asset health. New status indicators make it easy to spot missing and stale assets (even on large graphs!) and the UI updates in real-time as displayed assets are materialized.
  • The Asset Details page has been redesigned and features a new side-by-side UI that makes it easier to inspect event metadata. A color-coded timeline on the partitions view allows you to drag-select a time range and inspect the metadata and status quickly. The new view also supports assets that have been partitioned across multiple dimensions.
  • The new Workspace page helps you quickly find and navigate between all your Dagster definitions. It’s also been re-architected to load significantly faster when you have thousands of definitions.
  • The Overview page is the new home for the live run timeline and helps you understand the status of all the jobs, schedules, sensors, and backfills across your entire deployment. The timeline is now grouped by repository and shows a run status rollup for each group.

Integrations

  • dagster-dbt now supports generating software-defined assets from your dbt Cloud jobs.
  • dagster-airbyte and dagster-fivetran now support automatically generating assets from your ETL connections using load_assets_from_airbyte_instance and load_assets_from_fivetran_instance.
  • New dagster-duckdb integration: build_duckdb_io_manager allows you to build an I/O manager that stores and loads Pandas and PySpark DataFrames in DuckDB.

Database migration

  • Optional database schema migration, which can be run via dagster instance migrate:
    • Improves Dagit performance by adding database indexes which should speed up the run view as well as a range of asset-based queries.
    • Enables multi-dimensional asset partitions and asset versioning.

Breaking Changes and Deprecations

  • define_dagstermill_solid, a legacy API, has been removed from dagstermill. Use define_dagstermill_op or define_dagstermill_asset instead to create an op or asset from a Jupyter notebook, respectively.
  • The internal ComputeLogManager API is marked as deprecated in favor of an updated interface: CapturedLogManager. It will be removed in 1.2.0. This should only affect dagster instances that have implemented a custom compute log manager.

Dependency Changes

  • dagster-graphql and dagit now use version 3 of graphene

Since 1.0.17

New

  • The new UPathIOManager base class is now a top-level Dagster export. This enables you to write a custom I/O manager that plugs stores data in any filesystem supported by universal-pathlib and uses different serialization format than pickle (Thanks Daniel Gafni!).
  • The default fs_io_manager now inherits from the UPathIOManager, which means that its base_dir can be a path on any filesystem supported by universal-pathlib (Thanks Daniel Gafni!).
  • build_asset_reconciliation_sensor now works with support partitioned assets.
  • build_asset_reconciliation_sensor now launches runs to keep assets in line with their defined FreshnessPolicies.
  • The FreshnessPolicy object is now exported from the top level dagster package.
  • For assets with a FreshnessPolicy defined, their current freshness status will be rendered in the asset graph and asset details pages.
  • The AWS, GCS, and Azure compute log managers now take an additional config argument upload_interval which specifies in seconds, the interval in which partial logs will be uploaded to the respective cloud storage. This can be used to display compute logs for long-running compute steps.
  • When running dagit or dagster-daemon locally, environment variables included in a .env file in the form KEY=value in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily test environment variables during local development.
  • observable_source_asset decorator creates a SourceAsset with an associated observation_fn that should return a LogicalVersion, a new class that wraps a string expressing a version of an asset’s data value.
  • [dagit] The asset graph now shows branded compute_kind tags for dbt, Airbyte, Fivetran, Python and more.
  • [dagit] The asset details page now features a redesigned event viewer, and separate tabs for Partitions, Events, and Plots. This UI was previously behind a feature flag and is now generally available.
  • [dagit] The asset graph UI has been revamped and makes better use of color to communicate asset status, especially in the zoomed-out view.
  • [dagit] The asset catalog now shows freshness policies in the “Latest Run” column when they are defined on your assets.
  • [dagit] The UI for launching backfills in Dagit has been simplified. Rather than selecting detailed ranges, the new UI allows you to select a large “range of interest” and materialize only the partitions of certain statuses within that range.
  • [dagit] The partitions page of asset jobs has been updated to show per-asset status rather than per-op status, so that it shares the same terminology and color coding as other asset health views.
  • [dagster-k8s] Added an execute_k8s_job function that can be called within any op to run an image within a Kubernetes job. The implementation is similar to the build-in k8s_job_op , but allows additional customization - for example, you can incorporate the output of a previous op into the launched Kubernetes job by passing it into execute_k8s_job. See the dagster-k8s API docs for more information.
  • [dagster-databricks] Environment variables used by dagster cloud are now automatically set when submitting databricks jobs if they exist, thank you @zyd14!
  • [dagstermill] define_dagstermill_asset now supports RetryPolicy . Thanks @nickvazz!
  • [dagster-airbyte] When loading assets from an Airbyte instance using load_assets_from_airbyte_instance, users can now optionally customize asset names using connector_to_asset_key_fn.
  • [dagster-fivetran] When loading assets from a Fivetran instance using load_assets_from_fivetran_instance, users can now alter the IO manager using io_manager_key or connector_to_io_manager_key_fn, and customize asset names using connector_to_asset_key_fn.

Bugfixes

  • Fixed a bug where terminating runs from a backfill would fail without notice.
  • Executing a subset of ops within a job that specifies its config value directly on the job, it no longer attempts to use that config value as the default. The default is still presented in the editable interface in dagit.
  • [dagit] The partition step run matrix now reflects historical step status instead of just the last run’s step status for a particular partition.

Documentation

1.0.17

1 year ago

New

  • With the new asset_selection parameter on @sensor and SensorDefinition, you can now define a sensor that directly targets a selection of assets, instead of targeting a job.
  • materialize and materialize_to_memory now accept a raise_on_error argument, which allows you to determine whether to raise an Error if the run hits an error or just return as failed.
  • (experimental) Dagster now supports multi-dimensional asset partitions, through a new MultiPartitionsDefinition object. An optional schema migration enables support for this feature (run via dagster instance migrate). Users who are not using this feature do not need to run the migration.
  • You can now launch a run that targets a range of asset partitions, by supplying the "dagster/asset_partition_range_start" and "dagster/asset_partition_range_end" tags.
  • [dagit] Asset and op graphs in Dagit now show integration logos, making it easier to identify assets backed by notebooks, DBT, Airbyte, and more.
  • [dagit] a -db-pool-recycle cli flag (and dbPoolRecycle helm option) have been added to control how long the pooled connection dagit uses persists before recycle. The default of 1 hour is now respected by postgres (mysql previously already had a hard coded 1hr setting). Thanks @adam-bloom!
  • [dagster-airbyte] Introduced the ability to specify output IO managers when using load_assets_from_airbyte_instance and load_assets_from_airbyte_project.
  • [dagster-dbt] the dbt_cloud_resource resource configuration account_id can now be sourced from the environment. Thanks @sowusu-ba!
  • [dagster-duckdb] The DuckDB integration improvements: PySpark DataFrames are now fully supported, “schema” can be specified via IO Manager config, and API documentation has been improved to include more examples
  • [dagster-fivetran] Introduced experimental load_assets_from_fivetran_instance helper which automatically pulls assets from a Fivetran instance.
  • [dagster-k8s] Fixed an issue where setting the securityContext configuration of the Dagit pod in the Helm chart didn’t apply to one of its containers. Thanks @jblawatt!

Bugfixes

  • Fixed a bug that caused the asset_selection parameter of RunRequest to not be respected when used inside a schedule.
  • Fixed a bug with health checks during delayed Op retries with the k8s_executor and docker_executor.
  • [dagit] The asset graph now live-updates when assets fail to materialize due to op failures.
  • [dagit] The "Materialize" button now respects the backfill permission for multi-run materializations.
  • [dagit] Materializations without metadata are padded correctly in the run logs.
  • [dagster-aws] Fixed an issue where setting the value of task_definition field in the EcsRunLauncher to an environment variable stopped working.
  • [dagster-dbt] Add exposures in load_assets_from_dbt_manifest. This fixed then error when load_assets_from_dbt_manifest failed to load from dbt manifest with exposures. Thanks @sowusu-ba!
  • [dagster-duckdb] In some examples, the duckdb config was incorrectly specified. This has been fixed.

Breaking Changes

  • The behavior of the experimental asset reconciliation sensor, which is accessible via build_asset_reconciliation_sensor has changed to be more focused on reconciliation. It now materializes assets that have never been materialized before and avoids materializing assets that are “Upstream changed”. The build_asset_reconciliation_sensor API no longer accepts wait_for_in_progress_runs and wait_for_all_upstream arguments.

Documentation

All Changes

https://github.com/dagster-io/dagster/compare/1.0.16...1.0.17

See All Contributors
  • 7e9d8f6 - [jog] host_representation_tests, instance_tests, selector_tests (#10256) by @dpeng817
  • ec07488 - [jog] resource_tests (#10257) by @dpeng817
  • 8e44cf5 - [dagstermill] notebook backed assets (#10277) by @jamiedemaria
  • ce52573 - notebook and assets example project (#10315) by @jamiedemaria
  • 10187df - [dagster-airbyte][managed-elements] Explicit handling of reconciling secret values (#10195) by @benpankow
  • faece2a - [duckdb] integration improvements (#10114) by @jamiedemaria
  • b094d26 - Fix Helm package skipping (#10313) by @jmsanders
  • 0500534 - Remove special casing for graphql tests (#10327) by @jmsanders
  • 0c89026 - Add client ID to dagit (#10316) by @dpeng817
  • 2cc3545 - [dagster-airflow] add airflow 2 support to make_dagster_job_from_airflow_dag + xcom mock option (#10337) by @Ramshackle-Jamathon
  • a403cfc - 1.0.16 changelog (#10340) by @alangenfeld
  • 3bbca03 - disentangle asset reconciliation sensor from MultiAssetSensorDefinition (#10258) by @sryza
  • 1130ec0 - Mock tqdm in tests to avoid segfaults (#10343) by @jmsanders
  • fa4a445 - Automation: versioned docs for 1.0.16 by @elementl-devtools
  • 3b07b06 - update 1.0.16 changelog (#10346) by @yuhan
  • 8b44cda - Better fix for tooltips around disabled buttons + gitlab icon (#10338) by @salazarm
  • 08f4e2d - Back-compat fix for setting the task definition in the EcsRunLauncher to an env var (#10341) by @gibsondan
  • 8289da9 - Mock each callsite of tqdm (#10368) by @gibsondan
  • 15f1b09 - Build integration steps before test project steps (#10367) by @jmsanders
  • 7fa824b - Packages with test changes only shouldn't run deps (#10309) by @jmsanders
  • a8d457a - Remove test-project-core (#10310) by @jmsanders
  • 369fb96 - chore: add project urls to dagster pypi (#10328) by @rexledesma
  • 5690af5 - fix: add comma (#10373) by @rexledesma
  • 22cf4db - add securityContext to dagit init-user-deployment initContainers (#10369) by @gibsondan
  • adf924f - JB - Docs Typography Follow-ups (#10344) by @braunjj
  • 49081bd - [dagit] Add more logos to tags (#10355) by @jamiedemaria
  • 713d873 - [Multi-dimensional partitions 1/n] Add schema migration for asset event tags table (#10001) by @clairelin135
  • 0244116 - Revert "Mock each callsite of tqdm" (#10379) by @jmsanders
  • a34ffc5 - [Multi-dimensional partitions 2/n] Read/write from asset event tags table (#10056) by @clairelin135
  • a8f8be0 - Prevent segfaults (#10372) by @jmsanders
  • d709c11 - [apidocs] clarify usage of create_databricks_job_op (#10299) by @sryza
  • cefab82 - Mark celery-k8s-test-suite as flaky (#10204) by @jmsanders
  • 6d1429c - [Multi-dimensional partitions 3/n] Output multi-dimensional materializations (#10082) by @clairelin135
  • 454a570 - fix duckdb configs in examples (#10378) by @jamiedemaria
  • 79d1f12 - Revert "Mark celery-k8s-test-suite as flaky" (#10384) by @jmsanders
  • b4ccac3 - Unpin grpc (#10386) by @gibsondan
  • 22d94cd - some refactorings to asset reconciliation sensor (#10233) by @sryza
  • 3065bad - fix mysql heading level + typo (#10336) by @domsj
  • 620d6cf - [dagster-fivetran] Enable loading of Fivetran connection assets from instance (#10290) by @benpankow
  • edcdcba - Resolver -> AssetGraph (#10401) by @sryza
  • a35654b - Enable accessing base jobs off of repository definitions (#10405) by @sryza
  • ef4a4bb - add pyrightconfig.json to .gitignore (#10403) by @sryza
  • e9fcb96 - [dagster-dbt] add exposures in load_assets_from_dbt_manifest (#10395) by @sowusu-ba
  • 339d14c - operate on keys instead of strings in AssetSelection and AssetGraph (#10402) by @sryza
  • 9607192 - Updated "Loaded with Errors" link (#10421) by @gibsondan
  • 7c205d5 - [dagster-io/ui] Add stroke color to TextArea (#10422) by @hellendag
  • 4900d22 - Install project-fully-featured (#10419) by @jmsanders
  • 353ed2a - More process cleanup in grpc tests (#10423) by @gibsondan
  • b6aea90 - Add methods to AssetGraph for traversing asset partitions graph (#10413) by @sryza
  • 29b24f1 - [dagit] Update partner compute_kind tags to latest designs (#10394) by @bengotow
  • 4b207e5 - [dagit] Fix materialization log padding when no metadata is present (#10350) by @bengotow
  • cd5af37 - [dagit] Respect backfill permission in asset backfill modal (#10387) by @bengotow
  • 174a958 - [freshness-policy] add latency freshness policy (#10393) by @OwenKephart
  • 50026a1 - Revert "Unpin grpc (#10386)" (#10428) by @gibsondan
  • 44c9ddb - support RunRequest.asset_selection in schedules (#10400) by @sryza
  • 4df96a7 - support raise_on_error in materialize and materialize_to_memory (#10397) by @sryza
  • 42c47a8 - docs(cacheable-assets): mark unimplemented methods as abstract (#10424) by @rexledesma
  • c694526 - [easy] fix incorrect header in docker guide (#10427) by @gibsondan
  • 36a8383 - [dagit] Fix bug in asset live reloading after step failures, clean up AssetView (#10349) by @bengotow
  • 3f75ae6 - allow sensors to target asset selections instead of jobs (#10417) by @sryza
  • 1365944 - Implement abstract methods (#10434) by @jmsanders
  • 46e41ad - [graphql] handle null int metadata val (#10436) by @alangenfeld
  • dbbfffb - [jog] convert runtime_types_tests (#10317) by @dpeng817
  • bbe1378 - [jog] convert core_test/selector_tests (#10318) by @dpeng817
  • 649ea0e - [easy][dagster-airbyte] Treat SSH key as secret for managed stack (#10390) by @benpankow
  • dba7b8b - [jog] Convert snap tests (#10319) by @dpeng817
  • 9e28855 - Expose sqlalchemy pool recycle option for mysql/postgres (#10416) by @adam-bloom
  • 1e7e9a7 - cached_method decorator (#10398) by @sryza
  • 7f6a458 - Enable action logging in dagit (#10342) by @dpeng817
  • 1e9af55 - Temporarily skip celery-k8s tests (#10438) by @jmsanders
  • b6237b6 - Run builds on changes to .txt files (#10433) by @jmsanders
  • ff77d5e - Hook for loading secrets in grpc/run/step entry points (#10089) by @gibsondan
  • f03a491 - Read account_idfrom an environment variable indbt_cloud_resource (#10324) by @sowusu-ba
  • 7bfe9f1 - Add container_name to EcsContainerContext (#10446) by @gibsondan
  • 15f0f2d - Make asset reconciliation sensor more reconcile-y (#10435) by @sryza
  • 198d234 - [dagster-airbyte] Generate src/dst Python classes for managed stacks (#10272) by @benpankow
  • 28739e8 - [docs] - Guides for configuring environment variables (#10034) by @erinkcochran87
  • e579a21 - Revert "specify schema in pandas to_sql" (#10450) by @jamiedemaria
  • 79eaa72 - feat: bind command-K to search (#10449) by @rexledesma
  • 91487d7 - [dagster-airbyte] Add ability to specify output IO managers (#10217) by @benpankow
  • 7d0cc3e - [dagster-airbyte] When loading managed connections, ignore those which are not managed (#10388) by @benpankow
  • efab894 - enable launching a run that targets a range of asset partitions (#10441) by @sryza
  • 356f768 - chore: remove support for github dark mode (#10329) by @rexledesma
  • f7da04d - [dagit] Clarify notebook buttons (#10451) by @jamiedemaria
  • 2a3f8f2 - [dagstermill] allow custom io manager key specification (#10448) by @jamiedemaria
  • 87c8389 - [dagster-airbyte][docs] Reorganize Airbyte guide (#10215) by @benpankow
  • 00e4491 - Fix check_step_health with delayed op retries (#10458) by @johannkm
  • fd6411f - changelog 1.0.17 (#10464) by @yuhan
  • 4c51207 - 1.0.17 by @elementl-devtools

1.0.16

1 year ago

New

  • [dagit] The new Overview and Workspace pages have been enabled for all users, after being gated with a feature flag for the last several releases. These changes include design updates, virtualized tables, and more performant querying.
    • The top navigation has been updated to improve space allocation, with main nav links moved to the left.
    • “Overview” is the new Dagit home page and “factory floor” view, were you can find the run timeline, which now offers time-based pagination. The Overview section also contains pages with all of your jobs, schedules, sensors, and backfills. You can filter objects by name, and collapse or expand repository sections.
    • “Workspace” has been redesigned to offer a better summary of your repositories, and to use the same performant table views, querying, and filtering as in the Overview pages.
  • @asset and @multi_asset now accept a retry_policy argument. (Thanks Adam Bloom!)
  • When loading an input that depends on multiple partitions of an upstream asset, the fs_io_manager will now return a dictionary that maps partition keys to the stored values for those partitions. (Thanks andrewgryan!).
  • JobDefinition.execute_in_process now accepts a run_config argument even when the job is partitioned. If supplied, the run config will be used instead of any config provided by the job’s PartitionedConfig.
  • The run_request_for_partition method on jobs now accepts a run_config argument. If supplied, the run config will be used instead of any config provided by the job’s PartitionedConfig.
  • The new NotebookMetadataValue can be used to report the location of executed jupyter notebooks, and Dagit will be able to render the notebook.
  • Resolving asset dependencies within a group now works with multi-assets, as long as all the assets within the multi-asset are in the same group. (Thanks @peay!)
  • UPathIOManager, a filesystem-agnostic IOManager base class has been added - (Thanks @danielgafni!)
  • A threadpool option has been added for the scheduler daemon. This can be enabled via your dagster.yaml file; check out the docs.
  • The default LocalComputeLogManager will capture compute logs by process instead of by step. This means that for the in_process executor, where all steps are executed in the same process, the captured compute logs for all steps in a run will be captured in the same file.
  • [dagster-airflow] make_dagster_job_from_airflow_dag now supports airflow 2, there is also a new mock_xcom parameter that will mock all calls to made by operators to xcom.
  • [helm] volume and volumeMount sections have been added for the dagit and daemon sections of the helm chart.

Bugfixes

  • For partitioned asset jobs whose config is a hardcoded dictionary (rather than a PartitionedConfig), previously run_request_for_partition would produce a run with no config. Now, the run has the hardcoded dictionary as its config.
  • Previously, asset inputs would be resolved to upstream assets in the same group that had the same name, even if the asset input already had a key prefix. Now, asset inputs are only resolved to upstream assets in the same group if the input path only has a single component.
  • Previously, asset inputs could get resolved to outputs of the same AssetsDefinition, through group-based asset dependency resolution, which would later error because of a circular dependency. This has been fixed.
  • Previously, the “Partition Status” and “Backfill Status” fields on the Backfill page in dagit were always incomplete and showed missing partitions. This has been fixed to accurately show the status of the backfill runs.
  • [dagit] When viewing the config dialog for a run with a very long config, scrolling was broken and the “copy” button was not visible. This has been fixed.
  • Executors now compress step worker arguments to avoid CLI length limits with large DAGs.
  • [dagster-msteams] Longer messages can now be used in Teams HeroCard - (Thanks @jayhale!)

Documentation

  • API docs for InputContext have been improved - (Thanks @peay!)
  • [dagster-snowflake] Improved documentation for the Snowflake IO manager

All Changes

https://github.com/dagster-io/dagster/compare/1.0.15...1.0.16

See All Contributors
  • ee39fcd - rm all_types from config_context (#10203) by @alangenfeld
  • bad2e44 - Also skip docs next (#10178) by @jmsanders
  • bd6e269 - Make handleLaunchResult agnostic to the query that returned the data (#10179) by @salazarm
  • b5212c7 - Make PythonPackages.get() more flexible to _/- (#10184) by @jmsanders
  • 16bc4cf - [dagit] Invert stored state for expand/collapse in Overview pages (#10212) by @hellendag
  • af58bf9 - [config] memoize ConfigType snap creation (#10210) by @alangenfeld
  • 2eebc3c - [config] avoid double init on field cache objects (#10214) by @alangenfeld
  • fcda5f3 - Override default io manager in more places (#10202) by @johannkm
  • 59ae17e - Fix black (#10216) by @johannkm
  • d4334f1 - type annotations on backfill-related code paths (#9402) by @sryza
  • b56013e - Fix lint (#10218) by @johannkm
  • 3176dcc - Add docs for customizing the serverless base image (#9571) by @petehunt
  • a24298a - Skip test-project image builds (#10099) by @jmsanders
  • f7c058a - [dagit] Updated asset details event view (#10143) by @bengotow
  • 4ef9d27 - [dagit] Fix AssetView.test key warnings (#10226) by @hellendag
  • 2bbf998 - Also trigger builds when tox.ini changes (#10209) by @jmsanders
  • c1ff910 - 1.0.15 changelog (#10227) by @jamiedemaria
  • 688a5d8 - Automation: versioned docs for 1.0.15 by @elementl-devtools
  • 0478723 - add materialization property on dagster event (#10230) by @prha
  • cf8ad4a - Parse HEAD commit in addition to BUILDKITE_MESSAGE (#10208) by @jmsanders
  • 58ee17c - Allow default io_manager load_input to support partitions of differing frequencies (#10172) by @andrewgryan
  • 67313e8 - Install the correct dagster-buildkite CLI (#10211) by @jmsanders
  • cf84896 - Type annotations in dagster-graphql (#10005) by @smackesey
  • 174dd99 - fix black, mypy (#10235) by @prha
  • 3c89b70 - [freshness-policy] [1/n] FreshnessPolicy object (#10024) by @OwenKephart
  • 281595b - Break up sensor_definition.py (#10181) by @sryza
  • ffa01e1 - Highlight config entry when being hovered in yaml editor (#10239) by @salazarm
  • a5f0fa9 - Resolve multi-asset deps when they have the same group (#10222) by @peay
  • 77faa3c - update apidoc for postgres (#10241) by @prha
  • 35b8f8b - fix schedules threading config (#10247) by @alangenfeld
  • 8296cf7 - [dagit] Ship Overview/Workspace (#10245) by @hellendag
  • 24af700 - Fix stray references to define_assets_job (#10199) by @bengotow
  • 631c3ef - captured log manager (#9429) by @prha
  • 65b47a0 - Allow for longer messages in Teams HeroCard (#10234) by @jayhale
  • ee43f93 - [dagit] Clean up code post-Overview changes (#10249) by @hellendag
  • a097c43 - [dagit] Move the refresh indicator on asset details pages to avoid flicker (#10253) by @bengotow
  • ce7afb7 - [dagit] Fix issues with embedding fonts in downloaded DAG SVGs (#10252) by @bengotow
  • 37f2c95 - [dagit] Fix daylight savings issue in humanCronString.test (#10263) by @hellendag
  • 4ca8da0 - [dagit] Make 50-level colors opaque (#10186) by @hellendag
  • 997fdbb - [dagit] Add query countdown/refresh to Timeline (#10250) by @hellendag
  • da7555c - Support overriding run config for partitions with execute_in_process (#10246) by @sryza
  • a2bb0cf - Add calendar icon (#10267) by @salazarm
  • 5de98f5 - UPathIOManager - filesystem-agnostic IOManager base (#10193) by @danielgafni
  • 40bf6c7 - rename logKey to fileKey / logFileKey for disambiguation with new API (#9956) by @prha
  • dc667ef - fix some cases in group-based asset dep resolution (#10266) by @sryza
  • ef678e6 - dagster-io/ui release notes (#10269) by @salazarm
  • 966bfb7 - fix UnresolvedAssetJobDefinition.run_request_for_partition when confi… (#10238) by @sryza
  • 36f7c38 - Improve doc for InputContext.{dagster_type,metadata} (#10242) by @peay
  • 69bfd0c - Fetch asset materialization planned event from index shard in Sqlite (#10248) by @clairelin135
  • cb62bc4 - fix #9193: add retry policy to @asset and @multi_asset (#10150) by @adam-bloom
  • ca18259 - adds dagster-aws install to dockerfile in docker deployment guide (#10225) by @jamiedemaria
  • e6444be - keep storage name for serialized event backcompat (#10280) by @prha
  • 16b3cc3 - graphql for captured log subscription (#9957) by @prha
  • 9ccfe7c - Add NotebookMetadataValue (#10278) by @jamiedemaria
  • b88f309 - Add volumes and volumeMounts to dagit and daemon in OSS helm chart (#10285) by @gibsondan
  • 1830880 - support new captured log API for process-based execution (#9958) by @prha
  • a0a1340 - add new capture APIs in frontend queries (#9959) by @prha
  • 621b57a - specify schema in pandas to_sql (#10289) by @jamiedemaria
  • 7dd241f - [dagit] Fix AssetView flakiness (#10293) by @hellendag
  • fc6ea04 - [dagit] Middle-truncate asset key path in virtualized list (#10275) by @hellendag
  • b4c0930 - Snowflake IO Manager API docs (#10175) by @jamiedemaria
  • 2b3280f - Compress execute_step args (#10244) by @johannkm
  • 1dd602a - fix black formatting (#10298) by @prha
  • 9a3b5bc - fix backfill table status (#10295) by @prha
  • f7f1e13 - run_config argument on run_request_for_partition (#10279) by @sryza
  • 0e5386b - [dagit] NotebookMetadataValue support (#10287) by @jamiedemaria
  • f966490 - [dagit] Fix very tall configs in run config dialog (#10301) by @hellendag
  • f466eb7 - Observable source asset decorator (#9899) by @smackesey
  • 70c5f8d - convert NoOpComputeLogManager to support captured log API (#9960) by @prha
  • 4df6e55 - convert S3 compute log manager to support new captured log API (#9961) by @prha
  • c8dee2d - Test createdBefore run filter (#10270) by @benpankow
  • 7c38a8c - fix build; isort; black (#10303) by @prha
  • 082b407 - Fix example scaffold (#10003) by @smackesey
  • 6459b0f - Add version to asset decorator (#10167) by @smackesey
  • 3f99335 - Fix logic to run lints on builds (#10304) by @jmsanders
  • 98fe688 - Don't repeat skip logs (#10307) by @jmsanders
  • 0fc9f84 - [jog] Add a utility method to execute op inside of graph (#10255) by @dpeng817
  • 6b333d5 - [jog] execution_tests,hook_tests (#10254) by @dpeng817
  • 1283e63 - Revert "convert S3 compute log manager to support new captured log API (#9961) (#10311) by @prha
  • 834abba - [docs] Automatically toggle tab components to display URL hash/anchor (#10231) by @benpankow
  • 5b0d959 - [dagstermill] notebook backed assets (#10277) by @jamiedemaria
  • 345f0b3 - notebook and assets example project (#10315) by @jamiedemaria
  • 9157ebb - [dagster-airbyte][managed-elements] Explicit handling of reconciling secret values (#10195) by @benpankow
  • 251b666 - [dagster-airflow] add airflow 2 support to make_dagster_job_from_airflow_dag + xcom mock option (#10337) by @Ramshackle-Jamathon
  • 89948a3 - 1.0.16 changelog (#10340) by @alangenfeld
  • afb20aa - 1.0.16 by @elementl-devtools