An orchestration platform for the development, production, and observation of data assets.
The new @graph_asset and @graph_multi_asset decorators make it more ergonomic to define graph-backed assets.
Dagster will auto-infer dependency relationships between single-dimensionally partitioned assets and multipartitioned assets, when the single-dimensional partitions definition is a dimension of the MultiPartitionsDefinition.
A new Test sensor / Test schedule button that allows you to perform a dry-run of your sensor / schedule. Check out the docs on this functionality here for sensors and here for schedules.
[dagit] Added (back) tag autocompletion in the runs filter, now with improved query performance.
[dagit] The Dagster libraries and their versions that were used when loading definitions can now be viewed in the actions menu for each code location.
New bigquery_pandas_io_manager can store and load Pandas dataframes in BigQuery.
[dagster-snowflake, dagster-duckdb] SnowflakeIOManagers and DuckDBIOManagers can now default to loading inputs as a specified type if a type annotation does not exist for the input.
[dagster-dbt] Added the ability to use the “state:” selector
[dagster-k8s] The Helm chart now supports the full kubernetes env var spec for Dagit and the Daemon. E.g.
dagit:
env:
- name: “FOO”
valueFrom:
fieldRef:
fieldPath: metadata.uid
ruff, pyright).https://github.com/dagster-io/dagster/compare/1.1.19...1.1.20
disable check_same_thread on in-memory sqlite storage (#12229) by @alangenfeld[direct invoke] yield implicit Nothing Output (#12309) by @alangenfeldFix multipartitions run length encoding error (#12329) by @clairelin135[freshness-policies] Allow setting freshness policies when using graph-backed assets (#12357) by @OwenKephartAdd skip reason to asset sensor (#12343) by @OwenKephartFix partitions backfill deserialization error (#12238) by @clairelin135Move CachingRepositoryData.from_list and from_dict into standalone function (#12321) by @schrocknrefactor(databricks): divest from databricks_api in favor of databricks-cli (#12153) by @rexledesma nullsafe array index access (#12362) by @salazarmChange schedule button text (#12361) by @dpeng817[dagit] add full serialized error to graphql errors (#12228) by @alangenfeld[dagster-pandas][dagster-pandera] assign a typing_type for generated pandas dataframe DagsterTypes (#12363) by @OwenKephart[typing/static] Execution API types (#12330) by @smackesey[refactor] DependencyDefinition renames (#12338) by @smackesey[refactor] execute_step renames (#12354) by @smackesey[typing/runtime] Standardize StepInputSource.load_input_object (#12342) by @smackesey[refactor] Delete NodeInput.solid_name (#12339) by @smackesey[refactor] NodeDefiniton.iterate_solid_defs -> iterate_op_defs (#12336) by @smackesey[refactor] GraphDefinition method renames (#12335) by @smackesey[2/n][structured config] Enable struct config resources, IO managers to depend on other resources (#11645) by @benpankow[refactor] local var/private arg solid -> node (#12337) by @smackeseyDagsterLibraryRegistry (#12266) by @alangenfeld[refactor] misc core solid -> node renames (#12368) by @smackesey[refactor] dagster._core.definitions.solid_container -> node_container (#12369) by @smackesey[refactor] Assorted local var solid -> node (#12370) by @smackeseyFix intermittent dynamic partitions table SQLite concurrency error (#12367) by @clairelin135change storage signature for run tags (#12348) by @prha1.1.19 Changelog (#12378) by @OwenKephartguide to how assets relate to ops and graphs (#12204) by @sryza[pythonic config] Rename pythonic config classes (#12235) by @benpankowupdates tests to handle new kubernetes resources field (#12395) by @alangenfeld[structured config] Migrate resources from project-fully-featured to struct config resources (#11785) by @benpankowrefactor(databricks): add types to databricks.py (#12364) by @rexledesmarefactor(databricks): consolidate types (#12366) by @rexledesmaadd dagster_libraries to ListRepositoriesResponse (#12267) by @alangenfeld[graphql] add RepositoryLocation.dagsterLibraryVersions (#12268) by @alangenfeld[dagit] add dagster libraries menu to code location row (#12315) by @alangenfeld1.1.19 changelog: reorder code block (#12402) by @yuhanrefactor(databricks): use databricks_cli's raw api client (#12377) by @rexledesma[docs] [snowflake] Add partitions to snowflake guide (#12231) by @jamiedemariaAdd valid start time check to materialized time partitions subsets (#12403) by @clairelin135Add api docs for some PartitionsDefinition and PartitionMapping classes (#12365) by @sryzaAdd text to timestamp dropdown (#12379) by @dpeng817[refactor] IExecutionStep.solid_handle -> node_handle (#12371) by @smackesey[asset-reconciliation] Factor in more run statuses (#12412) by @OwenKephart[refactor] *ExecutionContext.solid_config -> op_config (#12372) by @smackesey[refactor] ResolvedRunConfig.solids -> ops (#12373) by @smackeseyAutomation: versioned docs for 1.1.19 by @elementl-devtools[refactor] assorted Dagstermill renames (#12380) by @smackesey[refactor] Context solid renames (#12374) by @smackeseylambda_solid -> solid (#10816) by @smackesey[refactor] Assorted pipeline_run -> dagster_run (#12383) by @smackeseyMake a script to template out new dagster packages (#12389) by @jamiedemaria[library template] add registry call (#12418) by @alangenfeld[db io managers] connection refactor (#12258) by @jamiedemariaCode location alerting docs (#12411) by @dpeng817[refactor] remove @solid decorator (#10952) by @smackesey[refactor] Delete PipelineRunsFilter (#12384) by @smackesey[db io managers] add default_load_type (#12356) by @jamiedemaria[refactor] Delete RunRecord.pipeline_run (#12385) by @smackesey[refactor] pipeline_run_from_storage -> dagster_run_from_storage (#12386) by @smackesey[Docs RFC] Dynamic Partitions (#12227) by @clairelin135Auto infer multipartition <-> single dimension mapping (#12400) by @clairelin135[refactor] execution pipeline_run -> dagster_run (#12388) by @smackesey[test-api-update] execution_tests/dynamic_tests (#12427) by @smackeseyConsider the run worker unhealthy is the job has no active pods but the run is in a non-terminal state (#11510) by @gibsondanfix: only inspect schema when we may create tables (#12269) by @plaflammeMore helpful asset key mismatch errors (#12008) by @benpankowAdd docs for testing schedules/sensors via UI (#12381) by @dpeng817document missing breaking change in 1.1.19 changelog (#12424) by @sryzaBigQuery IO manager (#11425) by @jamiedemaria[dagster-gcp-pandas] API docs fix (#12450) by @jamiedemaria[graphql] launch backfills over assets with different partitionings, if all roots have same partitioning (#11827) by @sryzaFix resolution error with input manager key and custom dagster type (#12449) by @clairelin135[dagster-dbt] Add ability to use the "state:" selector (#12432) by @OwenKephartRevert "More helpful asset key mismatch errors (#12008)" (#12459) by @benpankow[dagit] Add missing React keys to prevent new warning toasts (#12210) by @bengotow[CustomConfirmationDialog] Allow overriding the button text (#12444) by @salazarmadd another todo to create_dagster_package (#12453) by @jamiedemaria[dagster-gcp-pandas] register library in init (#12469) by @jamiedemaria[bugfix] fix projected logical version resolution for asset downstream of self-dep (#12443) by @smackeseyfix: resolve correct legacy arguments for emr pyspark step launcher (#12419) by @rexledesma[dagster-dbt] Add missing files for tests (#12471) by @OwenKephartAdd task_role_arn and execution_role_arn to EcsContainerContext (#12358) by @gibsondandagster-census api docs (#12413) by @yuhangraph_asset and graph_multi_asset decorators (#10152) by @sryza[dagster-snowflake-pyspark] fix bug loading partitions (#12472) by @jamiedemariaAdd stale status causes (#11953) by @smackeseyUpdate Contributing doc with instructions for ruff/pyright (#12481) by @smackesey[bigquery] mark bigquery io manager experimental (#12479) by @jamiedemariafix (#12477) by @salazarmadd graphql fields for querying run tags (#12409) by @prhaadd partial tag autocomplete for run filter input (#12410) by @prhaUpdate GQL to expose StaleStatus and StaleStatusCause (#11952) by @smackeseySupport env valueFrom in Helm chart (#12425) by @johannkmFix state status logical version test (#12484) by @smackeseyuse opt_nullable_mapping for dagster library versions (#12487) by @alangenfeldfix ruff (#12486) by @alangenfeld[pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445) by @benpankow[structured config] Add support for Selectors w/ pydantic discriminated unions (#11280) by @benpankow[pythonic resources] Last set of class renames (#12490) by @benpankowAllow setting logical version inside op (#12189) by @smackeseyAdd example of customizing task role and execution role arn to the ECS agent docs (#12491) by @gibsondanRemove existing RunConfig class (#12488) by @benpankow[pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965) by @benpankow[draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260) by @benpankow[draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349) by @benpankow1.1.20 changelog (#12506) by @benpankow[fix] fix sphinx airflow version parsing (#12507) by @benpankow1.1.20 by @elementl-devtoolsFreshnessPolicy object now supports a cron_schedule_timezone argument.AssetsDefinition.from_graph now supports a freshness_policies_by_output_name parameter.@asset_sensor will now display an informative SkipReason when no new materializations have been created since the last sensor tick.AssetsDefinition now has a to_source_asset method, which returns a representation of this asset as a SourceAsset.from dagster import asset, job, op
@asset
def emails_to_send():
...
@op
def send_emails(emails) -> None:
...
@job
def send_emails_job():
send_emails(emails_to_send.to_source_asset())
--dagit-host/-h argument to the dagster dev command to allow customization of the host where Dagit runs.input_manager_key to an asset’s input, incorrect resource config could be used when loading that input. This has been fixed.runK8sConfig in the Dagster Helm chart would not pass configuration through to pods launched using the k8s_job_executor.execute_k8s_job op downstream of a dynamic output would result in k8s jobs with duplicate names being created. This has been fixed.asset_key, asset_partitions, and asset_partitions_defs arguments on Out.@multi_asset no longer accepts Out values in the dictionary passed to its outs argument. This was experimental and deprecated. Instead, use AssetOut.top_level_resources argument to the repository decorator has been renamed to _top_level_resources to emphasize that it should not be set manually.load_asset_values now accepts resource configuration (thanks @Nintorac!)UPathIOManager, paths with the "." character in them would be incorrectly truncated, which could result in multiple distinct objects being written to the same path. This has been fixed. (Thanks @spenczar!)https://github.com/dagster-io/dagster/compare/1.1.18...1.1.19
feat(dagster-dbt): support dbt-core 1.4.x (#11902) by @rexledesmaFixes/improvements for pyright script (#12175) by @smackeseynux examples in oss 1/: add dagster_cloud.yaml to prep cloud nux onboarding (#12172) by @yuhan[dagster-dbt] [1/2] Add streaming entrypoint for dbt cli execution (#12086) by @OwenKephartFix import (#12185) by @OwenKephart(tick-testing 3/6) type annotations to schedules gql (#12057) by @dpeng817(tick-testing 4/6) mutation to dry-run sensor (#11616) by @dpeng817[db io managers] support static partitions (#12129) by @jamiedemariaPass correct config to io manager when using input_manager_key (#12053) by @jamiedemaria[dagster-dbt] [2/2] Add ability to stream events while executing dbt assets (#12100) by @OwenKephartAgent downtime alert docs (#12186) by @johannkm(tick-testing 5/6) Add SensorType (#12021) by @dpeng817(tick-testing 6/6) Add dry run mutations for schedules (#11869) by @dpeng817[dagit] Add some icons for Cloud (#12187) by @hellendag[1/n][structured config] Add ability to runtime-configure struct-config resources (#11773) by @benpankowfix pyright (#12193) by @jamiedemaria[dagit] Storybooks for asset partition and event details + dayJS fix (#12178) by @bengotow[dagster-wandb] Integration with Weights & Biases (#10470) by @chrishisteStore the instance ref on the grpc server class if it's present, so that grpc api calls can use it if it's there (#12194) by @gibsondanW&B integration follow up 2/ update example to defs (#12059) by @yuhannux examples in oss 2/: add required env vars in README's front matter (#12154) by @yuhanSensor testing UI (#12148) by @salazarmvalidate time partition keys when adding to a TimeWindowPartitionsSubset (#12195) by @OwenKephartMake Partition GRPC calls with instance_ref args optional (#12196) by @clairelin135Add public api doc for create_repository_using_definitions_args, update typehints, and cleanup Definitions docstring (#12176) by @schrockn[dagster-airflow] [docs] migration guide updates/considerations (#12198) by @Ramshackle-Jamathon[dagster-airflow] refactor airflow_db resources (#12202) by @Ramshackle-JamathonFix alert docs (#12208) by @johannkmTemporarily disable pyright in BK (#12212) by @smackeseyRename "Loading multiple repositories" section in workspace files docs (#12201) by @gibsondanMake top_level_resources argument _top_level_resources (#12211) by @schrocknRevert "nux examples in oss 2/: add required env vars in README's front matter" (#12199) by @yuhanPyright config/script fixes and improvements (#12206) by @smackeseyAdd guide for observable source assets and versioning (#12118) by @schrockn1.1.18 Changelog (#12217) by @smackeseyFix dagster-wandb placeholder (#12222) by @smackeseyAdd "--dagit-host" arg to dagster dev (#12220) by @gibsondandiff against origin/master in quick_pyright (#12200) by @sryzaRe-enable pyright in BK (#12214) by @smackeseyRemove restart-on-failure from local docker agent guide (#12224) by @gibsondan[dagit] Storybooks for MetadataEntry rendering (#12177) by @bengotowAdd airbyte guide to experimental guide index (#12207) by @smackeseyAutomation: versioned docs for 1.1.18 by @elementl-devtoolsSchedules Testing UI (#12160) by @salazarmKeep dots in paths in UPathIOManager (#12174) by @spenczarremove Out.asset_key (#12221) by @sryza[structured config] Fix env vars not working with direct Resource instantiation (#11468) by @benpankowallow using SourceAssets to satisfy node inputs in non-asset jobs (#12091) by @sryzaAdd observe function (source asset analogue to materialize) (#11996) by @smackeseytreat pyright warnings as errors for buildkite purposes (#12239) by @gibsondanUpdate serverless docs with new run isolation default (#12225) by @johannkmdocs(dbt-cloud): add instructions to cache dbt Cloud compilation (#11793) by @rexledesmaAllow dagster api grpc --max-workers (#12246) by @smackeseyAllow Ursula to test dynamic partition writes (#12215) by @clairelin135[docs] fix dbt tutorial error (#12255) by @jamiedemaria[docs] - Correct name of freshness sensor context object (#12250) by @erinkcochran87document system tags on metadata/tags page (#12256) by @sryza[dagster-airflow] remove ref to cli (#12262) by @Ramshackle-JamathonRevamp intro of partitions concepts page (#12257) by @sryzaRemove @experimental from build_assets_job and build_source_asset_observation_job (#12252) by @smackesey[fix] Surface the correct description for multi-assets when a description is provided for the op (#12271) by @OwenKephartremove legacy APIs from dagster-k8s tests (#12020) by @sryzaUse Definitions in the body of materialize functions (and other public-facing execution functions) (#12038) by @schrockn[typing/static] assets (#12003) by @smackesey[typing/runtime] Make GraphDefinition.create_adjacency_lists internally public (#12291) by @smackesey[typing/runtime] add AssetsDefinition.partition_mappings (#12290) by @smackeseyin docs navigation, make partitions independent from schedules (#12264) by @sryza[typing/static] storage (#12283) by @smackesey[api docs] Show PartitionsDefinition doc strings on docs site (#12265) by @jamiedemaria[db io manager] Support MultiPartitions (#12165) by @jamiedemariaload asset value config (#10991) by @Nintoracremove some handling for solid in composition.py (#12278) by @sryzaAdd shortcuts for '--dagit-host' and '--dagit-port' to dagster dev (#12301) by @gibsondan[typing/runtime] [gql] eliminate resolver **kwargs (2) (#12241) by @smackesey[typing/runtime] [gql] eliminate resolver **kwargs (4) (#12243) by @smackesey[typing/runtime] [gql] eliminate resolver **kwargs (5) (#12244) by @smackesey[Runs Table] Open link in new tab (#12307) by @salazarm[typing/runtime] [gql] eliminate resolver **kwargs (3) (#12242) by @smackeseyadd types and break up big functions in dagstermill impl (#12274) by @sryza[typing/runtime] [gql] eliminate resolver **kwargs (1) (#11724) by @smackeseyadd cloud to scaffold (#12306) by @slopp[typing/runtime] Remove unnecessary isinstance checks (#12287) by @smackesey[typing/runtime] Make AssetKey.to_string always return AssetKey (#12285) by @smackesey[typing/runtime] Refactor cached_method (#12004) by @smackeseyRemove mypy from tox files (#12112) by @smackesey[typing/runtime] Asset graph comparisons (#12286) by @smackeseyrefactor(databricks): remove legacy Dagster definitions (#12150) by @rexledesmadocs: update scaffold to use dagster dev (#12297) by @rexledesma[db io managers] create schema if not exist (#11764) by @jamiedemariaAssetsDefinition.to_source_asset (#12203) by @sryzain tags doc, fix schedule tag and add sensor tag (#12313) by @sryzaSimplify buildkite-build-test-project-image (#12029) by @gibsondanMove repository_definition into subpackage (#12318) by @schrocknBreak up repository_definition subpackage into multiple files (#12319) by @schrockn[docs] - Build on local Dagster guide (#12168) by @erinkcochran87[freshness-policies] add a cron_schedule_timezone argument to the FreshnessPolicy class (#12263) by @OwenKephart[typing/runtime] Refactor subset selection tree (#12289) by @smackeseyFix unimported symbol BK (#12327) by @smackeseynux examples in oss 3/: sync dagster-io/quickstart-*/setup.py to example/* (#12234) by @yuhan[dagster-k8s] In the execute_k8s_job op, use step key to generate the k8s job name (#12344) by @OwenKephartInclude instance-level / code-location-level runK8sConfig in step pods (#12308) by @gibsondanremove unused lazy-repository docs snippet (#12277) by @sryza[asset-reconciliation][perf] Perf regression tests (#12230) by @OwenKepharts/_CacheingDefinitionIndex/CacheingDefinitionIndex/g (#12320) by @schrockn[typing/runtime] Massage set flattening (#12288) by @smackesey[asset-reconciliation][perf] Better caching of most recent materializations (#12237) by @OwenKephart[db io managers] Dynamic Partition tests (#12216) by @jamiedemariabump limit (#12355) by @OwenKephartFix multipartitions run length encoding error (#12329) by @clairelin135[freshness-policies] Allow setting freshness policies when using graph-backed assets (#12357) by @OwenKephart nullsafe array index access (#12362) by @salazarmChange schedule button text (#12361) by @dpeng817[dagster-pandas][dagster-pandera] assign a typing_type for generated pandas dataframe DagsterTypes (#12363) by @OwenKephartFix partitions backfill deserialization error (#12238) by @clairelin1351.1.19 Changelog (#12378) by @OwenKephartguide to how assets relate to ops and graphs (#12204) by @sryzaupdates tests to handle new kubernetes resources field (#12395) by @alangenfeld1.1.19 changelog: reorder code block (#12402) by @yuhan1.1.19 by @elementl-devtoolsPartitionsDefinitions (e.g. HourlyPartitionsDefinition, DailyPartitionsDefinition) may now have a FreshnessPolicy.load_assets_from_dbt_project or load_assets_from_dbt_manifest with dbt-core>=1.4, AssetMaterialization events will be emitted as the dbt command executes, rather than waiting for dbt to complete before emitting events.make_ephemeral_airflow_db_resource which returns a ResourceDefinition for a local only airflow database for use in migrated airflow DAGsdagster instance migrate.observable_source_asset incorrectly required its function to have a context argument.dagster-airbyte
dagster-dbt integration was incompatible with dbt-core>=1.4. This has been fixed.load_assets_from_dbt_cloud_job will now avoid unnecessarily generating docs when compiling a manifest for the job. Compile runs will no longer be kicked off for jobs not managed by this integration.context.asset_partition_key returned a string instead of a MultiPartitionKey. This has been fixed.k8s_job_executor would sometimes unexpectedly fail due to transient 401 errors in certain kubernetes clusters.load_assets_from_airflow_dag no longer creates airflow db resource definitions, as a user you will need to provide them on Definitions directlypartitions_fn argument of the DynamicPartitionsDefinition class is now deprecated and will be removed in 2.0.0.asset (rather than using @asset in decorator form) to create an asset. Thanks @ns-finkelstein!https://github.com/dagster-io/dagster/compare/1.1.17...1.1.18
Add google analytics tracking query param to slack link from OSS (#12024) by @salazarmAdd --working-directory argument to dagster dev (#12026) by @gibsondan[dagster-fivetran] Add option to force-create materializations for tables not in API response (#11972) by @benpankowdocs: removed typo from install.mdx (#12022) by @clayheaton[dagit] Fix use of fragments causing Apollo caching error in partition health (#12030) by @bengotow[dagster-airflow] use full timestamp for partition name (#12034) by @Ramshackle-Jamathon[dagster-airflow] airflow retry support (#11954) by @Ramshackle-JamathonChange endpoints to the ones that are used by airbyte UI (#12012) by @emilija-omnisendlint fix (#12044) by @benpankowfeat(dbt-cloud): compile run only if job has environment variable cache (#12042) by @rexledesmafix(dbt-cloud): inherit generate docs settings for compile run (#12043) by @rexledesmaskip import test on windows (#12048) by @alangenfeld1.1.15 changelog (#12051) by @jamiedemaria[pyright] [scripts] misc (#11923) by @smackeseyaccept UndefinedAssetJob in run status sensors (#12054) by @jamiedemariaEnable use of arguments when using the asset function directly instead of as a decorator (#11903) by @nsfinkelsteinAutomation: versioned docs for 1.1.15 by @elementl-devtoolsBump Postgres to 14.6 (#12015) by @DustyShapmake black (#12078) by @alangenfeldConstruct required resources correctly when materializing partial asset graphs (#12052) by @jamiedemariaFix context.asset_partition_key for multidimensional partitions (#12035) by @clairelin135Install dagster-managed-elements when appropriate (#12046) by @jmsandersBump http-cache-semantics from 4.1.0 to 4.1.1 in /js_modules/dagit (#12080) by @dependabot[bot][docs] [dagster-snowflake] update docs with private key auth info (#11746) by @jamiedemariain comment, clarify that scraping asset info off of In/Out is legacy (#12089) by @sryzaLazy load ExternalAssetGraph for CachingStaleStatusResolver (#12090) by @smackesey[telemetry] fix setting __TELEMETRY_ENABLED__ flag on Dagit (#12092) by @benpankow[pyright] misc (#12102) by @smackeseyAllow observable source assets to have no context argument (#11981) by @schrockn[dagster-airflow] remove api's for 1.x.x release (#12023) by @Ramshackle-Jamathon[dagster-airflow] extract utils into seperate file (#12065) by @Ramshackle-Jamathon[dagster-airflow] move schedule and asset functions into their own files (#12066) by @Ramshackle-Jamathon[dagster-airflow] remove vended airflow code (#12121) by @Ramshackle-JamathonFix pre-release core => library translation (#12109) by @gibsondanadd run job index (#12033) by @prhaAdd Databricks compute kind tag (#12117) by @braunjj[schedules] fix nth weekday of month cron handling (#12130) by @alangenfeld[dagit] Add tests for unpartitioned case, fix for console.error in partition health (#12085) by @bengotow[js] fix lint (#12131) by @alangenfeld[pythonic config] Add disabled tests for various more complex config schemas (#12105) by @benpankow[docs] Clairfy dagster-pagerduty docs (#12128) by @benpankowTest with a 64 bit after_cursor (#12093) by @jmsanders[dagit] Memoize some Intl behavior (#12125) by @hellendag[dagit] Add storybook coverage of Asset Table states (#12103) by @bengotow[dagit] Add Storybook coverage of PartitionHealthSummary rendering (#12116) by @bengotow[dagit] In development, render toasts for unhandled promise exceptions (#12082) by @bengotowInclude logs in failure message for ECS monitoring failures (#12113) by @gibsondan[dagster-airflow] 1.x.x api changes (#12067) by @Ramshackle-Jamathon[dagster-airflow] migration limitations (#12124) by @Ramshackle-Jamathon[dagit] Mocks, storybooks and a test for BackfillTable (#12095) by @bengotow[pythonic config] Correctly handle pydantic List types in config (#12106) by @benpankowcomment to explain what define_solid_dictionary_cls does (#12088) by @sryzaremove protobuf pin in dagster[test] (#11974) by @yuhan[dagster-airflow] drop unique_id param (#12127) by @Ramshackle-JamathonTest with a 64 bit after_cursor (#12137) by @jmsanders[dagit] Show partition status on the asset graph, catalog pages, add tests (#11914) by @bengotow[pythonic config] Correctly handle pydantic Dict/Mapping types in config (#12107) by @benpankowrelease 1.1.17 changelog (#12140) by @Ramshackle-Jamathon[docs] Add PySpark to Snowflake reference guide (#11814) by @jamiedemariaAdd bugfixes to 1.1.17 changelog (#12141) by @gibsondanAutomation: versioned docs for 1.1.17 by @elementl-devtoolsRuff 0.0.212 -> 0.0.241 (#12138) by @smackeseyAdopt pyright for typechecking (#10983) by @smackesey[1/n] Serialize top-level resources into repository data (#11529) by @benpankow[2/n] Expose top-level resources via GraphQL (#11553) by @benpankowFix serdes on FreshnessPolicy object (#12143) by @benpankowBetter error output for Pyright script (#12145) by @smackesey[3/n] Display top-level resources in Dagit sidebar (#11554) by @benpankowFix misc type errors (#12147) by @smackeseyDelete no-longer-needed type-ignores (#12156) by @smackesey[dagit] Fix build failure caused by mocks out of sync with query (#12152) by @bengotowSupport json output from pyright script and ignore build (#12155) by @smackeseyAdd pyright_rebuild Makefile target (#12159) by @smackesey[pyright] --find-links for gprcio wheels (#12158) by @alangenfeld[docs] - Remove Pagination component from layout (#12135) by @erinkcochran87Fix instructions for using serverless from the CLI without GitHub (#12161) by @shalabhcmaking erin the owner of all docs 👑 (#12139) by @tacastillo[dagit] Better Asset DAG states for source assets (#12142) by @bengotow[gql] Add tags to AssetEventMixin (#11973) by @smackeseytype annotations for input_bindings in composition.py (#12087) by @sryza[Dynamic Partitions 1] Storage changes (#11994) by @clairelin135[dagit] System tags (code + logical versions) on asset graph + details (#12151) by @bengotow[Dynamic Partitions 2] Update DynamicPartitionsDefinition to have name param (#12000) by @clairelin135hide download links until logs are available (#12164) by @prha(tick-testing 1/6) Rename GrapheneFutureInstigationTick to GrapheneDryRunInstigationTick (#12055) by @dpeng817[Dynamic Partitions 3] Display dynamic partitions in Dagit (#11900) by @clairelin135Avoid creating repo on each partitionStats gql call (#12166) by @clairelin135[Docs] Removed sneaky citation mark in title on docs (#12120) by @C0DKAdd 401 to the list of API codes that our k8s client retries on (#12074) by @gibsondan[freshness-policies] Add the ability to calculate the used_data_time of a TimeWindowPartitioned asset (#11607) by @OwenKephart(tick-testing 2/6) Revamp dry run tick behavior (#12056) by @dpeng817feat(dagster-dbt): support dbt-core 1.4.x (#11902) by @rexledesma[dagster-dbt] [1/2] Add streaming entrypoint for dbt cli execution (#12086) by @OwenKephart[dagster-dbt] [2/2] Add ability to stream events while executing dbt assets (#12100) by @OwenKephartFix import (#12185) by @OwenKephartStore the instance ref on the grpc server class if it's present, so that grpc api calls can use it if it's there (#12194) by @gibsondan[dagster-wandb] Integration with Weights & Biases (#10470) by @chrishisteW&B integration follow up 2/ update example to defs (#12059) by @yuhanvalidate time partition keys when adding to a TimeWindowPartitionsSubset (#12195) by @OwenKephartMake Partition GRPC calls with instance_ref args optional (#12196) by @clairelin135[dagster-airflow] [docs] migration guide updates/considerations (#12198) by @Ramshackle-Jamathon[dagster-airflow] refactor airflow_db resources (#12202) by @Ramshackle-JamathonMake top_level_resources argument _top_level_resources (#12211) by @schrocknAdd guide for observable source assets and versioning (#12118) by @schrockn1.1.18 Changelog (#12217) by @smackeseyFix dagster-wandb placeholder (#12222) by @smackesey1.1.18 by @elementl-devtoolsdagster-airflow library as been moved to 1.x.x to denote the stability of its api's going forward.make_schedules_and_jobs_from_airflow_dag_bag has been added to allow for more fine grained composition of your transformed airflow DAGs into Dagster.retries and retry_delay configuration are now converted to op RetryPolicies with all make_dagster_* apis.0 5 * * mon#1 to execute on a certain day of the week each month executed every week instead.use_airflow_template_context, mock_xcom and use_ephemeral_airflow_db params have been dropped, by default all make_dagster_* apis now use a run-scoped airflow db, similiar to how use_ephemeral_airflow_db worked.make_airflow_dag has been removed.make_airflow_dag_for_operator has been removed.make_airflow_dag_containerized has been removed.airflow_operator_to_op has been removed.make_dagster_repo_from_airflow_dags_path has been removed.make_dagster_repo_from_airflow_dag_bag has been removed.make_dagster_repo_from_airflow_example_dags has been removed.${dag_id}__${task_id} from airflow_${task_id}_${unique_int}.${dag_id} from airflow_${dag_id}.@multi_asset_sensor now accepts a request_assets parameter, which allows it to directly request that assets be materialized, instead of requesting a run of a job.Definitions when using large numbers of assets or many asset jobs.build_schedule_from_partitioned_job no longer needs to have a partitions_def directly assigned to it. Instead, Dagster will infer from the partitions from the assets it targets.OpExecutionContext.asset_partition_keys_for_output no longer requires an argument to specify the default output.pyproject.toml file that were made while Dagit was running. Previously, Dagit needed to be restarted in order for such changes to be shown.get_run_record_by_id has been added to DagsterInstance to provide easier access to RunRecord objects which expose the start_time and end_time of the run.docker_container_op op and execute_docker_container_op helper function for running ops that launch arbitrary Docker containers. See the docs for more information.python:3.x-slim base image.build_asset_reconciliation_sensor could time out when evaluating ticks over large selections of assets, or assets with many partitions. A series of performance improvements should make this much less likely.run_request_for_partition in a sensor that targeted multiple jobs created via define_asset_job.dagster has been reduced.PartitionsDefinition.k8s_job_op raised an Exception when running pods with multiple containers.asset_keys and asset_selection parameters of the experimental @multi_asset_sensor decorator have been replaced with a monitored_assets parameter. This helps disambiguate them from the new request_assets parameter.StaticPartitionMapping is now serializable, thanks @AlexanderVR!build_fivetran_assets now supports group_name , thanks @toddy86!AzureBlobComputeManager now supports authentication via DefaultAzureCredential, thanks @mpicard!load_assets_from_airflow_dag that creates graph-backed, partitioned, assets based on the provided Airflow DAG.https://github.com/dagster-io/dagster/compare/1.1.14...1.1.15
Make StaticPartitionMapping serializable. Add autodoc. (#11738) by @AlexanderVRPerformance improvements for large multi assets (#11782) by @OwenKephartmonitored_assets param for multi-asset sensor (#11567) by @sryza[fix] Fix Snowflake IO manager tests for project_fully_featured (#11781) by @benpankowUpdate docs to include wheel workarounds for m1 macs (#11777) by @gibsondanfix run_request_for_partition for sensors that target multiple unreso… (#11780) by @sryzaNew "running dagster locally" deployment guide that walks through dagster dev usage (#11741) by @gibsondanRestore grpcio pin for python 3.10 (#11784) by @gibsondanChangelog 1.1.11 (#11789) by @dpeng817docs: add clarification for helm migration guide (#10454) by @adam-bloom[dagit] Bump react-scripts to remove codegen plugin (#11769) by @hellendagfeat(dagster-dbt): use cached run id to fetch artifacts (#11744) by @rexledesmafeat(dbt-cloud): raise exception if cached compile has not completed (#11792) by @rexledesmachangelog 1.1.11 pt 2 (#11800) by @alangenfeldHeartbeat once at the beginning of every interval daemon (#11802) by @gibsondan1.1.11 changelog: fix link (#11804) by @yuhan[dagit] Add missing changelog entries for 1.1.11 (#11803) by @bengotowadd request_assets param to multi_asset_sensor (#11786) by @sryzaAutomation: versioned docs for 1.1.11 by @elementl-devtoolsLog to daemon output when running the schedule code fails or times out (#11805) by @gibsondan1.1.12 changelog (#11807) by @alangenfeld[dagit] When viewing an asset that is a root, do not fetch root used data (#11806) by @bengotow[dagit] Fix support for long descriptions in the asset catalog table (#11810) by @bengotowConvert accesses using '__ASSET_JOB' to instead use the supported implicit job methods (#11348) by @schrockn[dagster-azure] Add support for DefaultAzureCredential for AzureBlobComputeLogManager (#11310) by @mpicardadd dagster.yaml tests for nux key (#11820) by @gibsondanFix 'dagster dev' command with workspace.yaml passed in (#11819) by @gibsondanWhen can't deserialize asset backfill because asset is no longer partitioned, return it as empty (#11812) by @sryza1.1.13 changelog (#11818) by @alangenfeldadd after_cursor to get_materialization_counts_by_partition (#11759) by @OwenKephartAdds group_label to build_fivetran_assets (#11718) by @toddy86default argument to asset_partition_keys_for_output (#11811) by @sryzaSpecify credentials masking. (#11813) by @joel-olazagastiFix multidimensional partition backfills (#11788) by @clairelin135Automation: versioned docs for 1.1.13 by @yuhanMove pyproject.toml => origin generation to a WorkspaceLoadTarget class so that it can be reloaded in dagit (#11821) by @gibsondanfix tz in time_windows_for_partition_keys (#11825) by @sryzadynamic re-execution fixes (#11581) by @alangenfeld[instance] get_run_record_by_id (#11643) by @alangenfeld[asset-graph] keep track of source asset partitions defs (#11840) by @OwenKephart[dagster-airbyte] Fix issue using freshness policy w/ Airbyte + multiprocessing executor (#11837) by @benpankow[asset-reconciliation] pre-fetch the results of some queries (#11770) by @OwenKephart[asset-reconciliation] Fix case where the partitions definition does not have partitions for some subset of the past 24 hours (#11842) by @OwenKephartFix for backoff logic incorrectly storing state (#11848) by @gibsondanuse an AssetGraph to resolve asset selections in asset jobs (#11846) by @sryzafix(dbt-cloud): remove dbt selector when materializing subset (#11843) by @rexledesmapin sqlalchemy below 2.0.0 (#11871) by @benpankow[dagit] Allow materializing “All” without requiring large graphs are rendered (#11854) by @bengotowchore(dagster-dbt): pin dbt-core<1.4.0 (#11870) by @rexledesmastop using python semver parsing for mysql versions (#11868) by @prhaFix aws ssm tests (#11886) by @gibsondanunbreak backcompat tests after sqlalchemy upgrade (#11885) by @gibsondan[dagit] Fix jumpy code location status spinner (#11801) by @hellendagAdd apache-airflow test pin to <2.5.1 (#11888) by @gibsondanFix databricks tests running in release branch (#11887) by @gibsondanAdd pin for jupyter-client<8 (#11901) by @gibsondan[pyright] [core] storage (#11363) by @smackesey[pyright] [core] standardize is_context_provided (#11364) by @smackesey[pyright] [core] eliminate funcsigs (#11365) by @smackesey1.1.14 changelog (#11907) by @gibsondan[pyright] [core] remove builitins star import (#11366) by @smackesey[readme] update twitter badge (#11892) by @alangenfeld[pyright] [gql] DagsterPipelineRunMetadataValue -> DagsterRunMetadataValue (#11893) by @smackesey[pyright] [core] _core/definitions/asset_reconciliation_sensor (#11717) by @smackesey[pyright] [gql] implementation/events (#11894) by @smackesey[pyright] [gql] implementation/fetch_partition_sets (#11895) by @smackeseyAutomation: versioned docs for 1.1.14 by @elementl-devtools[pyright] examples/tutorial-notebook-assets (#11910) by @smackesey[pyright] examples/with-airflow (#11911) by @smackesey[pyright] [dagit] misc (#11727) by @smackesey[pyright] [dagster-dask] misc (#11944) by @smackesey[pyright] [dagster-databricks] misc (#11943) by @smackesey[pyright] [dagster-test] misc (#11940) by @smackesey[pyright] [dagster-mysql] misc (#11939) by @smackesey[pyright] [dagster-duckdb-pyspark] misc (#11937) by @smackesey[pyright] [dagster-pandas] misc (#11931) by @smackesey[pyright] [dagster-postgres] misc (#11929) by @smackesey[pyright] [dagster-snowflake] misc (#11926) by @smackesey[pyright] [examples/assets-dbt-python] misc (#11922) by @smackesey[pyright] [dagster-snowflake] misc (#11928) by @smackesey[pyright] [dagster-pandera] misc (#11930) by @smackesey[pyright] [dagster-aws] misc (#11730) by @smackesey[pyright] [dagster-airflow] misc (#11729) by @smackesey[pyright] examples/quickstart-gcp (#11909) by @smackesey[pyright] [dagster-dbt] misc (#11941) by @smackesey[pyright] [automation] misc (#11726) by @smackesey[pyright] [dagster-msteams] misc (#11932) by @smackesey[pyright] [dagster-snowflake-pandas] misc (#11927) by @smackesey[pyright] [dagstermill] misc (#11925) by @smackesey[pyright] [dagster-docker] misc (#11938) by @smackesey[pyright] [dagster-gcp] misc (#11936) by @smackeseyremove need to provide partitions_defto asset job targeted bybuild_schedule_from_partitioned_job (#11844) by @sryzafix GQL snapshot tests on py310 (#11945) by @smackesey[pyright] [dagster-airbyte] misc (#11728) by @smackesey[pyright] [dagster-mlflow] misc (#11948) by @smackeseyFix k8s_job_op with multiple containers (#11916) by @gibsondan[pyright] [gql] capture_error sig fix (#11722) by @smackesey[pyright] [dagster-azure] misc (#11946) by @smackeseyAdd code versions to AssetGraph (#11950) by @smackesey[pyright] [gql] schema/solids#resolve_required_resources (#11898) by @smackeseyrefactor GrpcServerProcess (#11960) by @smackeseyFix s3_resource docstring example (#11757) by @dpeng817[pyright] [core] create RunGroupInfo type (#11368) by @smackeseysnowflake connector pin (#11970) by @alangenfeld[pyright] [gql] schema/roots/mutation (#11897) by @smackesey[easy] List dagster_world.mp4 in static files (#11961) by @salazarm[pyright] [gql] AssetKey cleanup (#11721) by @smackeseyCachingProjectedLogicalVersionResolver -> CachingStaleStatusResolver (#11951) by @smackeseywhitelist EventLogRecord for serdes (#11978) by @OwenKephartdocument tags_for_partition_fn in _partitioned_config API doc (#11977) by @sryza[pyright] [gql] misc (#11720) by @smackesey[pyright] [gql] misc type errors 2 (#11947) by @smackesey[pyright] [gql] implementation/fetch_runs (#11896) by @smackeseyAllow bare executor in Definitions (#11795) by @schrocknAdd opt_iterable_param (#11796) by @schrocknUse opt_iterable_param in Definitions (#11797) by @schrockn[pyright] [gql] compute log manager (#11725) by @smackesey[dagster-snowflake-pandas] snowflake-sqlalchemy pin (#11984) by @smackesey[pyright] [gql] add ResolveInfo everywhere (#11723) by @smackeseyadd .ruff_cache to gitignore (#11987) by @Ramshackle-Jamathonsmall tweak to airflow integration page (#11985) by @sryzaRename _core.decorator_utils.is_context_provided (#11982) by @smackeseyAdd docker_container_op and execute_docker_container (#11831) by @gibsondan[perf] perf improvement for TimeWindowPartitionsSubset (#11850) by @OwenKephart[dagit] Add ErrorBoundary to Dagit to reduce severity of React errors (#11824) by @bengotow[dagit] Repair markdownToPlaintext test failure (#11995) by @hellendag[import perf] defer CachingInstanceQueryer imports to avoid storage imports (#11905) by @alangenfeldUpdate error messaging for DB IO managers (#11815) by @jamiedemaria[import perf] moves to prevent grpc import (#11906) by @alangenfeld[import perf] test to prevent regressions (#11969) by @alangenfeldremove legacy APIs from dagstermill tests (#11999) by @sryzaremove legacy APIs from dagster_shell_tests (#11998) by @sryza[dagit] UI support for launching a single asset run with a range of partition keys (#11866) by @bengotow[dagit] Add more truncation test cases for Firefox, change const (#12001) by @bengotowFix black pre-commit hook (#11979) by @smackesey[3/3 partition status cache] Update graphQL partition data (#10822) by @clairelin135Upgrade base image on Dagster Dockerfiles to latest python version (#11863) by @gibsondan[pyright] [examples/docs-snippets] misc (#11921) by @smackeseyAdd last_materialization_record to AssetEntry. (#11919) by @OwenKephart[asset-reconciliation][perf] Improve prefetch accuracy (#11991) by @OwenKephartfix dagstermill tests (#12002) by @sryzaType annotations for workspace code (#11958) by @smackeseyConsolidate GrpcServerRegistry/ProcessGrpcServerRegistry (#11990) by @smackeseyRefactor ProcessRegistryEntry (#11959) by @smackesey[pyright] examples/project-fully-featured (#11908) by @smackesey[pyright] [dagster-k8s] misc (#11935) by @smackesey[pyright] [core] tests (#11369) by @smackesey[pyright] Type-ignores for various errors related to managed elements (#12014) by @smackeseycleam up get_asset_events (#11913) by @alangenfeldPySpark type handler for snowflake io manager (#11542) by @jamiedemaria[asset-reconciliation] [perf] do not try to fetch materialization records for sources (#12011) by @OwenKepharttest(dagster-dbt): add testing for python 3.10 (#11912) by @rexledesmafix: use new GitHub graphql resolvers for issue automation (#11877) by @rexledesma[dagit] Export GhostDaggy with tooltip for Cloud usage (#11967) by @hellendag[dagster-airflow] load_assets_from_airflow_dag (#11876) by @Ramshackle-JamathonFix broken doc link for Snowflake credential setup (#12017) by @clayheaton[dagster-fivetran] Add option to force-create materializations for tables not in API response (#11972) by @benpankow[dagit] Fix use of fragments causing Apollo caching error in partition health (#12030) by @bengotowChange endpoints to the ones that are used by airbyte UI (#12012) by @emilija-omnisendlint fix (#12044) by @benpankowfeat(dbt-cloud): compile run only if job has environment variable cache (#12042) by @rexledesmafix(dbt-cloud): inherit generate docs settings for compile run (#12043) by @rexledesma1.1.15 changelog (#12051) by @jamiedemaria1.1.15 by @elementl-devtools* as the asset selection before materializing the assets.sqlalchemy package to <2.0.0 due to a breaking change in that version.dbt-core package to <1.4.0 due to breaking changes in that release that affected the Dagster dbt integration. We plan to remove this pin in the next release.jupyter-client package to <8.0 due to an issue with the most recent release causing hangs while running dagstermill ops.EcsRunLauncher sometimes waited much longer than intended before retrying after a failure launching a run.nux section of dagster.yaml config has been fixed.dagster dev command raised an errordagster dev command that can be used to run both Dagit and the Dagster daemon in the same process during local development. See the new Running Dagster Locally guide in the docs for more information.dagster package on M1 and M2 Macs that avoids installation errors when building the grpcio package. See the Installing Dagster guide in the docs for more information.create_repository_using_definitions_args has been added for users to backport their repository definitions to the new Definitions APInux:
enabled: false
load_assets_from_dbt* is now configurable, allowing you to pass in specific parameters to the underlying dbt command (such as --full-refresh). To do so, supply config of the form {"ops": {"run_dbt_<abcde>": {"config": {"full_refresh": True}}}} when defining your asset job, or in Dagit.build_asset_reconciliation_sensor will now only materialize partitions from the past day. Previously, if a new asset was added with a large number of missing partitions, a run for each of the missing partitions would be launched immediately.build_asset_reconciliation_sensor, which should help significantly speed up sensor evaluation in cases where there is a large number of assets or partitions.https://github.com/dagster-io/dagster/compare/1.1.10...1.1.11
telemetry: avoid race condition of dagit and daemon creating same dir (#11652) by @yuhan[dagit] Link to the failed run, not the last materialzation, on “Failed" nodes (#11593) by @bengotow[dagit] Stop foreign node links from creating sprawling asset graphs (#11588) by @bengotowrefactor: remove workspace.yaml in favor of pyproject.toml (#11665) by @rexledesmadocs: update license copyright (#11666) by @rexledesmaKill stray tqdms when starting up dagster (#11658) by @gibsondan[Community Nux] Don't use blueprint dialog (#11660) by @salazarmfix code location status when a new location is added in a loading state (#11654) by @prha[dagster-airflow] pass airflow dag timezone to dagster schedule definition (#11663) by @Ramshackle-Jamathonadd example and compose_fn argument doc to job decorator (#11655) by @sryzaRevert "test fixes for python 3.7 (#11636)" and "Fall back to a different port when 3000 is in use instead of failing (#11610)" (#11671) by @gibsondanRemove community nux for now (#11672) by @salazarmchore(dbt): remove unnecessary dependencies (#11668) by @rexledesmachore(phabricator): remove arcanist config (#11667) by @rexledesmafix a link on deployment/overview (#11674) by @yuhanUnbreak windows tests (and likely the windows multiprocess executor in general) (#11679) by @gibsondanAdd black hook to precommit (#11678) by @smackeseychangelog for 1.1.10 (#11682) by @sryza[dagit] fix clientTime arg (#11670) by @alangenfeldFix black precommit (#11683) by @smackeseyAutomation: versioned docs for 1.1.10 by @elementl-devtools[tick-alerting] Add properties to InstigationTick (#11140) by @dpeng817[tick-alerting] Add docs for tick alerting (#11417) by @dpeng817fix broken link to Airflow library page (#11685) by @sryzaRemove run launcher list from run monitoring docs (#11644) by @gibsondanMake create_repository_using_definitions_args public (#11684) by @gibsondanBring back dagit socket reuse (#11688) by @gibsondanStart event log consumer from latest event (#11681) by @johannkm[dagit] Restore some previous GraphQL codegen behavior (#11675) by @hellendagone more windows test fix for signals (#11680) by @gibsondanShut down the daemon faster when it is interrupted (#11657) by @gibsondanExample of usage of Branching I/O Manager (#11549) by @schrockninclude docs snippet for how to record metadata for non-output assets (#11686) by @sryzaFix stale asset schedule (#11708) by @smackesey[docs][dagit-screenshot] Add CLI command to generate asset SVGs from snippets (#9764) by @benpankowfeat(dbt-cloud): allow jobs to be retrieved from project id (#11673) by @rexledesmafix issue with k8s_job_op when combining command and container_config (#11713) by @gibsondaneliminate SensorExecutionContext, ScheduleExecutionContext (#11038) by @smackeseyMove string construction out of non-error path in check (#11701) by @schrockn[pyright] [core] misc (#11362) by @smackesey[pyright] [core] _utils/error (#11695) by @smackesey[pyright] [core] _utils/backcompat (#11696) by @smackesey[pyright] [core] _cli/api (#11697) by @smackesey[dagit] Rename a few partition data structures in Dagit (#11574) by @bengotowfeat(dbt-cloud): disable schedule only if it exists (#11690) by @rexledesma[dagit] Permissions cleanup (#11712) by @hellendag[dagit] Move User Settings into a Dialog (#11706) by @hellendag[dagit] Fix Definitions filter dialog (#11711) by @hellendag[dagit] Clean up a bunch of document titles (#11715) by @hellendag"dagster dev" command that spins up dagit and the daemon as subprocesses and requires no DAGSTER_HOME env var to be set (#11584) by @gibsondan[dagit] Empty state on Runs with better CTAs (#11714) by @hellendag[OSS Nux] only load an iframe for recaptcha after use agrees to submit email (#11743) by @salazarmBan invalid escape chars (#11443) by @clairelin135[@dagster-io/ui] Bump eslint-config (#11703) by @hellendag[dagit] Display source data used in specific asset materializations (#11736) by @bengotow[dagit] Add the Asset compute kind tag in more places #11185 (#11569) by @bengotow[dagit] Add backfill warnings and run tags options to the Materialize modal (#11656) by @bengotow[1/3 partition status cache] Time window subset perf improvements (#11640) by @clairelin135[dagster-dbt] add config schema for dbt asset op (#11710) by @OwenKephart[asset-reconciliation] Limit the partitions that can be reconciled with the sensor (#11582) by @OwenKephart[dagit] Strip down query for Scheduled Runs (#11751) by @hellendag[asset-reconciliation] Improve performance for never materialized root partitions (#11755) by @OwenKephart[dagit] Fix Backfills page locking up when displaying large backfills, improve "target" column (#11693) by @bengotow[airbyte] Consider keys containing credentials a secret (#11620) by @joel-olazagasti[asset-reconciliation] Perf improvements for reconciliation sensor (#11633) by @OwenKephart[OSS Nux] Add kill switch + persist Nux dismissal in temp folder (#11758) by @salazarm[pyright] [core] _core/definitions/scoped_resources_builder (#11698) by @smackeseyAdd back dagster_world.mp4 video (#11763) by @salazarmopen grpc servers in their own process group so they don't get interrupted along with whatever process opened them (#11737) by @gibsondanDon't spew "Aborted!" to the command-line when the Daemon is cleanly interrupted (#11739) by @gibsondanSwitch the grpcio dependency to use an internally built m1 wheel for m1 macs only (#11760) by @shalabhcInclude retry count in the job created by k8s_job_op (#11753) by @gibsondantest: unpin responses (#11768) by @rexledesmaSelectively show the multiple files/modules help text to the commands where it applies (#11689) by @gibsondanfeat(dbt-cloud): cache compile run id as env var for job (#11691) by @rexledesma[2/3 partition status cache] Serialize version, rebuild cache for unsupported version (#11687) by @clairelin135Fix airflow_op_to_dagster_op test (#11771) by @clairelin135DBT code version docs (#11692) by @smackeseyimprove performance of StaticPartitionsDefinition.__eq__ (#11775) by @sryza[dagster-snowflake] remove password check for private key auth for unencrypted keys (#11745) by @jamiedemariaRename "base asset job" to "implicit asset job" in RepositoryDefinition methods (#11344) by @schrocknmark PartitionMappings not experimental in IO manager doc (#11774) by @sryzadagit => dagster dev in docs (#11740) by @gibsondanRevert "Switch the grpcio dependency to use an internally built m1 wheel for m1 macs only (#11760)" (#11776) by @gibsondanPerformance improvements for large multi assets (#11782) by @OwenKephartUpdate docs to include wheel workarounds for m1 macs (#11777) by @gibsondanNew "running dagster locally" deployment guide that walks through dagster dev usage (#11741) by @gibsondanChangelog 1.1.11 (#11789) by @dpeng817Restore grpcio pin for python 3.10 (#11784) by @gibsondanfeat(dagster-dbt): use cached run id to fetch artifacts (#11744) by @rexledesmafeat(dbt-cloud): raise exception if cached compile has not completed (#11792) by @rexledesmachangelog 1.1.11 pt 2 (#11800) by @alangenfeld[dagit] Add missing changelog entries for 1.1.11 (#11803) by @bengotow1.1.11 by @elementl-devtoolsselection argument of define_asset_job now accepts lists of AssetKeys or AssetsDefinitions.RunRequest now takes a stale_assets_only flag that filters the full set of assets that would be materialized by a job to stale assets only. This can be used in schedules and sensors.dagit command and the default port 3000 is already in use.grpcio pin in Dagster to <1.48.1 has been removed for Python versions 3.10 and 3.11. Python 3.7, 3.8, and 3.9 are still pinned to <1.48.1 due to a bug in the grpc library that is causing the process to sometimes hang.stdout or stderr tab. The next time you view compute logs, you will see that tab first by default.executor and loggers arguments on Definitions are no longer experimental.json_log_format is set to False when using the dbt_cli_resource, logs will be emitted at the appropriate log level in some situations. Previously, all logs would be emitted at the INFO level.dagster-azure / dagster-aws / dagster-gcp)build_asset_reconciliation_sensor with asset graphs that contained references to source assets without the associated SourceAsset objects (which may happen when using load_assets_from_dbt_*).DbtManifestAssetSelection which could result in KeyErrors when selecting from projects with sources defined./instance and /workspace path prefixes were removed in previous version, but redirects were left in place. These redirects have now been removed.StaticPartitionMapping enables explicitly defining the dependencies between partitions in two StaticPartitionsDefinitions. Thanks Alexander VR!https://github.com/dagster-io/dagster/compare/1.1.9...1.1.10
docs for new tag_concurrency_limits feature on executor (#11499) by @gibsondan[dagster-airflow] warn on airflow dataset (#11498) by @Ramshackle-JamathonAdd typehints to snapshot arguments to create_run (#11454) by @schrocknTypehint and do invariants for root_run_id, parent_run_id, step_keys_to_execute and execution_plan_snapshot (#11455) by @schrocknTypehint and validate tags and status params to create_run (#11456) by @schrocknRequire kwargs on create_reexecuted_run (#11460) by @schrocknMake frozentags inherit from Mapping[str, str] and correctly type _check_execute_external_pipeline_args (#11461) by @schrocknAdd support for loading cacheable assets from module (#10389) by @benpankowenable filtering by asset tag when asset tags table is not present (#11509) by @sryzaonly count materializations within backfill (#11506) by @sryzatelemetry: add num_assets_in_repo in log_repo_stats metadata (#11513) by @yuhantelemetry: add location_name in repo-level metadata (#11514) by @yuhan[dagit] Clean up /instance and /workspace redirects (#11503) by @hellendagFix getting partition mappings when asset dependencies are resolved (#11483) by @sryzadocstring examples for AssetSelection.keys (#11477) by @sryza[dagster-airflow] add make_dagster_definitions_from_airflow_dags_pathandmake_dagster_definitions_from_airflow_dag_bag apis (#11441) by @Ramshackle-JamathonFix maxResumeRunAttempts being explicitly enabled even if you set it to 0 (#11511) by @gibsondandagster-graphql tests for permissions failures (#11486) by @gibsondanHook for Cloud dagit into PythonErrorInfo (#11467) by @salazarmChangelog 1.1.8 (#11526) by @clairelin135Add experimental examples directory (#11505) by @schrockn[dagster-airbyte] Allow specifying freshness policies on Airbyte assets (#11525) by @benpankowMigrate airflow guide to definitions (#11518) by @petehuntAutomation: versioned docs for 1.1.8 by @elementl-devtoolsRemove packaging v22 cap (#11212) by @smackeseyremove pragmas in docs snippets (#11335) by @smackeseyupdate black 22.3.0 -> 22.12.0 and enable --preview (#10900) by @smackeseyAdopt ruff as linter (#10901) by @smackeseyAdd .git to excludes for ruff in pyproject.toml (#11534) by @schrocknchore(ruff): add pre-commit config (#11538) by @rexledesmachore(dagit): remove unnecessary dependencies (#11532) by @rexledesmadocs: add links to follow LinkedIn page (#11533) by @rexledesma[dagit] Use cache-and-network as default watchQuery fetch policy (#11523) by @hellendag[docs] - Add GSC site verification file (#11547) by @erinkcochran87disable executing airflow_example_http_operator in tests (#11544) by @Ramshackle-JamathonAdd and apply ruff docstring rules (#10907) by @smackeseyRuff docstrings D209 (NewLineAfterLastParagraph) (#10993) by @smackeseyUse Config class instead of base class kwargs in structured config to not break early pydantic versions (#11546) by @schrockn[fix] Fix builtin type error related to resource annotations (#11551) by @benpankow[changelog] 1.1.9 (#11556) by @benpankowadd NullMetadataValue (#10969) by @smackeseyRuff docstrings D210 (NoSurroundingWhitespace) (#10994) by @smackesey[changelog] 1.1.9 round 2 (#11560) by @benpankowMake JsonMetadataValue accept lists (#11550) by @smackeseyRuff docstrings D212,D214 (MultilineSummaryFirstLine,SectionNotOverIndented) (#10995) by @smackeseyAutomation: versioned docs for 1.1.9 by @elementl-devtoolsRuff docstring D300 (UsesTripleQuotes) (#10996) by @smackeseyAdd a graphql field and API for location-scoped permissions (#11097) by @gibsondan[@dagster-io/eslint-config] Remove GraphQL type parameter rule (#11479) by @hellendagRuff docstrings D301 (Raw string for backslash in docstring)) (#11329) by @smackesey[dagit] Update eslint-config dep in dagster-io/ui (#11563) by @hellendag[dagit] Stop generating docblocks on schema.graphql and codegen (#11564) by @hellendag[dagit] Track most recent preference for computed log type view (#11527) by @hellendagFix graphql permisisons checks, fix black (#11566) by @gibsondan[docs] - Move ops + jobs tutorial to Guides (#11528) by @erinkcochran87[dynamic] fix how skips cascade (#11561) by @alangenfeld[dagit] Add GraphQL configuration for VSCode (#11565) by @hellendag[dagster-dbt] add DbtManifestAssetSelection to API docs (#11562) by @OwenKephartdefine_asset_job accept AssetKey and AssetsDefinition selections (#11568) by @sryzaRuff docstrings D403,D405 (FirstLineCapitalized,CapitalizeSectionName) (#10997) by @smackeseyfix remote captured log manager complete check (#11552) by @prhamisc docs fixes in tutorial / getting started / assets (#11575) by @gibsondandocs: fix typo Telementry -> Telemetry (#11571) by @chriszsfix docstring for static_partitioned_config (#11535) by @severoAdd tip when the multiprocess executor hits a likely OOM error (#11570) by @gibsondanRuff docstrings D414 (NonEmptySection) (#10999) by @smackeseyRuff docstrings D419 (NonEmpty) (#11000) by @smackesey[@dagster-io/ui] Output cjs (#11594) by @hellendagShow clearer grpc errors, as well as a custom failure for sensor timeouts in particular since those are so common (#11576) by @gibsondanAdds OSS email collection dialog (#11480) by @salazarmRuff docstrings D202 (NoBlankLineAfterFunction) (#10992) by @smackeseyRuff docstrings D411,D412 (BlankLineBeforeSection,NoBlankLinesBetweenHeaderAndContent) (#10998) by @smackesey Change exclude to extend-exclude in pyproject.toml settings for ruff (#11591) by @schrocknrefactor(dbt-cloud): remove v2 suffix (#11586) by @rexledesmafeat(dbt-cloud): allow env vars to be retrieved and set (#11592) by @rexledesmaRuff docstrings (misc fixes) (#11002) by @smackesey[docs] - Remove accidental comments (#11601) by @erinkcochran87docs: fix dead link to Slack community (#11602) by @EmilRexFix graphene enum not being instantiated (#11605) by @gibsondan[dagit] Add location-scoped permissions to Dagit (#11587) by @hellendag[dagster-dbt] parse log level when json_log_level is not set (#11596) by @OwenKephartuse Python type annotations in loading-from-config docs examples (#11609) by @sryza[dagit] Use updated react-scripts fork for codegen babel plugin (#11583) by @hellendagFall back to a different port when 3000 is in use instead of failing (#11610) by @gibsondan[dagit] Some Permissions tweaks to support Cloud (#11612) by @hellendagRuff docstrings D415 (EndsInPunctuation) (#11001) by @smackesey[dagster-dbt] allow for non-model nodes to be part of the DbtManifestAssetSelection (#11617) by @OwenKephartUpdate OSS Nux to receive size from iframe via postmessage (#11625) by @salazarm[freshness-policies] fix get_non_source_roots (#11619) by @OwenKephartremove reexecute_pipeline_iterator (#11608) by @alangenfeld[eslint-config] Enable ignoreExternal on no-cycle rule (#11632) by @hellendag[@dagster-io/ui] Bump eslint-config (#11634) by @hellendagadd example to TimeWindowPartitionMapping apidoc (#11615) by @sryzaFix typo in Dagster instance documentation (#11621) by @GrigoriiKushniruse run tags instead of asset tags to filter materializations within backfill (#11555) by @sryzatest fixes for python 3.7 (#11636) by @gibsondanRestrict the grpcio pin to only py3.7/3.8/3.9 (#11635) by @gibsondan Remove experimental flag on executor and loggers parameters of Definitions and add more detail to docblock (#11626) by @schrockn[dagster-snowflake-pandas] update doc string to use Definitions (#11606) by @jamiedemaria[dagster-snowflake] Add private key authentication for Snowflake (#10750) by @joshuataylorLoad OSS Nux iframe via https (#11638) by @salazarmtelemetry: add step start/success/failure event + client timestamp (#11624) by @yuhanAdd StaticPartitionMapping as a built-in PartitionMapping (#11521) by @AlexanderVR[dagster-airbyte] Enable setting a destination table prefix using managed ingestion (#11057) by @benpankowtelemetry: add is_known_ci_env (#11642) by @yuhanstale asset materialization for schedules and sensors (#11516) by @smackeseytelemetry: avoid race condition of dagit and daemon creating same dir (#11652) by @yuhan[Community Nux] Don't use blueprint dialog (#11660) by @salazarmfix code location status when a new location is added in a loading state (#11654) by @prha[dagster-airflow] pass airflow dag timezone to dagster schedule definition (#11663) by @Ramshackle-JamathonRevert "test fixes for python 3.7 (#11636)" and "Fall back to a different port when 3000 is in use instead of failing (#11610)" (#11671) by @gibsondanRemove community nux for now (#11672) by @salazarmUnbreak windows tests (and likely the windows multiprocess executor in general) (#11679) by @gibsondan1.1.10 by @elementl-devtools