An orchestration platform for the development, production, and observation of data assets.
build_asset_reconciliation_sensor
s , AutoMaterializePolicy
s work across code locations, as well as allow you to customize the conditions under which each asset is auto-materialized. [docs]FreshnessPolicy
s, Dagster’s UI now marks them as “overdue” instead of “late”.AthenaClientResource
, ECRPublicResource
, RedshiftClientResource
, S3Resource
, S3FileManagerResource
, ConfigurablePickledObjectS3IOManager
, SecretsManagerResource
follow Pythonic resource system. The existing APIs remain supported.DatadogResource
follows Pythonic resource system. The existing datadog_resource
remains supported.GEContextResource
follows Pythonic resource system. The existing ge_context_resource
remains supported.GithubResource
follows Pythonic resource system. The existing github_resource
remains supported.MSTeamsResource
follows Pythonic resource system. The existing msteams_resource
remains supported.SlackResource
follows Pythonic resource system. The existing slack_resource
remains supported.pdb.set_trace
no longer worked when running Dagster locally using dagster dev
or dagit
.@asset
or Out
caused an error to be thrown.pipelines
or jobs
could lead to blank pages.Definitions
are now automatically bound to jobs. This will only result in a change in behavior if you a) have a job with no "io_manager" defined in its resource_defs
and b) have supplied an IOManager
with key "io_manager" to the resource_defs
argument of your Definitions
. Prior to 1.3.0, this would result in the job using the default filesystem-based IOManager
for the key "io_manager". In 1.3.0, this will result in the "io_manager" supplied to your Definitions
being used instead. The BindResourcesToJobs
wrapper, introduced in 1.2 to simulate this behavior, no longer has any effect.CeleryK8sRunLauncher
is now the same namespace as the Helm chart, instead of the default
namespace. To restore the previous behavior, you can set the celeryK8sRunLauncher.jobNamespace
field to the string default
.SnowflakePandasIOManager
has historically converted all timestamp data to strings before storing it in Snowflake. Now, it will instead ensure that timestamp data has a timezone, and if not, attach the UTC timezone. This allows the timestamp data to be stored as timestamps in Snowflake. If you have been storing timestamp data using the SnowflakePandasIOManager
you can set the store_timestamps_as_strings=True
configuration to continue storing timestamps as strings. For more information, and instructions for migrating Snowflake tables to use timestamp types, see the Migration Guide.Changes to experimental APIs
RunConfig
to many APIs which previously would only accept a config dictionary.ResourceDefinition
.execution
config when constructing a RunConfig
object.minutes_late
and previous_minutes_late
properties on the experimental FreshnesPolicySensorContext
have been renamed to minutes_overdue
and previous_minutes_overdue
, respectively.Removal of deprecated APIs
metadata_entries
arguments to event constructors have been removed. While MetadataEntry
still exists and will only be removed in 2.0, it is no longer passable to any Dagster public API — users should always pass a dictionary of metadata values instead.latest_materialization_records_by_key
function.BigQueryPandasIOManager
and BigQueryPySparkIOManager
.SnowflakePandasIOManager
and SnowflakePySparkIOManager
.required_resource_keys
and Pythonic resources) are now supported in observable source assets.EcsRunLauncher
now allows you to customize volumes and mount points for the launched ECS task. See the API docs for more information.DuckDBPandasIOManager
and DuckDBPySparkIOManager
follow Pythonic resource system. The existing duckdb_pandas_io_manager
and duckdb_pyspark_io_manager
remain supported.BigQueryPandasIOManager
and BigQueryPySparkIOManager
follow Pythonic resource system. The existing bigquery_pandas_io_manager
and bigquery_pyspark_io_manager
remain supported.gcp_crentials
, a temporary file to store the credentials will be created and the GOOGLE_APPLICATION_CREDENTIALS
environment variable will be set to the temporary file. When the BigQuery resource is garbage collected, the environment variable will be unset and the temporary file deleted.SnowflakePandasIOManager
and SnowflakePySparkIOManager
follow Pythonic resource system. The existing snowflake_pandas_io_manager
and snowflake_pyspark_io_manager
remain supported.dagster dev
would periodically emit a harmless but annoying warning every few minutes about a gRPC server being shut down.RunRequest(partition_key=...)
object.PartitionsDefinition
.run_request_for_partition
would incorrectly raise an error for a job with a DynamicPartitionsDefinition
that was defined with a function.define_asset_job
would raise an error.__ASSET_JOB has no op named ...
errors when using automatic run retries.EcsRunLauncher
would fail to launch runs if the use_current_ecs_task_config
field was set to False
but no task_definition
field was set.EcsRunLauncher
now allows you to set a capacity provider strategy and customize the ephemeral storage used for launched ECS tasks. See the docs for details. Thanks AranVinkItility!AssetsDefinition.from_op
. Thanks @tghanken for the fix!minimum_interval_seconds
parameter to enable customizing the evaluation interval on the slack run failure sensor, thanks @ldnicolasmay!Resource
annotation for Pythonic resource inputs has been renamed to ResourceParam
in preparation for the release of the feature in 1.3.materialize
and materialize_to_memory
now both accept a selection
argument that allows specifying a subset of assets to materialize.MultiPartitionsDefinition
is no longer marked experimental.MultiPartitionsDefinition
s with a time dimension.load_assets_from_package_module
and similar methods now accept a freshness_policy
, which will be applied to all loaded assets.build_sensor_context
and build_multi_asset_sensor_context
can now take a Definitions
object in place of a RepositoryDefinition
s3_resource
now accepts use_ssl
and verify
configurations.selection
argument on define_asset_job
.MultiPartitionsDefinition
.MultiPartitionsMapping
(experimental) is loaded.TimeWindowPartitionMapping
s to not be bubbled up to the UI.pdb
stopped working when using dagster dev.AssetMaterialization
objects with a null AssetKey
.TimeWindowPartitionsDefinition
with a non-standard cron schedule, and also provided a minute_of_hour
or similar argument in build_schedule_from_partitioned_job
. Dagster would silently create the wrong cron expression. It now raises an error.PartitionsDefinition
. These partitions are now ignored.PersistentAirflowDatabase
where versions of airflow from 2.0.0 till 2.3.0 would not use the correct connection environment variable name.poll_sync_run
function ofdagster-census
that prevented polling from working correctly (thanks @ldincolasmay!)run_request_for_partition
method on JobDefinition
and UnresolvedAssetJobDefinition
is now deprecated and will be removed in 2.0.0. Instead, directly instantiate a run request with a partition key via RunRequest(partition_key=...)
.AssetsDefinition.to_source_assets
to method convert a set of assets to SourceAsset
objects.MultiPartitionsDefinitions
.mlflow
pin from the dagster-mlflow
package.When using build_asset_reconciliation_sensor
, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.
When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.
Fixed an issue where context.asset_partitions_time_window_for_output
threw an error when an asset was directly invoked with build_op_context
.
[dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.
[celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent
.
Fixed an issue where launching a large backfill while running dagster dev
would sometimes fail with a connection error after running for a few minutes.
Fixed an issue where dagster dev
would sometimes hang when running Dagster code that attempted to read in input via stdin.
Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.
Fixed an issue where AssetSelection.groups()
would simultaneously select both source and regular assets and consequently raise an error.
Fixed an issue where BindResourcesToJobs
would raise errors encapsulating jobs which had config specified at definition-time.
Fixed Pythonic config objects erroring when omitting optional values rather than specifying None
.
Fixed Pythonic config and resources not supporting Enum values.
DagsterInstance.local_temp
and DagsterInstance.ephemeral
now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.
Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:
class MyResource(ConfigurableResource):
pass
@op
def my_op(x: int, y: int, my_resource: MyResource) -> int:
return x + y
my_op(4, 5, my_resource=MyResource())
[dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential
(thanks @mpicard)
[ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.
environment_vars
argument to ScheduleDefinition
is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)[CHANGES.md](http://CHANGES.md)
(thanks @fridiculous)--path-prefix
can now be supplied via Helm chart (thanks @mpicard)https://github.com/dagster-io/dagster/compare/1.2.3...1.2.4
[refactor] delete Materialization (#13030)
by @smackeseyAdd in progress subsets to the partition cache (#13045)
by @johannkmAdd multipartitioned assets with dynamic dimension to toys (#13061)
by @clairelin135add docs example for multi-asset conditional materialization (#13054)
by @sryzaAdd docs for source asset observation jobs/schedules (#13062)
by @smackeseyRevert "Add in progress subsets to the partition cache (#13045)"
by @johannkm[asset-reconciliation][bug] Fix issue where overly-aggressive runs would be kicked off. (#13069)
by @OwenKepharttweaks to cross-repo-assets toy (#12973)
by @sryzaShow unauthorized error graphql error message (#13064)
by @salazarm[dagster-io/ui] Make Suggest component a bit more flexible (#13056)
by @hellendagFix disabled state for launchpad button submenu (#13078)
by @salazarmRe-enable in progress subsets in the partition cache (#13082)
by @johannkmAdd dynamic partitions name resolver to dimension type (#13070)
by @clairelin135asset sensor test docs (#13065)
by @prhaFix issue where backfill fails when gRPC server is replaced mid-backfill (#13085)
by @gibsondanUse instance from sensor/schedule context to instantiate resources, delay until accessed (#13041)
by @benpankowAdd materializing subset to asset gql (#13046)
by @johannkmfeat(helm): add path-prefix to dagit command (#13080)
by @mpicardUse dynamic partition definition name for dimension of multipartition definition (#13090)
by @salazarm[typing/static] Fix @repository decorator typing (#12295)
by @smackesey[instance] make local artifact directory scheme thread safe (#13043)
by @alangenfeld1.2.3 changelog (#13094)
by @jamiedemariaEnsure pyright venvs use statically legible editable installs (#13089)
by @smackesey[docs] - Remove finished code from dbt tutorial template (#13091)
by @erinkcochran87[ui] Upgrade react-markdown (#13092)
by @hellendagFix submitting backfills synchronously from graphql (#13093)
by @gibsondanTest get and set serialized_in_progress_partition_subset (#13063)
by @johannkmExitStack.pop_all -> close (#13050)
by @alangenfeldAutomation: versioned docs for 1.2.3
by @elementl-devtoolsFraser/rework readme (#12565)
by @frasermarlow[ui] Use DefaultLogLevels when there is no level state stored (#13109)
by @hellendagSet stdin to DEVNULL when opening dagster subprocesses (#13099)
by @gibsondanadd a vercel github action to build docs/storybook previews (#13052)
by @prha[refactor] Remove
frozen{list,dict,tags} classes (#12293)
by @smackeseyDocs for setting up Gitlab CI, branch deployment guide (#12998)
by @prhaAdd assets def to op context (#13088)
by @clairelin135make AssetsDefinition.to_source_assets public (#13073)
by @sryza[docs] New ML pipeline guide PR (#13100)
by @odette-elementl[freshness-refactor][3/n] Update methods on the CachingDataTimeResolver to work with scalar data time (#12906)
by @OwenKephartFixing refs to images in the README (#13126)
by @tacastillorestrict vercel builds based on paths (#13129)
by @prhafix missing snapshots (#13134)
by @OwenKephartTelemetry for dynamic partitions (#12605)
by @clairelin135fix missing snapshots (again) (#13136)
by @OwenKephart[freshness-refactor][4/n] Simplify scheduling algorithm (#13019)
by @OwenKephartDeprecate environment_vars argument to
ScheduleDefinition,
@schedule (#13044)
by @smackesey[refactor] delete hourly/daily/weekly/monthly schedule decorators, PartitionScheduleDefinition, build_schedule_from_partition (#13006)
by @smackesey[refactor] simplify dependency dict typing (#12521)
by @smackeseycelery-k8s executor: handle stdout that's valid json but not a dagster event (#13143)
by @johannkmfix typo in CHANGES.md (#13140)
by @fridiculousFix branch deployment docs (#13131)
by @dpeng817Update multiple agents docs (#13135)
by @dpeng817add toy for eager asset reconciliation (#13066)
by @sryzatrying again by moving the images to the same directory as the readme (#13127)
by @tacastillo[tech][templates] moving the scaffold project's asset loader outside of the defs (#13103)
by @tacastillo[docs][tutorial-revamp] Adding a section for scheduling to the tutorial (#13101)
by @tacastilloreplacing corrupt .png image (#13157)
by @frasermarlow[dagster-dbt] Fix bug when calculating transitive dependencies (#13128)
by @OwenKephartdeploy storybook to prod when landing pushes on master (#13159)
by @prha[dagster-azure] fix: AttributeError: 'coroutine' object has no attribute 'token' (#13110)
by @mpicardAdd support for more asset tags (#13153)
by @braunjj[caching-refactor] Remove use of get_and_update_asset_status_cache. (#13151)
by @OwenKephart[asset-reconciliation][perf] Cache common properties on the TimeWindowPartitionsDefinition (#12981)
by @OwenKephartMultiPartitionMapping (#12950)
by @clairelin135[asset-reconciliation] Fix issue with duplicate runs for partitions with in-progress materializations (#13130)
by @OwenKephartupdate telemetry documentation links (#13176)
by @Abbe98[structured config] respect field aliases for resources (#13177)
by @jamiedemaria[refactor] Remove PartitionSetDefinition (#13145)
by @smackeseyTableMetadataValue toy (#13124)
by @smackeseyfix repository decorator typing (#13180)
by @smackeseyadd storybook for @core in addition to @ui (#13184)
by @prhafeat(dbt): follow dbt Core's version support constraints (#13189)
by @rexledesmaFix case where a pipeline load takes long enough that run monitoring kills it (#13156)
by @gibsondanAllow newlines in error messages in Dagit (#13193)
by @gibsondanMake sure we always wait for all grpc server processes to spin down in tests (#13146)
by @gibsondanunpin mlflow from dagster-mlflow (#13194)
by @gibsondanGreatly reduce the number of gRPC calls when doing large asset backfills (#13086)
by @gibsondandon't run mlflow tests on python 3.11 (#13197)
by @gibsondan[pythonic config] Treat Optional as both not-required and Noneable (#12975)
by @benpankow[pythonic config] Add support for enums (#12979)
by @benpankow[pythonic resources] Make direct invocation of ops/assets w/ resources easier (#13002)
by @benpankow[bugfix] make AssetSelection.groups() resolve to only regular assets (#13196)
by @smackesey[fix] Fix behavior of BindResourcesToJobs when jobs have config specified (#13057)
by @benpankow[serdes] remove extraneous @cached_method use (#13206)
by @alangenfeld1.2.4 changelog (#13226)
by @smackesey1.2.4
by @elementl-devtools[refactor] delete Materialization (#13030)
by @smackeseyAdd in progress subsets to the partition cache (#13045)
by @johannkmAdd multipartitioned assets with dynamic dimension to toys (#13061)
by @clairelin135add docs example for multi-asset conditional materialization (#13054)
by @sryzaAdd docs for source asset observation jobs/schedules (#13062)
by @smackeseyRevert "Add in progress subsets to the partition cache (#13045)"
by @johannkm[asset-reconciliation][bug] Fix issue where overly-aggressive runs would be kicked off. (#13069)
by @OwenKepharttweaks to cross-repo-assets toy (#12973)
by @sryzaShow unauthorized error graphql error message (#13064)
by @salazarm[dagster-io/ui] Make Suggest component a bit more flexible (#13056)
by @hellendagFix disabled state for launchpad button submenu (#13078)
by @salazarmRe-enable in progress subsets in the partition cache (#13082)
by @johannkmAdd dynamic partitions name resolver to dimension type (#13070)
by @clairelin135asset sensor test docs (#13065)
by @prhaFix issue where backfill fails when gRPC server is replaced mid-backfill (#13085)
by @gibsondanUse instance from sensor/schedule context to instantiate resources, delay until accessed (#13041)
by @benpankowAdd materializing subset to asset gql (#13046)
by @johannkmfeat(helm): add path-prefix to dagit command (#13080)
by @mpicardUse dynamic partition definition name for dimension of multipartition definition (#13090)
by @salazarm[typing/static] Fix @repository decorator typing (#12295)
by @smackesey[instance] make local artifact directory scheme thread safe (#13043)
by @alangenfeld1.2.3 changelog (#13094)
by @jamiedemariaEnsure pyright venvs use statically legible editable installs (#13089)
by @smackesey[docs] - Remove finished code from dbt tutorial template (#13091)
by @erinkcochran87[ui] Upgrade react-markdown (#13092)
by @hellendagFix submitting backfills synchronously from graphql (#13093)
by @gibsondanTest get and set serialized_in_progress_partition_subset (#13063)
by @johannkmExitStack.pop_all -> close (#13050)
by @alangenfeldAutomation: versioned docs for 1.2.3
by @elementl-devtoolsFraser/rework readme (#12565)
by @frasermarlow[ui] Use DefaultLogLevels when there is no level state stored (#13109)
by @hellendagSet stdin to DEVNULL when opening dagster subprocesses (#13099)
by @gibsondanadd a vercel github action to build docs/storybook previews (#13052)
by @prha[refactor] Remove
frozen{list,dict,tags} classes (#12293)
by @smackeseyDocs for setting up Gitlab CI, branch deployment guide (#12998)
by @prhaAdd assets def to op context (#13088)
by @clairelin135make AssetsDefinition.to_source_assets public (#13073)
by @sryza[docs] New ML pipeline guide PR (#13100)
by @odette-elementl[freshness-refactor][3/n] Update methods on the CachingDataTimeResolver to work with scalar data time (#12906)
by @OwenKephartFixing refs to images in the README (#13126)
by @tacastillorestrict vercel builds based on paths (#13129)
by @prhafix missing snapshots (#13134)
by @OwenKephartTelemetry for dynamic partitions (#12605)
by @clairelin135fix missing snapshots (again) (#13136)
by @OwenKephart[freshness-refactor][4/n] Simplify scheduling algorithm (#13019)
by @OwenKephartDeprecate environment_vars argument to
ScheduleDefinition,
@schedule (#13044)
by @smackesey[refactor] delete hourly/daily/weekly/monthly schedule decorators, PartitionScheduleDefinition, build_schedule_from_partition (#13006)
by @smackesey[refactor] simplify dependency dict typing (#12521)
by @smackeseycelery-k8s executor: handle stdout that's valid json but not a dagster event (#13143)
by @johannkmfix typo in CHANGES.md (#13140)
by @fridiculousFix branch deployment docs (#13131)
by @dpeng817Update multiple agents docs (#13135)
by @dpeng817add toy for eager asset reconciliation (#13066)
by @sryzatrying again by moving the images to the same directory as the readme (#13127)
by @tacastillo[tech][templates] moving the scaffold project's asset loader outside of the defs (#13103)
by @tacastillo[docs][tutorial-revamp] Adding a section for scheduling to the tutorial (#13101)
by @tacastilloreplacing corrupt .png image (#13157)
by @frasermarlow[dagster-dbt] Fix bug when calculating transitive dependencies (#13128)
by @OwenKephartdeploy storybook to prod when landing pushes on master (#13159)
by @prha[dagster-azure] fix: AttributeError: 'coroutine' object has no attribute 'token' (#13110)
by @mpicardAdd support for more asset tags (#13153)
by @braunjj[caching-refactor] Remove use of get_and_update_asset_status_cache. (#13151)
by @OwenKephart[asset-reconciliation][perf] Cache common properties on the TimeWindowPartitionsDefinition (#12981)
by @OwenKephartMultiPartitionMapping (#12950)
by @clairelin135[asset-reconciliation] Fix issue with duplicate runs for partitions with in-progress materializations (#13130)
by @OwenKephartupdate telemetry documentation links (#13176)
by @Abbe98[structured config] respect field aliases for resources (#13177)
by @jamiedemaria[refactor] Remove PartitionSetDefinition (#13145)
by @smackeseyTableMetadataValue toy (#13124)
by @smackeseyfix repository decorator typing (#13180)
by @smackeseyadd storybook for @core in addition to @ui (#13184)
by @prhafeat(dbt): follow dbt Core's version support constraints (#13189)
by @rexledesmaFix case where a pipeline load takes long enough that run monitoring kills it (#13156)
by @gibsondanAllow newlines in error messages in Dagit (#13193)
by @gibsondanMake sure we always wait for all grpc server processes to spin down in tests (#13146)
by @gibsondanunpin mlflow from dagster-mlflow (#13194)
by @gibsondanGreatly reduce the number of gRPC calls when doing large asset backfills (#13086)
by @gibsondandon't run mlflow tests on python 3.11 (#13197)
by @gibsondan[pythonic config] Treat Optional as both not-required and Noneable (#12975)
by @benpankow[pythonic config] Add support for enums (#12979)
by @benpankow[pythonic resources] Make direct invocation of ops/assets w/ resources easier (#13002)
by @benpankow[bugfix] make AssetSelection.groups() resolve to only regular assets (#13196)
by @smackesey[fix] Fix behavior of BindResourcesToJobs when jobs have config specified (#13057)
by @benpankow[serdes] remove extraneous @cached_method use (#13206)
by @alangenfeld1.2.4 changelog (#13226)
by @smackesey1.2.4
by @elementl-devtoolsdefine_asset_job
now auto-infer their partitions definitions if not explicitly defined.define_asset_job
. This allows putting them on a schedule/sensor.instance
property to the HookContext
object that is passed into Op Hook functions, which can be used to access the current DagsterInstance
object for the hook.create_table_schema_metadata_from_dataframe
function to generate a TableSchemaMetadataValue
from a Pandas DataFrame. Thanks @AndyBys!dag_run
configuration on the integration’s database resources.dagster asset wipe-partitions-status-cache
to see the partitions displayed.1.1.18
, users with a gRPC server that could not access the Dagster instance on user code deployments would see an error when launching backfills as the instance could not instantiate. This has been fixed.build_asset_reconciliation_sensor
from kicking off runs of downstream assets. This has been fixed.build_asset_reconciliation_sensor
is now much more performant in cases where unpartitioned assets are upstream or downstream of static-partitioned assets with a large number of partitions.jobNamespace
field in the Dagster Helm chart when using the CeleryK8sRunLauncher
.en-US
browser locale. This has been fixed.Dagster is now tested on Python 3.11.
Users can now opt in to have resources provided to Definitions
bind to their jobs. Opt in by wrapping your job definitions in BindResourcesToJobs
. This will become the default behavior in the future.
@op(required_resource_keys={"foo")
def my_op(context)
print(context.foo)
@job
def my_job():
my_op()
defs = Definitions(
jobs=BindResourcesToJobs([my_job])
resources={"foo": foo_resource}
Added dagster asset list
and dagster asset materialize
commands to Dagster’s command line interface, for listing and materializing software-defined assets.
build_schedule_from_partitioned_job
now accepts jobs partitioned with a MultiPartitionsDefinition
that have a time-partitioned dimension.
Added SpecificPartitionsPartitionMapping
, which allows an asset, or all partitions of an asset, to depend on a specific subset of the partitions in an upstream asset.
load_asset_value
now supports SourceAsset
s.
[ui] Ctrl+K has been added as a keyboard shortcut to open global search.
[ui] In the run logs table, the timestamp column has been moved to the far left, which will hopefully allow for better visual alignment with op names and tags.
[dagster-dbt] A new node_info_to_definition_metadata_fn
to load_assets_from_dbt_project
and load_assets_from_dbt_manifest
allows custom metadata to be attached to the asset definitions generated from these methods.
[dagster-celery-k8s] The Kubernetes namespace that runs using the CeleryK8sRunLauncher
are launched in can now be configured by setting the jobNamespace
field in the Dagster Helm chart under celeryK8sRunLauncherConfig
.
[dagster-gcp] The BigQuery I/O manager now accepts timeout
configuration. Currently, this configuration will only be applied when working with Pandas DataFrames, and will set the number of seconds to wait for a request before using a retry.
[dagster-gcp] [dagster-snowflake] [dagster-duckdb] The BigQuery, Snowflake, and DuckDB I/O managers now support self-dependent assets. When a partitioned asset depends on a prior partition of itself, the I/O managers will now load that partition as a DataFrame. For the first partition in the dependency sequence, an empty DataFrame will be returned.
[dagster-k8s] k8s_job_op
now supports running Kubernetes jobs with more than one pod (Thanks @Taadas).
AssetValueLoader.load_asset_value()
didn’t load transitive resource dependencies correctly.RunConfig
object with optional config arguments would lead to an error.ScheduleEvaluationContext.scheduled_execution_time
to not be Optional
.OpExecutionContext.partition_time_window
****(thanks @elben10).InputContext.upstream_output.log
is no longer None
when loading a source asset.async def
ops are now invoked with asyncio.run
.TimeWindowPartitionDefinition
now throws an error at definition time when passed an invalid cron schedule instead of at runtime.cron_schedule_timezone
inside of the config for a dbt model would not result in that property being set on the generated FreshnessPolicy
. This has been fixed.GCSComputeLogManager
when the session does not have permissions to generate signed urls.validate_run_config
no longer accepts pipeline_def
or mode
arguments. These arguments refer to legacy concepts that were removed in Dagster 1.0, and since then there have been no valid values for them.Added experimental support for resource requirements in sensors and schedules. Resources can be specified using required_resource_keys
and accessed through the context or specified as parameters:
@sensor(job=my_job, required_resource_keys={"my_resource"})
def my_sensor(context):
files_to_process = context.my_resource.get_files()
...
@sensor(job=my_job)
def my_sensor(context, my_resource: MyResource):
files_to_process = my_resource.get_files()
...
https://github.com/dagster-io/dagster/compare/1.2.1...1.2.2
[dagster-gcp-pandas] add timeout config (#12637)
by @jamiedemaria[docs][tutorial-revamp] Basic tutorial revamp parts 1 through 4 (#12509)
by @tacastillofeat(dbt): add support for
--debug (#12722)
by @rexledesmaDefault useful Dagster helm chart features to on (#12737)
by @gibsondan[For 1.2] Allow both protobuf 3 and 4 in dagster (#12466)
by @gibsondan[toys repo] Export partitioned assets toys (#12733)
by @salazarm[pythonic config] Allow using 'resource_defs' with resource args in assets (#12679)
by @benpankow[refactor] Remove symbols deprecated until 1.2 (#12360)
by @smackesey[dagster-snowflake] fix inconsistencies in snowflake resource (#12633)
by @jamiedemaria[ui] Update permissions for launching jobs and materializing assets (#12681)
by @hellendag[For 1.2] Change default run monitoring settings (#11512)
by @gibsondan[For 1.2] Don't include run/job tags in k8s_job_ops k8s config computations (#12345)
by @gibsondanRemove Partitioned Schedules from docs, fix dagit staleStatusCauses mocks (#12742)
by @smackesey[Sensor Testing] Add "Test again" button (#12735)
by @salazarm[1.2.0] [refactor] Delete DagsterTypeMaterializer (#12516)
by @smackeseychange DynamicPartitionsDefinition.__repr__ (#12754)
by @sryzaRecommended Project Structure Guide (#12656)
by @odette-elementl[pythonic config] Add test showcasing use of Pydantic validators (#12536)
by @benpankowfix serialization of TableRecord (#10731)
by @sryza[dagster-gcp-pandas] revert flaky test (#12748)
by @jamiedemaria[typing/static] PartitionsDefinition covariant type var (#12284)
by @smackesey[rename] LogicalVersion -> DataVersion (#12500)
by @smackesey[rename] LogicalVersion -> DataVersion in gql/dagit (#12501)
by @smackesey[rename] Rename LogicalVersion -> DataVersion in docs (#12503)
by @smackesey[dagster-airflow] add
make_persistent_airflow_db_resource (#12305)
by @Ramshackle-Jamathon[ui] Remove usePermissionsDEPRECATED (#12771)
by @hellendag[Asset Details Page] Auto-select partition in materialization dialog. (#12734)
by @salazarmfeat(databricks)!: remove
create_databricks_job_op (#12600)
by @rexledesmaMake permissions_for_location require keyword args (#12774)
by @gibsondanRemove DynamicPartitionsDefinitions API methods (#12744)
by @clairelin135Set default memory/CPU for new ECS tasks based on runtime platform (#12767)
by @gibsondancron_schedule validation for time window partitions (#12761)
by @prhaupdate some guide titles and descriptions (#12775)
by @sryzaadd primary keys to all run tables (#12711)
by @prhadocument PartitionMappings in partitions concepts page (#12768)
by @sryzamake sure kv/daemon_heartbeat queries explicitly enumerate columns (#12789)
by @prhaupdate API docs for freshness policies (#12788)
by @OwenKephart[apidoc] asset_selection repository -> Definitions (#11303)
by @yuhanFix typing on mem_io_manager (#12791)
by @dpeng817[docs] add note about biquery temp tables (#12747)
by @jamiedemariaRemove deprecated MetadataEntry constructors, update
entry_data internal refs (#12724)
by @smackesey[dagster-airflow] persistent db docs (#12485)
by @Ramshackle-Jamathon[pythonic resources][fix] Fix inheriting attributes when extending Pythonic resources (#12781)
by @benpankowMetadataEntryUnion (#12725)
by @smackesey[docs] - Update RBAC docs for Cloud (#12752)
by @erinkcochran87[dagit] Update the asset partitions / events view after run failures (#12798)
by @bengotow[dagit] Asset node rendering tweaks (#12764, #12765) (#12794)
by @bengotowReflect dynamically-added partitions in asset sidebar immediately (#12778)
by @salazarm[Create Partition Dialog] Autofocus input (#12777)
by @salazarmRemove PartitionMetadataEntry (#12726)
by @smackeseyMaintain a separate cursor in the asset partition cache pointing back to the earliest in progress run (#12782)
by @johannkm[ui] Run logs: make Timestamp the first column (#12805)
by @hellendag[snowflake-pyspark] refactor spark session creation in tests (#12738)
by @jamiedemaria[db io managers] refactor tests to test table creation (#12739)
by @jamiedemaria[dagit] Fix yaml editor help context when indented on a new line (#12537) (#12804)
by @bengotowmake context.upstream_output.log work for source assets (#12787)
by @sryza[typing/runtime] serdes (#12523)
by @smackesey[dagit] Show “Observe” on source asset details pages, not “Materialize" (#12803)
by @bengotow[ui] Disable "Open in launchpad" based on permissions (#12817)
by @hellendagAdd a way to cause partition health data to refetch without passing around refetch callbacks (#12822)
by @salazarm[ui] Fix space oddities on Schedules overview table (#12810)
by @hellendag[ESLint] Update missing graphql variable type rule to check mutations and subscriptions too (#12779)
by @salazarmTest get and set new asset cache fields (#12820)
by @johannkm[duckdb-polars] fix readme (#12499)
by @jamiedemaria[freshness-refactor][1/n] Separate part of CachingInstanceQueryer into CachingDataTimeResolver (#12809)
by @OwenKephart[ui] Affordance for copying asset key (#12829)
by @hellendag[fix][dagit] Allow resources page to scroll (#12827)
by @benpankow[async op] use asyncio.run (#12785)
by @alangenfeld[dagster-airflow] remove double encoding of timezones (#12811)
by @Ramshackle-Jamathon[ui] Support Ctrl+K to trigger search (#12840)
by @hellendagVirtualize TagSelector component (#12841)
by @salazarm[Launchpad] Allow adding new dynamic partition from launchpad UI (#12757)
by @salazarm[tox] download latest pip (#12821)
by @alangenfeld[dagster-snowflake] fix permissions issue when creating schemas (#12802)
by @jamiedemaria[pythonic config] Special rendering for env vars in UI (#12446)
by @benpankownothing input toy (#12849)
by @sryzalatestRunForPartition gql (#12845)
by @johannkm[dagster-dbt] add support for node_info_to_definition_metadata_fn (#12831)
by @OwenKephartRemove nothing types from asset graph (#12848)
by @salazarm[TagSelector] Allow filtering tags in dropdown (#12844)
by @salazarmAdd a tooltip on asset nodes in the graph (#12853)
by @salazarm[pythonic resources] Keep track of utilized env vars in external repo data (#12557)
by @benpankowChangelog 1.2.0 (#12851)
by @dpeng817add a local download url fallback for GCS (#12815)
by @prha[freshness-refactor][2/n] Reorganize/Rename/Remove/Document CachingInstanceQueryer methods (#12813)
by @OwenKephartfix type annotation for partition_time_window (#12850)
by @sryzaRevert "fix type annotation for partition_time_window" (#12860)
by @sryzaAutomation: versioned docs for 1.2.0
by @elementl-devtoolsAdd volatility property to CachingStaleStatusResolver (#12855)
by @smackeseyUpdate CHANGES.md (#12864)
by @Ramshackle-Jamathon[docs] - Add doc checklist to PR template (#12861)
by @erinkcochran87Fix partition_time_window type error (#12838)
by @elben10Add 1.2.0 changes to migration.md (#12866)
by @clairelin135Fix config w/ dynamic partitions error (#12710)
by @clairelin135[1.2] make ScheduleEvaluationContext.scheduled_execution_time non-optional (#12623)
by @sryzaasset materialize CLI (#12691)
by @sryza[refactor] serdes pack/deserialize APIs (#12524)
by @smackesey[refactor] MayHaveInstanceWeakRef error on access non-existent _instance (#12553)
by @smackeseyasset list command (#12790)
by @sryzafixups to config_schedule.py (#12874)
by @sryzaFix some upcoming lint errors in docstrings (#12704)
by @charliermarsh[db io managers] support self dependent assets in db io managers (#12700)
by @jamiedemariaFix markdown links in changelog (#12877)
by @johannkmFix dagster-k8s job execution when a job has more than 1 pod (#12731)
by @Taadastake grpcio-health-checking pins out of backcompat suite (#12876)
by @gibsondanFix ruff (#12879)
by @gibsondanneed to explicitly return a known existing column for updates (#12883)
by @prhaFix docs build (#12884)
by @smackeseyadd 1.2.1 changelog (#12887)
by @prhaadd SpecificPartitionsPartitionMapping (#12878)
by @sryzaAutomation: versioned docs for 1.2.1
by @elementl-devtools[docs] - Apply style guide to Asset versioning and caching guide (#12895)
by @erinkcochran87[dagster-ui] Sync the search filter to the query string on most pages (#12899)
by @bengotowon asset backfills, propagate backfill tags to runs (#12886)
by @sryzaSupport SourceAssets in load_asset_value (#12859)
by @sryzaFix transitive resource deps for asset value loader (#12872)
by @dpeng817[typing/static] telemetry (#12714)
by @smackeseyadd docs page on asset selection syntax (#12871)
by @sryzaAdd dropdown for materializing using launchpad (#12852)
by @salazarm[ui] Refactor permissions objects (#12903)
by @hellendagavoid custom PartitionMapping on IO manager concept page (#12902)
by @sryza[bk] Add changelog check step (#12865)
by @benpankowtutorial intro tweaks (#12780)
by @sryzaMove docs style guidelines from pull request template to contribution guide (#12875)
by @sryzaremove support for legacy APIs from validate_run_config (#12881)
by @sryzamake reconciliation perf test names easier to scan, and turn them all on (#12908)
by @sryzaMake fetch_flattened_time_window_ranges support arbitrary statuses (#12863)
by @johannkmBuild schedule from multipartitioned job (#12907)
by @clairelin135[ui] Clean up temporary permission code (#12923)
by @hellendag[op] error on multiple values for argument (#12473)
by @alangenfeldtest assets in dynamic re-execution (#12651)
by @alangenfeldSupport requiring resources in sensors, experimentally (#12401)
by @benpankowSupport requiring resources in schedules, experimentally (#12697)
by @benpankow[pythonic config] support Pydantic constrained types (#12703)
by @benpankow[docs] Add env var docs to Pythonic config experimental docs (#12740)
by @benpankow[pythonic resources] Surface utilized env vars in GraphQL (#12558)
by @benpankow[dagit] Allow the lineage tab to appear before the asset definition has loaded (#12191) (#12814)
by @bengotowSupport defaulting to selected partitions for Multi-partitioned Assets (#12856)
by @salazarmfix execution plan step output collisions (#12799)
by @alangenfeld[pythonic config] Fix using RunConfig with optional config fields (#12933)
by @benpankow[Partitions] Use OrdinalPartitionSelector for STATIC partitioned assets (#12919)
by @salazarmfix ruff (#12936)
by @alangenfeldUnpin grpcio on dagster for python 3.11 (#12917)
by @gibsondanAllow setting the job_namespace for the celery k8s run launcher (#12911)
by @gibsondan[dagster-dbt] add cron_schedule_timezone to set of properties parsed from dbt config (#12904)
by @OwenKephartRedesign partition backfill modal (#12928)
by @salazarmFix error reporter infinite loop (#12941)
by @salazarmAdd Python 3.11 to dagster buildkite (#12723)
by @gibsondanBind top-level resources to jobs at Definitions-time (#12430)
by @benpankow[asset-reconciliation] Do not reconcile partitions involved in active asset backfills (#12926)
by @OwenKephart[asset-reconciliation] Fix behavior that could cause overly-aggressive updates (#12940)
by @OwenKephartimprove execution plan snapshot use in execute_run (#12660)
by @alangenfeldopen backfill options section by default (#12957)
by @salazarm[Partition Backfill Dialog] Update backfill options spacing (#12959)
by @salazarmResolve transitive resource deps for schedules and sensors (#12946)
by @benpankowchangelog 1.2.2 (#12960)
by @yuhanFix pandera tests (#12947)
by @smackesey[dagit] Use pure-asset backfills instead of hidden asset job backfills (#12948)
by @bengotow1.2.2
by @elementl-devtoolsdagster dev
command that can be used to run both Dagit and the Dagster daemon in the same process during local development. [docs]DynamicPartitionsDefinition
allows partitioning assets dynamically - you can add and remove partitions without reloading your definitions (experimental). [docs]TimeWindowPartitionMapping
now accepts start_offset
and end_offset
arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets [docs].dagster-wandb
with Weights & Biases allows you to orchestrate your MLOps pipelines and maintain ML assets with Dagster. [docs]dagster-snowflake-pyspark
allows you to store and load PySpark DataFrames as Snowflake tables using the snowflake_pyspark_io_manager
. [docs]dagster-gcp-pandas
and dagster-gcp-pyspark
allow you to store and load Pandas and PySpark DataFrames as BigQuery tables using the bigquery_pandas_io_manager
and bigquery_pyspark_io_manager
. [docs]dagster-airflow
integration library was bumped to 1.x.x, with that major bump the library has been refocused on enabling migration from Airflow to Dagster. Refer to the docs for an in-depth migration guide.create_databricks_run_now_op
), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op
).create_databricks_job_op
op factory is now deprecated.Stay tuned, as this is only the first part of the overhaul. We’ll be adding more chapters - including automating materializations, using resources, using I/O managers, and more - in the next few weeks.
@graph_asset
and @graph_multi_asset
.project_fully_featured
example now uses the built in DuckDB and Snowflake I/O managers.make_persistent_airflow_db_resource
this enables support for Airflow features like pools and cross-dagrun state sharing. In particular retry-from-failure now works for jobs generated from Airflow DAGs.BigQueryPandasTypeHandler
now uses google.bigquery.Client
methods load_table_from_dataframe
and query
rather than the pandas_gbq
library to store and fetch DataFrames.args
instead of both command
and args
for user code deployments, allowing to include a custom ENTRYPOINT in your the Dockerfile that loads your code.protobuf<4
pin in Dagster has been removed. Installing either protobuf 3 or protobuf 4 will both work with Dagster.@graph_asset
and @graph_multi_asset
now support passing metadata (thanks @askvinni)!@graph_asset
and @graph_multi_asset
to be ignored.TableRecord
.@multi_asset
and other functions would register as type errors for mypy and other static analyzers.start_date
would not be converted correctly causing runs to fail.dagster instance migrate
:
id
column to the kvs
, daemon_heartbeats
and instance_info
tables, enforcing that all tables have a primary key.The minimum grpcio
version supported by Dagster has been increased to 1.44.0 so that Dagster can support both protobuf
3 and protobuf
4. Similarly, the minimum protobuf
version supported by Dagster has been increased to 3.20.0. We are working closely with the gRPC team on resolving the upstream issues keeping the upper-bound grpcio
pin in place in Dagster, and hope to be able to remove it very soon.
Prior to 0.9.19, asset keys were serialized in a legacy format. This release removes support for querying asset events serialized with this legacy format. Contact #dagster-support for tooling to migrate legacy events to the supported version. Users who began using assets after 0.9.19 will not be affected by this change.
[dagster-snowflake] The execute_query
and execute_queries
methods of the SnowflakeResource
now have consistent behavior based on the values of the fetch_results
and use_pandas_result
parameters. If fetch_results
is True, the standard Snowflake result will be returned. If fetch_results
and use_pandas_result
are True, a pandas DataFrame will be returned. If fetch_results
is False and use_pandas_result
is True, an error will be raised. If both are False, no result will be returned.
[dagster-snowflake] The execute_queries
command now returns a list of DataFrames when use_pandas_result
is True, rather than appending the results of each query to a single DataFrame.
[dagster-shell] The default behavior of the execute
and execute_shell_command
functions is now to include any environment variables in the calling op. To restore the previous behavior, you can pass in env={}
to these functions.
[dagster-k8s] Several Dagster features that were previously disabled by default in the Dagster Helm chart are now enabled by default. These features are:
dagster/max_retries
tag set. You can configure a global number of retries in the Helm chart by setting run_retries.max_retries
to a value greater than the default of 0.Each of these features can be disabled in the Helm chart to restore the previous behavior.
[dagster-k8s] The experimental k8s_job_op
op and execute_k8s_job
functions no longer automatically include configuration from a dagster-k8s/config
tag on the Dagster job in the launched Kubernetes job. To include raw Kubernetes configuration in a k8s_job_op
, you can set the container_config
, pod_template_spec_metadata
, pod_spec_config
, or job_metadata
config fields on the k8s_job_op
(or arguments to the execute_k8s_job
function).
[dagster-databricks] The integration has now been refactored to support the official Databricks API.
create_databricks_job_op
is now deprecated. To submit one-off runs of Databricks tasks, you must now use the create_databricks_submit_run_op
.databricks_client
resource must now begin with https://
.LogicalVersion
has been renamed to DataVersion
and LogicalVersionProvenance
has been renamed to DataProvenance
.DynamicPartitionsDefinition
to add, remove, and check for existence of partitions have been removed. Refer to documentation for updated API methods.MetadataEntry
have been removed.DagsterTypeMaterializer
, DagsterTypeMaterializerContext
, and @dagster_type_materializer
have been removed.PartitionScheduleDefinition
has been removed.RunRecord.pipeline_run
has been removed (use RunRecord.dagster_run
).DependencyDefinition.solid
has been removed (use DependencyDefinition.node
).pipeline_run
argument to build_resources
has been removed (use dagster_run
)iteritems
usage was removed and changed to the recommended items
within dagster-snowflake-pandas
(thanks @sethkimmel3)!@asset_graph
decorator (thanks @simonvanderveldt)!DataVersions
can now be returned on Output
OpExecutionContext.get_asset_provenance
Picking a github issue
was not correctly rendering, this has been fixed (thanks @Sedosa)!build_asset_reconciliation_sensor
.TimeWindowPartitionMapping
s with offsets. (Thanks Sean Han!)runtimePlatform
field for the task definitions of the runs that it launches, allowing it to launch runs using Windows Docker images.create_databricks_run_now_op
), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op
). See the new Databricks guide for more details.DuckDBPolarsTypeHandler
for use with build_duckdb_io_manager
, which allows loading / storing Polars DataFrames from/to DuckDB. (Thanks Pezhman Zarabadi-Poor!)bigquery_pyspark_io_manager
.DataFrame
.AssetSelection
which matched no assets was passed into define_asset_job
, the resulting job would target all assets in the repository. This has been fixed.build_schedule_from_partitioned_job
.fs_io_manager
to store multipartitioned materializations in directory levels by dimension. This resolves a bug on windows where multipartitioned materializations could not be stored with the fs_io_manager
.context.partition_key
would raise an error when executing on a partition range within a single run via Dagit.Definitions reloaded
messages on fresh page loads.location
information.ProgrammingError
was imported from the wrong library, this has been fixed. Thanks @herbert-allium!Output
objects rather than using Dagster’s auto-generated versions.get_asset_provenance
method on OpExecutionContext
allows fetching logical version provenance for an arbitrary asset key.examples/assets_dynamic_partitions
folderhttps://github.com/dagster-io/dagster/compare/1.1.20...1.1.21
dagster-census api docs (#12413)
by @yuhangraph_asset and graph_multi_asset decorators (#10152)
by @sryza[dagster-snowflake-pyspark] fix bug loading partitions (#12472)
by @jamiedemariaadd graphql fields for querying run tags (#12409)
by @prhaAdd stale status causes (#11953)
by @smackeseyfix (#12477)
by @salazarmUpdate Contributing doc with instructions for ruff/pyright (#12481)
by @smackesey[bigquery] mark bigquery io manager experimental (#12479)
by @jamiedemariaadd partial tag autocomplete for run filter input (#12410)
by @prhaSupport env valueFrom in Helm chart (#12425)
by @johannkmUpdate GQL to expose StaleStatus and StaleStatusCause (#11952)
by @smackeseyremove timestamp comparisons of code location entries to reduce OSS dagit replica spam (#12407)
by @prhaFix state status logical version test (#12484)
by @smackeseyfix ruff (#12486)
by @alangenfelduse opt_nullable_mapping for dagster library versions (#12487)
by @alangenfeldclarify error when op is missing argument for In (#12456)
by @sryzaRemove existing RunConfig class (#12488)
by @benpankow[pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445)
by @benpankow[structured config] Add support for Selectors w/ pydantic discriminated unions (#11280)
by @benpankowAllow setting logical version inside op (#12189)
by @smackeseyReplace usages of
nslookupwith
nc for user deployments (#11033)
by @michaeljguarinoRevert "Replace usages of
nslookupwith
nc for user deployments (#11033)"
by @johannkm[pythonic resources] Last set of class renames (#12490)
by @benpankow[dagster-azure] Add support for DefaultAzureCredential for adls2_resource (#11309)
by @mpicardAdd example of customizing task role and execution role arn to the ECS agent docs (#12491)
by @gibsondanadd dagster-duckdb-polars library (#12197)
by @pzarabadip[draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260)
by @benpankow[pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965)
by @benpankow[draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349)
by @benpankow1.1.20 changelog (#12506)
by @benpankowrefactor(databricks): lift polling methods up to the client (#12382)
by @rexledesma[fix] fix sphinx airflow version parsing (#12507)
by @benpankowAutomation: versioned docs for 1.1.20
by @elementl-devtools[dagit] Expose range-based asset health, use it for partition status rendering (#12302)
by @bengotow[dagit] Use range-based asset health for asset partitions / job partitions pages (#12434)
by @bengotowFix 1.1.20 changelog codeblock (#12525)
by @johannkm[dagit] Delete generated GraphQL types before regenerating (#12518)
by @hellendag[refactor] Delete build_solid_context (#12513)
by @smackesey[typing/static] serdes (#12522)
by @smackesey[refactor] make ResolvedRunConfig.to_dict use "ops" (#12514)
by @smackeseyImprove multipartition performance for
get_partition (#12431)
by @clairelin135feat(databricks): override user agent in resource (#12526)
by @rexledesmaBump typing-extensions dep to >=4.4.0 (#12529)
by @smackesey[docs] Snowflake reference page fixes (#12455)
by @jamiedemaria[bugfix] Make assets downstream of partitions never stale in dagit (#12528)
by @smackeseyEnable internal testing for writing asset cached status data (#12497)
by @clairelin135[typing] fix typing in daemon tests (#12475)
by @dpeng817dynamic partitions toy (#12533)
by @sryzapare down multi-partition runtime type checking in upath IO manager (#12508)
by @sryza[dagit] Upgrade to Jest 29, allow more time for coverage collection (#12534)
by @bengotowAdd description to invariant check (#12496)
by @CodeMySky[fix] Correctly resolve asset jobs with empty selections (#12531)
by @OwenKephart[dagit] Support backfills on partition-mapped asset selections (#12458)
by @bengotow[dagit] Remove unnecessary usage of <TestProvider> (#12519)
by @bengotowFix multipartitions w/ fs_io_manager on windows (#12414)
by @clairelin135Fix master (#12550)
by @clairelin135Booleans that check permissions for specific objects and automatically incorporate per-location checks (#12548)
by @gibsondandocs(dagster-dbt): add example to retrieve asset keys from dbt selection (#12323)
by @rexledesmaFix IO manager doc snippets (#12545)
by @clairelin135fix setup.cfg in create package (#12559)
by @jamiedemariaFetch multipartition key from context methods (#12512)
by @clairelin135Include partition in asset materialization planned event (#12333)
by @johannkmhandle multiple non-time-window partitions in db io managers (#12517)
by @sryzaAdd runtime_platform to EcsRunLauncher (#12566)
by @gibsondanTag Selector Component (#12564)
by @salazarmStorybook for LaunchAssetChoosePartitionsDialog (#12567)
by @salazarmUpdate setting-up-alerts.mdx (#12582)
by @johannkmMove useViewport to
uiform
core (#12578)
by @salazarm[ui] Always show "Reload all" button (#12588)
by @hellendagBump minimist from 1.2.5 to 1.2.8 in /js_modules/dagit (#12595)
by @dependabot[bot]Expose partition definition name (#12585)
by @salazarmadd dynamic partitions example (#12494)
by @sryza[dagit] Do not capitalize tag keys in the Dagit op metadata panel (#12591)
by @bengotow[graphql] empty numPartitions and partitionNames for cross-partitioning backfills (#12462)
by @sryza[dagit] Use last path component of linked assets on the asset graph + add tooltips (#12590)
by @bengotow[dagster-snowflake] Fix ProgrammingError import in snowflake_io_manager (#12576)
by @herbert-alliumremove dagster-cloud dep from dynamic partitions example (#12603)
by @sryzafix future ticks for partitioned schedules (#12601)
by @sryzafix loading partitioned source assets in non-partitioned non-asset jobs (#12586)
by @sryzaUpdate stale reasons to remove UNKNOWN, add MISSING (#12584)
by @smackeseyGQL Add dynamic partition mutation (#12562)
by @clairelin135Add asset provenance access to OpExecutionContext (#12467)
by @smackeseyfeat(databricks): add basic op implementations (#12492)
by @rexledesmadocs(databricks): add guide for Databricks integration (#12326)
by @rexledesmamigrate some tests away from AssetGroup (#12606)
by @sryzadivest permissions for editing sensor from updating sensor cursor (#12589)
by @dpeng817[dagit] Fix tests pinned to February (#12616)
by @bengotowMove tests, fixtures, and storybooks into folders (#12614)
by @salazarmAdd root stale causes for assets (#12619)
by @smackeseydocs for graphs that depend on assets (#12597)
by @sryza[UI] Dynamic partition creation and selection (#12615)
by @salazarm[asset-reconciliation][perf] Do not get/set known_used_data in asset reconciliation loop (#12433)
by @OwenKephart[dagster-gcp-pyspark] Add BigQuery PySpark type handler (#12398)
by @jamiedemariaAutomating pipeline guide (#12547)
by @odette-elementlCapture db timeout errors for tag key queries (#12596)
by @prhaGet partitions with planned but not completed materializations, the event log query (#12404)
by @johannkmAdd failures to partition status cache (#12599)
by @johannkm[bigquery-pandas] pipe gcp location through to type handler (#12587)
by @jamiedemaria[docs, bigquery] tutorial and reference guide (#12452)
by @jamiedemariamake freshness policies work with graph_asset and graph_multi_asset (#12630)
by @sryzaRevert "Add failures to partition status cache (#12599)"
by @johannkmRevert "Get partitions with planned but not completed materializations, the event log query (#12404)"
by @johannkmModify
get_first_partition_window to account for offset (#12504)
by @clairelin135[dagit] Switch to staleStatus for stale tags, display causes in tooltips (#12611)
by @bengotowChange format of staleness root causes and remove old resolvers (#12628)
by @smackeseyHelm: Support valueFrom in user deployments (#12644)
by @johannkmUpdate asset versioning guide (#12659)
by @smackeseyupdate CHANGES.md for 1.1.21 (#12662)
by @sryzadocs(databricks): update guide based on feedback (#12669)
by @rexledesma1.1.21
by @elementl-devtools