Astro Projects Astro Versions Save

Misc

• Replace openlineage-airflow with Apache Airflow OSS provider apache-airflow-providers-openlineage #2103
• Bump up minimum version of apache-airflow to 2.7 #2103
• Bump up minimum version of Python to 3.8 #2103

Bug Fixes

• Limit pandas version to <2.2.0 due to an open issue #57053 for the release pandas==2.2.0 #2105

Feature

• Allow users to disable schema check and creation on load_file #1922
• Allow users to disable schema check and creation on transform #1925
• Add support for Excel files #1978
• Support loading metadata columns from stage into table for Snowflake #2023

Bug Fixes

• Add openlineage_dataset_uri in databricks #1919
• Fix QueryModifier issue on Snowflake #1962
• Fix AstroCustomXcomBackend circular import issue #1943

Misc

• Add an example DAG for using dynamic task with dataframe #1912
• Improve example_load_file DAG tasks names #1958
• Limit databricks-sql-connector<2.9.0 #2013

Docs

• Add docs about using dtype #1903
• Make cleanup operator summary docs smaller #2017

Bug Fixes

• Fix Snowflake QueryModifier issue #1962
• Add support for Pandas 2, Airflow 2.6.3 and Python 3.11 #1989
• Update the WASB connection #1994

Bug Fixes

• Fix AstroCustomXcomBackend circular import issue. https://github.com/astronomer/astro-sdk/pull/1943

Bug Fixes

• Fix AstroCustomXcomBackend circular import issue. #1943

Feature

• Allow users to disable schema check and creation on transform #1925
• Allow users to disable schema check and creation on load_file #1922

Documentation: https://astro-sdk-python.readthedocs.io/en/latest/configurations.html#configuring-if-schemas-existence-should-be-checked-and-if-the-sdk-should-create-them

Feature

• Allow users to disable schema check and creation on load_file #1922

Feature

• Add MySQL support #1801
• Add support to load from Azure blob storage into Databricks #1561
• Add argument skip_on_failure to CleanupOperator #1837 by @scottleechua
• Add query_modifier to raw_sql, transform and transform_file, which allow users to define SQL statements to be run before the main query statement #1898. Example of how to use this feature can be used to add Snowflake query tags to a SQL statement:

  from astro.query_modifier import QueryModifier
  @aql.run_raw_sql(
      results_format="pandas_dataframe",
      conn_id="sqlite_default",
      query_modifier=QueryModifier(pre_queries=["ALTER team_1", "ALTER team_2"]),
  )
  def dummy_method():
      return "SELECT 1+1"
  

• Upgrade astro-runtime to 7.4.2 #1878

Bug fix

• Raise exception in case larger dataframes than expected are passed to aql.dataframe #1839
• Revert breaking change introduced in 1.5.0, re-allowing aql.transform to receive `sql filepath #1879

Docs

• Update open lineage documentation #1881

Misc

• Support Apache Airflow 2.6 #1899, with internal serialization changes
• Add basic tiltifle for local dev #1819

Feature

• Add query_modifier to raw_sql, which users can use to define SQL statements to be run before and after the main query statement #1898

Example:

    from astro.query_modifier import QueryModifier
    @aql.run_raw_sql(
        results_format="pandas_dataframe",
        conn_id="sqlite_default",
        query_modifier=QueryModifier(pre_queries=["ALTER team_1", "ALTER team_2"]),
    )
    def dummy_method():
        return "SELECT 1+1"

Bug fix:

• Support using SQL operators (run_raw_sql, transform, dataframe) to convert a Pandas dataframe into a table when using a DuckDB in-memory database. #1831

Thanks to @pgzmnk for reporting the issue!