the portable Python dataframe library
date types are now returned as object dtype containing datetime.date objects when executing with the pandas backend.ibis.show_sql(expr) calls with print(ibis.to_sql(expr)) or if using Jupyter or IPython ibis.to_sql(expr)
nullifzero is removed; use nullif(0) insteadzeroifnull is removed; use fillna(0) insteadlist_databases is removed; use list_schemas insteadcurrent_database method returns the data_project instead of the dataset_id. Use current_schema to retrieve dataset_id. To explicitly list tables in a given project and dataset, you can use f"{con.current_database}.{con.current_schema}"
RegexSplit operation and re_split API (07beaed)RegexSplit (e3c507e)ops.RegexSplit using pyarrow UDF (37b6b7f)RegexSplit (229a1f4)zones geojson example (#8040) (2d562b7), closes #7958
ops.RegexSplit using pyarrow UDF (a3bed10)RegexSplit (c955b6a)RegexSplit (cfe0329)RegexSplit (2c1a726)RegexSplit (9d1295f)order_by position in window function when subsequent expressions are duplicated (#7943) (89056b9), closes #7940
to_numpy() (6fd4550)filelock from required dependencies (76dded5)UNION ALL to work around sqlglot bug (399a5ef)mutateandselect APIs (#8014) (8ed19ea), closes #8013
Table.sample (6a652ec)_name, _dtype (#7911) (e58be2e)ibis.connect using a URL (d6049f8)ibis.connect (33c855a)show_sql in favor of print(to_sql) (36da8c1)list_databases (22e5ada)nullifzero (8447b9a)zeroifnull (8be3c25)pins an optional dependency through an examples extra (#7878) (3d6c3f1), closes #7844
raw_sql over _exec_sql (0b66b94)ArrayValue.flatten method and operation (e6e995c)ibis.range function for generating sequences (f5a0a5a)to_pandas method to BaseBackend (3d1cf66)node.replace() now supports mappings for quick lookup-like substitutions (bbc93c7)node.find_topmost() method to locate matching nodes without descending further to their children (15acf7d)node.__children__ property to access the flattened list of children of a node (2e91476)ParquetWriter and CSVWriter (40558fd)streaming kwarg to to_pandas (703507f).substitute (291a290)array_sort (9624341)table.info() nullable column (3b3bd7b)None literals with dt.Null type to the requested type during value coercion (d51ec4e)Concrete.copy() receives unexpected arguments (442199a).sql metadata results are in column-definition order (26a3c1f)InValues list (b26aa55)ValueError and hiding legitimate failures (b7f650c)str when checking for a prefix (c5f884c)None keys (4b28ff1)ir.Table docstring (e3b9611)ibis.param API (2f9ec90)Value.name API (dd66af2)relabels with renames (6bc9e15)raw_sql cursor only when using SELECT statements (74379a8)BigQueryType to use sqlglot for type parsing and generation (6e3219f)traverse() function's filter argument since it can be expressed using the visitor (e4e2993)node.find() and node.match() methods to transparently support types and patterns (3c14091)execute and to_pyarrow implementations (c572eab)TableProxy object to formats from the operations (05964b1)node.__recreate__() method (f3da926)bucket method for timestamps (ca0f7bc)Table.sample method for sampling rows from a table (3ce2617)order_by (359fd5e)signature in udf definitions (764977e)location parameter (d652dbb)read_csv, read_json, read_parquet support (ff83110)bucket (10a5916)Table.fillna (5633660)Self annotations for Annotable (0c60146)Table.sample (09a7626)attach and detach methods for adding and removing databases to the current duckdb session (162b058)ntile support (bf08a2a)Table.sample as a TABLESAMPLE query (3a80f3a)ops.TryCast operation (752e587)to_pyarrow (6b33454)Table.sample (8e78dfc)bucket (1ffac11)ntile (9a14ba3)Table.sample (77215be)bucket (c59518c)bucket (4d34afc)Table.sample (6aa897e)ntile (39eed1a)Table.sample as a random() filter across several SQL backends (e1870ea)Table.sample as a TABLESAMPLE query (f3d044c)ntile (2978d1a)to_pyarrow instead of to_pandas in the interactive repr (72aa573)GroupConcat operation (ed3b017)ops.ApproxCountDistinct (2e3a5a0)ibis.parse_sql() removes where clause (522f3a4)Value[dt.Floating] annotated values as dt.float64 (b8a924a)__index_level_N__ cols before applying schema (b53feac)Object pattern should match on positional arguments first (96c796f)PatternList should keep the original pattern's type (6552639).sql calls works (a5e3062)pyarrow_hotfix where necessary (0fa1e5d)Table.sample (51027d9)BasePandasBackend (20fd120)Literal annotation to display possible units for delta method (ee94cb5)_rewrite_filter() in favor of using replacement patterns (4c0ac2e)is_reduction() (2acc31f)pushdown_aggregation_filters() (cf95ff7)sub_for(), substitute(), find_toplevel_aggs() (492b296)substitute_parents() (cd91a7e)substitute_unbound() since it is used at a single place (6a6ad19)pushdown_selection_filters() (2e47738)read_delta to base io handler (3d5a684)BigQueryTable object (b83e60e)lit usage (1bc6cee)lit with builtin sqlglot functions (221b630)create_table implementation (ea0826d)ibis.common.bases.Base in favor of Abstract (8ed313c)FILTER syntax when counting (815c12f)ibis.expr.streaming (70df318)ibis.common.pattern builders and ibis.expr.deferred (652ceab)*args in udf functions (e22236c)Temporal* classes (d3bcf73)list_schemas implementation (84d38a1)ibis.dtype top-level API (867e5f1)table.nunique() for counting unique table rows (adcd762)ibis.array (3355dd8)__dataframe__ protocol (15e27da)concat method equivalent to __add__/__radd__ (0ed0ab1)repeat method equivalent to __mul__/__rmul__ (b457c7b)current_schema API (955a9d0)CREATE TABLE DDL options including support for overwrite (5dac7ec)to_sql a proper str subclass (084bdb9)ops.SelfReference (983e393).sql table expression method (479bc60)current_database implementation and API consistent across all backends (eeeeee0)has_operation (acb7f3f)list_schemas() (172520e)__dataframe__ protocol (50f3be9)ops.SelfReference (9068aca)pivot_longer works with more than one column (822c912)has_operation method a [@classmethod](https://github.com/classmethod) (c1b7dbc)regexp_instr != 0 instead of REGEXP keyword (06e2be4)DataType.is_* methods (ed40fdb)dtype function (39b7a24)schema.empty_table() instead of manually constructing empty tables (c099302)result_handler in favor of expression specific methods (3dc7143)distinct API for dropping duplicates based on column subsets (3720ea5)Table.relabel (0583959)to_pyarrow_batches() smarter (42f5987)ops.Arbitrary (ee8dbab)pivot_longer (f600c90)ops.Arbitrary (9bcdf77)pivot_wider API (60e7731)ibis.expr.selectors to ibis.selectors and deprecate for removal in 6.0 (0ae639d)resolve method (e12c4df)make_url usage (ea10a27)create_table implementation (5a54489)match (762f4d6)SIMILAR TO instead of tilde to workaround sqlglot issue (434da27)time_zone and time_unit properties (3a2c4df)having filter (bd81a9f)ArrayRemove operation (772668b)pivot_longer semantics in the presence of multiple unnests (6bc301a)notany/notall aggregations work (c2e985f)how='first' with arbitrary reduction (315b5e7)NULL values (8618789).projection in favor of the shorter .select (73df8df)to_pyarrow_batches implementation (d6235ee)_meta instance attributes (523e198)Table.set_column in favor of Table.mutate (954a6b7)None (5cf186a)SUMMARY.md (0bdd093)ibis.common.exceptions.UnsupportedOperationError in more places during compilation. You may need to catch this error type instead of the previous type, which differed between backends.Table.info now returns an expressionTable.drop is removed. Replace drop(cols) with drop(*cols).spark plugin alias is removed. Use pyspark insteadibis.expr.scope and ibis.expr.timecontext modules, access them under ibis.backends.base.df.<module>
ibis.<backend> namespaces, access them on a connected backend instance instead.ibis.common.geospatial, import the functions from ibis.backends.base.sql.registry.geospatial
JSON is no longer a subtype of String
Category, CategoryValue/Column/Scalar are removed. Use string types instead.metric_name argument to value_counts is removed. Use Table.relabel to change the metric column's name.parsy is now 2.0ibis.backends.duckdb.parse_type() functionibis.backends.impala.Backend.set_database() methodibis.backends.pyspark.Backend.set_database() methodibis.backends.impala.ImpalaConnection.ping() methodibis.expr.operations.DatabaseTable.change_name() methodibis.expr.operations.ParseURL classibis.expr.operations.Value.to_projection() methodibis.expr.types.Table.get_column() methodibis.expr.types.Table.get_columns() methodibis.expr.types.StringValue.parse_url() methodSchema.from_dict(), .delete() and .append() methods are removedstruct_type.pairs is removed, use struct_type.fields insteadStruct(names, types) is not supported anymore, pass a dictionary to Struct constructor insteadmax_columns option for table repr (a3aa236)map/array accessors for easy conversion of JSON to stronger-typed values (d1e9d11)ibis.map (d365fd4)to_pandas() API for ecosystem compatibility (cad316c)cache evaluate only once per session per expression (5a8ffe9)rlz.table rule (8dcfb8d)ops.Time for sqlalchemy backends (713cd33)BIGNUMERIC type support (5c98ea4)ArrayFilter translation (f2144b6)ops.ArrayMap (45000e7)ops.MapLength (fc82eaa)Callable arguments and return types in Validator.from_annotable() (ae57c36)Decimal type (8761732)Mapping abstract base class for StructType (5df2022)ArrayFilter translation (5f35d5c)ops.ArrayMap (063602d)read_json_auto when reading json (4193867)ArrayFilter operation (e719d60)ArrayMap operation (49e5f7a)ops.Union (17c6011)schema (39c9ea8)argmin/argmax (82668ec)ops.ArgMin and ops.ArgMax (a3fa57c)Mapping abstract base class for Schema (167d85a)pivot_longer API (11c5736)MapLength operation (a7ad1db)ArrayFilter translation (50f6fcc)ops.ArrayMap (657bf61)ops.Between (d70b9c0)Table.drop (325140f)between, rows and range keyword arguments in value.over() (5763063)Limit subqueries (62f6e14)name attribute to backend proxy modules (d6d8e7e)__radd__ array concat operation (121d9a0)clickhouse_driver (ab8f064)ibis.<backend> directly (5a16431)null to duckdb datatype parser (07d2a86)temp_directory exists (00ba6cb)to_pyarrow/to_pyarrow_batches (0e8b066)struct_pack (6e598cc)g re_replace flag as literal text (aa3c31c)read_parquet and read_csv to base backend mixin (ce80d36), closes #5420
find_subqueries return in topological order (3587910)rlz.map_to() rule instead of isin to normalize temporal units (a1c46a2)wrap_case_result (a3b82f7)concat_str separator argument (ced5a61)reverse/descending argument (f067d81)ops.EndsWith translation rule (4c0a5a2)array_agg preserve nulls (24b95bf)sa.Column (af4db5c)NULL when trying to call map functions on non-object JSON (d85fb28)_flatten to avoid overriding unrelated function in other backends (8c31594)strip family of functions behave like Python (dd0a04c)sa.true instead of Python literal (8423eba)value_counts aggregate column a better name (abab1d7)duckdb: faster to_parquet/to_csv implementations (6071bb5)
fix duckdb insert-from-dataframe performance (cd27b99)
deps: bump minimum required version of parsy (22020cb)
remove spark alias to pyspark and associated cruft (4b286bd)
find_subqueries() (ab3712f)frozendict and DotDict to ibis.common.collections (4451375)geospatial module to the base SQL backend (3e7bfa3)Category type and related APIs (bb0ee78)StructType.pairs property in favor of identical fields attribute (6668122)String parent type from JSON type (34f3898)StructType fields rather than names and types tuples (84455ac)limit kwarg from to_parquet/to_csv (a54460c)list<...> parsing rule (f040b86)scope and timecontext utilities (80bd494)ArrayMap to use the new callable_with validation rule (560474e)non_negative_integer and pair rules (e00920a)Schema.from_dict(), .delete() and .append() methods (8912b24)parsy (c53403a)rules test file to the ibis.expr subpackage (47f0909)schema test file to the ibis.expr subpackage (40033e1)Table.info (71cc0e0)collect method API documentation (b4fcef1)ibis-examples repo in the README (1205636)__dunder__ method API documentation (b532c63)ibis.get_backend function (2d27df8)ibis.set_backend function (e7fabaf)connect() with no arguments (67eed42)as_struct method to convert schemas to structs (64be7b1)read_json function for consuming newline-delimited JSON files (65e65c1)DATETIME2 and DATETIMEOFFSET (aa9f151)connect() with no arguments (78cbbdd)Map(string, string) support via the built-in HSTORE extension (f968f8f)ArrayCollect (c425f68)NthValue (0dca57c)ops.Arbitrary (45f4f05)ops.StructColumn (41698ed)StringSplit (e6acc09)StructField and struct literals (286a5c3)TimestampFromUNIX (314637d)TimestampFromYMDHMS (1eba8be)typeof operation (029499c)_get_schema_using_query in SQLite backend (7ff84c8)ops.ExistsSubquery and ops.NotExistsSubquery (9b9b315)_default_backend (11252af)StructType and Schema (c9e7078)read_csv on URLs work (9e61816)to_pyarrow/to_pyarrow_batches (a1a6902)order_by on nonexistent column (57b1dd8)TimestampNow (42d95b0)ops.TableArrayView (a1ad2b7)typeof test and bring back implementations (7dc5356)exec_driver_sql in view teardown (2599c9b)AlchemyCompiler (9f4ff54)ibis.map (045edc7)pathlib.Path correctly in attach (0415bd3)to_pyarrow_batches documentation (ec1ffce)StructType fields rather than names and types tuples (ff34c7b).raw_sql call (abc939e)Schema fields rather than names and types tuples (318179a)_find_backend implementation (60f1a1b)parse_json call in ops.StructField impl (9e80231)unary instead of fixed_arity(..., 1) (4a1c7c9)_get_schema_using_query to base class (296cd7d)isnot (4ec53a4)exec_driver_sql everywhere (e8f96b6)_CorrelatedRefCheck (f49e429)