Polars Versions Save

🏆 Highlights

• Full plan CSE (#15264)

⚠️ Deprecations

• Rename parameter by to group_by in DataFrame.upsample/group_by_dynamic/rolling (#14840)
• Rename from_repr parameter from tbl to data (#15156)

🚀 Performance improvements

• load bits one word at a time for BitmapIter (#15333)
• Ipc exec multiple paths (#15040)
• add SIMD support for if-then-else kernels (#15131)

✨ Enhancements

• Remove 'FileCacher' optimization (#15357)
• check input type in entropy (#15351)
• Implements arr.n_unique (#15296)
• CSE don't scan share if predicate pushdown predicates don't match (#15328)
• Add read_database support for SurrealDB ("ws" and "http") (#15269)
• Only allow inputs of type Sequence in from_records (#15329)
• In hypothesis testing strategies, enable Decimal strategy by default (#15321)
• Remove cached nodes when finished (#15310)
• Full plan CSE (#15264)
• More robust handling of async database calls (#15202)
• Add name parameter to GroupBy.len method (#15235)
• Add IR for expressions. (#15168)
• Improve read_database when reading from Kùzu graph database (#15218)
• Warn if map_elements is called without return_dtype specified (#15188)
• Add support for async SQLAlchemy connections to read_database (#15162)
• Infer time_unit in pl.duration when nanoseconds is specified (#14987)
• Add strict parameter to from_dict/from_records (#15158)

🐞 Bug fixes

• Return correct dtype for s.clear() when dtype is Object (#15315)
• ensure first datapoint is always included in group_by_dynamic (#15312)
• Non-exhaustive patterns: arrow-schema::DataType in polars-arrow (#15250)
• use dynamic stacks for problematic recursive functions (#15355)
• Adding default ddof for Series.list.std and Series.list.var (#15267)
• Raise properly for slices not supported by LazyFrame (#15331)
• Propagate strictness in from_dicts (#15344)
• Raise error when schema_overrides contains nonexistent column name (#15290)
• Enforce integer dtype input for int_range and int_ranges (#15339)
• Preserve Decimal precision when constructing empty Series (#15320)
• Fix cache dot visualization (#15311)
• Handle special case correctly when slicing a LazyFrame (#15297)
• Properly propagate strict flag when constructing a Struct Series from any values (#15302)
• Consistent expansion of nested struct data during DataFrame init from dict (#15217)
• Raise when join projects name with suffix that doesn't exist (#15256)
• Ensure ooc_start is set (#15255)
• Fix bug where rolling operations were ignoring check_sorted in some cases (#15227)
• Fix lazy schema for rle expression (#15248)
• incorrect negative offset in multi-byte string slicing (#15140)
• do not clamp negative offsets to start of array prematurely (#15242)
• allow null index in list.get and array.get (#15239)
• Avoid loading all columns in read_parquet when columns parameter is specified (#15229)
• properly support nulls_last + descending (#15212)
• fix nested runtime panic (#15216)
• Block rounding/truncating to negative durations (#15175)
• Ensure the cs.temporal() selector uses wildcard time zone matching for Datetime (#13683)
• Consistently raise TypeError on constructor failure (#15178)
• Properly propagate strictness in some constructor cases (#15166)
• Fix constructing a Series from a list of Series with given dtype (#15144)

📖 Documentation

• Fix time unit in timestamp example (#15281)
• Fix link to renamed method (.list.lengths -> .list.len) (#15228)
• Update Excel and database pages in user guide (#14721)
• Add examples for Series.search_sorted (#14737)
• Correct the definition of an expression in the user guide (#14750)
• Add a note about the behaviour of lower/upper bounds for is_between, and add an example (#15197)

📦 Build system

• Update Cargo lock (#15370)

🛠️ Other improvements

• Memory usage test infrastructure, plus a test for #15098 (#15285)
• Update CODEOWNERS (#15352)
• remove try_apply_values (#15336)
• always use non-legacy float_sum for mean (#15343)
• remove legacy bitmap module (#15335)
• Fix test not writing to temporary directory (#15318)
• Reorganize tests for clear operation (#15304)
• Rename Cache[count] to Cache[cache_hits] (#15300)
• Cleanup file_caching optimization call (#15299)
• Minor refactor of PyDataFrame.from_dicts (#15274)
• remove 'fast-projection' node (#15253)
• cleanup column names in optimizer (#15252)
• remove left_most_input_name from expr ir (#15251)
• add AlignedBitmapSlice (#15171)
• Run non-benchmark tests in benchmark workflow (#15207)
• Add wrapping_abs to arithmetic kernel (#15210)
• remove raw buffers from BinViewArray (#15206)
• Enable RUST_BACKTRACE=1 in the CI test suite (#15204)
• Split read_database functionality into cleaner module structure (#15201)
• Clean up some of the AnyValue conversion logic (#15190)
• remove parts of legacy bit_util (#15169)
• remove legacy arrow compute (#15164)
• Split up dataframe module in PyO3 bindings (#15165)
• Remove unused private constructors (#15160)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @braaannigan, @c-peters, @cojmeister, @deanm0000, @dependabot, @dependabot[bot], @itamarst, @kszlim, @mbuhidar, @mcrumiller, @mickvangelderen, @orlp, @petrosbar, @reswqa, @ritchie46, @rob-sil, @sportfloh, @stinodego and @yutannihilation

🚀 Performance improvements

• add new when-then-otherwise kernels (#15089)
• Coerce sorted flag of unit arrays during concat (#15104)
• Use sorted flag for (first|last)_non_null (#15050)
• OOC sort improvements (#14994)

✨ Enhancements

• raise if both closed and by are passed to rolling_* aggregations (#15108)
• raise informative error for rolling_* aggs with by of invalid dtype (#15088)
• add non_existent arg to replace_time_zone (#15062)
• Support single nested row encodings (#15105)
• make ooc sort configurable (#15084)
• Make register_plugin a standalone function and include shared lib discovery (#14804)
• Async parquet: Decode parquet on a blocking thread pool (#15083)
• let "ambiguous" take "null" value (#14961)
• Raise informative error message when join would introduce duplicate column name (#15042)
• Allow cast of decimal to boolean (#15015)
• Return error when no supertype can be determined in AnyValue constructor when strict=false (#15025)
• Implement IpcReaderAsync (#14984)
• Support Array statistics in parquet (#15031)
• Support decimal groupby (#15000)
• Add thread names to rayon thread pool (#15024)
• Support decimal uniq (#15001)
• expose timings in verbose state of OOC sort (#14979)

🐞 Bug fixes

• Fix Series construction from nested list with mixed data types (#15046)
• Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
• Binview chunked gather; don't modify inlined view (#15124)
• Fix chunked_id gather for binview buffers (#15123)
• Don't cache HTTP object stores as they maintain URL state (#15121)
• use wrapping_add in csv line snooping (#15109)
• Output u32 when sum_horizontal provided with single boolean column (#15114)
• Ensure eprintln! is only called within debug/verbose context (#15100)
• Propagate error instead of panicking when calling product on an invalid type (#15093)
• Raise error when casting Array to different width (#14995)
• Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
• Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
• Return largest non-NaN value for max() on sorted float arrays if it exists instead of NaN (#15060)
• return NaN for all-NaN min/max (#15066)
• Prevent "index out of range for slice" error in parquet reader (#15021)
• Respect nulls_last in streaming sort (#15061)
• Fix Series construction from nested list with mixed data types (#15046)
• Don't count nulls in streaming count agg (#15051)
• agg_list on decimal lost scale (#15054)
• Block predicate pushdown on equality that are use in join (#15055)
• Enum equality based on categories (#15053)
• Strict cast in when/then/otherwise operation (#15052)
• Don't panic in string_addition_to_linear_concat (#15006)
• CSV do utf8-validation after escaping fields (#15004)
• Use primitive constructors to create a Series of lists when dtype is provided (#15002)
• replace_time_zone with single-null-element "ambiguous" was panicking (#14971)

📖 Documentation

• Fix typo in comment (#14997)

🛠️ Other improvements

• Extend and speed up scan tests (#15127)
• always assert on ChunkedArray::get (#15120)
• Use ObjectStore instead of AsyncRead in parquet get metadata (#15069)
• Minor refactor of Rust any value constructors (#15077)
• Simplify streaming execution (#15039)
• Ensure we hit the spilled source path in ooc sort test (#15010)
• Refactor constructor code (#15009)
• Apply clippy:assigning_clones lint (#14999)
• fix features (#14977)

Thank you to all our contributors for making this release possible! @JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2

🚀 Performance improvements

• add new when-then-otherwise kernels (#15089)
• Coerce sorted flag of unit arrays during concat (#15104)
• Use sorted flag for (first|last)_non_null (#15050)
• OOC sort improvements (#14994)

✨ Enhancements

• improved dtype inference/refinement for read_database results (#15126)
• raise if both closed and by are passed to rolling_* aggregations (#15108)
• raise informative error for rolling_* aggs with by of invalid dtype (#15088)
• add non_existent arg to replace_time_zone (#15062)
• Support single nested row encodings (#15105)
• make ooc sort configurable (#15084)
• Make register_plugin a standalone function and include shared lib discovery (#14804)
• Expose infer_schema_length parameter on read_database (#15076)
• Async parquet: Decode parquet on a blocking thread pool (#15083)
• let "ambiguous" take "null" value (#14961)
• Raise informative error message when join would introduce duplicate column name (#15042)
• Allow cast of decimal to boolean (#15015)
• Add strict parameter to DataFrame constructor to allow non-strict construction (#15034)
• Support Array statistics in parquet (#15031)
• Support decimal groupby (#15000)
• Add thread names to rayon thread pool (#15024)
• Support decimal uniq (#15001)
• expose timings in verbose state of OOC sort (#14979)

🐞 Bug fixes

• Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
• Binview chunked gather; don't modify inlined view (#15124)
• Fix chunked_id gather for binview buffers (#15123)
• Don't cache HTTP object stores as they maintain URL state (#15121)
• Output u32 when sum_horizontal provided with single boolean column (#15114)
• Propagate error instead of panicking when calling product on an invalid type (#15093)
• Raise error when casting Array to different width (#14995)
• Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
• Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
• Return largest non-NaN value for max() on sorted float arrays if it exists instead of NaN (#15060)
• return NaN for all-NaN min/max (#15066)
• Prevent "index out of range for slice" error in parquet reader (#15021)
• Respect nulls_last in streaming sort (#15061)
• Fix Series construction from nested list with mixed data types (#15046)
• Don't count nulls in streaming count agg (#15051)
• agg_list on decimal lost scale (#15054)
• Block predicate pushdown on equality that are use in join (#15055)
• Enum equality based on categories (#15053)
• Don't panic in string_addition_to_linear_concat (#15006)
• CSV do utf8-validation after escaping fields (#15004)
• Use primitive constructors to create a Series of lists when dtype is provided (#15002)
• replace_time_zone with single-null-element "ambiguous" was panicking (#14971)

📖 Documentation

• Update write_database code blocks in user guide (#15106)
• Add missing docstring examples in the Struct namespace (#15071)
• Improve API reference landing page (#14888)
• improve join_asof example (#14993)
• Fix inadvertent swap of new and old parameters in replace description (#15019)

🛠️ Other improvements

• Extend and speed up scan tests (#15127)
• Add parameterized-scan-tests (#15057)
• Simplify streaming execution (#15039)
• Ensure we hit the spilled source path in ooc sort test (#15010)
• Refactor constructor code (#15009)
• fix features (#14977)
• Revert pinning PyPI publish action (#14975)

Thank you to all our contributors for making this release possible! @JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2

🚀 Performance improvements

• Use sorted flag for (first|last)_non_null (#15050)
• OOC sort improvements (#14994)

✨ Enhancements

• let "ambiguous" take "null" value (#14961)
• Raise informative error message when join would introduce duplicate column name (#15042)
• Allow cast of decimal to boolean (#15015)
• Add strict parameter to DataFrame constructor to allow non-strict construction (#15034)
• Support Array statistics in parquet (#15031)
• Support decimal groupby (#15000)
• Add thread names to rayon thread pool (#15024)
• Support decimal uniq (#15001)
• expose timings in verbose state of OOC sort (#14979)

🐞 Bug fixes

• Prevent "index out of range for slice" error in parquet reader (#15021)
• Respect nulls_last in streaming sort (#15061)
• Fix Series construction from nested list with mixed data types (#15046)
• Don't count nulls in streaming count agg (#15051)
• agg_list on decimal lost scale (#15054)
• Block predicate pushdown on equality that are use in join (#15055)
• Enum equality based on categories (#15053)
• Don't panic in string_addition_to_linear_concat (#15006)
• CSV do utf8-validation after escaping fields (#15004)
• Use primitive constructors to create a Series of lists when dtype is provided (#15002)
• replace_time_zone with single-null-element "ambiguous" was panicking (#14971)

📖 Documentation

• Improve API reference landing page (#14888)
• improve join_asof example (#14993)
• Fix inadvertent swap of new and old parameters in replace description (#15019)

🛠️ Other improvements

• Add parameterized-scan-tests (#15057)
• Simplify streaming execution (#15039)
• Ensure we hit the spilled source path in ooc sort test (#15010)
• Refactor constructor code (#15009)
• fix features (#14977)
• Revert pinning PyPI publish action (#14975)

Thank you to all our contributors for making this release possible! @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @petrosbar, @ritchie46, @stinodego and @trueb2

🏆 Highlights

• Streaming outer joins (#14828)

🚀 Performance improvements

• Ensure parallel encoding/compression in sink_parquet (#14964)
• hoist errors out of iterators in parquet (#14945)
• add basic AVX-512 filters (#14892)
• improve join-asof materialization (#14884)
• Optimize chunked-id gather for binaryviews (#14878)
• rework scalar filter kernels (#14865)
• Reduce size of optional join-indexes (#14856)
• Streaming outer joins (#14828)
• Set sorted flag for cum_count on columns (#14849)

✨ Enhancements

• Support writing Array type in parquet (#14943)
• Sort decimal fields (#14649)
• Import NamedFrom in df! macro (#14860)
• try-improve concurrency tuner (#14827)
• Streaming outer joins (#14828)
• Set sorted flag for cum_count on columns (#14849)
• Ensure binview types are rle-encoded in parquet write (#14818)
• Implement strict/nonstrict conversion for primitive AnyValues (#14186)
• Disable timeouts (#14809)
• cleanup spill disks in process (#14807)

🐞 Bug fixes

• Fix invalid partitionable query (#14966)
• allow nonstrict cast of categorical/enum to enum (#14910)
• count_rows multi-threaded under-counting in parser.rs (#14963)
• raise proper error instead of panicking when result of truncation is non-existent datetime (#14958)
• ooc-sort issues (#14959)
• Do not raise when constructing from a list of Series with Nones (#14942)
• Don't access out-of-bounds for null indices in bitmap gather (#14932)
• std when ddof>=n_values returns None even in rolling context (#11750)
• Don't rechunk categoricals when moving to physical (#14934)
• parquet rle boolean decoder (#14931)
• boolean filter gave overly large buffers to Bitmap::from_u8_vec (#14924)
• Fix sliced dictionary state in parquet (#14917)
• Fix possibly incorrect order of columns when using ipc stream with_columns (#14859)
• Fully qualify polars_bail! in polars_ensure! (#14901)
• Fix DataFrame.min/max for decimals (#14890)
• Assert chunks are equal after physical cast to prevent OOB (#14873)
• not all cpu feature flag tests were mocked (#14864)

📖 Documentation

• Remove some repetition in comments/docstrings (#14912)
• Update contributing link (#14882)
• Fix some word-repetition in code comments (#14825)
• Seperate asof from join strategy, change parameter from strategy to how in user guide (#14793)

🛠️ Other improvements

• fix features (#14977)
• fix chrono deprecation warnings (#14928)
• Update Cargo.lock and remove cmake limit workaround (#14905)
• Simplify streaming placeholder replacement. (#14915)
• Optional deps should include fastexcel (#14907)
• Deduplicate POLARS_FORCE_ASYNC env var parsing (#14909)
• Make assumption about column name to index conversion having occurred explicit (#14894)
• Make assumption about wildcards having been resolved explicit (#14899)
• reactivate argminmax simd (#14679)
• sort by 'idx' after outer join (#14867)
• Simplify computation of with_columns attribute in physical csv scanner of default engine. (#14837)
• centrally define IdxSize (#14854)
• run and fix pext64_polyfill test (#14852)
• introduce partitioned table (#14819)
• add missing deprecation directive in groupby.count (#14817)
• Extract key value construction (#14812)
• Fix Makefile build commands (#14806)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @Sol-Hee, @alexander-beedie, @ambidextrous, @battmdpkq, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @flisky, @geekvest, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego

🚀 Performance improvements

• Ensure parallel encoding/compression in sink_parquet (#14964)
• hoist errors out of iterators in parquet (#14945)
• add basic AVX-512 filters (#14892)

✨ Enhancements

• Support writing Array type in parquet (#14943)
• Add drop_first parameter to Series.to_dummies (#14846)
• Add "execute_options" support for read_database_uri (#14682)

🐞 Bug fixes

• Fix invalid paritionable query (#14966)
• allow nonstrict cast of categorical/enum to enum (#14910)
• count_rows multi-threaded under-counting in parser.rs (#14963)
• raise proper error instead of panicking when result of truncation is non-existent datetime (#14958)
• ooc-sort issues (#14959)
• Do not raise when constructing from a list of Series with Nones (#14942)
• Don't access out-of-bounds for null indices in bitmap gather (#14932)
• std when ddof>=n_values returns None even in rolling context (#11750)
• Don't rechunk categoricals when moving to physical (#14934)
• Ensure consistent read_database behaviour with empty ODBC "iter_batches" (#14918)
• parquet rle boolean decoder (#14931)
• Fix frame init from single RecordBatch objects when pyarrow <= 12 (#14922)
• boolean filter gave overly large buffers to Bitmap::from_u8_vec (#14924)
• Fix sliced dictionary state in parquet (#14917)
• read_database now properly handles empty result sets from arrow-odbc (#14916)
• Fix possibly incorrect order of columns when using ipc stream with_columns (#14859)

📖 Documentation

• Add note about include_index in from_pandas regarding "default indices" (#14920)
• Remove some repetition in comments/docstrings (#14912)

🛠️ Other improvements

• Update Cargo.lock and remove cmake limit workaround (#14905)
• Simplify streaming placeholder replacement. (#14915)
• Optional deps should include fastexcel (#14907)
• Deduplicate POLARS_FORCE_ASYNC env var parsing (#14909)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @alexander-beedie, @ambidextrous, @battmdpkq, @mcrumiller, @mickvangelderen, @orlp, @petrosbar and @ritchie46

🏆 Highlights

• Streaming outer joins (#14828)

⚠️ Deprecations

• Deprecate overwrite_schema parameter for DataFrame.write_delta (#14879)

🚀 Performance improvements

• improve join-asof materialization (#14884)
• Optimize chunked-id gather for binaryviews (#14878)
• rework scalar filter kernels (#14865)
• Reduce size of optional join-indexes (#14856)
• Streaming outer joins (#14828)
• Set sorted flag for cum_count on columns (#14849)

✨ Enhancements

• Sort decimal fields (#14649)
• Revert addition of __slots__ to Polars classes (#14857)
• Add fastexcel to show_versions (#14869)
• try-improve concurrency tuner (#14827)
• Streaming outer joins (#14828)
• Set sorted flag for cum_count on columns (#14849)
• support use of KùzuDB via pl.read_database (#14822)
• Ensure binview types are rle-encoded in parquet write (#14818)
• Disable timeouts (#14809)
• cleanup spill disks in process (#14807)
• Implement compression and skipping for binview IPC (#14789)

🐞 Bug fixes

• Fix DataFrame.min/max for decimals (#14890)
• Assert chunks are equal after physical cast to prevent OOB (#14873)
• not all cpu feature flag tests were mocked (#14864)
• Remove custom __reduce__ implementation on DataType object (#14778)
• Allow non-strict construction / initialization of Enum columns (#14728)
• Fix streaming parquet limit (#14783)

📖 Documentation

• Update contributing link (#14882)
• update to use ambiguous instead of use_earliest (#14820)
• Seperate asof from join strategy, change parameter from strategy to how in user guide (#14793)

🛠️ Other improvements

• Pin PyPI publish action to commit (#14896)
• reactivate argminmax simd (#14679)
• sort by 'idx' after outer join (#14867)
• run and fix pext64_polyfill test (#14852)
• add missing deprecation directive in groupby.count (#14817)
• Fix Makefile build commands (#14806)
• Bump ruff from 0.2.0 to 0.3.0 in /py-polars (#14800)
• Rename utils module to _utils to explicitly mark it as private (#14772)
• Add test coverage for _cpu_check module (#14768)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @Sol-Hee, @alexander-beedie, @c-peters, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @flisky, @geekvest, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego

🚀 Performance improvements

• Elide utf8/binary cast in Parquet reading (#14757)

✨ Enhancements

• Implement compression and skipping for binview IPC (#14789)

🐞 Bug fixes

• fix feature flags (#14802)
• Allow non-strict construction / initialization of Enum columns (#14728)
• Fix streaming parquet limit (#14783)

📦 Build system

• bump rayon from 1.8.1 to 1.9.0 (#14797)

Thank you to all our contributors for making this release possible! @alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @ritchie46 and @stinodego

🚀 Performance improvements

• Elide utf8/binary cast in Parquet reading (#14757)

🐞 Bug fixes

• Add missing "pclmulqdq" instruction to _cpu_check ("read_cpu_flags") (#14758)

🛠️ Other improvements

• Test release wheels on x86-64 (#14761)

Thank you to all our contributors for making this release possible! @alexander-beedie, @ritchie46 and @stinodego

[!WARNING]
This release was deleted from PyPI. Please use the 0.20.13 release instead.

🚀 Performance improvements

• auto-tune concurrency budget (#14753)
• Don't materialize for broadcasting fill_null value and default value of replace (#14736)
• Improve performance of boolean filters 1-100x. (#14746)

🐞 Bug fixes

• fix hashing specialization (#14754)
• Sum after filter in aggregation context sometimes returned NULL (#14752)
• Allow list.contains() for list of categoricals (#14744)
• Fix bug where alias was ignored in COUNT(*) optimization (#14738)
• Fix DataFrame.sum for decimals (#14732)

📖 Documentation

• Link to plugins tutorial more prominently (#14727)

📦 Build system

• update ahash (#14731)

🛠️ Other improvements

• update ahash (#14731)
• Use datetime_to_int util for AnyValue conversion (#14743)
• Refactor utils/convert.py module (#14739)

Thank you to all our contributors for making this release possible! @MarcoGorelli, @c-peters, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego