Aws Data Wrangler Versions Save

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

3.7.2

4 weeks ago

Features/Enhancements 🚀

Add support for DeltaLake's DynamoDB lock mechanism by @LeonLuttenberger in #2705

Bug fixes 🐛

wr.athena.to_iceberg - Insert query has mismatched column types #2678 by @GalvFionic in #2715
allow s3_output in athena.to_iceberg by @jaidisido in #2727
replace deprecated np.split_array by @jaidisido in #2735
Athena to_iceberg fails with non-lowercase column names by @LeonLuttenberger in #2736
Support Ray 2.10 by @kukushking in #2741

New Contributors

@GalvFionic made their first contribution in #2715

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.7.1...3.7.2

3.7.1

1 month ago

Bug fixes 🐛

fix breaking change in _create_table by @jaidisido in https://github.com/aws/aws-sdk-pandas/pull/2711
pin pyarrow to version 8 and above by @jaidisido in https://github.com/aws/aws-sdk-pandas/pull/2709

Documentation 📚

fix redshift.to_sql doc indentation error by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2706

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.7.0...3.7.1

3.7.0

1 month ago

Breaking changes 💥

Lake Formation Governed tables are being phased out and we are dropping support (#2692).

Features/Enhancements 🚀

support parquet client encryption (#2642) by @Marwen94 in https://github.com/aws/aws-sdk-pandas/pull/2674

Bug fixes 🐛

Index columns removed on s3.to_parquet by @robert-schmidtke in https://github.com/aws/aws-sdk-pandas/pull/2655
Missing timezone metadata by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2682
remove enforced openpyxl engine constraint by @jaidisido in https://github.com/aws/aws-sdk-pandas/pull/2696
Iceberg partitioning not working with partition transform functions by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2694
remove awswrangler README from site-packages folder by @AlJohri in https://github.com/aws/aws-sdk-pandas/pull/2698
indent categories in pyarrow_additional_kwargs correctly by @jaidisido in https://github.com/aws/aws-sdk-pandas/pull/2701

New Contributors

@Marwen94 made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2674
@AlJohri made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2698

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.6.0...3.7.0

3.6.0

2 months ago

Features/Enhancements 🚀

Enable Iceberg row deletion & add mode parameter to to_iceberg by @LeonLuttenberger in #2632
Add support for pyarrow type large_string by @joakibo in #2663
Add max_results to athena.list_query_executions by @LeonLuttenberger in #2665

Bug fixes 🐛

Pyarrow 15 imports & remove unused code by @kukushking in #2649

New Contributors

@joakibo made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2663

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.5.2...3.6.0

3.5.2

3 months ago

Bug fixes 🐛

DynamoDB key & filter expressions attribute overwrite by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2615
Allow PostgreSQL reserved keywords as column names by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2619
Add to_iceberg support for filling missing columns in the DataFrame with None by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2616
Forward ignore_nulls for container types by @raaidarshad in #2636

Documentation 📚

Add s3_additional_kwargs to docstrings by @malachi-constant in https://github.com/aws/aws-sdk-pandas/pull/2627
Fix outdated hyperlinks in documentation by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2634

Other 🤖

Enable dependabot to upgrade GitHub actions by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2618
Update badges in README by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2628
Add vulnerability label to dependabot PRs with alert state by @jaidisido in https://github.com/aws/aws-sdk-pandas/pull/2629

New Contributors

@raaidarshad made their first contribution in #2636

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.5.1...3.5.2

3.5.1

3 months ago

Bug fixes 🐛

Deserialization error when reading from DynamoDB using KeyConditionExpression by @LeonLuttenberger in #2607
Reading of chunked parquet when columns parameter is specified by @rchromik in #2599

Documentation 📚

Add show_create_table to Athena API page by @MikeSchriefer in #2610

Other 🤖

chore: Replace bump2version with bump-my-version by @LeonLuttenberger in #2608
chore(deps-dev): bump jinja2 from 3.1.2 to 3.1.3 by @dependabot in #2609
chore(deps): bump grpcio from 1.51.3 to 1.53.0 by @dependabot in #2612

New Contributors

@MikeSchriefer made their first contribution in #2610
@rchromik made their first contribution in #2599

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.5.0...3.5.1

3.5.0

3 months ago

Breaking changes 💥

Due to CVEs, Ray is capped to patched version 2.9.x. As a result, the latest version of the library cannot be used on the Glue for Ray runtime. We have raised the CVEs issue to the Glue team

Features/Enhancements 🚀

Add spark_properties to athena spark by @rajagurunath in https://github.com/aws/aws-sdk-pandas/pull/2508
Add MERGE INTO support for Iceberg by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2527
Support partitioning by index cols by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2528
Add analysis_template_arn to cleanrooms.read_sql_query by @jaidisido in https://github.com/aws/aws-sdk-pandas/pull/2584
Python 3.12 support by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2559
- Note: Ray currently does not support Python 3.12. As such, distributed operations on data frames will not work yet.
- Relevant Ray issue
Upgrade to Ray 2.9.0+ and refactor Ray datasources to the new API by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2570

Bug fixes 🐛

Athena/Neptune minor fixes by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2526
Reset index and handle last index by @Antropath in https://github.com/aws/aws-sdk-pandas/pull/2531
Oracle failed import message by @matthewdeanmartin in https://github.com/aws/aws-sdk-pandas/pull/2537
Add parameterized queries where possible to address the risk of SQL injection by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2540
SQL identifiers by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2543
coerce_timestamps - allow None by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2556
Add validation for table and schema params for Redshift by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2551
Redshift VARBYTE support by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2573

Documentation 📚

Add SSM Public Param usage to docs by @malachi-constant in https://github.com/aws/aws-sdk-pandas/pull/2521

Other 🤖

refactor: Remove usage of boto3 resources by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2525
chore(deps): bump aiohttp from 3.8.5 to 3.8.6 by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2519
chore(deps): bump aiohttp from 3.8.6 to 3.9.0 by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2535
chore(deps): bump cryptography from 41.0.4 to 41.0.6 by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2538
chore(deps-dev): bump jupyter-server from 2.7.2 to 2.11.2 by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2545
chore: Upgrade test infrastructure dependencies by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2562
chore: Prepare 3.5.0 release by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2560
chore: Upgrade deltalake dependency by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2563
chore: Replace black formatter with ruff format by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2568
chore: ruff improvements by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2571
chore: upgrade oracledb to 2.0 by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2574
chore(deps-dev): bump the development-dependencies group with 8 updates by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2577
chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2583
chore(deps-dev): bump the development-dependencies group with 3 updates by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2590
chore(deps): bump the production-dependencies group with 5 updates by @dependabot in https://github.com/aws/aws-sdk-pandas/pull/2591
chore: type annotations by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2585
chore: Replace PyLint with Ruff by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2588
chore: Update gremlinpython & add aiohttp by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2595

New Contributors

@rajagurunath made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2508
@Antropath made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2531
@matthewdeanmartin made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2537

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.4.2...3.5.0

3.4.2

5 months ago

Features/Enhancements 🚀

Update pyarrow to 14.0.1 to fix arbitrary code execution security vulnerability

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.4.1...3.4.2

3.4.1

6 months ago

Features/Enhancements 🚀

feat: Add schema evolution to athena.to_iceberg by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2465
feat: Athena - add client_request_token by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2474
feat: Redshift data api - allow all auth combinations by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2475
feat: add columns comments to iceberg by @frenchytheasian in https://github.com/aws/aws-sdk-pandas/pull/2482
feat: Add Python 3.11 layers in cn-north-1 & cn-northwest-1 by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2514

Bug fixes 🐛

fix: Add missing call to sanitize_column_name in create_*_table by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2464
fix: Hyphenated Iceberg table names by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2466
fix: requests_aws4auth not being treated as an optional dependency by @LeonLuttenberger in https://github.com/aws/aws-sdk-pandas/pull/2471
fix: KeyError exception in athena wrangler by @rabingaire in https://github.com/aws/aws-sdk-pandas/pull/2483
fix: column names and apply map by @LumberjackUsingMath in https://github.com/aws/aws-sdk-pandas/pull/2492
fix: Gremlin batch size calc by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2496

Documentation 📚

docs: Update layers.rst - add cn-north-1 & cn-northwest-1 by @kukushking in https://github.com/aws/aws-sdk-pandas/pull/2477

New Contributors

@rabingaire made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2483
@frenchytheasian made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2482
@LumberjackUsingMath made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2492

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.4.0...3.4.1

3.4.0

7 months ago

Features/Enhancements 🚀

Geospatial - parse Athena geospatial types via geopandas by @kukushking in #2346
Allow group identifiers to be used in wr.cloudwatch queries by @LeonLuttenberger in #2430
Add ignore null store parquet metadata by @raaidarshad in #2450

Bug fixes 🐛

Add missing boto3 session in athena.to_iceberg wait_query by @jaidisido in #2428
Add catalog ID in athena.to_iceberg by @jaidisido in #2446
Return None for missing column and partition key comment by @robert-schmidtke in #2449
Fix urllib3 error when building AWS Lambda Layers by @LeonLuttenberger in #2447
Duplicate schema argument in wr.s3.to_parquet by @kukushking in #2455

Tests 🧪

Test dependabot groups feature by @jaidisido in #2426

New Contributors

@raaidarshad made their first contribution in https://github.com/aws/aws-sdk-pandas/pull/2450

Full Changelog: https://github.com/aws/aws-sdk-pandas/compare/3.3.0...3.4.0