Deequ Versions Save

What's Changed

• NEW: Exact Quantile Check
  • Creation of Exact Quantile Check by @jmilis2000 in https://github.com/awslabs/deequ/pull/512
• Data Synchronization/Matching fixes
  • Delegate to Spark for checking existence of columns in the given dataframes by @rdsharma26 in https://github.com/awslabs/deequ/pull/515
  • Verify that non key columns exist in each dataset by @rdsharma26 in https://github.com/awslabs/deequ/pull/517
• Addition of tests
  • Test that exceptions within a check's constraints do not affect other… by @tylermcdaniel0 in https://github.com/awslabs/deequ/pull/516

New Contributors

• @jmilis2000 made their first contribution in https://github.com/awslabs/deequ/pull/512
• @tylermcdaniel0 made their first contribution in https://github.com/awslabs/deequ/pull/516

Full Changelog: https://github.com/awslabs/deequ/compare/2.0.5...2.0.6

What's Changed

• Spark 3.4 Update
  • Add Spark 3.4 support by @jhchee in https://github.com/awslabs/deequ/pull/505
  • Update minor version for Spark 3.4 maven release by @eycho-am in https://github.com/awslabs/deequ/pull/513
• NEW: Custom SQL analyzer
  • Custom SQL Analyzer by @mentekid in https://github.com/awslabs/deequ/pull/509
  • Fail when CustomSql has syntax errors by @mentekid in https://github.com/awslabs/deequ/pull/510
  • Fix CustomSQL test syntax by @eycho-am in https://github.com/awslabs/deequ/pull/511
• Analyzer Improvements
  • Allow all DQ constraints to be generated from an Analyzer by @mentekid in https://github.com/awslabs/deequ/pull/508

New Contributors

• @jhchee made their first contribution in https://github.com/awslabs/deequ/pull/505

Full Changelog: https://github.com/awslabs/deequ/compare/2.0.4...2.0.5

What's Changed

• Row-Level Results:
  • MinLength by @eycho-am in https://github.com/awslabs/deequ/pull/465
  • Uniqueness by @eycho-am in https://github.com/awslabs/deequ/pull/471
  • ColumnValues by @zixianzh1 in https://github.com/awslabs/deequ/pull/476
  • ReferentialIntegrity by @rdsharma26 in https://github.com/awslabs/deequ/pull/466
  • [Experimental] DataSynchronization by @rdsharma26 in https://github.com/awslabs/deequ/pull/473
• Referential Integrity:
  • Updated Referential Integrity to support multiple columns by @rdsharma26 in https://github.com/awslabs/deequ/pull/463
• Constraints and Condition Changes:
  • Add population stability index (PSI) to distance methods by @bevhanno in https://github.com/awslabs/deequ/pull/480
  • Fix chi-square test conditions by @bevhanno in https://github.com/awslabs/deequ/pull/482
  • Missing Column Precondition for Compliance Check - issue fix 467 by @samarth-c1 in https://github.com/awslabs/deequ/pull/478
  • Addition of HasMax/HasMin/HasStandardDeviation/HasMean constraint suggestions by @rdsharma26 in https://github.com/awslabs/deequ/pull/489
  • Alternative aggregate functions to calculate histogram values. by @akalotkin in https://github.com/awslabs/deequ/pull/475

New Contributors

• @zixianzh1 made their first contribution in https://github.com/awslabs/deequ/pull/476
• @samarth-c1 made their first contribution in https://github.com/awslabs/deequ/pull/478
• @akalotkin made their first contribution in https://github.com/awslabs/deequ/pull/475

Full Changelog: https://github.com/awslabs/deequ/compare/2.0.3...2.0.4

What's Changed

• Adding chi-square distance method for categorical variables by @bevhanno in https://github.com/awslabs/deequ/pull/444
• [WIP] Row Level Results by @mentekid in https://github.com/awslabs/deequ/pull/451
• [Experimental] Addition of dataset comparison utilities by @rdsharma26 in https://github.com/awslabs/deequ/pull/449

New Contributors

• @rdsharma26 made their first contribution in https://github.com/awslabs/deequ/pull/447
• @bevhanno made their first contribution in https://github.com/awslabs/deequ/pull/444
• @mentekid made their first contribution in https://github.com/awslabs/deequ/pull/451

Full Changelog: https://github.com/awslabs/deequ/compare/2.0.2...2.0.3

Adds Spark 3.3 compatibility.

What's Changed

• Upgrade to Spark 3.3.0 by @eycho-am in https://github.com/awslabs/deequ/pull/442

New Contributors

• @eycho-am made their first contribution in https://github.com/awslabs/deequ/pull/442

Full Changelog: https://github.com/awslabs/deequ/compare/2.0.1...2.0.2

Adds Spark 3.2 compatibility.

Add Spark 3.1 compatibility.

Note: this version is no longer compatible with Spark <=3.0. Use previous versions and branch legacy-spark-3.0 instead.

This release updates the build setup (i.e. the pom.xml and the publishing process) so that the artefacts published to maven can now be imported using maven or sbt. There are four branches associated with this new release:

• for spark 2.2: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.2
• for spark 2.3: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.3
• for spark 2.4: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.4
• for spark 2.5: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.5

Changes to the build setup to support Spark 2.2.x to 2.4.x and 3.0.x. There now is one maven release available per Spark version:

• spark-3.0-scala-2.12
• spark-2.4-scala-2.11
• spark-2.3-scala-2.11
• spark-2.2-scala-2.11