Scio Versions Save

A Scala API for Apache Beam and Google Cloud Dataflow.

v0.14.4

2 weeks ago

Includes Beam 2.55.1 support.

πŸš€ Enhancements

  • Robust throwable kryo coder (#5318) @RustedBones
  • Use JacksonJsonpMapper as default for Elasticsearch (#5306) @kellen
  • Allow String key type to transform SMB sources with CharSequence key (#5297) @clairemcginty

πŸ› Bug Fixes

  • [scio-avro] fix: allow conversions field in record (#5332) @RustedBones
  • Add all avro logical type conversions to model (#5301) @RustedBones
  • fix: safer implementation of distinctBy (#5299) @RustedBones

πŸ“œ Scalafix Migrations

  • scalafix: Rule to migrate deprecated query to queryRaw (#5302) @RustedBones

πŸ“— Documentation

  • Prepare release v0.14.4 (#5342) @RustedBones
  • [doc] Fix scaladoc link to class declared in package objects (#5316) @RustedBones
  • Add staging dir to REPL docs (#5308) @kellen

πŸ§ͺ Test Improvements

  • Move test code in respective projects (#5330) @RustedBones
  • [integration] Use literals for fromSchemaFile integration test (#5334) @RustedBones
  • [integration] Fix avro integration test (#5333) @RustedBones

πŸ—οΈ Build Improvements

  • Update sbt-site, sbt-site-paradox to 1.7.0 (#5324) @scala-steward
  • Update sbt-paradox to 0.10.7 (#5325) @scala-steward
  • Update scalafmt-core to 3.8.1 (#5320) @scala-steward
  • Update sbt-buildinfo to 0.12.0 (#5310) @scala-steward
  • Update sbt-assembly to 2.2.0 (#5304) @scala-steward

🌱 Dependency Updates

  • Update scalacheck to 1.17.1 (#5341) @scala-steward
  • Update cassandra-all to 3.11.17 (#5340) @scala-steward
  • Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.0 (#5339) @scala-steward
  • Update pprint to 0.9.0 (#5336) @scala-steward
  • Update scala-collection-compat to 2.12.0 (#5337) @scala-steward
  • Update beam to 2.55.1 (#5322) @RustedBones
  • Update elasticsearch-java to 8.13.2 (#5323) @scala-steward
  • Update neo4j-java-driver to 4.4.15 (#5328) @scala-steward
  • Update magnolify-avro, magnolify-bigtable, ... to 0.7.2 (#5319) @scala-steward
  • Update elasticsearch v7 to 7.17.19 (#5317) @RustedBones
  • Update beam to 2.55 (#5307) @RustedBones
  • Update sbt-site, sbt-site-paradox to 1.6.0 (#5311) @scala-steward
  • Update voyager to 2.0.6 (#5313) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @kellen and @scala-steward

v0.14.3

1 month ago

πŸ› Bug Fixes

  • fix: Execute SmbIO output assertions (#5289) @RustedBones
  • (fix #5290) Support empty input in TransformOverride.ofSource (#5293) @clairemcginty
  • (Fix #5285) Allow String key type to read SMB sources written with CharSequence key type (#5291) @clairemcginty

🌱 Dependency Updates

  • Update jedis to 5.1.2 (#5292) @scala-steward
  • Update neo4j-java-driver to 4.4.14 (#5288) @scala-steward

πŸ—οΈ Build Improvements

  • Update sbt-scalafix to 0.12.0 (#5287) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty and @scala-steward

v0.14.2

2 months ago

Contains a bugfix for duplicated SMB Transforms.

πŸ› Bug Fixes

  • (fix) Remove duplicated transform application (#5283) @RustedBones

πŸš€ Enhancements

  • Make checkVersion more dynamic (#5282) @RustedBones
  • Improve implicit coder not found message (#5281) @RustedBones

πŸ“—Documentation

  • (doc) fix 0.14 migration guide broken links (#5278) @RustedBones

πŸ—οΈ Build Improvements

  • Update scala to 2.13.13 and 2.12.19 (#5276) @RustedBones

Contributors to this release

@RustedBones

v0.14.1

2 months ago

Includes Beam 2.54.0 support.

πŸš€ Enhancements

  • Add layer for low priority coder conflict (#5274) @RustedBones
  • Add support of BIGNUMERIC in BigQueryIO (#5225) @shnapz
  • Change dataflow runner check for parquet splittable do fn (#5264) @RustedBones
  • Make sure Coder.gen produces informative error (#5258) @RustedBones
  • Add default coder for ResourceId (#5244) @clairemcginty
  • Add type bound for parquet-avro SCollection ops (#5230) @RustedBones

πŸ› Bug Fixes

  • (fix) Update SmbIO to support absolute path in test (#5277) @RustedBones
  • Simplify BucketedInput serialization (#5270) @clairemcginty
  • Fix byte[] equality issue for MockByteArraySparkeyReader (#5262) @clairemcginty
  • Fix scio-repl assembly after bream 2.54 upgrade (#5263) @RustedBon
  • Always set Parquet-Avro projection (#5234) @clairemcginty
  • Fix SmbIO's testId naming (#5228) @clairemcginty
  • Sort Path IDs in SmbIO (#5242) @clairemcginty
  • Fix SmbIO side input support (fix #5240) (#5241) @clairemcginty

πŸ“œ Scalafix Migrations

  • Make FixLogicalTypeSupplier more permissive (#5233) @clairemcginty
  • Fix MatchError for FixAvroCoder (#5269) @clairemcginty
  • Broaden use cases to FixAvroCoder (#5267) @clairemcginty
  • Update FixAvroCoder to include SMB Avro reads (#5245) @clairemcginty
  • Fix Scalafix rule for Avro package import (#5239) @clairemcginty
  • Add Scalafix rule for SMB CharSequence key (#5236) @clairemcginty
  • Scalafix: Add Avro coder import for SpecificRecord JobTest IOs (#5237) @clairemcginty
  • Check options in scalafix (#5231) @kellen
  • Inline scalafix (#5229) @kellen

πŸ“— Documentation

  • Update copyright year for scio site (#5261) @RustedBones

πŸ§ͺ Test Improvements

  • Fix populate it data (#5260) @RustedBones
  • Add SortMergeBucketParityIT case for sortMergeTransform (#5224) @clairemcginty

πŸ—οΈ Build Improvements

  • Update sbt-typelevel to 0.6.7 (#5275) @RustedBones
  • Update sbt, sbt-dependency-tree to 1.9.9 (#5272) @scala-steward
  • Update sbt-scoverage to 2.0.11 (#5273) @scala-steward
  • Use released 0.14.0 scio version in scalafix (#5259) @RustedBones
  • Update sbt-paradox-material-theme to 0.7.0 (#5251) @scala-steward
  • Update scalafmt-core to 3.8.0 (#5254) @scala-steward
  • Update scalactic to 3.2.18 (#5253) @scala-steward
  • Update scalatest to 3.2.18 (#5255) @scala-steward
  • Update sbt-scoverage to 2.0.10 (#5256) @scala-steward
  • Cache TensorFlow Metadata proto files (#5246) @Duhemm
  • Fix cache step for scalafix GHA (#5238) @RustedBones
  • Bump release-drafter/release-drafter from 5 to 6 (#5226) @dependabot

🌱 Dependency Updates

  • Update beam to 2.54.0 (#5235) @RustedBones
  • Update elasticsearch-java to 8.12.2 (#5271) @scala-steward
  • Update mssql-jdbc to 12.6.1.jre11 (#5268) @scala-steward
  • Update cloud-sql-connector-jdbc-sqlserver, ... to 1.16.0 (#5249) @scala-steward
  • Update testcontainers-scala-elasticsearch, ... to 0.41.3 (#5248) @scala-steward

Contributors to this release

@Duhemm, @RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @kellen, @scala-steward and @shnapz

v0.14.0

3 months ago

What's Changed

Includes Beam 2.53.0 support.

Breaking Changes

  • avro removed from scio-core. scalafix rules helping: FixAvroCoder, FixAvroSchemasPackage, FixDynamicAvro
  • some avro API changes . scalafix rules helping: FixGenericAvro.
  • fallback kryo coder requires explicit import
  • use of official tensorflow metadata
  • BigQuery error-info and result handling API change
  • sio-smb module not pulling implementation dependencies
  • sio-smb in JobTest expecting SmbIO test input/output

See the Migration Guide for more information.

πŸš€ Enhancements

πŸ› Bug Fixes

πŸ“œ Scalafix Migrations

πŸ“— Documentation

πŸ§ͺ Test Improvements

πŸ—οΈ Build Improvements

πŸ”§ Refactorings

🌱 Dependency Updates

New Contributors

Full Changelog: https://github.com/spotify/scio/compare/v0.13.6...v0.14.0

v0.13.6

5 months ago

Includes Beam 2.52 support.

πŸš€ Enhancements

  • Add a ShardRangeString for sqlserver's uniqueidentifier type (#5062) @avandel

πŸ› Bug Fixes

  • Don't overwrite Configured projection in scio-smb (#5083) @clairemcginty
  • Allow None for JDBC password (#5081) @kellen
  • Fix sending empty request when batch is empty (#5060) @senegalo
  • Exclude deprecated dropwizard artifact (#5052) @RustedBones

πŸ—οΈ Build improvements

  • Reworked excluded libs (#5091) @RustedBones
  • Cleanup mima filters (#5090) @RustedBones
  • Enable jUnitSettings for scio-smb (#5088) @clairemcginty
  • Increase BQ read timeout for integration test (#5087) @RustedBones
  • Update testing dataset (#5086) @RustedBones
  • Enable header plugin in IntegrationTest (#5074) @RustedBones
  • Allow manual release (#5072) @RustedBones
  • Update scalafmt-core to 3.7.17 (#5085) @scala-steward
  • Update sbt-mdoc to 2.5.1 (#5065) @scala-steward
  • Update sbt-assembly to 2.1.4 (#5051) @scala-steward
  • Downgrade socco-ng to 0.1.8. (#5047) @RustedBones
  • Downgrade scala version to last supported socco-ng (#5046) @RustedBones

🌱 Dependency Updates

  • Update beam to 2.52 (#5054) @RustedBones
  • Update mysql-socket-factory-connector-j-8 to 1.15.0 (#5071) @scala-steward
  • Update elasticsearch-java to 8.11.1 (#5070) @scala-steward
  • Update elasticsearch to 7.17.14 (#5059) @RustedBones
  • Update metrics-core to 4.2.22 (#5057) @scala-steward
  • Update elasticsearch-java to 8.11.0 (#5056) @scala-steward
  • Update magnolify-avro, magnolify-bigtable, ... to 0.6.4 (#5049) @scala-steward
  • Update mysql-connector-j to 8.2.0 (#5048) @scala-steward

Contributors to this release

@RustedBones, @avandel, @clairemcginty, @kellen, @scala-steward and @senegalo

v0.13.5

6 months ago

Updates newly introduced scio-extra Voyager experimental API to v2

πŸ—οΈ Build improvements

  • Update scalafmt-core to 3.7.15 (#5044) @scala-steward
  • Update sbt, sbt-dependency-tree to 1.9.7 (#5043) @scala-steward

🌱 Dependency Updates

  • Update voyager to 2.0.2 (#5030) @scala-steward
  • Update jakarta.json-api to 2.1.3 (#5041) @scala-steward
  • Update jedis to 4.4.6 (#5042) @scala-steward

Contributors to this release

@RustedBones and @scala-steward

v0.13.4

6 months ago

Includes Beam 2.51 support.

πŸš€ Enhancements

  • Voyager support in Scio (#4996) @patrickwmcgee
  • Support a transform() API for ScioContext (#5035) @clairemcginty
  • Use readNextFilteredRowGroup instead of readNextRowGroup (#5025) @RustedBones
  • Simplify ParquetBucketMetadata (#5024) @RustedBones
  • Adds GrpcBatchDoFn (#4977) @senegalo

πŸ› Bug Fixes

  • Make macro print debug logs if bigquery.types.debug enabled (#5033) @RustedBones
  • Pass avro reader schema for SMB operation (#5032) @RustedBones
  • Remove TF model from the resource cache before closing (#5011) @jrglee
  • Move log4j-over-slf4 to compile scope (#5018) @RustedBones
  • Restrict implicit dynamic extensions for generic avro SCollection (#5021) @RustedBones
  • Propagate ParquetOutputFormat options to ParquetWriter (#4980) @clairemcginty
  • Route all logging to slf4j (#4981) @RustedBones

πŸ“— Documentation

  • fix some typos (#5039) @vuittont60
  • Fix scaladoc links (#5022) @kellen
  • Update dev docs (#5003) @RustedBones

πŸ—οΈ Build improvements

  • Update sbt-mdoc to 2.4.0 (#5038) @scala-steward
  • Set outputStrategy for run scope (#5019) @RustedBones
  • Update sbt, sbt-dependency-tree to 1.9.6 (#5005) @scala-steward
  • Update sbt-assembly to 2.1.3 (#5007) @scala-steward
  • Update sbt-scalafix to 0.11.1 (#4997) @scala-steward
  • Update sbt-jmh to 0.4.6 (#4992) @scala-steward
  • Update sbt-scalafmt to 2.5.2 (#4989) @scala-steward
  • Update sbt-bloop to 1.5.11 (#4982) @scala-steward
  • Update testcontainers-scala-elasticsearch, ... to 0.41.0 (#4985) @scala-steward
  • Update scalafmt-core to 3.7.14 (#4990) @scala-steward
  • Update scalatest to 3.2.17 (#4991) @scala-steward
  • Update sbt-scoverage to 2.0.9 (#4994) @scala-steward
  • Bump actions/checkout from 3 to 4 (#4978) @dependabot

🌱 Dependency Updates

  • Update beam to version 2.51 (#5023) @RustedBones
  • Update elasticsearch-java to 8.10.4 (#5036) @scala-steward
  • Update mysql-socket-factory-connector-j-8 to 1.14.1 (#5029) @scala-steward
  • Update jedis to 4.4.5 (#5016) @scala-steward
  • Update elasticsearch to v7.17.13 (#5008) @RustedBones
  • Update magnolify-avro, magnolify-bigtable, ... to 0.6.3 (#5000) @scala-steward
  • Update scala-compiler, scala-library, ... to 2.13.12 (#5001) @scala-steward
  • Update cassandra-driver-core to 3.11.5 (#4984) @scala-steward
  • Update scalactic to 3.2.17 (#4988) @scala-steward
  • Update circe-core, circe-generic, ... to 0.14.6 (#4987) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @jrglee, @kellen, @patrickwmcgee, @scala-steward, @senegalo and @vuittont60

v0.13.3

8 months ago

Includes Beam 2.50.0 support.

πŸš€ Enhancements

  • (fix #4970) Default to Parquet-SplittableDoFn if RunnerV2 is enabled (#4973) @clairemcginty

πŸ› Bug Fixes

  • Patch datum factory for specific data in IOs (#4975) @RustedBones
  • Fix typeValidation in TransformOverride (#4967) @RustedBones
  • Exclude logger implementation (#4969) @RustedBones

πŸ“— Documentation

  • Fix broken link (#4974) @saveriogzz
  • Update jdbc doc to new API (#4954) @RustedBones

🌱 Dependency Updates

  • Update beam to v2.50.0 (#4968) @RustedBones
  • Update sbt, sbt-dependency-tree to 1.9.4 (#4965) @scala-steward
  • Update sbt-avro to 3.4.3 (#4961) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @saveriogzz and @scala-steward

v0.13.2

8 months ago

πŸš€ Enhancements

  • Set typed table description when using scio BQ client (#4860) @RustedBones
  • Added abstract JdbcDoFn to execute lookup (#4932) @shnapz

πŸ› Bug Fixes

  • Fix nonworking scalafix (#4931) @kellen
  • Allow tuple coder extraction from wrapped RecordCoder (#4946) @RustedBones
  • Don't re-use ReflectData in AvroCoder (#4951) @clairemcginty

🌱 Dependency Updates

  • Update elasticsearch-java to 8.9.1 (#4956) @scala-steward
  • Update cassandra-all to 3.11.16 (#4958) @scala-steward
  • Update scalafmt-core to 3.7.12 (#4950) @scala-steward
  • Update cassandra-driver-core to 3.11.4 (#4941) @scala-steward
  • Update mysql-socket-factory-connector-j-8 to 1.13.1 (#4933) @scala-steward
  • Update sbt, sbt-dependency-tree to 1.9.3 (#4935) @scala-steward

Full Changelog: https://github.com/spotify/scio/compare/v0.13.1...v0.13.2