Scio Versions Save

A Scala API for Apache Beam and Google Cloud Dataflow.

v0.13.1

9 months ago

Includes Beam 2.49.0 support.

🚀 Enhancements

📗 Documentation

🌱 Dependency Updates

Full Changelog: https://github.com/spotify/scio/compare/v0.13.0...v0.13.1

v0.13.0

10 months ago

Includes Beam 2.48.0 support.

🔧 Breaking Changes

  • Support is dropped for Elasticsearch 6.
  • Elasticsearch 7 is migrated to new elastic search java client
  • File Taps API changed to accept read params
  • :warning: avro write default suffix is now ".avro" (previously was ""). Extension is not appended automatically anymore.

🚀 Enhancements

  • Clean up Configuration initialization in scio-parquet (#4791) @clairemcginty
  • Support EmptyMatchTreatment on TextIO (#4644) @turb
  • Remove scio-elasticsearch v6 (#4802) @RustedBones
  • Added optional header to saveAsDynamicTextFile (#4463) @ruchirsachdeva
  • CoderTest changes for v0.13.x (#4806) @shnapz
  • Remove deprecated avro specific coder macro (#4807) @RustedBones
  • Relax avro record type bound (#4803) @RustedBones
  • Mark Kryo coders nondeterminisitic (#4814) @kellen
  • Provide module specific kryo coder registrar (#4753) @RustedBones
  • Move TableRow API from core to GCP module (#4821) @RustedBones
  • Add a better worded exception of unsupported SpecificRecord coder case (#4815) @shnapz
  • Factorize scalactic and scalatest versions (#4836) @RustedBones
  • Make ZSTD default compression for Parquet writes (#4726) @clairemcginty
  • Added new JdbcIO read/write params to Scio (#4820) @shnapz
  • Add prefix & suffix param for all IO APIs (#4809) @RustedBones
  • Support EmptyMatchTreatment on JsonIO (#4818) @RustedBones
  • Remove old skewed-join API (#4808) @RustedBones
  • Support missing BQ write parameters (#4810) @RustedBones
  • Use transform finder in JdbcIOTest (#4865) @RustedBones
  • Cleanup context from fanout API (#4873) @RustedBones
  • Move to avro extensions (#4883) @kellen
  • Group grpc transforms (#4879) @RustedBones
  • Safe coder unwrap (#4887) @RustedBones
  • Add helper methods for Parquet ReadFiles transforms (#4801) @clairemcginty

🐛 Bug Fixes

  • Fix site generation (#4891) @RustedBones
  • Fix stray copypaste error (#4892) @kellen
  • Fix spelling in LowPriorityCoderDerivation (#4844) @shnapz
  • Add newline in .sbtopts (#4835) @RustedBones
  • Correct shardNameTemplate in tests (#4817) @kellen
  • Fix build badge (#4798) @RustedBones

📗 Documentation

  • Fix GHA Build Documnetation step (#4833) @RustedBones
  • Fix site settings (#4834) @RustedBones
  • Add new sbt-site-paradox plugin (#4837) @RustedBones
  • Documentation revamp (#4871) @kellen
  • Update doc for 0.13 release (#4893) @RustedBones

🏗️ Build improvements

  • Clean dependencies and add CI checks (#4732) @RustedBones
  • Ignore checkerframework update (#4843) @RustedBones
  • Move scalafix rules in proper package (#4855) @RustedBones
  • Move scalafix rules in proper package (cont'd) (#4856) @RustedBones
  • Sync apache commons lib with beam (#4863) @RustedBones
  • Set io implementation as provided for smb and parquet (#4857) @RustedBones
  • Leverage sbt-project-matrix to test all scalafix rules in isolation (#4874) @RustedBones
  • Fix scio version for scalafix (#4875) @RustedBones
  • Fix semanticdb (#4877) @RustedBones
  • Drop java 8 in CI, set java 11 default, test java 17 (#4747) @RustedBones
  • Fix dependency issues (#4876) @RustedBones

🌱 Dependency Updates

  • Update scala-collection-compat to 2.10.0 (#4790) @scala-steward
  • Update sbt-ci-release to 1.5.12 (#4796) @scala-steward
  • Update elasticsearch-java to 8.7.1 (#4795) @scala-steward
  • Update cassandra-all to 3.11.15 (#4799) @scala-steward
  • Update elasticsearch-java to 8.8.0 (#4822) @scala-steward
  • Update kryo-shaded to 4.0.3 (#4823) @scala-steward
  • Update sbt-explicit-dependencies to 0.3.1 (#4824) @scala-steward
  • Update sbt-ghpages to 0.8.0 (#4825) @scala-steward
  • Update sbt-site to 1.5.0 (#4827) @scala-steward
  • Update sbt, sbt-dependency-tree to 1.8.3 (#4830) @scala-steward
  • Update scalatest to 3.2.16 (#4832) @scala-steward
  • Update scalactic to 3.2.16 (#4831) @scala-steward
  • Update commons-io to 2.12.0 (#4828) @scala-steward
  • Update jedis to 4.4.1 (#4841) @scala-steward
  • Update jakarta.json-api to 2.0.2 (#4839) @scala-steward
  • Update testcontainers-scala-elasticsearch, ... to 0.40.16 (#4838) @scala-steward
  • Update scalafmt-core to 3.7.4 (#4847) @scala-steward
  • Update jakarta.json-api to 2.1.2 (#4845) @scala-steward
  • Update sbt-scoverage to 2.0.8 (#4854) @scala-steward
  • Update sbt to 1.9.0 (#4850) @scala-steward
  • Update sbt-scalafix to 0.11.0 (#4853) @scala-steward
  • Update flink to v1.16 (#4858) @RustedBones
  • Update sbt-jmh to 0.4.5 (#4861) @scala-steward
  • Update scala versions to 2.12.18 and 2.13.11 (#4864) @RustedBones
  • Update jedis to 4.4.2 (#4870) @scala-steward
  • Update sbt-header to 5.10.0 (#4867) @scala-steward
  • Update elasticsearch-java to 8.8.1 (#4866) @scala-steward
  • Update mysql-socket-factory to 1.11.2 (#4826) @scala-steward
  • Update mysql-socket-factory-connector-j-8 to 1.12.0 (#4884) @scala-steward
  • Update jedis to 4.4.3 (#4885) @scala-steward
  • Update testcontainers-scala-elasticsearch, ... to 0.40.17 (#4889) @scala-steward
  • Update scala-collection-compat to 2.11.0 (#4888) @scala-steward
  • Update beam-runners-core-construction-java, ... to 2.48.0 (#4846) @scala-steward
  • Bump JamesIves/github-pages-deploy-action from 4.4.1 to 4.4.2 (#4842) @dependabot

Contributors to this release

@RustedBones, @clairemcginty, @dependabot, @kellen, @ruchirsachdeva, @scala-steward, @turb, and @shnapz

Full Changelog: https://github.com/spotify/scio/compare/v0.12.8...v0.13.0

v0.11.15

1 year ago

This is a patch release for 0.11 adding opt-in logical type support for Parquet reads and writes.

🚀 Enhancements

Full Changelog: https://github.com/spotify/scio/compare/v0.11.14...v0.11.15

v0.12.8

1 year ago

🚀 Enhancements

  • Parquet supports logical types on an opt-in basis (#4772, #4782) @clairemcginty
  • Relax transform coder equality (#4780) @RustedBones
  • Added more examples on Join types (#4769) @shnapz
  • Add java sql Date and Time coder (#4781) @RustedBones
  • Add deduplicate option to all AsyncLookupDoFn impl (#4775) @RustedBones

🐛 Bug Fixes

  • Ensures guava callback to complete chained futures (#4778) @RustedBones
  • Fix skewedJoin chill side with normal join instead of hashJoin (#4787) @RustedBones
  • Set provided name on transforms with side outputs (#4779) @RustedBones

🌱 Dependency Updates

  • Update mysql-connector-java to 8.0.33 (#4786) @scala-steward
  • Update testcontainers-scala-elasticsearch, ... to 0.40.15 (#4785) @scala-steward
  • Update mysql-socket-factory to 1.11.1 (#4776) @scala-steward
  • Sync joda-time with beam (#4771) @RustedBones
  • update elasticsearch7 to 7.17.9 (#4770) @RustedBones
  • Update elasticsearch-java to 8.7.0 (#4760) @scala-steward
  • Update scalafmt-core to 3.7.3 (#4758) @scala-steward
  • Update testcontainers-scala-elasticsearch, ... to 0.40.14 (#4754) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @scala-steward and @shnapz

v0.12.7

1 year ago

🚀 Enhancements

  • Add support for DML statements to MockBigQuery (#4759) @fallonchen
  • Allow F type param in tuple coders (#4767) @RustedBones

🐛 Bug Fixes

  • Generate proper getCoderArguments on tuple coders (#4766) @RustedBones
  • Fix avro coder regression on custom SpecificCoder (#4764) @RustedBones

🏗️ Build Improvements

  • Remove add-to-project GHA due to SSO token issue (#4765) @RustedBones

🌱 Dependency Updates

  • Update sbt-mima-plugin to 1.1.2 (#4761) @scala-steward

Contributors to this release

@RustedBones, @fallonchen and @scala-steward

v0.12.6

1 year ago

This release includes an upgrade to Beam 2.46.

🚀 Enhancements

  • Add fanout for CMS hot key computation in skewed joins (#4752) @RustedBones
  • Delay deprecation of old skewed API (#4755) @RustedBones
  • Expose cache in scio-grpc (#4734) @RustedBones
  • Add magnolify imports to docs for methods using Funnel (#4739) @kellen
  • Distribute CMS hot key set as side input (#4724) @RustedBones
  • Support Parquet in BQ ExtractOps (#4728) @clairemcginty
  • Neo4j parametrized from existing SCollection (#4719) @sumitsu
  • Make SeqExampleSCollectionOps API consistent (#4731) @clairemcginty
  • Add skewed job methods with topN and top percent (#4718) @RustedBones
  • Modernize scalafix rules (#4687) @RustedBones

🐛 Bug Fixes

  • Don't forget to fan out (#4740) @kellen
  • Don't re-use Configuration objects in Parquet SMB (#4727) @clairemcginty
  • apply extendedErrorInfo input parameter (#4723) @farzad-sedghi

🏗️ Build Improvements

  • Fork all tests. Default to java 11 (#4748) @RustedBones
  • Factorize java options and propagate GCP settings (#4746) @RustedBones
  • Add java 17 configuration for integration tests (#4745) @RustedBones
  • Remove test matrix for integration test (#4737) @RustedBones
  • Setup MiMa filter for planed breaking changes (#4733) @RustedBones

🌱 Dependency Updates

  • Update beam to 2.46.0 (#4744) @RustedBones
  • Update mysql-socket-factory to 1.11.0 (#4730) @scala-steward
  • Update circe-core, circe-generic, ... to 0.14.5 (#4735) @scala-steward
  • Bump actions/add-to-project from 0.4.0 to 0.4.1 (#4729) @dependabot
  • Update scalafmt-core to 3.7.2 (#4720) @scala-steward
  • Update elasticsearch-java to 8.6.2 (#4715) @scala-steward

Contributors to this release

@RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @farzad-sedghi, @kellen, @scala-steward and @sumitsu

v0.12.5

1 year ago

This release adds support for Java 17 and contains coder bug fixes. It also includes Beam 2.45.0 dependency update.

🚀 Enhancements

🐛 Bug Fixes

📗 Documentation

🏗️ Build Improvements

🌱 Dependency Updates

Contributors to this release

@RustedBones, @clairemcginty, @farzad-sedghi, @scala-steward and @shnapz

Full Changelog: https://github.com/spotify/scio/compare/v0.12.4...v0.12.5

v0.12.4

1 year ago

🚀 Enhancements

🐛 Bug Fixes

📗 Documentation

🌱 Dependency Updates

Full Changelog: https://github.com/spotify/scio/compare/v0.12.3...v0.12.4

v0.12.3

1 year ago

🐛 Bug Fixes

🌱 Dependency Updates

Full Changelog: https://github.com/spotify/scio/compare/v0.12.2...v0.12.3

v0.12.2

1 year ago

This release contains a bugfix and a bump to Beam 2.44.0.

🚀 Enhancements

🐛 Bug Fixes

📗 Documentation

🌱 Dependency Updates

Full Changelog: https://github.com/spotify/scio/compare/v0.12.1...v0.12.2