Spark Bigquery Connector Versions Save

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

0.36.2

3 weeks ago
  • Issue #1215: Support predicate pushdown for DATETIME

0.38.0

1 month ago
  • PR #1205: Sending Identity token in the read API header
  • Issue #1195: Support map type with complex value
  • Issue #1215: Support predicate pushdown for DATETIME
  • BigQuery API has been upgraded to version 2.39.0
  • BigQuery Storage API has been upgraded to version 3.5.0
  • GAX has been upgraded to version 2.47.0
  • Arrow has been upgraded to version 16.0.0
  • gRPC has been upgraded to version 1.63.0
  • Netty has been upgraded to version 4.1.109.Final

0.37.0

2 months ago
  • :warning: Starting version 0.38.0 of the connector, the spark-2.4-bigquery version won't be released as Spark 2.4 is well-supported by the spark-bigquery-with-dependencies connectors.
  • PR #1156: Propagate stats for BigLake Managed tables
  • PR #1181: Add caching during protobuf generation
  • PR #1190: Enable connection sharing for atLeastOnce writes
  • Issue #1182: Fix query check logic
  • BigQuery API has been upgraded to version 2.38.1
  • BigQuery Storage API has been upgraded to version 3.3.1
  • GAX has been upgraded to version 2.45.0
  • Arrow has been upgraded to version 15.0.1
  • gRPC has been upgraded to version 1.62.2
  • Netty has been upgraded to version 4.1.107.Final
  • Protocol Buffers has been upgraded to version 3.25.3

0.36.1

4 months ago
  • PR #1176: fix timestamp filter translation issue

0.36.0

4 months ago
  • PR #1155: allow lazy materialization of query on load
  • PR #1163: Added config to set the BigQuery Job timeout
  • PR #1166: Fix filters by adding surrounding parenthesis. Thanks @tom-s-powell !
  • PR #1171: fix read, write issues with Timestamp
  • Issue #1116: BigQuery write fails with MessageSize is too large
  • BigQuery API has been upgraded to version 2.36.0
  • GAX has been upgraded to version 2.40.0
  • gRPC has been upgraded to version 1.61.0
  • Netty has been upgraded to version 4.1.106.Final
  • Protocol Buffers has been upgraded to version 3.25.2

0.35.1

5 months ago
  • PR #1153: allow writing spark string to BQ datetime

0.35.0

5 months ago
  • PR #1115: Added new connector, spark-3.5-bigquery aimed to be used in Spark 3.5. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API.
  • PR #1117: Make read session caching duration configurable
  • PR #1118: Improve read session caching key
  • PR #1122: Set traceId on write
  • PR #1124: Added SparkListenerEvents for Query and Load jobs running on BigQuery
  • PR #1127: Fix job labeling for mixed case Dataproc job names
  • PR #1136: Consider projections for biglake stats
  • PR #1143: Enable async write for default stream
  • BigQuery API has been upgraded to version 2.35.0
  • BigQuery Storage API has been upgraded to version 2.47.0
  • GAX has been upgraded to version 2.38.0
  • gRPC has been upgraded to version 1.60.0
  • Netty has been upgraded to version 4.1.101.Final
  • Protocol Buffers has been upgraded to version 3.25.1

0.34.0

7 months ago
  • PR #1057: Enable async writes for greater throughput
  • PR #1094: CVE-2023-5072: Upgrading the org.json:json dependency
  • PR #1095: CVE-2023-4586: Upgrading the netty dependencies
  • PR #1104: Fixed nested field predicate pushdown
  • PR #1109: Enable read session caching by default for faster Spark planning
  • PR #1111: Enable retry of failed messages
  • Issue #103: Support for Dynamic partition overwrite for time and range partitioned table
  • Issue #1099: Fixing the usage of ExternalAccountCredentials
  • BigQuery API has been upgraded to version 2.33.2
  • BigQuery Storage API has been upgraded to version 2.44.0
  • GAX has been upgraded to version 2.35.0
  • gRPC has been upgraded to version 1.58.0
  • Protocol Buffers has been upgraded to version 3.24.4

0.33.0

7 months ago
  • Added new connector, spark-3.4-bigquery aimed to be used in Spark 3.4 and above. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API.
  • PR #1008: Adding support to expose BigQuery metrics using Spark custom metrics API.
  • PR #1038: Logical plan now shows the BigQuery table of DirectBigQueryRelation. Thanks @idc101 !
  • PR #1058: View names will appear in query plan instead of the materialized table
  • PR #1061: Handle NPE case when reading BQ table with NUMERIC fields. Thanks @hayssams !
  • PR #1069: Support TimestampNTZ datatype in spark 3.4
  • Issue #453: fix comment handling in query
  • Issue #144: allow writing Spark String to BQ TIME type
  • Issue #867: Support writing with RangePartitioning
  • Issue #1046: Add a way to disable map type support
  • Issue #1062: Adding dataproc job ID and UUID labels to BigQuery jobs

0.32.2

9 months ago
  • CVE-2023-34462: Upgrading netty to verision 4.1.96.Final