Apache Beam is a unified programming model for Batch and Streaming data processing.
We are happy to present the new 2.55.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.55.0, check out the detailed release notes.
--enableBundling
pipeline option for BigQueryIO DIRECT_READ is replaced by --enableStorageReadApiV2
. Both were considered experimental and subject to change (Java) (#26354).sdks/java/fn-execution
and runners/core-construction-java
into the main SDK. These artifacts were never meant for users, but noting
that they no longer exist. These are steps to bring portability into the core SDK alongside all other core functionality.--max_cache_memory_usage_mb
pipeline option from 100 to 0. This option was first introduced in the 2.52.0 SDK version. This change restores the behavior of the 2.51.0 SDK, which does not use the state cache. If your pipeline uses iterable side inputs views, consider increasing the cache size by setting the option manually. (#30360).SpannerIO.readChangeStream
to support propagating credentials from pipeline options
to the getDialect
calls for authenticating with Spanner (Java) (#30361).According to git shortlog, the following people contributed to the {$RELEASE_VERSION} release. Thank you to all contributors!
Ahmed Abualsaud
Anand Inguva
Andrew Crites
Andrey Devyatkin
Arun Pandian
Arvind Ram
Chamikara Jayalath
Chris Gray
Claire McGinty
Damon Douglas
Dan Ellis
Danny McCormick
Daria Bezkorovaina
Dima I
Edward Cui
Ferran Fernández Garrido
GStravinsky
Jan Lukavský
Jason Mitchell
JayajP
Jeff Kinard
Jeffrey Kinard
Kenneth Knowles
Mattie Fu
Michel Davit
Oleh Borysevych
Ritesh Ghorse
Ritesh Tarway
Robert Bradshaw
Robert Burke
Sam Whittle
Scott Strong
Shunping Huang
Steven van Rossum
Svetak Sundhar
Talat UYARER
Ukjae Jeong (Jay)
Vitaly Terentyev
Vlado Djerek
Yi Hu
akashorabek
case-k
clmccart
dengwe1
dhruvdua
hardshah
johnjcasey
liferoad
martin trieu
tvalentyn
We are happy to present the new 2.54.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.54.0, check out the detailed release notes.
According to git shortlog, the following people contributed to the 2.54.0 release. Thank you to all contributors!
Ahmed Abualsaud
Alexey Romanenko
Anand Inguva
Andrew Crites
Arun Pandian
Bruno Volpato
caneff
Chamikara Jayalath
Changyu Li
Cheskel Twersky
Claire McGinty
clmccart
Damon
Danny McCormick
dependabot[bot]
Edward Cheng
Ferran Fernández Garrido
Hai Joey Tran
hugo-syn
Issac
Jack McCluskey
Jan Lukavský
JayajP
Jeffrey Kinard
Jerry Wang
Jing
Joey Tran
johnjcasey
Kenneth Knowles
Knut Olav Løite
liferoad
Marc
Mark Zitnik
martin trieu
Mattie Fu
Naireen Hussain
Neeraj Bansal
Niel Markwick
Oleh Borysevych
pablo rodriguez defino
Rebecca Szper
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Sam Whittle
Shunping Huang
Svetak Sundhar
S. Veyrié
Talat UYARER
tvalentyn
Vlado Djerek
Yi Hu
Zechen Jian
We are happy to present the new 2.53.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.53.0, check out the detailed release notes.
collections.abc.Collections
types properly. Some type hints that were erroneously allowed by the SDK may now fail. (#29272)According to git shortlog, the following people contributed to the 2.53.0 release. Thank you to all contributors!
Ahmed Abualsaud
Ahmet Altay
Alexey Romanenko
Anand Inguva
Arun Pandian
Balázs Németh
Bruno Volpato
Byron Ellis
Calvin Swenson Jr
Chamikara Jayalath
Clay Johnson
Damon
Danny McCormick
Ferran Fernández Garrido
Georgii Zemlianyi
Israel Herraiz
Jack McCluskey
Jacob Tomlinson
Jan Lukavský
JayajP
Jeffrey Kinard
Johanna Öjeling
Julian Braha
Julien Tournay
Kenneth Knowles
Lawrence Qiu
Mark Zitnik
Mattie Fu
Michel Davit
Mike Williamson
Naireen
Naireen Hussain
Niel Markwick
Pablo Estrada
Radosław Stankiewicz
Rebecca Szper
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Sam Rohde
Sam Whittle
Shunping Huang
Svetak Sundhar
Talat UYARER
Tom Stepp
Tony Tang
Vlado Djerek
Yi Hu
Zechen Jiang
clmccart
damccorm
darshan-sj
gabry.wu
johnjcasey
liferoad
lrakla
martin trieu
tvalentyn
We are happy to present the new 2.52.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.52.0, check out the detailed release notes.
beam-sdks-java-extensions-avro
instead. This will allow to easily update Avro version in user code without
potential breaking changes in Beam "core" since the Beam Avro extension already supports the latest Avro versions and
should handle this. (#25252).UseDataStreamForBatch
pipeline option to the Flink runner. When it is set to true, Flink runner will run batch
jobs using the DataStream API. By default the option is set to false, so the batch jobs are still executed
using the DataSet API.upload_graph
as one of the Experiments options for DataflowRunner is no longer required when the graph is larger than 10MB for Java SDK (PR#28621.--max_cache_memory_usage_mb=X
to provide cache size for the user state API and side inputs. (Python) (#28770).org.apache.beam.sdk.io.CountingSource.CounterMark
uses custom CounterMarkCoder
as a default coder since all Avro-dependent
classes finally moved to extensions/avro
. In case if it's still required to use AvroCoder
for CounterMark
, then,
as a workaround, a copy of "old" CountingSource
class should be placed into a project code and used directly
(#25252).host
to firestoreHost
in FirestoreOptions
to avoid potential conflict of command line arguments (Java) (#29201).watch_file_pattern
arg of the RunInference arg had no effect prior to 2.52.0. To use the behavior of arg watch_file_pattern
prior to 2.52.0, follow the documentation at https://beam.apache.org/documentation/ml/side-input-updates/ and use WatchFilePattern
PTransform as a SideInput. (#28948)MLTransform
doesn't output artifacts such as min, max and quantiles. Instead, MLTransform
will add a feature to output these artifacts as human readable format - #29017. For now, to use the artifacts such as min and max that were produced by the eariler MLTransform
, use read_artifact_location
of MLTransform
, which reads artifacts that were produced earlier in a different MLTransform
(#29016)According to git shortlog, the following people contributed to the 2.52.0 release. Thank you to all contributors!
Ahmed Abualsaud Ahmet Altay Aleksandr Dudko Alexey Romanenko Anand Inguva Andrei Gurau Andrey Devyatkin BjornPrime Bruno Volpato Bulat Chamikara Jayalath Damon Danny McCormick Devansh Modi Dominik Dębowczyk Ferran Fernández Garrido Hai Joey Tran Israel Herraiz Jack McCluskey Jan Lukavský JayajP Jeff Kinard Jeffrey Kinard Jiangjie Qin Jing Joar Wandborg Johanna Öjeling Julien Tournay Kanishk Karanawat Kenneth Knowles Kerry Donny-Clark Luís Bianchin Minbo Bae Pranav Bhandari Rebecca Szper Reuven Lax Ritesh Ghorse Robert Bradshaw Robert Burke RyuSA Shunping Huang Steven van Rossum Svetak Sundhar Tony Tang Vitaly Terentyev Vivek Sumanth Vlado Djerek Yi Hu aku019 brucearctor caneff damccorm ddebowczyk92 dependabot[bot] dpcollins-google edman124 gabry.wu illoise johnjcasey jonathan-lemos kennknowles liferoad magicgoody martin trieu nancyxu123 pablo rodriguez defino tvalentyn
We are happy to present the new 2.51.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.51.0, check out the detailed release notes.
mypy
on user pipelines (#27906)t reflect.Type
from parquetio.Write
. The element type is derived from the input PCollection (Go) (#28490)fastavro
dependency to 1.8.3
or earlier: #28811
According to git shortlog, the following people contributed to the 2.50.0 release. Thank you to all contributors!
Adam Whitmore
Ahmed Abualsaud
Ahmet Altay
Aleksandr Dudko
Alexey Romanenko
Anand Inguva
Andrey Devyatkin
Arvind Ram
Arwin Tio
BjornPrime
Bruno Volpato
Bulat
Celeste Zeng
Chamikara Jayalath
Clay Johnson
Damon
Danny McCormick
David Cavazos
Dip Patel
Hai Joey Tran
Hao Xu
Haruka Abe
Jack Dingilian
Jack McCluskey
Jeff Kinard
Jeffrey Kinard
Joey Tran
Johanna Öjeling
Julien Tournay
Kenneth Knowles
Kerry Donny-Clark
Mattie Fu
Melissa Pashniak
Michel Davit
Moritz Mack
Pranav Bhandari
Rebecca Szper
Reeba Qureshi
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Ruwann
Ryan Tam
Sam Rohde
Sereana Seim
Svetak Sundhar
Tim Grein
Udi Meiri
Valentyn Tymofieiev
Vitaly Terentyev
Vlado Djerek
Xinyu Liu
Yi Hu
Zbynek Konecny
Zechen Jiang
bzablocki
caneff
dependabot[bot]
gDuperran
gabry.wu
johnjcasey
kberezin-nshl
kennknowles
liferoad
lostluck
magicgoody
martin trieu
mosche
olalamichelle
tvalentyn
xqhu
Łukasz Spyra
We are happy to present the new 2.50.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.50.0, check out the detailed release notes.
--staging_location
at pipeline submission. Custom container images that are not based on Beam's default image must include Apache Beam installation.(#26996)func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }
direct_num_workers!=1
.(#27373)According to git shortlog, the following people contributed to the 2.50.0 release. Thank you to all contributors!
Abacn
acejune
AdalbertMemSQL
ahmedabu98
Ahmed Abualsaud
al97
Aleksandr Dudko
Alexey Romanenko
Anand Inguva
Andrey Devyatkin
Anton Shalkovich
ArjunGHUB
Bjorn Pedersen
BjornPrime
Brett Morgan
Bruno Volpato
Buqian Zheng
Burke Davison
Byron Ellis
bzablocki
case-k
Celeste Zeng
Chamikara Jayalath
Clay Johnson
Connor Brett
Damon
Damon Douglas
Dan Hansen
Danny McCormick
Darkhan Nausharipov
Dip Patel
Dmytro Sadovnychyi
Florent Biville
Gabriel Lacroix
Hai Joey Tran
Hong Liang Teoh
Jack McCluskey
James Fricker
Jeff Kinard
Jeff Zhang
Jing
johnjcasey
jon esperanza
Josef Šimánek
Kenneth Knowles
Laksh
Liam Miller-Cushon
liferoad
magicgoody
Mahmud Ridwan
Manav Garg
Marco Vela
martin trieu
Mattie Fu
Michel Davit
Moritz Mack
mosche
Peter Sobot
Pranav Bhandari
Reeba Qureshi
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
RyuSA
Saba Sathya
Sam Whittle
Steven Niemitz
Steven van Rossum
Svetak Sundhar
Tony Tang
Valentyn Tymofieiev
Vitaly Terentyev
Vlado Djerek
Yichi Zhang
Yi Hu
Zechen Jiang
We are happy to present the new 2.49.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.49.0, check out the detailed release notes.
BigtableIO.ReadChangeStream
(#27183).--prebuild_sdk_container_engine=cloud_build
, like images depending on tensorflow
or torch
(#27023).pip
cache when installing packages on the workers. This reduces the size of prebuilt Python container images (#27035).NullPointerException
when a progress check is made before the reader is started (IO) (#23868)According to git shortlog, the following people contributed to the 2.49.0 release. Thank you to all contributors!
Abzal Tuganbay
AdalbertMemSQL
Ahmed Abualsaud
Ahmet Altay
Alan Zhang
Alexey Romanenko
Anand Inguva
Andrei Gurau
Arwin Tio
Bartosz Zablocki
Bruno Volpato
Burke Davison
Byron Ellis
Chamikara Jayalath
Charles Rothrock
Chris Gavin
Claire McGinty
Clay Johnson
Damon
Daniel Dopierała
Danny McCormick
Darkhan Nausharipov
David Cavazos
Dip Patel
Dmitry Repin
Gavin McDonald
Jack Dingilian
Jack McCluskey
James Fricker
Jan Lukavský
Jasper Van den Bossche
John Casey
John Gill
Joseph Crowley
Kanishk Karanawat
Katie Liu
Kenneth Knowles
Kyle Galloway
Liam Miller-Cushon
MakarkinSAkvelon
Masato Nakamura
Mattie Fu
Michel Davit
Naireen Hussain
Nathaniel Young
Nelson Osacky
Nick Li
Oleh Borysevych
Pablo Estrada
Reeba Qureshi
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Rouslan
Saadat Su
Sam Rohde
Sam Whittle
Sanil Jain
Shunping Huang
Smeet nagda
Svetak Sundhar
Timur Sultanov
Udi Meiri
Valentyn Tymofieiev
Vlado Djerek
WuA
XQ Hu
Xianhua Liu
Xinyu Liu
Yi Hu
Zachary Houfek
alexeyinkin
bigduu
bullet03
bzablocki
jonathan-lemos
jubebo
magicgoody
ruslan-ikhsan
sultanalieva-s
vitaly.terentyev
We are happy to present the new 2.48.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.48.0, check out the detailed release notes.
Note: The release tag for Go SDK for this release is sdks/v2.48.2 instead of sdks/v2.48.0 because of incorrect commit attached to the release tag sdks/v2.48.0.
According to git shortlog, the following people contributed to the 2.48.0 release. Thank you to all contributors!
Abzal Tuganbay
Ahmed Abualsaud
Alexey Romanenko
Anand Inguva
Andrei Gurau
Andrey Devyatkin
Balázs Németh
Bazyli Polednia
Bruno Volpato
Chamikara Jayalath
Clay Johnson
Damon
Daniel Arn
Danny McCormick
Darkhan Nausharipov
Dip Patel
Dmitry Repin
George Novitskiy
Israel Herraiz
Jack Dingilian
Jack McCluskey
Jan Lukavský
Jasper Van den Bossche
Jeff Zhang
Jeremy Edwards
Johanna Öjeling
John Casey
Katie Liu
Kenneth Knowles
Kerry Donny-Clark
Kuba Rauch
Liam Miller-Cushon
MakarkinSAkvelon
Mattie Fu
Michel Davit
Moritz Mack
Nick Li
Oleh Borysevych
Pablo Estrada
Pranav Bhandari
Pranjal Joshi
Rebecca Szper
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Rouslan
RuiLong J
RyujiTamaki
Sam Whittle
Sanil Jain
Svetak Sundhar
Timur Sultanov
Tony Tang
Udi Meiri
Valentyn Tymofieiev
Vishal Bhise
Vitaly Terentyev
Xinyu Liu
Yi Hu
bullet03
darshan-sj
kellen
liferoad
mokamoka03210120
psolomin
We are happy to present the new 2.47.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.47.0, check out the detailed release notes.
df | (Transform1 | Transform2 | ...)
to avoid excessive conversions.)protobuf <4.23.0
(#24599)key
and reverse
. (#25888).According to git shortlog, the following people contributed to the 2.47.0 release. Thank you to all contributors!
Ahmed Abualsaud
Ahmet Altay
Alexey Romanenko
Amir Fayazi
Amrane Ait Zeouay
Anand Inguva
Andrew Pilloud
Andrey Kot
Bjorn Pedersen
Bruno Volpato
Buqian Zheng
Chamikara Jayalath
ChangyuLi28
Damon
Danny McCormick
Dmitry Repin
George Ma
Jack Dingilian
Jack McCluskey
Jasper Van den Bossche
Jeremy Edwards
Jiangjie (Becket) Qin
Johanna Öjeling
Juta Staes
Kenneth Knowles
Kyle Weaver
Mattie Fu
Moritz Mack
Nick Li
Oleh Borysevych
Pablo Estrada
Rebecca Szper
Reuven Lax
Reza Rokni
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Saadat Su
Saifuddin53
Sam Rohde
Shubham Krishna
Svetak Sundhar
Theodore Ni
Thomas Gaddy
Timur Sultanov
Udi Meiri
Valentyn Tymofieiev
Xinyu Liu
Yanan Hao
Yi Hu
Yuvi Panda
andres-vv
bochap
dannikay
darshan-sj
dependabot[bot]
harrisonlimh
hnnsgstfssn
jrmccluskey
liferoad
tvalentyn
xianhualiu
zhangskz