• Multistep Adam Optimizer many thanks to @AgoloCuongHoang for contributing in #1773 !
• Residual Shuffle-Exchange Network thanks to @EmilsOzolins in #1805 !
• Not pinning the gym version.

Added basic support for TF2 modeling in f65b5e4e0be50b284f9b21d56d3d2a46792cdecf thanks to @rjpower !

Other misc fixes:

• Fixing feature encoder for tf.string variable length features.
• adding hparam to make encoder self-attention optional.
• Documentation update, thanks @w-hat

PRs:

• #1788 by @mzilinec adding an option to select different TPU zone.

Some more code cleanups, regarding tf.compat.v1

• Flush out some more contrib remnants.

• Some changes to handle 1.x to 2.x for tf contrib
• TODO(afrozm): Write more

Some changes needed to be able to import problems with TF 2.0

• Move away from tf.flags to absl-py's flags.
• Move away from std::string to tensorflow::string

Final T2T major release

It is now in maintenance mode — we keep it running and welcome bug-fixes, but encourage users to use the successor library Trax.

PRs Merged

• #1724 by @Separius - use batch_size in _test_img2img_transformer thanks!
• #1726 by @senarvi - Fix decoding in prepend mode thanks!
• #1733 by @prasastoadi - En-Id untokenized parallel corpora thanks!
• #1748 by @gabegrand adding a Text2RealProblem class -- thanks a lot @gabegrand

Bug Fixes

• Fix features and decoding on TPUs by @mts42000
• @iansimon and Kristy Choi around shape assertions and modalities
• @superbobry fixed cases where tf.TensorShape was constructed with float dimensions

Misc

• Trax was moved into its own repo: https://github.com/google/trax

PRs Merged

• #1720 thanks @przemb
• #1698 #1699 test/util file fixes thanks to @Vooblin
• Fix serving response from Cloud ML Engine (#1688) thanks to @evalphobia
• Refine automatic mixed precision support via hyper param (#1681) thanks @vinhngx
• correct return shape of rel_pos2abs_pos() (#1686) thanks to @Separius
• save attention weights for relative attention v2 (#1682) thanks to @Ghostvv
• Update generator_utils.py (#1674) thanks to @TanguyUrvoy

Docs

• Transformer tutorial (#1675) many thanks to @Styleoshin

Problems

• 4 new dialog problems by @ricsinaruto in #1642

Models

• Extend NeuralStack to support Dequeu by reading/writing in both directions, thanks @narphorium

TRAX

• Lots of work on SimPLe tuning hyperparameters by @koz4k , @lukaszkaiser and @afrozenator
• async data collection for RL in TRAX
• New memory efficient Transformer using Reversible layers, thanks to Nikita Kitaev, @lukaszkaiser and Anselm Levskaya
• Losses and metrics are layers now in trax, thanks to @lukaszkaiser
• Activations in TRAX thanks to @joaogui1 in #1684 and #1666

Models / Layers:

• NeuralStack and NeuralQueue added, in https://github.com/tensorflow/tensor2tensor/commit/838aca4960f851cd759307481ea904038c1a1ab5 - thanks @narphorium !
• Open Sourcing the Search Space used in EvolvedTransformer - https://github.com/tensorflow/tensor2tensor/commit/4ce366131ce69d1005f035e14677609f7dfdb580
• Masked local n-D attention added in - https://github.com/tensorflow/tensor2tensor/commit/2da59d24eb9367cbed20c98df559beccd11b7582

Problems:

• Add English-Spanish translation problem (#1626) thanks @voluntadpear !
• MovingMNist added in https://github.com/tensorflow/tensor2tensor/commit/121ee60a3b57a092264aa5b5bf69ad194cafb118 thanks @MechCoder !

Bug Fixes:

• Loss twice multiplied with loss_coef (#1627) by @davidmrau - thanks a lot David!
• Fix log_prob accumulation during decoding, thanks @lmthang !
• Fixed high usage of TPU HBM "Arguments" during serving in https://github.com/tensorflow/tensor2tensor/commit/d38f3435ded822e585d1fc7136f3ece857a41c8d thanks @ziy !
• Should not generate summary during decoding in dot_product_relative_atention (#1618) thanks @phamthuonghai !

Misc changes:

• Implement sequence packing as a tf.data.Dataset transformation - https://github.com/tensorflow/tensor2tensor/commit/560c008f7d87502174765fac5ae3d822bbf6b243 thanks @robieta !
• Lots of work on t2t_distill and model exporting by @ziy - thanks @ziy !

RL:

Introduce Rainbow. (#1607) by @konradczechowski in #1607 Changes to MBRL by @konradczechowski , @koz4k in multiple PRs.

PRs:

• Adding automatic mixed precision support (#1637) thanks a lot to @vinhngx !
• Documentation for creating own model #1589 thanks @hbrylkowski !
• Adding extra linear to semantic hashing discretization bottleneck. #1578 thanks @martiansideofthemoon !
• Using partial targets at inference time. (#1596) thanks @EugKar !
• Updated link to DeepMind Math dataset (#1583) thanks @MaxSobolMark !
• Only strip end of line (#1577) thanks @funtion !
• correct typo in add_timing_signal_nd (#1651) many thanks to @Separius !
• fix decode bug (#1645) many thanks to @dong-s !
• Change confusing function name (#1669) thanks @lazylife7157 !

TRAX:

Base

• Forked optimizers from JAX and make them objects in https://github.com/tensorflow/tensor2tensor/commit/1c7c10c60abc31308b40ae6c850e5c9e363dd4a9
• Trax layers are now stateful and support custom gradients.
• Multi-device capability added.
• Memory efficient trainer added in https://github.com/tensorflow/tensor2tensor/commit/b2615aab938af99418ac0d1318338bf3030357fa ! Thanks Nikita Kitaev!
• Adafactor optimizer added in TRAX - https://github.com/tensorflow/tensor2tensor/commit/63c015f964c1166d181d8efd232abd856574fd83
• Demo Colab added in https://github.com/tensorflow/tensor2tensor/commit/cec26dbd782ea7e4c07377e8d1f9391eb0c5a65c thanks @levskaya
• Demo colab for trax layers - https://github.com/tensorflow/tensor2tensor/commit/7632ed01e739cd124c8bac85f121f0f49ddd86cf
• Transformer, TransformerLM, Reversible Transformer, PositionLookupTransformer and Resnet50 are some of the models that TRAX now supports.

RL

• Many PPO changes to be able to work on Atari.
• Distributed PPO where the envs can run in multiple parallel machines using gRPC
• SimulatedEnvProblem by @koz4k - a gym env that simulates a step taken by a trainer of a Neural Network in https://github.com/tensorflow/tensor2tensor/commit/2c761783a7aacd6800d445d10ad3676a56365514
• Implement SerializedSequenceSimulatedEnvProblem by @koz4k

• https://github.com/tensorflow/tensor2tensor/commit/f7f8549a6421723154b366996b2c6559048ac3fb

• Transformer can be used as a policy now, thanks to @koz4k in https://github.com/tensorflow/tensor2tensor/commit/33783fd63bd0debe2138c5569698b31d9af350f6 !