Spacy Transformers Versions Save

• Extend support for transformers up to v4.26.x.

• For fast tokenizers, use the offset mapping provided by the tokenizer (#338).

  Using the offset mapping instead of the heuristic alignment from spacy-alignments resolves unexpected and missing alignments such as those discussed in https://github.com/explosion/spaCy/discussions/6563, https://github.com/explosion/spaCy/discussions/10794 and https://github.com/explosion/spaCy/discussions/12023.

  :warning: Slow and fast tokenizers will no longer give identical results due to potential differences in the alignments between transformer tokens and spaCy tokens. We recommend retraining all models with fast tokenizers for use with spacy-transformers v1.2.

• Serialize the tokenizer use_fast setting (#339).

• Extend support for transformers up to v4.25.x.
• Add support for Python 3.11 (currently limited to linux due to supported platforms for PyTorch v1.13.x).

• Extend support for transformers up to v4.20.x.
• Convert all transformer outputs to XP arrays at once (#330).
• Support alternate model loaders in HFShim and HFWrapper (#332).

• Extend support for transformers up to v4.21.x.
• Support MPS device in HFShim (#328).
• Track seen docs during alignment to improve speed (#337).
• Don't require examples in Transformer.initialize (#341).

• Extend support for transformers up to v4.19.x.
• Fix issue #324: Skip backprop for transformer if not available, for example if the transformer is frozen.

✨ New features and improvements

• Extend support for transformers up to v4.17.x.

👥 Contributors

@adrianeboyd

✨ New features and improvements

• Extend support for transformers up to v4.15.x.

👥 Contributors

@adrianeboyd, @danieldk

✨ New features and improvements

• Extend support for transformers up to v4.12.x.

👥 Contributors

@adrianeboyd

🔴 Bug fixes

• Fix #315: Enable loading of v1.0.x pipelines in windows.

👥 Contributors

@adrianeboyd, @ryndaniels, @svlandeg