Spacy Transformers Versions Save

🔴 Bug fixes

• Fix #309: Fix parameter ordering and defaults for new parameters in TransformerModel architectures.
• Fix #310: Fix config and model issues when replacing listeners.

👥 Contributors

@adrianeboyd, @svlandeg

✨ New features and improvements

• Refactor and improve transformer serialization for better support of inline transformer components and replacing listeners.
• Provide the transformer model output as ModelOutput instead of tuples in TransformerData.model_output and FullTransformerBatch.model_output. For backwards compatibility, the tuple format remains available under TransformerData.tensors and FullTransformerBatch.tensors. See more details in the transformer API docs.
• Add support for transformer_config settings such as output_attentions. Additional output is stored under TransformerData.model_output. More details in the TransformerModel docs.
• Add support for mixed-precision training.
• Improve training speed by streamlining allocations for tokenizer output.
• Extend support for transformers up to v4.11.x.

🔴 Bug fixes

• Fix support for GPT2 models.

⚠️ Backwards incompatibilities

• The serialization format for transformer components has changed in v1.1 and is not compatible with spacy-transformers v1.0.x. Pipelines trained with v1.0.x can be loaded with v1.1.x, but pipelines saved with v1.1.x cannot be loaded with v1.0.x.
• TransformerData.tensors and FullTransformerBatch.tensors return a tuple instead of a list.

👥 Contributors

@adrianeboyd, @bryant1410, @danieldk, @honnibal, @ines, @KennethEnevoldsen, @svlandeg

• Fix copying of grad_factor when replacing listeners.

• Fix replacing listeners: #277
• Require spaCy 3.1.0 or higher

• Extend transformers support to <4.10.0
• Enable pickling of span getters and annotation setters, which is required for multiprocessing with spawn

• Allow spaCy 3.1
• Extend transformers to <4.7.0

✨ New features and improvements

• Add support for transformers v4.3-v4.5
• Add extra for CUDA 11.2

🔴 Bug fixes

• Fix #264, #265: Improve handling of empty docs
• Fix #269: Add trf_data extension in Transformer.__call__ and Transformer.pipe to support distributed processing

👥 Contributors

Thanks to @bryant1410 for the pull requests and contributions!

🔴 Bug fixes

• Fix listener initialization when model width is unset.

This release requires spaCy v3.

✨ New features and improvements

• Rewrite library from scratch for spaCy v3.0.
• Transformer component for easy pipeline integration.
• TransformerListener to share transformer weights between components.
• Built-in registered functions that are available in spaCy if spacy-transformers is installed in the same environment.

📖 Documentation

• Embeddings, Transformers and Transfer Learning: How to use transformers in spaCy
• Training Pipelines and Models: Train and update components on your own data and integrate custom models
• Layers and Model Architectures: Power spaCy components with custom neural networks
• Transformer: Pipeline component API reference
• Transformer architectures: Architectures and registered functions

🌙 This release is a pre-release and requires spaCy v3 (nightly).

✨ New features and improvements

• Add support for Python 3.9
• Add support for transformers v4

🔴 Bug fixes

• Fix #230: Add upstream argument to TransformerListener.v1
• Fix #238: Skip special tokens during alignment
• Fix #246: Raise error if model max length exceeded