This version is built for Hugging Face Transformers v4.39.x.

New

• Add support for QLoRA/ QAdapter training via bitsandbytes (@calpt via #663): Notebook Tutorial
• Add dropout to bottleneck adapters (@calpt via #667)

Changed

• Upgrade supported Transformers version (@lenglaender via #654; @calpt via #686)
• Deprecate Hub repo in docs (@calpt via #668)
• Switch resolving order if source not specified in load_adapter() (@calpt via #681)

Fixed

• Fix DataParallel training with adapters (@calpt via #658)
• Fix embedding Training Bug (@hSterz via #655)
• Fix fp16/ bf16 for Prefix Tuning (@calpt via #659)
• Fix Training Error with AdapterDrop and Prefix Tuning (@TimoImhof via #673)
• Fix default cache path for adapters loaded from AH repo (@calpt via #676)
• Fix skipping composition blocks in not applicable layers (@calpt via #665)
• Fix Unipelt Lora default config (@calpt via #682)
• Fix compatibility of adapters with HF Accelerate auto device-mapping (@calpt via #678)
• Use default head dropout prob if not provided by model (@calpt via #685)

This version is built for Hugging Face Transformers v4.36.x.

New

• Add MT5 support (@sotwi via #629)

Changed

• Upgrade supported Transformers version (@calpt via #617)
• Simplify XAdapterModel implementations (@calpt via #641)

Fixed

• Fix prediction head loading for T5 (@calpt via #640)

This version is built for Hugging Face Transformers v4.35.x.

New

• Add leave_out to LoRA and (IA)³ (@calpt via #608)

Fixed

• Fix error in push_adapter_to_hub() due to deprecated args (@calpt via https://github.com/adapter-hub/adapters/pull/613)
• Fix Prefix-Tuning for T5 models where d_kv != d_model / num_heads (@calpt via https://github.com/adapter-hub/adapters/pull/621)
• [Bart] Move CLS rep extraction from EOS tokens to head classes (@calpt via https://github.com/adapter-hub/adapters/pull/624)
• Fix adapter activation with skip_layers/ AdapterDrop training (@calpt via #634)

Docs & Notebooks

• Update notebooks & add new complex configuration demo notebook (@hSterz & @calpt via https://github.com/adapter-hub/adapters/pull/614)

Blog post: https://adapterhub.ml/blog/2023/11/introducing-adapters/

With the new Adapters library, we fundamentally refactored the adapter-transformers library and added support for new models and adapter methods.

This version is compatible with Hugging Face Transformers version 4.35.2.

For a guide on how to migrate from adapter-transformers to Adapters have a look at https://docs.adapterhub.ml/transitioning.md. Changes are given compared to the latest adapters-transformers v3.2.1.

New Models & Adapter Methods

• Add LLaMA model integration (@hSterz)
• Add X-MOD model integration (@calpt via #581)
• Add Electra model integration (@hSterz via #583, based on work of @amitkumarj441 and @pauli31 in #400)
• Add adapter output & parameter averaging (@calpt)
• Add Prompt Tuning (@lenglaender and @calpt via #595)
• Add Composition Support to LoRA and (IA)³ (@calpt via #598)

Breaking Changes

• Renamed bottleneck adapter configs and config strings. The new names can be found here: https://docs.adapterhub.ml/overview.html (@calpt)
• Removed the XModelWithHeads classes (@lenglaender) (XModelWithHeads have been deprecated since adapter-transformers version 3.0.0)

Changes Due to the Refactoring

• Refactored the implementation of all already supported models (@calpt, @lenglaender, @hSterz, @TimoImhof)
• Separate the model config (PretrainedConfig) from the adapters config (ModelAdaptersConfig) (@calpt)
• Updated the whole documentation, Jupyter Notebooks and example scripts (@hSterz, @lenglaender, @TimoImhof, @calpt)
• Introduced the load_model function to load models containing adapters. This replaces the Hugging Face from_pretrained function used in the adapter-transformers library (@lenglaender)
• Sharing more logic for adapter composition between different composition blocks (@calpt via #591)
• Added Backwards Compatibility Tests which allow for testing if adaptations of the codebase, such as Refactoring, impair the functionality of the library (@TimoImhof via #596)
• Refactored the EncoderDecoderModel by introducing a new mixin (ModelUsingSubmodelsAdaptersMixin) for models that contain other models (@lenglaender)
• Rename the class AdapterConfigBase into AdapterConfig (@hSterz via #603)

Fixes and Minor Improvements

• Fixed EncoderDecoderModel generate function (@lenglaender)
• Fixed deletion of invertible adapters (@TimoImhof)
• Automatically convert heads when loading with XAdapterModel (@calpt via #594)
• Fix training T5 adapter models with Trainer (@calpt via #599)
• Ensure output embeddings are frozen during adapter training (@calpt #537)

This is the last release of adapter-transformers. See here for the legacy codebase: https://github.com/adapter-hub/adapter-transformers-legacy.

Based on transformers v4.26.1

Fixed

• Fix compacter init weights (@hSterz via #516)
• Restore compatibility of GPT-2 weight initialization with Transformers (@calpt via #525)
• Restore Python 3.7 compatibility (@lenglaender via #510)
• Fix LoRA & (IA)³ implementation for Bart & MBart (@calpt via #518)
• Fix resume_from_checkpoint in AdapterTrainer class (@hSterz via #514)

Based on transformers v4.26.1

New

New model integrations

• Add BEiT integration (@jannik-brinkmann via #428, #439)
• Add GPT-J integration (@ChiragBSavani via #426)
• Add CLIP integration (@calpt via #483)
• Add ALBERT integration (@lenglaender via #488)
• Add BertGeneration (@hSterz via #480)

Misc

• Add support for adapter configuration strings (@calpt via #465, #486) This enables you to easily configure adapter configs. To create a Pfeiffer adapter with reduction factor 16 you can know use pfeiffer[reduction_factor=16]. Especially for experiments using different hyperparameters or the example scripts, this can come in handy. Learn more
• Add for Stack, Parallel & BatchSplit composition to prefix tuning (@calpt via #476) In previous adapter-transformers versions, you could combine multiple bottleneck adapters. You could use them in parallel or stack them. Now, this is also possible for prefix-tuning adapters. Add multiple prefixes to the same model to combine the functionality of multiple adapters (Stack) or perform several tasks simultaneously (Parallel, BatchSplit) Learn more
• Enable parallel sequence generation with adapters (@calpt via #436)

Changed

• Removal of the MultiLingAdapterArguments class. Use the AdapterArguments class and setup_adapter_training method instead. Learn more.
• Upgrade of underlying transformers version to 4.26.1 (@calpt via #455, @hSterz via #503)

Fixed

• Fixes for GLUE & dependency parsing example script (@calpt via #430, #454)
• Fix access to shared parameters of compacter (e.g. during sequence generation) (@calpt via #440)
• Fix reference to adapter configs in T5EncoderModel (@calpt via #437)
• Fix DeBERTa prefix tuning with enabled relative attention (@calpt via #451)
• Fix gating for prefix tuning layers (@calpt via #471)
• Fix input to T5 adapter layers (@calpt via #479)
• Fix AdapterTrainer hyperparameter tuning (@dtuit via #482)
• Move loading best adapter to AdapterTrainer class (@MaBeHen via #487)
• Make HuggingFace Hub Mixin work with newer utilities (@Helw150 via #473)
• Only compute fusion reg loss if fusion layer is trained (@calpt via #505)

Based on transformers v4.21.3

New

New adapter methods

• Add LoRA implementation (@calpt via #334, #399): Documentation
• Add (IA)^3 implementation (@calpt via #396): Documentation
• Add UniPELT implementation (@calpt via #407): Documentation

New model integrations

• Add Deberta and DebertaV2 integration(@hSterz via #340)
• Add Vision Transformer integration (@calpt via #363)

Misc

• Add adapter_summary() method (@calpt via #371): More info
• Return AdapterFusion attentions using output_adapter_fusion_attentions argument (@calpt via #417): Documentation

Changed

• Upgrade of underlying transformers version (@calpt via #344, #368, #404)

Fixed

• Infer label names for training for flex head models (@calpt via #367)
• Ensure root dir exists when saving all adapters/heads/fusions (@calpt via #375)
• Avoid attempting to set prediction head if non-existent (@calpt via #377)
• Fix T5EncoderModel adapter integration (@calpt via #376)
• Fix loading adapters together with full model (@calpt via #378)
• Multi-gpu support for prefix-tuning (@alexanderhanboli via #359)
• Fix issues with embedding training (@calpt via #386)
• Fix initialization of added embeddings (@calpt via #402)
• Fix model serialization using torch.save() & torch.load() (@calpt via #406)

Based on transformers v4.17.0

New

• Support float reduction factors in bottleneck adapter configs (@calpt via #339)

Fixed

• [AdapterTrainer] add missing preprocess_logits_for_metrics argument (@stefan-it via #317)
• Fix save_all_adapters such that with_head is not ignored (@hSterz via #325)
• Fix inferring batch size for prefix tuning (@calpt via #335)
• Fix bug when using compacters with AdapterSetup context (@calpt via #328)
• [Trainer] Fix issue with AdapterFusion and load_best_model_at_end (@calpt via #341)
• Fix generation with GPT-2, T5 and Prefix Tuning (@calpt via #343)

Based on transformers v4.17.0

New

Efficient Fine-Tuning Methods

• Add Prefix Tuning (@calpt via #292)
• Add Parallel adapters & Mix-and-Match adapter (@calpt via #292)
• Add Compacter (@hSterz via #297)

Misc

• Introduce XAdapterModel classes as central & recommended model classes (@calpt via #289)
• Introduce ConfigUnion class for flexible combination of adapter configs (@calpt via #292)
• Add AdapterSetup context manager to replace adapter_names parameter (@calpt via #257)
• Add ForwardContext to wrap model forward pass with adapters (@calpt via #267, #295)
• Search all remote sources when passing source=None (new default) to load_adapter() (@calpt via #309)

Changed

• Deprecate XModelWithHeads in favor of XAdapterModel (@calpt via #289)
• Refactored adapter integration into model classes and model configs (@calpt via #263, #304)
• Rename activation functions to match Transformers' names (@hSterz via #298)
• Upgrade of underlying transformers version (@calpt via #311)

Fixed

• Fix seq2seq generation with flexible heads classes (@calpt via #275, @hSterz via #285)
• Parallel composition for XLM-Roberta (@calpt via #305)

Based on transformers v4.12.5

New

• Allow adding, loading & training of model embeddings (@hSterz via #245). See https://docs.adapterhub.ml/embeddings.html.

Changed

• Unify built-in & custom head implementation (@hSterz via #252)
• Upgrade of underlying transformers version (@calpt via #255)

Fixed

• Fix documentation and consistency issues for AdapterFusion methods (@calpt via #259)
• Fix serialization/ deserialization issues with custom adapter config classes (@calpt via #253)