Awesome Adapter Resources Save Abandoned
Collection of Tools and Papers related to Adapters (aka Parameter-Efficient Transfer Learning/ Fine-Tuning)
Awesome Adapter Resources
This repository collects important tools and papers related to adapter methods for recent large pre-trained neural networks.
Adapters (aka Parameter-Efficient Transfer Learning (PETL) or Parameter-Efficient Fine-Tuning (PEFT) methods) include various parameter-efficient approaches of adapting large pre-trained models to new tasks.
Content
- Why Adapters?
- Frameworks and Tools
- Surveys
- Natural Language Processing
- Computer Vision
- Audio Processing
- Multi-Modal
- Contributing
Why Adapters?
Large pre-trained (Transformer-based) models have become the foundation of various ML domains in recent years. While the most prevalent method of adapting these models to new tasks involves costly full fine-tuning of all model parameters, a series of parameter-efficient and lightweight alternatives, adapters, have been established in recent time.
Using adapters provides multiple benefits. They are ...
- ... parameter-efficient, i.e. they only update a very small subset (e.g. under 1%) of a model's parameters.
- ... modular, i.e. the updated parameters can be extracted and shared independently of the base model parameters
- ... easy to share and easy to deploy at scale due to their small file sizes. E.g. requiring only ~3MB per task instead of ~500MB for sharing a full model.
- ... often composable, i.e. can be stacked, fused or mixed to leverage their combined knowledge.
- ... often on-par in terms of performance with full fine-tuning.
Frameworks and Tools
-
AdapterHub: A Framework for Adapting Transformers
Conference on Empirical Methods in Natural Language Processing
Jonas Pfeiffer, Andreas Rücklé, Clifton A. Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych (2020)
TLDR
AdaptersHub is proposed, a framework that allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages that enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. -
OpenDelta
-
PEFT: State-of-the-art Parameter-Efficient Fine-Tuning
-
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
arXiv.org
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, R. Lee, Lidong Bing, Soujanya Poria (2023)
TLDR
LLM-Adapters is presented, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks, and provides a promising framework for fine-tuning large LLMs on downstream tasks. -
Alpaca-LoRA
Surveys
-
Modular Deep Learning
arXiv.org
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, E. Ponti (2023)
TLDR
A survey of modular architectures is offered, providing a unified view over several threads of research that evolved independently in the scientific literature, and various additional purposes of modularity are explored, including scaling language models, causal inference, programme induction, and planning in reinforcement learning. -
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
arXiv.org
Vladislav Lialin, Vijeta Deshpande, Anna Rumshisky (2023)
TLDR
A taxonomy that covers a broad range of methods and present a detailed method comparison with a specific focus on real-life efficiency and fine-tuning multibillion-scale language models is provided. -
PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques
arXiv.org
Mohammed Sabry, Anya Belz (2023)
TLDR
A reference framework is presented which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules.
Natural Language Processing
Methods
-
Parameter-Efficient Transfer Learning for NLP
International Conference on Machine Learning
N. Houlsby, A. Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, S. Gelly (2019)
TLDR
To demonstrate adapter's effectiveness, the recently proposed BERT Transformer model is transferred to 26 diverse text classification tasks, including the GLUE benchmark, and adapter attain near state-of-the-art performance, whilst adding only a few parameters per task. -
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
Conference on Empirical Methods in Natural Language Processing
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder (2020)
TLDR
MAD-X is proposed, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations and introduces a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new language. -
AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Conference of the European Chapter of the Association for Computational Linguistics
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych (2020)
TLDR
This work proposes AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks by separating the two stages, i.e., knowledge extraction and knowledge composition, so that the classifier can effectively exploit the representations learned frommultiple tasks in a non-destructive manner. -
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Findings
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, Ming Zhou (2020)
TLDR
K-Adapter is proposed, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion and captures richer factual and commonsense knowledge than RoBERTa. -
Parameter-Efficient Transfer Learning with Diff Pruning
Annual Meeting of the Association for Computational Linguistics
Demi Guo, Alexander M. Rush, Yoon Kim (2020)
TLDR
Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task and scales favorably in comparison to popular pruning approaches. -
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Annual Meeting of the Association for Computational Linguistics
Xiang Lisa Li, Percy Liang (2021)
TLDR
Prefix-tuning is proposed, a lightweight alternative to fine- Tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which is called the prefix. -
The Power of Scale for Parameter-Efficient Prompt Tuning
Conference on Empirical Methods in Natural Language Processing
Brian Lester, Rami Al-Rfou, Noah Constant (2021)
TLDR
This work explores “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “Prompt ensembling.” -
Towards a Unified View of Parameter-Efficient Transfer Learning
International Conference on Learning Representations
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig (2021)
TLDR
This paper re-frame state-of-the-art parameter-efficient transfer learning methods as modifications to specific hidden states in pre-trained models, and defines a set of design dimensions along which different methods vary, achieving comparable results to fine-tuning all parameters on all four tasks. -
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Neural Information Processing Systems
Rabeeh Karimi Mahabadi, James Henderson, Sebastian Ruder (2021)
TLDR
Compacter is proposed, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work, and accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. -
LoRA: Low-Rank Adaptation of Large Language Models
International Conference on Learning Representations
J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Weizhu Chen (2021)
TLDR
Low-Rank Adaptation, or LoRA, is proposed, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. -
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
Annual Meeting of the Association for Computational Linguistics
Rabeeh Karimi Mahabadi, Sebastian Ruder, M. Dehghani, J. Henderson (2021)
TLDR
This paper shows that one can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. -
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer
Conference on Empirical Methods in Natural Language Processing
Alan Ansell, E. Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavas, Ivan Vulic, A. Korhonen (2021)
TLDR
MAD-G is proposed, which contextually generates language adapters from language representations based on typological features and remains competitive with more expensive methods for language-specific adapter training across the board, particularly on the NER task in low-resource African languages. -
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Annual Meeting of the Association for Computational Linguistics
Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg (2021)
TLDR
BitFit is introduced, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified, which shows that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. -
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Neural Information Processing Systems
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel (2022)
TLDR
This paper rigorously compares few-shot ICL and PEFT and demonstrates that the latter offers better accuracy as well as dramatically lower computational costs, and introduces a new PEFT method called (IA)$^3$ that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. -
AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning
arXiv.org
Han Zhou, Xingchen Wan, Ivan Vulic, A. Korhonen (2023)
TLDR
AutoPEFT, a novel framework to traverse this configuration space: it automatically configures multiple PEFT modules via high-dimensional Bayesian optimisation, and shows the resource scalability and task transferability of AutoPEFT-found configurations, outperforming existing PEFT methods on average on the standard GLUE benchmark. -
Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
arXiv.org
Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao (2023)
TLDR
The proposed AdaLoRA adaptively allocates the parameter budget among weight matrices according to their importance score, which allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations.
%5E3-blue){kind=icon}
Analysis and Evaluation
-
Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers
Workshop on Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out
Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, N. Rozanov, Goran Glavavs (2020)
TLDR
A deeper analysis reveals that the adapter-based models substantially outperform BERT on inference tasks that require the type of conceptual knowledge explicitly present in ConceptNet and its corresponding Open Mind Common Sense corpus. -
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
Annual Meeting of the Association for Computational Linguistics
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si (2021)
TLDR
It is demonstrated that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates. -
Robust Transfer Learning with Pretrained Language Models through Adapters
Annual Meeting of the Association for Computational Linguistics
Wenjuan Han, Bo Pang, Y. Wu (2021)
TLDR
This work inserts small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, leading to improved stability and adversarial robustness in transfer learning to various downstream tasks. -
AdapterDrop: On the Efficiency of Adapters in Transformers
Conference on Empirical Methods in Natural Language Processing
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych (2020)
TLDR
This paper proposes AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions and can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. -
What to Pre-Train on? Efficient Intermediate Task Selection
Conference on Empirical Methods in Natural Language Processing
Clifton A. Poth, Jonas Pfeiffer, Andreas Ruckl'e, Iryna Gurevych (2021)
TLDR
This work provides a comprehensive comparison of different methods for efficiently identifying beneficial tasks for intermediate transfer learning, focusing on parameter and computationally efficient adapter settings, highlight different data-availability scenarios, and provide expense estimates for each method. -
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
arXiv.org
M. Vidoni, Ivan Vulic, Goran Glavas (2020)
TLDR
This work proposes orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer that are trained to encode language- and task-specific information that is complementary to the knowledge already stored in the pretrained transformer's parameters. -
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Annual Meeting of the Association for Computational Linguistics
Xiao Liu, Kaixuan Ji, Yicheng Fu, W. Tam, Zhengxiao Du, Zhilin Yang, Jie Tang (2022)
TLDR
The method P-Tuning v2 is an implementation of Deep Prompt Tuning (CITATION) optimized and adapted for NLU and can serve as an alternative to finetuning and a strong baseline for future research. -
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
arXiv.org
Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, Maosong Sun (2022)
TLDR
Though initially proposed as an efficient method to steer large models, some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. -
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Annual Meeting of the Association for Computational Linguistics
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, Madian Khabsa (2021)
TLDR
A unified framework, UniPELT, is proposed, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism, indicating that a mixture of multiple P ELT methods may be inherently more effective than single methods.
Applications
-
Simple, Scalable Adaptation for Neural Machine Translation
Conference on Empirical Methods in Natural Language Processing
Ankur Bapna, N. Arivazhagan, Orhan Firat (2019)
TLDR
The proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model, which adapt the model to multiple individual tasks simultaneously, paving the way towards universal machine translation. -
Monolingual Adapters for Zero-Shot Neural Machine Translation
Conference on Empirical Methods in Natural Language Processing
Jerin Philip, Alexandre Bérard, Matthias Gallé, L. Besacier (2020)
-
UDapter: Language Adaptation for Truly Universal Dependency Parsing
Conference on Empirical Methods in Natural Language Processing
A. Ustun, Arianna Bisazza, G. Bouma, Gertjan van Noord (2020)
TLDR
A novel multilingual task adaptation approach based on recent work in parameter-efficient transfer learning, which allows for an easy but effective integration of existing linguistic typology features into the parsing network, and consistently outperforms strong monolingual and multilingual baselines on both high-resource and low-resource languages. -
Single-dataset Experts for Multi-dataset Question Answering
Conference on Empirical Methods in Natural Language Processing
Dan Friedman, Ben Dodge, Danqi Chen (2021)
TLDR
This work trains a collection of lightweight, dataset-specific adapter modules that share an underlying Transformer model, and finds that these Multi-Adapter Dataset Experts (MADE) outperform all the authors' baselines in terms of in-distribution accuracy, and simple methods based on parameter-averaging lead to better zero-shot generalization and few-shot transfer performance. -
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts
Conference on Empirical Methods in Natural Language Processing
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, Sebastian Ruder (2020)
TLDR
This work proposes a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts and demonstrates that they can yield improvements for low- resource languages written in scripts covered by the pretrained model. -
Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
Conference on Machine Translation
Asa Cooper Stickland, Alexandre Berard, Vassilina Nikoulina (2021)
TLDR
This work study the compositionality of language and domain adapters in the context of Machine Translation, and aims to study parameter-efficient adaptation to multiple domains and languages simultaneously and cross-lingual transfer in domains where parallel data is unavailable for certain language pairs. -
Multilingual Unsupervised Neural Machine Translation with Denoising Adapters
Conference on Empirical Methods in Natural Language Processing
A. Ustun, Alexandre Berard, L. Besacier, Matthias Gallé (2021)
TLDR
This paper proposes to use _denoising adapters_, adapter layers with a denoising objective, on top of pre-trained mBART-50, and shows that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally. -
Efficient Test Time Adapter Ensembling for Low-resource Language Varieties
Conference on Empirical Methods in Natural Language Processing
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, Graham Neubig (2021)
TLDR
This paper finds that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters, and proposes EMA, a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions. -
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
arXiv.org
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, Y. Qiao (2023)
TLDR
A zero-init attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge.
Computer Vision
Methods
-
Learning multiple visual domains with residual adapters
NIPS
Sylvestre-Alvise Rebuffi, Hakan Bilen, A. Vedaldi (2017)
TLDR
This paper develops a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains and introduces the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very differentVisual domains and measures their ability to recognize well uniformly. -
Efficient Parametrization of Multi-domain Deep Neural Networks
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Sylvestre-Alvise Rebuffi, Hakan Bilen, A. Vedaldi (2018)
TLDR
This paper proposes to consider universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters, and shows that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques. -
Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets
arXiv.org
Hao Chen, R. Tao, Han Zhang, Yidong Wang, Weirong Ye, Jindong Wang, Guosheng Hu, M. Savvides (2022)
TLDR
Conv-Adapter, a PET module designed for ConvNets, is light-weight, domain-transferable, and architecture-agnostic with generalized performance on different tasks with comparable or surpasses the performance of full fine-tuning on 23 classification tasks of various domains. -
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Neural Information Processing Systems
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo (2022)
TLDR
AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks.
Audio Processing
Applications
-
Lightweight Adapter Tuning for Multilingual Speech Translation
Annual Meeting of the Association for Computational Linguistics
Hang Le, J. Pino, Changhan Wang, Jiatao Gu, D. Schwab, L. Besacier (2021)
TLDR
This paper proposes a comprehensive analysis of adapters for multilingual speech translation (ST) and shows that adapters can be used to efficiently specialize ST to specific language pairs with a low extra cost in terms of parameters. -
Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition
IEEE International Conference on Acoustics, Speech, and Signal Processing
Bethan Thomas, Samuel Kessler, S. Karout (2022)
TLDR
Adapters are applied to wav2vec 2.0 to reduce the number of parameters required for downstream ASR tasks, and increase scalability of the model to multiple tasks or languages, and support the theory that higher pre-trained layers encode more phonemic information.
Multi-Modal
Methods
-
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Computer Vision and Pattern Recognition
Yi-Lin Sung, Jaemin Cho, Mohit Bansal (2021)
TLDR
The results demonstrate that training the adapter with the weight-sharing technique can match the performance of fine-tuning the entire model, and enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. -
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Neural Information Processing Systems
Yi-Lin Sung, Jaemin Cho, Mohit Bansal (2022)
TLDR
LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections, and achieves higher accuracy than Adapter and LoRA in a low-memory regime. -
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
arXiv.org
Tao Lei, Junwen Bai, Siddhartha Brahma, J. Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang (2023)
TLDR
This work proposes Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency and achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approach with moderate to no accuracy loss and the same parameter efficiency.
Contributing
Contributions of new awesome adapter-related resources are very welcome! Before contributing, make sure to read this repository's contributing guide.
Acknowledgments
Paper metadata is partially retrieved via Semantic Scholar's API. Paper TLDRs are provided by Semantic Scholar's TLDR feature.
Open Source Agenda Badge
