An Open-Source Framework for Prompt-Learning.
An Open-Source Framework for Prompt-learning.
Overview • Installation • How To Use • Docs • Paper • Citation • Performance •
./tutorial/9_UltraChat.py
.pip install openprompt
Prompt-learning is the latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks, which modifies the input text with a textual template and directly uses PLMs to conduct pre-trained tasks. This library provides a standard, flexible and extensible framework to deploy the prompt-learning pipeline. OpenPrompt supports loading PLMs directly from huggingface transformers. In the future, we will also support PLMs implemented by other libraries. For more resources about prompt-learning, please check our paper list.
Note: Please use Python 3.8+ for OpenPrompt
Our repo is tested on Python 3.8+ and PyTorch 1.8.1+, install OpenPrompt using pip as follows:
pip install openprompt
To play with the latest features, you can also install OpenPrompt from the source.
Clone the repository from github:
git clone https://github.com/thunlp/OpenPrompt.git
cd OpenPrompt
pip install -r requirements.txt
python setup.py install
Modify the code
python setup.py develop
A PromptModel
object contains a PLM
, a (or multiple) Template
and a (or multiple) Verbalizer
, where the Template
class is defined to wrap the original input with templates, and the Verbalizer
class is to construct a projection between labels and target words in the current vocabulary. And a PromptModel
object practically participates in training and inference.
With the modularity and flexibility of OpenPrompt, you can easily develop a prompt-learning pipeline.
The first step is to determine the current NLP task, think about what’s your data looks like and what do you want from the data! That is, the essence of this step is to determine the classes
and the InputExample
of the task. For simplicity, we use Sentiment Analysis as an example. tutorial_task.
from openprompt.data_utils import InputExample
classes = [ # There are two classes in Sentiment Analysis, one for negative and one for positive
"negative",
"positive"
]
dataset = [ # For simplicity, there's only two examples
# text_a is the input text of the data, some other datasets may have multiple input sentences in one example.
InputExample(
guid = 0,
text_a = "Albert Einstein was one of the greatest intellects of his time.",
),
InputExample(
guid = 1,
text_a = "The film was badly made.",
),
]
Choose a PLM to support your task. Different models have different attributes, we encourge you to use OpenPrompt to explore the potential of various PLMs. OpenPrompt is compatible with models on huggingface.
from openprompt.plms import load_plm
plm, tokenizer, model_config, WrapperClass = load_plm("bert", "bert-base-cased")
A Template
is a modifier of the original input text, which is also one of the most important modules in prompt-learning.
We have defined text_a
in Step 1.
from openprompt.prompts import ManualTemplate
promptTemplate = ManualTemplate(
text = '{"placeholder":"text_a"} It was {"mask"}',
tokenizer = tokenizer,
)
A Verbalizer
is another important (but not necessary) in prompt-learning,which projects the original labels (we have defined them as classes
, remember?) to a set of label words. Here is an example that we project the negative
class to the word bad, and project the positive
class to the words good, wonderful, great.
from openprompt.prompts import ManualVerbalizer
promptVerbalizer = ManualVerbalizer(
classes = classes,
label_words = {
"negative": ["bad"],
"positive": ["good", "wonderful", "great"],
},
tokenizer = tokenizer,
)
Given the task, now we have a PLM
, a Template
and a Verbalizer
, we combine them into a PromptModel
. Note that although the example naively combine the three modules, you can actually define some complicated interactions among them.
from openprompt import PromptForClassification
promptModel = PromptForClassification(
template = promptTemplate,
plm = plm,
verbalizer = promptVerbalizer,
)
A PromptDataLoader
is basically a prompt version of pytorch Dataloader, which also includes a Tokenizer
, a Template
and a TokenizerWrapper
.
from openprompt import PromptDataLoader
data_loader = PromptDataLoader(
dataset = dataset,
tokenizer = tokenizer,
template = promptTemplate,
tokenizer_wrapper_class=WrapperClass,
)
Done! We can conduct training and inference the same as other processes in Pytorch.
import torch
# making zero-shot inference using pretrained MLM with prompt
promptModel.eval()
with torch.no_grad():
for batch in data_loader:
logits = promptModel(batch)
preds = torch.argmax(logits, dim = -1)
print(classes[preds])
# predictions would be 1, 0 for classes 'positive', 'negative'
Please refer to our tutorial scripts, and documentation for more details.
We provide a series of download scripts in the dataset/
folder, feel free to use them to download benchmarks.
There are too many possible combinations powered by OpenPrompt. We are trying our best to test the performance of different methods as soon as possible. The performance will be constantly updated into the Tables. We also encourage the users to find the best hyper-parameters for their own tasks and report the results by making pull request.
Major improvement/enhancement in future.
Please cite our paper if you use OpenPrompt in your work
@article{ding2021openprompt,
title={OpenPrompt: An Open-source Framework for Prompt-learning},
author={Ding, Ning and Hu, Shengding and Zhao, Weilin and Chen, Yulin and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong},
journal={arXiv preprint arXiv:2111.01998},
year={2021}
}
We thank all the contributors to this project, more contributors are welcome!