The implementation of DeBERTa
DeBERTa pre-trained models
base is the model with same config as BERT base, e.g. 12 layers, 12 heads, 768 hidden dimension
base_mnli is the base model fine-tuned with MNLI data
large is the model with same config as BERT large, e.g. 24 layers, 16 heads, 1024 hidden dimension
large_mnli is the large model fine-tuned with MNLI data
xlarge is the model with 48 layers, 16 heads, 1024 hidden dimension
xlarge_mnli is the xlarge model fine-tuned with MNLI data
bpe_encoder is the GPT2 vocabulary package