mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections. (EMNLP 2022)

https://arxiv.org/abs/2205.12005

Introduction

We presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross modal alignment. To address both problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross modal skip-connections. mPLUG achieves state-of-the-art results on a wide range of vision language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering.

News

• 2023.5.08: Moved from AliceMind repo for further update.
• 2022.8.28: Released mPLUG downstream tasks!

Pre-trained models and datasets

• Pre-trained models

For VQA and image captioning tasks, we do an additional continue pre-training on 4M image-text pairs based mplug.en.large to get mplug.en.large.v2.

• Pre-train Datasets

Results

• Image-text

• Video-text

Requirements

• PyTorch version >= 1.11.0

• Install other libraries via

pip install -r requirements.txt

Pre-training

Comming soon.

Fine-tuning

Download json files of downstream tasks

Visual Question Answering

1. Download VQA v2 dataset and Visual Genome dataset from the original websites VQA 2.0.
2. Download and extract the provided dataset json files.
3. In configs/vqa_mplug_base.yaml, set the paths for the json files and the image paths.
4. Finetune the pre-trained mplug_base or large model using 8 A100 GPUs:

sh scripts/vqa_mplug_base.sh

sh scripts/vqa_mplug_large.sh

5. Evaluate the result using the official evaluation server.

Image Captioning

1. Download COCO Caption dataset from the original websites.
2. Download and extract the provided dataset json files.
3. Download language evalution tool(language_evalution).
4. In configs/caption_mplug_base.yaml, set the paths for the json files and the image paths.
5. Finetune the pre-trained mplug_base or large model using 8 A100 GPUs:

sh scripts/caption_mplug_base.sh

sh scripts/caption_mplug_large.sh

Image-text Retrieval

1. Download MSCOCO or Flickr30k datasets from the original websites.
2. Download and extract the provided dataset json files.
3. In configs/retrieval_flickr30k_mplug_large.yaml or configs/retrieval_coco_mplug_large.yaml, set the paths for the json files and the image path.
4. Finetune the pre-trained checkpoint using 8 A100 GPUs:

sh scripts/retrieval_flickr30k_mplug_large.sh

sh scripts/retrieval_coco_mplug_large.sh

Visual Grounding

1. Download RefCOCO datasets from the original websites.
2. Download and extract the provided dataset json files.
3. In configs/grounding_mplug_large.yaml, set the paths for the json files and the image path. Data preparation can follow TransVG
4. Finetune the pre-trained checkpoint using 8 A100 GPUs:

 sh scripts/grounding_mplug_base.sh 

Zero-shot Video-text Retrieval

1. Download MSRVTT datasets from the original websites.
2. In configs/retrieval_msrvtt_mplug_large.yaml, set the paths for the json files and the video paths.
3. To perform zero-shot evaluation, run：

sh scripts/retrieval_msrvtt_mplug_large.sh

Zero-shot Video Question Answering

1. Download MSRVTT-QA datasets from the original websites.
2. In configs/videoqa_msrvtt_mplug_base.yaml, set the paths for the json files and the video paths.
3. To perform zero-shot evaluation, run：

sh scripts/videoqa_msrvtt_mplug_base.sh

Zero-shot Video Captioning

1. Download VATEX datasets from the original websites.
2. In configs/videocap_vatex_mplug_large.yaml, set the paths for the json files and the video paths.
3. To perform zero-shot evaluation, run：

sh scripts/videocap_vatex_mplug_large.sh

Citation

If you use our work, please cite:

@article{li2022mplug,
  title={mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections},
  author={Li, Chenliang and Xu, Haiyang and Tian, Junfeng and Wang, Wei and Yan, Ming and Bi, Bin and Ye, Jiabo and Chen, Hehong and Xu, Guohai and Cao, Zheng and others},
  journal={arXiv preprint arXiv:2205.12005},
  year={2022}
}

Acknowledgement

The implementation of mPLUG relies on resources from ALBEF, BLIP, and timm. We thank the original authors for their open-sourcing.