Multilingual Nmt Save

Multilingual Translation

This codebase was used for the multilingual translation experiments for the paper "Parameter Sharing Methods for Multilingual Self-Attentional Translation Models, WMT-EMNLP 2018".

The multilingual model is based on the Transformer model and also contains the following features:

• positional encoding
• multi-head dot-product attention
• label smoothing
• warm-up steps based training of Adam optimizer
• shared weights of the embedding and softmax layers
• beam search with length normalization
• exponential moving average checkpoint of parameters

Requirements

One can install the required packages from the requirements file.

pip install -r requirements.txt

Dataset

• Download the TED talks dataset as:

bash download_teddata.sh

This command will download, decompress, and will save the train, dev, and test splits of the TED talks under data directory.

• One can use the script ted_reader.py to specify language pairs for both bilingual/multilingual translation tasks.

• For bilingual/multilingual translation, just specify the source and target languages as

python ted_reader.py -s ja en zh fr ro -t en zh fr ro ja

• For multilingual translation, by default the training data will consist of the cartesian product of all the source and target language pairs.
• If all possible combinations of the language pairs are not needed, then just use the option of -ncp

python ted_reader.py -s ja en zh fr ro -t en zh fr ro ja -ncp

• Above command will only create training data for the corresponding language pairs, i.e. [(ja, en), (en, zh), (zh, fr), (fr, ro), (fr, ja)]

• For evaluating the multiingual model, one can generate the test set for each bilingual pair using the above command.

Instructions

For convenience, there are some example shell scripts under tools directory

• Bilingual Translation (NS)

bash tools/bpe_pipeline_bilingual.sh src_lang tgt_lang

• Fully Shared Multilingual Translation (FS)

bash tools/bpe_pipeline_fully_shared_multilingual.sh src_lang tgt_lang1 tgt_lang2 

• Partial Sharing Multilingual Translation (PS)

bash tools/bpe_pipeline_MT.sh src_lang tgt_lang1 tgt_lang2 share_sublayer share_attn

An example of sharing the Key(k), Query(q) in both the attention layers (Self, Source)

bash tools/bpe_pipeline_MT.sh src_lang tgt_lang1 tgt_lang2 k,q self,source

Experiments

• Dataset Statistics

• Bilingual Translation Tasks

• Multilingual Translation Tasks

Citation

If you find this code useful, please consider citing our paper as:

@InProceedings{devendra2018multilingual,
  author = 	"Sachan, Devendra
		and Neubig, Graham,
  title = 	"Parameter Sharing Methods for Multilingual Self-Attentional Translation Models",
  booktitle = 	"Proceedings of the Third Conference on Machine Translation",
  year = 	"2018",
  publisher = 	"Association for Computational Linguistics",
  location = 	"Brussels, Belgium"
}