An implementation of RNNsearch using TensorFlow
An open source Neural Machine Translation toolkit developed by the NLPLAB of Xiamen University.
This tutorial describes how to train an NMT model on WMT17's EN-DE data using this repository.
You must install TensorFlow (>=1.4.0) first to use this library.
The preprocessed data can be found at here.
python subword-nmt/learn_joint_bpe_and_vocab.py --input corpus.tc.en corpus.tc.de -s 32000 -o bpe32k --write-vocabulary vocab.en vocab.de
python subword-nmt/apply_bpe.py -c bpe32k --vocabulary vocab.en --vocabulary-threshold 50 < corpus.tc.en > corpus.bpe32k.en
python subword-nmt/apply_bpe.py -c bpe32k --vocabulary vocab.de --vocabulary-threshold 50 < corpus.tc.de > corpus.bpe32k.de
python subword-nmt/apply_bpe.py -c bpe32k --vocabulary vocab.en --vocabulary-threshold 50 < newstest2016.tc.en > newstest2016.bpe32k.en
python subword-nmt/apply_bpe.py -c bpe32k --vocabulary vocab.de --vocabulary-threshold 50 < newstest2016.tc.de > newstest2016.bpe32k.de
python subword-nmt/apply_bpe.py -c bpe32k --vocabulary vocab.en --vocabulary-threshold 50 < newstest2017.tc.en > newstest2017.bpe32k.en
PYTHONPATH=/PATH/TO/XMUNMT:$PYTHONPATH
cat corpus.bpe32k.en corpus.bpe32k.de > corpus.bpe32k.all
python XMUNMT/xmunmt/scripts/build_vocab.py corpus.bpe32k.all vocab.shared32k.txt
python XMUNMT/xmunmt/scripts/shuffle_corpus.py --corpus corpus.bpe32k.en corpus.bpe32k.de --seed 1234
python XMUNMT/xmunmt/bin/trainer.py
--model rnnsearch
--output train
--input corpus.bpe32k.en.shuf corpus.bpe32k.de.shuf
--vocabulary vocab.shared32k.txt vocab.shared32k.txt
--validation newstest2016.bpe32k.en
--references newstest2016.bpe32k.de
--parameters=device_list=[0],eval_steps=5000,train_steps=75000,
learning_rate_decay=piecewise_constant,
learning_rate_values=[5e-4,25e-5,125e-6],
learning_rate_boundaries=[25000,50000]
python XMUNMT/xmunmt/bin/translator.py
--models rnnsearch
--checkpoints train/eval
--input newstest2017.bpe32k.en
--output test.txt
--vocabulary vocab.shared32k.txt vocab.shared32k.txt
The benchmark is performed on 1 GTX 1080Ti GPU with default parameters.
Dataset | BLEU | BLEU (cased) |
---|---|---|
WMT17 En-De | 22.81 | 22.30 |
WMT17 De-En | 29.01 | 27.69 |
This code is written by Zhixing Tan. If you have any problems, feel free to send an email.
BSD