Machine Comprehension Train on MSMARCO with S-NET Extraction Modification
Here are some required libraries for training and evaluations.
This repo provides pretrained model and pre-processed validation dataset for testing the performance
Please download pretrained model and
pre-processed data and put them on
the MSMARCO/data
and MSMARCO
root directory respectively, then decompress them at the right places.
The code structure should be like
MSMARCO
├── data
│ ├── elmo_embedding.bin
│ ├── test.tsv
│ ├── vocabs.pkl
│ ├── data.tar.gz
│ └── ... others
├── model
│ ├── pm.model
│ ├── pm.model.ckp
│ └── pm.model_out.json
└── ... others
After decompressing,
cd Evaluation
sh eval.sh
then you should get the generated answer and rough-l score.
Download MSMARCO v1 dataset, GloVe embedding.
cd data
python3.6 download.py v1
Convert raw data to tsv format
python3.6 convert_msmarco.py v1 --threads=`nproc`
Convert tsv format to ctf(CNTK input) format and build vocabs dictionary
python3.6 tsv2ctf.py
Generate elmo embedding
sh elmo.sh
Download MSMARCO v2 dataset, GloVe embedding.
cd data
python3.6 download.py v2
Convert raw data to tsv format
python3.6 convert_msmarco.py v2 --threads=`nproc`
Convert tsv format to ctf(CNTK input) format and build vocabs dictionary
python3.6 tsv2ctf.py
Generate elmo embedding
sh elmo.sh
cd ../script
mkdir log
sh run.sh
cd Evaluation
sh eval.sh v1
cd Evaluation
sh eval.sh v2
rouge-l | bleu_1 | |
---|---|---|
S-Net (Extraction) | 41.45 | 44.08 |
S-Net (Extraction, Ensemble) | 42.92 | 44.97 |
rouge-l | bleu_1 | |
---|---|---|
MSMARCO v1 w/o elmo | 38.43 | 39.14 |
MSMARCO v1 w/ elmo | 39.42 | 39.47 |
MSMARCO v2 w/ elmo | 43.66 | 44.44 |