[AAAI 2019] Generating Distractors for Reading Comprehension Questions from Real Examinations
Dataset for our AAAI 2019 paper: Generating Distractors for Reading Comprehension Questions from Real Examinations https://arxiv.org/abs/1809.02768
If you use our data or code, please cite our paper as follows:
@inproceedings{gao2019distractor,
title="Generating Distractors for Reading Comprehension Questions from Real Examinations",
author="Yifan Gao and Lidong Bing and Piji Li and Irwin King and Michael R. Lyu",
booktitle="AAAI-19 AAAI Conference on Artificial Intelligence",
year="2019"
}
In the task of Distractor Generation (DG), we aim at generating reasonable distractors
(wrong options) for multiple choices questions (MCQs) in reading comprehension.
The generated distractors should:
Here is an example from our dataset. The question, options and their relevant sentences in the article are marked with the same color.
The data used in our paper is transformed from RACE Reading Comprehension Dataset.
We prune the distractors which have no semantic relevance
with the article or require some world knowledge
to generate.
The processed data is put in the /data/
directory. Please uncompress it first.
Here is the dataset statistics.
Note
Due to a bug in spacy, actually more examples in RACE dataset should be filtered by our rule. But we were not aware of this issue when we did this project. Here we release both the original dataset race_train/dev/test_original.json
and the updated dataset race_train/dev/test_updated.json
. Because of the smaller dataset size, the performance will be worse if the model is trained on the updated dataset.
Our implementation is based on OpenNMT-py.
GloVe vectors are required, please download glove.840B.300d first.
run scripts/preprocess.sh
for preprocessing and getting corresponding word embedding.
run scripts/train.sh
for training, scripts/generate.sh
for generation and evaluation