SemEval-2018 Task 12: The Argument Reasoning Comprehension Task
This repository provides supplementary code for the SemEval-2018 Task 12 competition (The Argument Reasoning Comprehension Task). It contains the CodaLab Competition definition, the official scorer, a baseline random system, as well as mirrored training/dev data.
Source codes and datasets regarding the The Argument Reasoning Comprehension Task itself are located in a separate GitHub repository.
The output format is shown in submission/answer.txt
. Replace its content with your predictions, see for instance the baseline system which implement a random classifier in baseline-system
.
submission.zip
Prepare submission ZIP file and submit via the Web interface
$ make submission.zip -B
The following text describes how to make predictions using the baseline system - a random classifier which gives the lower-bound accuracy of 50%.
$ cd baseline-system
$ mvn package
data
folder.$ wget https://raw.githubusercontent.com/UKPLab/argument-reasoning-comprehension-task/master/mturk/annotation-task/data/exported-SemEval2018-train-dev-test/train-full.txt
dev-only-data.txt
) from GitHub
$ wget https://raw.githubusercontent.com/UKPLab/argument-reasoning-comprehension-task/master/mturk/annotation-task/data/exported-SemEval2018-train-dev-test/dev-only-data.txt
$ java -jar target/baseline-system-1.0-SNAPSHOT.jar train-full.txt dev-only-data.txt /tmp/pred.txt
As you can see in the following snippet, the output file contains only instance IDs and the predicted label. This is the output you will have to submit to the CodaLab scorer (details later on).
$ head /tmp/pred.txt
#id correctLabelW0orW1
14085524_189_A34QZDSTKZ3JO9 0
16629299_15_A1CF6U3GF7DZEJ 1
14164520_257_A34QZDSTKZ3JO9 1
16967643_582_A1DJNUJZN8FE7N 0
15769481_321_A138BU1VWM2RKN 0
16157609_550_APW9F8OTJ4KXO 0
14106155_221_A3TKD7EJ6BM0M5 1
14228063_268_A1I4CYG5YDFTYM 0
17024278_160_A1J0GU26323WVA 0
You can evaluate your system on the dev
data locally without submitting it to CodaLab in the first place.
$ cd codalab/scorer
$ mvn package
dev-only-labels.txt
) from GitHub
$ wget https://raw.githubusercontent.com/UKPLab/argument-reasoning-comprehension-task/master/mturk/annotation-task/data/exported-SemEval2018-train-dev-test/dev-only-labels.txt
pred.txt
$ java -jar target/scorer-1.0-SNAPSHOT.jar -local dev-only-labels.txt /tmp/pred.txt
Accuracy: 0.503
Which gives us the expected accuracy of a random classifier.
We should get the very same results also on CodaLab.
answer.txt
(this name is required by CodaLab)$ cd codalab/competition
$ cat /tmp/pred.txt > submission/answer.txt
submission.zip
file using make
$ make submission.zip -B
(If you don't have make
, you can alternatively zip it using cd submission && zip ../submission.zip * && cd ..
)
submission.zip
to CodaLabDownload evaluation output from scoring step
which contains scores.txt
with a single line: Accuracy: 0.503
competition.zip
Build and upload using the Web interface
$ make competition.zip -B