[CVPR 2017] AMT chat interface code used to collect the Visual Dialog dataset
Source for the two-person chat interface used to collect the VisDial dataset (arxiv.org/abs/1611.08669) on Amazon Mechanical Turk.
If you find this code useful, consider citing our work:
@inproceedings{visdial,
title={{V}isual {D}ialog},
author={Abhishek Das and Satwik Kottur and Khushi Gupta and Avi Singh
and Deshraj Yadav and Jos\'e M.F. Moura and Devi Parikh and Dhruv Batra},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
year={2017}
}
The real-time chat interface is built using Node.js and Socket.io. We use Redis to maintain a pool of images for live HITs and all data finally resides in a MySQL database.
A database table stores images from the COCO dataset each with a randomly picked caption. A batch of images from this table are then pushed to a Redis list for launching HITs. The web server corresponding to the chat interface pairs two AMT workers, assigns them roles (questioner or answerer) and shows corresponding interface, picks an image from the Redis list to collect data on and saves their conversation in the database, also marking that image as 'complete' once the HIT is done. This happens in parallel so workers aren't left waiting, and the server ensures workers have unique ids. Disconnects are handled gracefully — remaining worker is asked to continue asking questions or providing facts (captions) up to 10 messages. Once the HITs are complete, scripts in mturk_scripts/approve
can be used to review, approve, reject HITs and pay workers.
Pops images from the Redis list, retrieves relevant data from the MySQL tables, renders the chat interface and pairs workers on AMT, saves submitted data to MySQL tables
npm install
(from the nodejs
folder)example.config.js
to config.js
, and set MySQL and Redis credentialsstatic/dataset
to /path/to/mscoco/images/
index.html
line 276node index.js
should now serve the interface at 127.0.0.1:5000Scripts to set up MySQL database, populate Redis list, and approve/reject HITs
example.config.json
to config.json
, and set MySQL credentials. from_timestamp
is unix timestamp before launching a batch of HITscreateDatabase.py
and fillDatabase.py
create MySQL tables, populate it with COCO images and captions and generate the Redis list for a batch of HITscreateHits.py
launches HITs on AMTmturk_scripts/approve
)example.config.json
to config.json
, and set MySQL credentialsexample.constants.py
to constants.py
, and set AMT credentialsreviewHits.py
gets completed HITs and saves them to amthitsQues.csv
and amthitsAns.csv
for reviewapproveHits.py
to mark approved HITs for payment and reviewRejectedHits.py
, rejectHits.py
to reject HITs, and finally payWorkers.py
to process paymentsParts of this code (MTurk scripts) are adapted from @jcjohnson's simple-amt project.
BSD