AttentionMask: Attentive, Efficient Object Proposal Generation Focusing on Small Objects (ACCV 2018, accepted as oral)
We propose a novel approach for class-agnostic object proposal generation, which is efficient and especially well-suited to detect small objects. Efficiency is achieved by scale-specific objectness attention maps which focus the processing on promising parts of the image and reduce the amount of sampled windows strongly. This leads to a system, which is 33% faster than the state-of-the-art and clearly outperforming state-of-the-art in terms of average recall. Secondly, we add a module for detecting small objects, which are often missed by recent models. We show that this module improves the average recall for small objects by about 53%.
The system is based on FastMask.
If you find this software useful in your research, please cite our paper.
@inproceedings{WilmsFrintropACCV2018,
title = {{AttentionMask}: Attentive, Efficient Object Proposal Generation Focusing on Small Objects},
author = {Christian Wilms and Simone Frintrop},
booktitle = {Asian Conference on Computer Vision (ACCV)},
year = {2018}
}
For the results in the paper we used the following hardware:
We assume Ubuntu 16.04 with Cuda 9.0, Python 2.7 and pip already installed
First, install OpenCV-Python:
$ sudo apt-get install python-opencv
Then, clone and install COCOApi as described here.
Now, clone this repository, install the Python packages from requirements.txt
if necessary and install the requirements of Caffe (PyCaffe) following the official instructions. Edit the Makefile.config
according to your system settings.
$ git clone https://github.com/chwilms/AttentionMask
$ cd AttentionMask
$ pip install -r requirements.txt
$ cd caffe
$ make pycaffe -j6
$ cd ..
Create new subdirectories for weights params
and results results
:
$ mkdir params results
After sucessfull installation, AttentionMask can immediatly be used without any training. Just download the weights (and the COCO dataset). However, you can also train AttentionMask with your own data. Note however, that your own data should be in the COCO format.
Download the train2014
and val2014
splits from COCO dataset. The train2014
split is exclusively used for training, while the first 5000 image from the val2014
split are used for testing. After downloading, extract the data in the following structure:
AttentionMask
|
---- data
|
---- coco
|
---- annotations
| |
| ---- instances_train2014.json
| |
| ---- instances_val2014.json
|
---- train2014
| |
| ---- COCO_train2014_000000000009.jpg
| |
| ---- ...
|
---- val2014
|
---- COCO_val2014_000000000042.jpg
|
---- ...
For inference, you have to download the model weights for one of the final AttentionMask models: AttentionMask-8-128, AttentionMask-8-192, AttentionMask-16-192
If you want to do training yourself, download the initial ImageNet weights for the ResNet. Weight files should be moved into the params
subdirectory.
There are two options for inference. You can either generate proposals for the COCO dataset (or any other dataset following that format) or you can generate proposals for one image.
For inference on the COCO dataset, use the testAttentionMask.py
script with the gpu id, the model name, the weights and the dataset you want to test on (e.g., val2014):
$ python testAttentionMask.py 0 attentionMask-8-128 --init_weights attentionmask-8-128final.caffemodel --dataset val2014 --end 5000
By default, only the first 5000 images of a dataset are used.
If you want to test AttentionMask on one of your images, call the demo.py
script with the path to your image, the gpu id, the model name and the weights:
$ python demo.py 0 attentionMask-8-128 <your image path here> --init_weights=attentionmask-8-128final.caffemodel
As a result you get an image with the best 20 proposals as overlays. If you want to dive deeper into the set of proposals, you can store them all using the ret_masks
variable in the script with the ret_scores
variable for the objectness scores.
For evaluation on the COCO dataset, you can use the evalCOCO.py
script with the model name and the dataset used. --useSegm
is a flag for using segmentation masks instead of bounding boxes.
$ python evalCOCO.py attentionMask-8-128 --dataset val2014 --useSegm True --end 5000
By default, only the first 5000 images of a dataset are used.
To train AttentionMask on a dataset, you can use the train.sh
script. It iterates over several epochs and saves as well as evaluates the result of each epoch (outputs in trainEval.txt
). For validation, currently the frist 5000 images of the training set are used, however, we encourage you to use a split of val2014
that is disjunct to the first 5000 images as your own validation set. Lowering the learning rate has to be done manually. We lowered the learning rate after 3 consecutive epochs of not improving results on our validation set. The environmental variable EPOCH
determines the next epoch to be run and is automatically incremented.
$ export EPOCH=1
$ ./train.sh
If you want to change the dataset form COCO to something else you have to follow the subsequent steps.
You have to provide the annotations in COCO-style. COCO-style means the annotation file has to be a json file similar to the COCO annotations. There are many tools on the web to change or create annotations accordingly.
Change the shuffledData.txt
. The only purpose of this file is to keep the data preprocessing in the data layer and the box selection layer in sync (loading the identical image, determining if the image should be flipped or slightly zoomed). Therefore this file keeps a randomly shuffled list of all indices of the dataset. In case of COCO dataset it is a list of numbers from 0 to 82080 (82081 images in training set). Additionally for each number there is a random flag (0 or 1) for horizontally flipping the image for training as well as a number between 0 and 69 as a tiny zoom. Tiny zoom is a small number that is added on top of the max edge length to get some more variety in image sizes (check fetch()
and fetch_image()
in base_coco_ssm_spider.py
or boxSelectionLayerMP.py
for details). All three values are separated by a semicolon and each line has one entry.
In config.py
adjust ANNOTATION_TYPE
and IMAGE_SET
according to your new dataset. Furthermore, you may have to adjust ANNOTATION_FILE_FORMAT
for the path to the annotations or IMAGE_PATH_FORMAT
for the path to the images. The image format strings are used in alchemy/data/coco.py
for locating the images. Changes may have to be applied there as well, e.g., if the image file name does not start with the dataset name.
The solver (models/*.solver.prototxt
) has to be adapted if the dataset is of different length than COCO. Change snapshot, display and average_loss according to the number of images in your dataset.
Change the value of the --step
parameter when calling the training script to the number of images in your dataset.
For inference, you may have to change the extraction of image_id
in the testAttentionMask.py
script according to the image IDs in your dataset.
To speed up the training or testing phase as well as decreasing the network's memory footprint, removing one or multiple scales is the straightforward solution. However, this comes at the cost of (slightly) decreased performance in terms of AR. Usually removing scales 24
, 48
and 96
results only in little changes in terms of AR. Removing scale 8
on the other hand results in large gains in terms of speed up and memory footprint, however, decreases the performance on small objects significantly. For removing one or multiple scales follow the subsequent steps.
Usually it is no problem to test with fewer scales than you used for training. We will show here how to remove scale 128
from myModel
. Removing other scales works accordingly.
configs/myModel.json
file remove 128
from the list RFs
.models/myModel.test.prototxt
remove the following elements:
bottom: "sample_128"
from sample_concat layer
.layers extractor_128
and conv_feat_1_128s
.top: "obj128_flags"
and bottom: "obj128_checked"
from the SplitIndices
layer.bottom: "obj128_checked"
and bottom: "nonObj128_checked"
from concatFlattendObj
and concatFlattendNonObj
layer, respectively.objectness attention at scale 128
block. This step is not necessary for scale 8
, scale 16
and scale 24
.shared neck at scale 128
block. This step is only necessary, if you remove a final scale from one of the branches (e.g. scale 96
or scale 128
from attentionMask-8-128
). This step is not necessary for scale 8
, scale 16
and scale 24
.models/myModel.train.prototxt
file that have been removed from the models/myModel.test.prototxt
file.top: "objAttMask_128"
and top: "objAttMask_128_org"
from the data
layer.128
from the attr
in spider/coco_ssm_spider.py
.If you remove scales 24
, 48
, 96
(, 192
), you can also remove the div3 branch
from the base net (layers marked with _div3
).
Check failed: error == cudaSuccess (4 vs. 0) driver shutting down
at the end of executing AttentionMaskMost likely, everything is just fine. You can ignore the error. Check, if a json-file with your chosen model name was created in the results
subdirectory in case you ran testAttentionMask.py
. The model name is the second parameter in testAttentionMask.py
.