Motion R-CNN: Mask R-CNN with support for 3D motion estimation (prototype)
This repository contains the TensorFlow prototype implementation of my bachelor thesis Motion R-CNN: Instance-level 3D Motion Estimation with Region-based CNNs.
In addition to the functionality provided by the TensorFlow Object Detection API (at the time of writing), the code supports:
Note that the code only supports training on the Virtual KITTI dataset,
but it is easy to adapt it to other datasets.
Motion prediction and frame pair input is fully optional and the code can be used as a Mask R-CNN
implementation with single image input.
Support for cityscapes is implemented, but using the records created with create_citiscapes_tf_record.py
may require adapting the data_decoder
or the record writing as the record interface changed.
Motion R-CNN is released under the MIT License (refer to the LICENSE file for details).
pip install opencv-python pandas pillow lxml matplotlib
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
object_detection/data
vkitti
.protoc object_detection/protos/*.proto --python_out=.
python create_vkitti_tf_record.py --data_dir=<data_parent_dir> --output_dir=data/records --set val
python create_vkitti_tf_record.py --data_dir=<data_parent_dir> --output_dir=data/records --set train
Note that <data_parent_dir>
is the directory containing the vkitti
directory.
Use
python train.py --logtostderr --pipeline_config_path=data/configs/motion_rcnn_vkitti_cam.config --train_dir=output/train/motion_rcnn_vkitti_cam --gpu 0
python eval.py --logtostderr --pipeline_config_path=data/configs/motion_rcnn_vkitti_cam.config --checkpoint_dir=output/train/motion_rcnn_vkitti_cam --eval_dir=output/eval/motion_rcnn_vkitti_cam
to train and evaluate a model with camera and instance motion prediction.
You can adapt the configurations found in data/configs/
. For a description of the configuration parameters, see object_detection/protos
.
The following files were added or modified from the original Object Detection API code
Additionally, some proto params and builders were modified, and extensions were made to eval_util.py, eval.py, evaluator.py, train.py, trainer.py.
The following tests were added or modified:
This repository is based on the TensorFlow Object Detection API.