MVSNet: Depth Inference for Unstructured Multi-view Stereo using pytorch-lightning
Unofficial implementation of MVSNet: Depth Inference for Unstructured Multi-view Stereo using pytorch-lightning
An improved version of MVSNet: CasMVSNet is available!
Official implementation: MVSNet
A pytorch implementation: MVSNet_pytorch. This code is heavily borrowed from his implementation. Thank xy-guo for the effortful contribution! Two main difference w.r.t. his repo:
homo_warping
function is rewritten in a more concise and slightly faster way.conda create -n mvsnet_pl python=3.6
to create a conda environment and activate it by conda activate mvsnet_pl
)pip install -r requirements.txt
pip install git+https://github.com/mapillary/[email protected]
Download the preprocessed DTU training data from original MVSNet repo and unzip. For the description of how the data is created, please refer to the original paper.
Run
python train.py \
--root_dir $DTU_DIR \
--num_epochs 6 --batch_size 1 \
--n_depths 192 --interval_scale 1.06 \
--optimizer adam --lr 1e-3 --lr_scheduler cosine
Note that the model consumes huge GPU memory, so the batch size is generally small. For reference, the above command requires 5901MB of GPU memory.
IMPORTANT : the combination of --n_depths
and --interval_scale
is important: you need to make sure 2.5 x n_depths x interval_scale
is roughly equal to 510
. The reason is that the actual depth ranges from 425 to 935mm, which is 510mm wide. Therefore, you need to make sure all the depth can be covered by the depth planes you set. Some common combinations are: --n_depths 256 --interval_scale 0.8
, --n_depths 192 --interval_scale 1.06
and --n_depths 128 --interval_scale 1.6
.
See opt.py for all configurations.
The metrics are collected on the DTU val set.
abs_err | 1mm acc | 2mm acc | 4mm acc | |
---|---|---|---|---|
Paper | 7.25mm* | N/A | N/A | N/A |
This repo | 6.374mm | 54.43% | 74.23% | 85.8% |
*From P-MVSNet Table 2.
n_depths
theoretically gives better results, but requires larger GPU memory, so basically the batch_size
can just be 1
or 2
. However at the meanwhile, larger batch_size
is also indispensable. To get a good balance between n_depths
and batch_size
, I found that n_depths 128 batch_size 2
performs better than n_depths 192 batch_size 1
given a fixed GPU memory of 11GB. Of course to get even better results, you'll definitely want to scale up the batch_size
by using more GPUs, and that is easy under pytorch-lightning's framework!The repo is only for training purpose for now. Please refer to the other repositories mentioned at the beginning if you want to evaluate the model.