[CoRL 2022] SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
[CoRL 2022] SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
Yi Wei*, Linqing Zhao*, Wenzhao Zheng, Zheng Zhu, Yongming Rao, Guan Huang, Jiwen Lu, Jie Zhou
Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR. The temporal photometric consistency enables self-supervised depth estimation without labels, further facilitating its application. However, most existing methods predict the depth solely based on each monocular image and ignore the correlations among multiple surrounding cameras, which are typically available for modern self-driving vehicles. In this paper, we propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras.
type | dataset | Abs Rel | Sq Rel | delta < 1.25 | download |
---|---|---|---|---|---|
scale-ambiguous | DDAD | 0.200 | 3.392 | 0.740 | model |
scale-aware | DDAD | 0.208 | 3.371 | 0.693 | model |
scale-ambiguous | nuScenes | 0.245 | 3.067 | 0.719 | model |
scale-aware | nuScenes | 0.280 | 4.401 | 0.661 | model |
git clone https://github.com/weiyithu/SurroundDepth.git
conda create -n surrounddepth python=3.8
conda activate surrounddepth
pip install -r requirements.txt
Since we use dgp codebase to generate groundtruth depth, you should also install it.
Datasets are assumed to be downloaded under data/<dataset-name>
.
data/ddad/raw_data
. You may refer to official DDAD repository for more info and instructions.datasets/ddad
.data/ddad/mask
.cd tools
python export_gt_depth_ddad.py val
conda create -n sift python=3.6
conda activate sift
pip install opencv-python==3.4.2.16 opencv-contrib-python==3.4.2.16
python sift_ddad.py
python match_ddad.py
SurroundDepth
├── data
│ ├── ddad
│ │ │── raw_data
│ │ │ │── 000000
| | | |── ...
| | |── depth
│ │ │ │── 000000
| | | |── ...
| | |── match
│ │ │ │── 000000
| | | |── ...
| | |── mask
│ │ │ │── 000000
| | | |── ...
data/nuscenes/raw_data
cd tools
python export_gt_depth_nusc.py val
conda activate sift
python sift_nusc.py
python match_nusc.py
SurroundDepth
├── data
│ ├── nuscenes
│ │ │── raw_data
│ │ │ │── samples
| | | |── sweeps
| | | |── maps
| | | |── v1.0-trainval
| | |── depth
│ │ │ │── samples
| | |── match
│ │ │ │── samples
Take DDAD dataset as an example. Train scale-ambiguous model.
python -m torch.distributed.launch --nproc_per_node 8 --num_workers=8 run.py --model_name ddad --config configs/ddad.txt
Train scale-aware model. First we should conduct SfM pretraining.
python -m torch.distributed.launch --nproc_per_node 8 run.py --model_name ddad_scale_pretrain --config configs/ddad_scale_pretrain.txt
Then we select the best pretrained model.
python -m torch.distributed.launch --nproc_per_node 8 run.py --model_name ddad_scale --config configs/ddad_scale.txt --load_weights_folder=${best pretrained}
We observe that the training on nuScenes dataset is unstable and easy to overfit. Also, the results with 4 GPUs are much better than 8 GPUs. Thus, we set fewer epochs and use 4 GPUs for nuScenes experiments. We also provide SfM pretrained model on DDAD and nuScenes.
python -m torch.distributed.launch --nproc_per_node ${NUM_GPU} run.py --model_name test --config configs/${TYPE}.txt --models_to_load depth encoder --load_weights_folder=${PATH} --eval_only
Our code is based on Monodepth2.
If you find this project useful in your research, please consider cite:
@article{wei2022surround,
title={SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation},
author={Wei, Yi and Zhao, Linqing and Zheng, Wenzhao and Zhu, Zheng and Rao, Yongming and Huang ,Guan and Lu, Jiwen and Zhou, Jie},
journal={arXiv preprint arXiv:2204.03636},
year={2022}
}