[ICCV 2023] Official implementation of "SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields"
ICCV 2023
Anh-Quan Cao
Raoul de Charette
Inria, Paris, France.
If you find this work or code useful, please cite our paper and give this repo a star:
@InProceedings{cao2023scenerf,
author = {Cao, Anh-Quan and de Charette, Raoul},
title = {SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields},
booktitle = {ICCV},
year = {2023},
}
Outdoor scenes | Indoor scenes |
$ conda create -y -n scenerf python=3.7
$ conda activate scenerf
$ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch
$ cd scenerf/
$ pip install -r requirements.txt
$ conda install -c bioconda tbb=2020.2
$ pip install torchmetrics==0.6.0
$ pip install -e ./
Make sure the docker daemon is installed and running on your local machine.
$ docker build -t scene-rf .
$ docker run -it scene-rf
If the container should be deleted after usage, the -rm
flag can be used.
If GPUs are available, the --gpus all
flag can be used.
For more information, follow this LINK
To train and evaluate novel depths/views synthesis, please download on KITTI Odometry website the following data:
To evaluate scene reconstruction, please download the SemanticKITTI voxel data (700 MB) and all extracted data for the training set (3.3 GB) on Semantic KITTI download website.
Create a folder to store preprocess data at /path/to/kitti/preprocess/folder
.
Store paths in environment variables for faster access (Note: folder 'dataset' is in /path/to/kitti):
$ export KITTI_PREPROCESS=/path/to/kitti/preprocess/folder
$ export KITTI_ROOT=/path/to/kitti
/gpfsdswork/dataset/bundlefusion
(change to your dataset directory).$ export BF_ROOT=/gpfsdswork/dataset/bundlefusion
Create folders to store training logs at /path/to/kitti/logdir.
Store in an environment variable:
$ export KITTI_LOG=/path/to/kitti/logdir
Train scenerf using 4 v100-32g GPUs with batch_size of 4 (1 item per GPU):
$ cd scenerf/
$ python scenerf/scripts/train_kitti.py \
--bs=4 --n_gpus=4 \
--enable_log=True \
--preprocess_root=$KITTI_PREPROCESS \
--root=$KITTI_ROOT \
--logdir=$KITTI_LOG \
--n_gaussians=4 --n_pts_per_gaussian=8 \
--max_epochs=50 --exp_prefix=Train
Create folders to store training logs at /gpfsscratch/rech/kvd/uyl37fq/logs/monoscene2/bundlefusion (Change to your directory).
Store in an environment variable:
$ export BF_LOG=/gpfsscratch/rech/kvd/uyl37fq/logs/monoscene2/bundlefusion
Train scenerf using 4 v100-32g GPUs with batch_size of 4 (1 item per GPU):
$ cd scenerf/
$ python scenerf/scripts/train_bundlefusion.py --bs=4 --n_gpus=4 \
--n_rays=2048 --lr=2e-5 \
--enable_log=True \
--root=$BF_ROOT \
--logdir=$BF_LOG
Create folders to store intermediate evaluation data at /path/to/evaluation/save/folder
and reconstruction data at /path/to/reconstruction/save/folder
.
$ export EVAL_SAVE_DIR=/path/to/evaluation/save/folder
$ export RECON_SAVE_DIR=/path/to/reconstruction/save/folder
Please download the pretrained model.
Supposed we obtain the model from the training step at /path/to/model/checkpoint/last.ckpt
. We follow the steps below to evaluate the novel depths synthesis performance.
$ cd scenerf/
$ python scenerf/scripts/evaluation/save_depth_metrics.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--model_path=/path/to/model/checkpoint/last.ckpt
$ cd scenerf/
$ python scenerf/scripts/evaluation/agg_depth_metrics.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS
Given the trained model at /path/to/model/checkpoint/last.ckpt
, the novel views synthesis performance is obtained as followed:
$ cd scenerf/
$ python scenerf/scripts/evaluation/render_colors.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--model_path=/path/to/model/checkpoint
$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_color.py --eval_save_dir=$EVAL_SAVE_DIR
$ cd scenerf/
$ python scenerf/scripts/reconstruction/generate_novel_depths.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--model_path=/path/to/model/checkpoint \
--angle=10 --step=0.5 --max_distance=10.1
$ cd scenerf/
$ python scenerf/scripts/reconstruction/depth2tsdf.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--angle=10 --step=0.5 --max_distance=10.1
$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_sr.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS
Create folders to store intermediate evaluation data at /gpfsscratch/rech/kvd/uyl37fq/to_delete/eval
and reconstruction data at /gpfsscratch/rech/kvd/uyl37fq/to_delete/recon
.
$ export EVAL_SAVE_DIR=/gpfsscratch/rech/kvd/uyl37fq/to_delete/eval
$ export RECON_SAVE_DIR=/gpfsscratch/rech/kvd/uyl37fq/to_delete/recon
Please download the pretrained model.
Supposed we obtain the model from the training step at /gpfsscratch/rech/kvd/uyl37fq/to_delete/last.ckpt
(Change to your location). We follow the steps below to evaluate the novel depths synthesis performance.
$ cd scenerf/
$ python scenerf/scripts/evaluation/save_depth_metrics_bf.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$BF_ROOT \
--model_path=/gpfsscratch/rech/kvd/uyl37fq/to_delete/last.ckpt
$ cd scenerf/
$ python scenerf/scripts/evaluation/agg_depth_metrics_bf.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$BF_ROOT
Given the trained model at /gpfsscratch/rech/kvd/uyl37fq/to_delete/last.ckpt
, the novel views synthesis performance is obtained as followed:
$ cd scenerf/
$ python scenerf/scripts/evaluation/render_colors_bf.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$BF_ROOT \
--model_path=/gpfsscratch/rech/kvd/uyl37fq/to_delete/last.ckpt
$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_color_bf.py --eval_save_dir=$EVAL_SAVE_DIR
$ cd scenerf/
$ python scenerf/scripts/reconstruction/generate_novel_depths_bf.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$BF_ROOT \
--model_path=/gpfsscratch/rech/kvd/uyl37fq/to_delete/last.ckpt \
--angle=30 --step=0.2 --max_distance=2.1
$ cd scenerf/
$ python scenerf/scripts/reconstruction/depth2tsdf_bf.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$BF_ROOT \
--angle=30 --step=0.2 --max_distance=2.1
$ cd scenerf/
$ python scenerf/scripts/reconstruction/generate_sc_gt_bf.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$BF_ROOT
$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_sc_bf.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$BF_ROOT
Mesh can be obtained from this line for KITTI and from this line for Bundlefusion , and drawed with open3d as following:
import open3d as o3d
mesh = o3d.geometry.TriangleMesh()
mesh.triangle_normals = o3d.utility.Vector3dVector(data['norms'])
mesh.vertices = o3d.utility.Vector3dVector(data['verts'])
mesh.triangles = o3d.utility.Vector3iVector(data['faces'])
mesh.vertex_colors = o3d.utility.Vector3dVector(data['colors'].astype(np.float) / 255.0)
o3d.visualization.draw_geometries([mesh])
The work was partly funded by the French project SIGHT (ANR-20-CE23-0016) and conducted in the SAMBA collaborative project, co-funded by BpiFrance in the Investissement d’Avenir Program. It was performed using HPC resources from GENCI–IDRIS (Grant 2021-AD011012808, 2022-AD011012808R1, and 2023-AD011014102). We thank Fabio Pizzati and Ivan Lopes for their kind proofreading and all Astra-vision group members of Inria Paris for the insightful discussions.