✨ SHINE-Mapping: Large-Scale 3D Mapping Using Sparse Hierarchical Implicit Neural Representations

Xingguang Zhong* · Yue Pan* · Jens Behley · Cyrill Stachniss

University of Bonn

(* Equal Contribution)

Arxiv | Paper | Video

Abstract

Accurate mapping of large-scale environments is an essential building block of most outdoor autonomous systems. Challenges of traditional mapping methods include the balance between memory consumption and mapping accuracy. This paper addresses the problems of achieving large-scale 3D reconstructions with implicit representations using 3D LiDAR measurements. We learn and store implicit features through an octree-based hierarchical structure, which is sparse and extensible. The features can be turned into signed distance values through a shallow neural network. We leverage binary cross entropy loss to optimize the local features with the 3D measurements as supervision. Based on our implicit representation, we design an incremental mapping system with regularization to tackle the issue of catastrophic forgetting in continual learning. Our experiments show that our 3D reconstructions are more accurate, complete, and memory-efficient than current state-of-the-art 3D mapping methods.

Installation

1. Clone SHINE Mapping repository

git clone [email protected]:PRBonn/SHINE_mapping.git
cd SHINE_mapping

2. Set up conda environment

conda create --name shine python=3.7
conda activate shine

3. Install the key requirement kaolin

Kaolin depends on Pytorch (>= 1.8, <= 1.13.1), please install the corresponding Pytorch for your CUDA version (can be checked by nvcc --version). You can find the installation commands here.

For example, for CUDA version >=11.6, you can use:

pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116

Kaolin now supports installation with wheels. For example, to install kaolin 0.12.0 over torch 1.12.1 and cuda 11.6:

pip install kaolin==0.12.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.12.1_cu116.html

git clone --recursive https://github.com/NVIDIAGameWorks/kaolin
cd kaolin

python setup.py develop

4. Install the other requirements

pip install open3d scikit-image wandb tqdm natsort pyquaternion

Containerized installation

Note that you CUDA version must be >=11.6.2 to be compatible with the container.

1. Install docker

https://docs.docker.com/engine/install/ubuntu/

2. Install nvidia container runtime

https://developer.nvidia.com/nvidia-container-runtime

3. Clone SHINE Mapping repository

git clone [email protected]:PRBonn/SHINE_mapping.git
cd SHINE_mapping

4. Build container

docker build --tag shine .

5. Run container with example

mkdir /tmp/shine_test_data
docker run --rm -v ${pwd}/:/repository -v /tmp/shine_test_data:/data -it --gpus all shine

Results will be produced in /tmp/shine_test_data/results.

6. Run container on your own data

docker run --rm -v .:/repository -v ${MY_DATA_DIR}:/data -it --gpus all shine bash

where ${MY_DATA_DIR} is the directory on the host with data in the format described in config/kitti/docker_kitti_batch.yaml. Once inside the container RUN as described below. Results will be found on host in ${MY_DATA_DIR}/results .

Prepare data

Generally speaking, you only need to provide:

1. pc_path : the folder containing the point cloud (.bin, .ply or .pcd format) for each frame.
2. pose_path : the pose file (.txt) containing the transformation matrix of each frame.
3. calib_path : the calib file (.txt) containing the static transformation between sensor and body frames (optional, would be identity matrix if set as '').

They all follow the KITTI odometry data format.

After preparing the data, you need to correctly set the data path (pc_path, pose_path and calib_path) in the config files under config folder. You may also set a path output_root to store the experiment results and logs.

Here, we provide the link to several publicly available datasets for testing SHINE Mapping:

MaiCity synthetic LiDAR dataset

Download the dataset from here or use the following script to download (3.4GB):

sh ./scripts/download_maicity.sh

KITTI real-world LiDAR dataset

Download the full dataset from here.

If you want to use an example part of the dataset (seq 00) for the test, you can use the following script to download (117 MB):

sh ./scripts/download_kitti_example.sh

Newer College real-world LiDAR dataset

Download the full dataset from here.

If you want to use an example part of the dataset (Quad) for the test, you can use the following script to download (634 MB):

sh ./scripts/download_ncd_example.sh

RGB-D datasets

SHINE Mapping also supports the mapping on RGB-D datasets. You may first try the synthetic dataset from NeuralRGB-D. You can download the full dataset from here or use the following script to download (7.25 GB).

sh ./scripts/download_neural_rgbd_data.sh

After downloading the data, you need to convert the dataset to the KITTI format by using for each sequence:

sh ./scripts/convert_rgbd_to_kitti_format.sh

Mapping without ground truth pose

pip install kiss-icp

kiss_icp_pipeline <pc_path>

Generate colorized mesh

Check the repository Color-SHINE-MAPPING (thanks @ZorAttC for the contribution) for using SHINE Mapping to reconstruct colorized mesh using colorized point cloud.

Run

We take the MaiCity dataset as an example to show how SHINE Mapping works. You can simply replace maicity with your dataset name in the config file path, such as ./config/[dataset]/[dataset]_[xxx].yaml.

The results will be stored with your experiment name with the starting timestamp in the output_root directory as what you set in the config file. You can find the reconstructed mesh (*.ply format) and optimized model in mesh and model folder, respectively. If the save_map option is turned on, then you can find the grid sdf map in map folder.

For mapping based on offline batch processing, use:

python shine_batch.py ./config/maicity/maicity_batch.yaml

For incremental mapping with replay strategy (within a local bounding box), use:

python shine_incre.py ./config/maicity/maicity_incre_replay.yaml

An interactive visualizer would pop up if you set o3d_vis_on: True (by default) in the config file. You can press space to pause and resume.

For incremental mapping with a regularization strategy, use:

python shine_incre.py ./config/maicity/maicity_incre_reg.yaml

The logs can be monitored via Weights & Bias online if you turn the wandb_vis_on option on. If it's your first time to use Weights & Bias, you would be requested to register and login to your wandb account.

Evaluation

To evaluate the reconstruction quality, you need to provide the (reference) ground truth point cloud and your reconstructed mesh. The ground truth point cloud can be found (or sampled from) the downloaded folder of MaiCity, Newer College and Neural RGBD datasets.

Please change the data path and evaluation set-up in ./eval/evaluator.py and then run:

python ./eval/evaluator.py

to get the reconstruction metrics such as Chamfer distance, completeness, F-score, etc.

As mentioned in the paper, we also compute a fairer accuracy metric using the ground truth point cloud masked by the intersection of the reconstructed meshes of all the compared methods. To generate such masked ground truth point clouds, you can configure the data path in ./eval/crop_intersection.py and then run it.

To reproduce the quantitative results on MaiCity and Newer College dataset in the paper, you can use the config files in ./config/config_icra2023/. The reconstructed meshes can also be downloaded from here. Note that these numbers are obtained using the batch mapping mode. You can achieve similar results using the incremental mapping mode with the replay strategy.

Tips

Citation

If you use SHINE Mapping for any academic work, please cite our original paper.

@inproceedings{zhong2023icra,
  title={SHINE-Mapping: Large-Scale 3D Mapping Using Sparse Hierarchical Implicit NEural Representations},
  author={Zhong, Xingguang and Pan, Yue and Behley, Jens and Stachniss, Cyrill},
  booktitle = {Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},
  year={2023}
}

Contact

If you have any questions, please contact:

• Xingguang Zhong {[email protected]}
• Yue Pan {[email protected]}

Acknowledgment

This work has partially been funded by the European Union’s HORIZON programme under grant agreement No 101070405 (DigiForest) and grant agreement No 101017008 (Harmony).

Additionally, we thank greatly for the authors of the following opensource projects:

• NGLOD (octree based hierarchical feature structure built based on kaolin)
• VDBFusion (comparison baseline)
• Voxblox (comparison baseline)
• Puma (comparison baseline and the MaiCity dataset)
• KISS-ICP (simple yet effective pose estimation)