3D OWIS Save

https://github.com/aminebdj/3D-OWIS/assets/95423980/f889b307-ab97-4047-ad45-d50152d82e5b

News

• 25 September 2023: 3D-OWIS paper released on arXiv. 📝
• 25 September 2023: Code and splits released. 💻
• 22 September 2023: Paper accepted into NeurIPS2023. 🥳

Abstract

We formulate the problem of 3D point cloud instance segmentation in the open world, where we relax the assumption of a closed setting of classes during the training. We propose an auto-labeling scheme to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance.

Qualitative results

Navigate in the scenes

Kindly click on the icon to navigate in the scenes

	Scene 1	Scene 2	Scene 3
3D-OWIS
Ground truth
Input scene

Comparative visualization

Installation guide

Kindly check Installation guide on how to setup the Conda environment and how to preprocess ScanNet200 dataset.

Use our splits with another 3D instance segmentation model

You can import the labels of the PREVIOUSLY_KNOWN, CURRENTLY_KNOWN, and UNKNOWN classes from ./datasets/scannet200/owis_splits.py for usage with another codebase.

Example: the following code will load the previously known, currently known, and unknown classes for split A task1

from datasets.scannet200.owis_splits import PREV_KNOWN_CLASSES_LABELS, KNOWN_CLASSES_LABELS, UNKNOWN_CLASSES_LABELS

OW_task = 'task1'
split = 'A'

PREVIOUSLY_KNOWN = PREV_KNOWN_CLASSES_LABELS[split][OW_task]
CURRENTLY_KNOWN = KNOWN_CLASSES_LABELS[split][OW_task]
UNKNOWN = UNKNOWN_CLASSES_LABELS[split][OW_task]

Training and testing :train2:

For training a model <MODEL> with a split <SPLIT> please run the following command:

NB: <SPLIT> takes the values 'A', 'B', or 'C'.

We provide the scripts for reproducing the results for the models 3d_owis, and oracle.

sh scripts/train_<MODEL>/<SPLIT>/train.sh

For testing a task <TASK> for a specific split <SPLIT> please run the following command:

NB: <TASK> takes the values 'task1', 'task2', or 'task3'

Important for inference: please make sure to not forget to specify the <EXPERIMENT_NAME>, <SPLIT>, and <TASK> when running the inference, as the prototypes are loaded from the path saved/<EXPERIMENT_NAME>/<SPLIT>/<TASK>/ to perform the probability correction.

python main_instance_segmentation.py \
general.checkpoint='saved/<EXPERIMENT_NAME>/<SPLIT>/<TASK>/CHECKPOINT_NAME.ckpt' \
general.experiment_name=<EXPERIMENT_NAME> \
general.split=<SPLIT> \
general.OW_task=<TASK> \
general.margin=<CLUSTERING_MARGIN_USED_DURING_TRAINING> \
general.correct_unknown_cls_prob=True \
general.train_mode=false

Trained checkpoints and output predictions

	Checkpoints	Clustering Margin
3D-OWIS		A :1.0, B :3.0, C :1.0
3D-OWIS - PC - CT		A :1.0, B :3.0, C :1.0
Oracle		A :1.0, B :3.0, C :1.0

BibTeX :pray:

@inproceedings{boudjoghra20233d,
 author = {Mohamed El Amine Boudjoghra and Salwa K. Al Khatib and Jean Lahoud and Hisham Cholakkal and Rao Muhammad Anwer and Salman Khan and Fahad Khan},
 booktitle = {Advances in Neural Information Processing Systems},
 title = {3D Indoor Instance Segmentation in an Open-World},
 year = {2023}
}