PointCloud C Save

Benchmarking and Analyzing Point Cloud Perception Robustness under Corruptions

Jiawei Ren,  Lingdong Kong,  Liang Pan,  Ziwei Liu
S-Lab, Nanyang Technological University

About

PointCloud-C is the very first test-suite for point cloud perception robustness analysis under corruptions. It includes two sets: ModelNet-C (ICML'22) for point cloud classification and ShapeNet-C (arXiv'22) for part segmentation.

Fig. Examples of point cloud corruptions in PointCloud-C.

Visit our project page to explore more details. 🌱

Updates

• [2024.03] - We add Leaderboard to this page. We welcome pull requests to submit your results!
• [2024.01] - The toolkit tailored for The RoboDrive Challenge has been released. :hammer_and_wrench:
• [2023.12] - We are hosting The RoboDrive Challenge at ICRA 2024. :blue_car:
• [2023.03] - Intend to test the robustness of your 3D perception models on real-world point clouds? Check our recent work, Robo3D, a comprehensive suite that enables OoD robustness evaluation of 3D detectors and segmentors on our newly established datasets: KITTI-C, SemanticKITTI-C, nuScenes-C, and WOD-C.
• [2022.11] - The preprint of the PointCloud-C paper (ModelNet-C + ShapeNet-C) is available here.
• [2022.10] - We have successfully hosted the 2022 PointCloud-C Challenge. Congratulations to the winners: 🥇 Antins_cv, 🥈 DGPC & DGPS, and 🥉 BIT_gdxy_xtf.
• [2022.07] - Try a Gradio demo for PointCloud-C corruptions at Hugging Face Spaces! :hugs:
• [2022.07] - Competition starts! Join now at our CodaLab page.
• [2022.06] - PointCloud-C is now live on Paper-with-Code. Join the benchmark today!
• [2022.06] - The 1st PointCloud-C challenge will be hosted in conjecture with the ECCV'22 SenseHuman workshop. 🚀
• [2022.06] - We are organizing the 1st PointCloud-C challenge! Click here to explore the competition details.
• [2022.05] - ModelNet-C is accepted to ICML 2022. Click here to check it out! 🎉

Overview

• Highlight
• Data Preparation
• Getting Started
• Leaderboard
• Benchmark Results
• Evaluation
• Customize Evaluation
• Build PointCloud-C
• TODO List
• License
• Acknowledgement
• Citation

Highlight

Corruption Taxonomy

ModelNet-C (Classification)

ShapeNet-C (Part Segmentation)

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare the ModelNet-C and ShapeNet-C datasets.

Getting Started

Please refer to GET_STARTED.md to learn more usage about this codebase.

Leaderboard

Benchmark Results

ModelNet-C (Classification)

ShapeNet-C (Part Segmentation)

*Note: Standalone indicates whether or not the method is a standalone architecture or a combination with augmentation or pretrain.

Evaluation

Evaluation commands are provided in EVALUATE.md.

Customize Evaluation

We have provided evaluation utilities to help you evaluate on ModelNet-C using your own codebase. Please follow CUSTOMIZE.md.

Build PointCloud-C

You can manage to generate your own "PointCloud-C"! Follow the instructions in GENERATE.md.

TODO List

• Initial release. 🚀
• Add license. See here for more details.
• Release test sets. Download ModelNet-C and ShapeNet-C from our project page.
• Add evaluation scripts for classification models.
• Add evaluation scripts for part segmentation models.
• Add competition details.
• Clean and retouch codebase.

License

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Acknowledgement

We acknowledge the use of the following public resources during the course of this work: ¹SimpleView, ²PCT, ³GDANet, ⁴CurveNet, ⁵PAConv, ⁶RSMix, ⁷PointMixUp, ⁸PointCutMix, ⁹PointWOLF, ¹⁰PointTransformers, ¹¹OcCo, ¹²PointMLP, ¹³PointBERT, and ¹⁴PointMAE.

Citation

If you find this work helpful, please kindly consider citing our papers:

@article{ren2022pointcloud-c,
  title = {PointCloud-C: Benchmarking and Analyzing Point Cloud Perception Robustness under Corruptions},
  author = {Jiawei Ren and Lingdong Kong and Liang Pan and Ziwei Liu},
  journal = {Preprint},
  year = {2022}
}

@inproceedings{ren2022modelnet-c,
  title = {Benchmarking and Analyzing Point Cloud Classification under Corruptions},
  author = {Jiawei Ren and Liang Pan and Ziwei Liu},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2022}
}