Awesome-Implicit-NeRF-Robotics

This repo contains a curative list of Implicit Representations and NeRF papers relating to Robotics/RL domain, inspired by awesome-computer-vision

Please feel free to send me pull requests or email to add papers!

If you find this repository useful, please consider citing and STARing this list. Feel free to share this list with others!

For an overview of NeRFs, checkout the Survey (Neural Volume Rendering: NeRF And Beyond), Blog post (NeRF Explosion 2020) and Collection (awesome-NeRF)

Overview

• Object Pose Estimation
• SLAM
• Manipulation/RL
• Object Reconstruction
• Physics
• Planning/Navigation
• Citation

Object Pose Estimation

• BundleSDF: "Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects", CVPR, 2023. [Paper] [Webpage]

• ShAPO: "Implicit Representations for Multi Object Shape Appearance and Pose Optimization", ECCV, 2022. [Paper] [Pytorch Code] [Webpage] [Video]

• NCF: "Neural Correspondence Field for Object Pose Estimation", ECCV, 2022. [Paper] [Pytorch Code] [Webpage]

• Neural-Sim: "Learning to Generate Training Data with NeRF", ECCV 2022. [Paper] [Pytorch Code] [Webpage]

• DISP6D: "Disentangled Implicit Shape and Pose Learning for Scalable 6D Pose Estimation", ECCV 2022. [Paper] [Pytorch Code] [Webpage] [Video]

• SNAKE: "SNAKE: Shape-aware Neural 3D Keypoint Field", NeurIPS, 2022. [Paper] [Pytorch Code]

• NeRF-RPN: "A general framework for object detection in NeRFs", arXiv. [Paper] [Video]

• nerf2nerf: "Pairwise Registration of Neural Radiance Fields", arXiv. [Paper] [Pytorch Code] [Webpage] [Dataset]

• iNeRF: "Inverting Neural Radiance Fields for Pose Estimation", IROS, 2021. [Paper] [Pytorch Code] [Website] [Dataset]

• NeRF-Pose: "A First-Reconstruct-Then-Regress Approach for Weakly-supervised 6D Object Pose Estimation", arXiv. [Paper]

• PixTrack: "Precise 6DoF Object Pose Tracking using NeRF Templates and Feature-metric Alignment", arXiv. [Paper] [Pytorch Code]

• "Parallel Inversion of Neural Radiance Fields for Robust Pose Estimation", arXiv. [Paper] [Website]

• NARF22: "Neural Articulated Radiance Fields for Configuration-Aware Rendering", IROS, 2022. [Paper] [Website]

• FroDO: "From Detections to 3D Objects", CVPR, 2020. [Paper]

• SDFEst: "Categorical Pose and Shape Estimation of Objects From RGB-D Using Signed Distance Fields", RA-L, 2022. [Paper] [Pytorch Code]

• SSC-6D: "Self-Supervised Category-Level 6D Object Pose Estimation with Deep Implicit Shape Representation", AAAI, 2022. [Paper] [Pytorch Code]

• Style2NeRF: "An Unsupervised One-Shot NeRF for Semantic 3D Reconstruction", BMVC, 2022. [Paper]

• "Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion", CVPR, 2023. [Paper] [Code]

• TexPose: "Neural Texture Learning for Self-Supervised 6D Object Pose Estimation", CVPR 2023. [Paper][Code]

• Canonical Fields: "Self-Supervised Learning of Pose-Canonicalized Neural Fields", arXiv. [Paper]

• NeRF-Det: "Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection", arXiv. [Paper] [[Page] https://chenfengxu714.github.io/nerfdet/] [[Code] https://github.com/facebookresearch/NeRF-Det]

SLAM

• iSDF: "Real-Time Neural Signed Distance Fields for Robot Perception", RSS, 2022. [Paper] [Pytorch Code] [Website]

• LENS: "LENS: Localization enhanced by NeRF synthesis", CORL, 2021. [Paper]

• NICE-SLAM: "Neural Implicit Scalable Encoding for SLAM", CVPR, 2021. [Paper] Pytorch Code] [Website]

• iMAP: "Implicit Mapping and Positioning in Real-Time", ICCV, 2021. [Paper] [Website]

• BNV-Fusion: "BNV-Fusion: Dense 3D Reconstruction using Bi-level Neural Volume Fusion", CVPR, 2022. [Paper] Pytorch Code]

• NeRF-SLAM: "Real-Time Dense Monocular SLAM with Neural Radiance Fields", arXiv. [Paper]

• NICER-SLAM: "Neural Implicit Scene Encoding for RGB SLAM", arXiv. [Paper] [Video]

• Nerfels: "Renderable Neural Codes for Improved Camera Pose Estimation", CVPR 2022 Workshop. [Paper]

• GO-Surf: "A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping", 3DV, 2022. [Paper] [[Website(https://jingwenwang95.github.io/go_surf/)] [Pytorch Code]

• Orbeez-SLAM: "Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction", arXiv, 2022. [Paper]

• ESLAM: "Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields", arXiv, 2022. [Paper]

• Panoptic Multi-TSDFs: "a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency", ICRA, 2022. [Paper] [Pytorch Code]

• SHINE-Mapping: "Large-Scale 3D Mapping Using Sparse Hierarchical Implicit Neural Representations", ICRA, 2023. [Paper] [Code]

• "SDF-based RGB-D Camera Tracking in Neural Scene Representations", ICRA Workshop, 2022. [Paper]

• Loc-NeRF: "Monte Carlo Localization using Neural Radiance Fields", ICRA, 2023. [Paper] [Code] [Video]

• Vox-Fusion: "Dense Tracking and Mapping with Voxel-based Neural Implicit Representation", ISMAR, 2022. [Paper] [Website] [Pytorch Code] [Video]

• NodeSLAM: "Dense Tracking and Mapping with Voxel-based Neural Implicit Representation", 3DV, 2020. [Paper]

• iLabel: "Revealing Objects in Neural Fields", RA-L, 2023. [Paper]

• Nerf–: "Neural radiance fields without known camera parameters", arXiv. [Paper]

• L2G-NeRF: "Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields", CVPR, 2023. [Paper] [Website] [code]

• H2-Mapping: "Real-time Dense Mapping Using Hierarchical Hybrid Representation", RA-L, 2023. [Paper] [code]

• Continual Neural Mapping: "Learning An Implicit Scene Representation from Sequential Observations", ICCV, 2021. [Paper]

• LATITUDE: Robotic Global Localization with Truncated Dynamic Low-pass Filter in City-scale NeRF, ICRA, 2023. [Paper] [Pytorch Code]

• "Dense RGB SLAM with neural implicit maps", ICLR, 2023. [Paper]

• NOCaL: Calibration-free semi-supervised learning of odometry and camera intrinsics, ICRA, 2023. [Paper] [Website]

• IRMCL: Implicit Representation-based Online Global Localization, arXiv. [Paper] [Code]

• Efficient Implicit Neural Reconstruction Using LiDAR, ICRA, 2023. [Paper] [Website] [Pytorch Code] [Video]

• vMAP: "Vectorised Object Mapping for Neural Field SLAM", CVPR, 2023. [Paper] [Website]

• "An Algorithm for the SE(3)-Transformation on Neural Implicit Maps for Remapping Functions", RA-L, 2022. [Paper]

• "Implicit Object Reconstruction With Noisy Data", RSS Workshop, 2021. [Paper]

• NeuSE: "Neural SE(3)-Equivariant Embedding for Consistent Spatial Understanding with Objects", arXiv. [Paper] [Website]

• ObjectFusion: "Accurate object-level SLAM with neural object priors", Graphical Models, 2022. [Paper]

• NDF_Change: "Robust Change Detection Based on Neural Descriptor Fields", IROS, 2022. [Paper]

• LNDF: "Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation", ICRA, 2023. [Paper] [Webpage]

• NeRF-LOAM: Neural Implicit Representation for Large-Scale Incremental LiDAR Odometry and Mapping, arXiv. [Paper] [Code]

• NeuralBlox: "Real-Time Neural Representation Fusion for Robust Volumetric Mapping", 3DV, 2021. [Paper] [Code]

• Di-fusion: Online implicit 3d reconstruction with deep priors, CVPR, 2021.[Paper] [Pytorch Code]

• CROSSFIRE: "Camera Relocalization On Self-Supervised Features from an Implicit Representation", arXiv. [Paper]

• SDF-Loc: "Signed Distance Field based 2D Relocalization and Map Update in Dynamic Environments", ACC, 2019. [Paper]

• iDF-SLAM: "End-to-End RGB-D SLAM with Neural Implicit Mapping and Deep Feature Tracking", arXiv. [Paper]

• "Implicit Map Augmentation for Relocalization", ECCV Workshop, 2022. [Paper]

• "Visual-Inertial Odometry Priors for Bundle-Adjusting Neural Radiance Fields", ICCAS, 2022. [Paper]

• "Towards Open World NeRF-Based SLAM", arXiv, 2023. [Paper]

• Uni-Fusion: Universal Continuous Mapping, arXiv, 2023.[Paper] [Website]

• NEWTON: Neural View-Centric Mapping for On-the-Fly Large-Scale SLAM, arXiv, 2023.[Paper]

• Point-SLAM: Dense Neural Point Cloud-based SLAM, arXiv, 2023. [Paper] [Code]

• "ActiveRMAP: Radiance Field for Active Mapping And Planning", *arXiv, 2022". [Paper]

• NF-Atlas: Multi-Volume Neural Feature Fields for Large Scale LiDAR Mapping, arXiv, 2023. [Paper]

• RO-MAP: Real-Time Multi-Object Mapping with Neural Radiance Fields, arXiv, 2023. [Paper] [Code] [Video]

• Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM, CVPR, 2023. [Paper] [Website]

• Neural Implicit Dense Semantic SLAM, arXiv, 2023. [Paper]

• 2D-SDF-SLAM: "A Signed Distance Function based SLAM Frontend for Laser Scanners", IROS, 2015. [Paper]

• LONER: "LiDAR Only Neural Representations for Real-Time SLAM", RA-L, 2023. [paper] [code] [website]

Manipulation/RL

• GNFactor: "GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields", CoRL 2023 Oral Presentation. [Paper/PDF] [Code] [Website]

• D³Fields: "D³Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation", arXiv. [Paper] [Webpage] [Code] [Video]

• SNeRL: "Semantic-aware Neural Radiance Fields for Reinforcement Learning", ICML, 2023. [Paper] [Webpage]

• Ditto: "Building Digital Twins of Articulated Objects from Interaction", CVPR, 2022. [Paper] [Pytorch Code] [Website]

• Relational-NDF: "SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields", CORL 2022. [Paper] [Pytorch Code] [Website]

• Neural Descriptor Fields: "SE(3)-Equivariant Object Representations for Manipulation", arXiv. [Paper] [Pytorch Code] [Website]

• Evo-NeRF: "Evolving NeRF for Sequential Robot Grasping of Transparent Objects", CORL 2022. [Paper] [Website]

• NeRF-RL: "Reinforcement Learning with Neural Radiance Fields", arXiv. [Paper] [Website]

• Neural Motion Fields: "Encoding Grasp Trajectories as Implicit Value Functions", RSS 2022. [Paper] [Video]

• Grasping Field: "Learning Implicit Representations for Human Grasps", 3DV 2020. [Paper] [Pytorch Code] [Video]

• Dex-NeRF: "Using a Neural Radiance Field to Grasp Transparent Objects", CORL, 2021. [Paper] [Website]

• NeRF-Supervision: "Learning Dense Object Descriptors from Neural Radiance Fields", ICRA, 2022. [Paper] [Pytorch Code] [Website]

• GIGA: "Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations", RSS, 2021. [Paper] [Pytorch Code] [Website]

• NeuralGrasps: "Learning Implicit Representations for Grasps of Multiple Robotic Hands", CORL, 2022. [Paper] [Website]

• "Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter", CORL, 2022. [Paper] [Website]

• ObjectFolder: "A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations"", CORL, 2021. [Paper] [Pytorch Code] [Website]

• ObjectFolder 2.0: "A Multisensory Object Dataset for Sim2Real Transfer"", CVPR, 2022. [Paper] [Pytorch Code] [Website]

• "Template-Based Category-Agnostic Instance Detection for Robotic Manipulation"", RA-L, 2022. [Paper]

• NeRF2Real: "Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields"", arXiv. [Paper] [Website]

• NeRF-Frenemy: "Co-Opting Adversarial Learning for Autonomy-Directed Co-Design", RSS Workshop, 2022. [Paper] [Website]

• CLA-NeRF: "Category-Level Articulated Neural Radiance Field", ICRA, 2022. [Paper] [Website]

• VIRDO: "Visio-tactile Implicit Representations of Deformable Objects", ICRA, 2022. [Paper] [Website]

• VIRDO++:: "Real-World, Visuo-Tactile Dynamics and Perception of Deformable Objects", CORL, 2022. [Paper] [Website]

• SceneCollisionNet: "Object Rearrangement Using Learned Implicit Collision Functions", ICRA, 2021. [Paper] [Website]

• "RGB-D Local Implicit Function for Depth Completion of Transparent Objects", CVPR, 2021. [Paper] [Website]

• "Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning", CORL, 2021. [Paper] [Video]

• ContactNets: "Learning Discontinuous Contact Dynamics with Smooth, Implicit Representations", CORL, 2020. [Paper] [Pytorch Code]

• "Learning Implicit Priors for Motion Optimization", IROS, 2022. [Paper] [Website]

• MIRA: "Mental Imagery for Robotic Affordances", CORL, 2022. [Paper] [Website]

• NiFR: "Neural Fields for Robotic Object Manipulation from a Single Image", ICRA, 2023. [Paper]

• NIFT: "Neural Interaction Field and Template for Object Manipulation", ICRA, 2023. [Paper]

• "Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance", "IROS, 2022". [Paper]

• GraspNeRF: "Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF", ICRA, 2023. [Paper]

• Touching a NeRF: "Leveraging Neural Radiance Fields for Tactile Sensory Data Generation ", CORL, 2022. [Paper]

• SE(3)-DiffusionFields: "Learning smooth cost functions for joint grasp and motion optimization through diffusion", ICRA, 2023. [Paper] [Pytorch Code]

• Equivariant Descriptor Fields: "SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning ", ICLR, 2023. [Paper]

• KP-NERF: "Dynamical Scene Representation and Control with Keypoint-Conditioned Neural Radiance Field", CASE, 2022. [Paper]

• ACID: "Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation", RSS, 2022. [Paper] [Pytorch Code]

• TRITON: "Neural Neural Textures Make Sim2Real Consistent", CORL, 2022. [Paper] [Website] [Pytorch Code]

• "Perceiving Unseen 3D Objects by Poking the Objects", ICRA, 2023. [Paper] [Website] [Pytorch Code]

• "Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding", ICRA, 2023. [Paper] [Website]

• CGF: "Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations", arXiv. [Paper] [Website]

• NGDF: "Neural Grasp Distance Fields for Robot Manipulation", arXiv. [Paper] [Website] [Pytorch Code]

• NCF: "Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing", arXiv. [Paper] [Pytorch Code]

• SPARTN: "NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis", arXiv. [Paper]

• "RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control", arXiv. [Paper] [Website]

• ""Affordance Transfer based on Self-Aligning Implicit Representations of Local Surfaces", "RSS Workshop, 2022". [Paper][https://imrss2022.github.io/contributions/tekden.pdf]

• "A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstruction", arXiv. [Paper] [Video]

• EndoNeRF: "Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery", MICCAI, 2022. [Paper] [Pytorch Code]

Object Reconstruction

• "Self-supervised Neural Articulated Shape and Appearance Models", CVPR, 2022. [Paper] [Website]

• NeuS: "Learning Neural Implicit Surfacesby Volume Rendering for Multi-view Reconstruction", Neurips, 2021. [Paper] [Website]

• VolSDF: "Volume Rendering of Neural Implicit Surfaces", Neurips, 2021. [Paper] [Pytorch Code]

• UNISURF: "Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction", ICCV, 2021. [Paper] [Website] [Pytorch Code]

• ObjectSDF: "Object-Compositional Neural Implicit Surfaces", ECCV, 2022. [Paper] [Website] [Pytorch Code]

• IDR: "Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance", Neurips, 2020. [Paper] [Website] [Pytorch Code]

• DVR: "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision", CVPR, 2020. [Paper] [Pytorch Code]

• A-SDF: "Learning Disentangled Signed Distance Functions for Articulated Shape Representation", ICCV, 2021. [Paper] [Pytorch Code]

• CodeNeRF: "Disentangled Neural Radiance Fields for Object Categories", ICCV, 2021. [Paper] [Pytorch Code]

• DeepSDF: " Learning Continuous Signed Distance Functions for Shape Representation", CVPR, 2019. [Paper] [Pytorch Code]

• Occupancy networks: " Learning 3d reconstruction in function space", CVPR, 2019. [Paper] [Website]

Physics

• DANOs: "Differentiable Physics Simulation of Dynamics-Augmented Neural Objects", arXiv. [Paper] [Video]

• PAC-NeRF: "Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification", ICLR, 23. [Paper] [Website] [Video] [Pytorch Code]

• NeuPhysics: "Editable Neural Geometry and Physics from Monocular Videos", Neurips, 2022. [Paper] [Pytorch Code] [Website]

• NeRF-ysics: "A Differentiable Pipeline for Enriching NeRF-Represented Objects with Dynamical Properties", ICRA Workshop, 2022. [Paper]

• "Neural Implicit Representations for Physical Parameter Inference from a Single Video", WACV, 2023. [Paper] [Pytorch Code] [Website]

• NeuroFluid: "Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields", ICML, 2022. [Paper] [Pytorch Code]

Planning/Navigation

• NeRFlow: "Neural Radiance Flow for 4D View Synthesis and Video Processing", ICCV, 2021. [Paper] [Pytorch Code] [Website]

• NeRF-Navigation: "Vision-Only Robot Navigation in a Neural Radiance World", ICRA, 2022. [Paper] [Pytorch Code] [Website]

• RNR-Map: "Renderable Neural Radiance Map for Visual Navigation", CVPR, 2023. [Paper] [Website]

• "Uncertainty Guided Policy for Active Robotic 3D Reconstruction using Neural Radiance Fields", RAL, 2022. [[Paper (https://arxiv.org/pdf/2209.08409.pdf)] [Website]

• NeRF-dy: "3D Neural Scene Representations for Visuomotor Control", CORL, 2021. [Paper] [Website]

• CompNeRFdyn: "Learning Multi-Object Dynamics with Compositional Neural Radiance Fields", arXiv. [Paper] [Website]

• PIFO: "Deep Visual Constraints: Neural Implicit Models for Manipulation Planning from Visual Input", arXiv. [Paper] [Website]

• "Learning Continuous Environment Fields via Implicit Functions", ICLR, 2022. [Paper] [Website]

• "Learning Barrier Functions with Memory for Robust Safe Navigation", RA-L, 2021. [Paper]

• RedSDF: "Regularized Deep Signed Distance Fields for Reactive Motion Generation", IROS, 2022. [Paper] [Website]

• AutoNeRF: "Training Implicit Scene Representations with Autonomous Agents", arxiv. [Paper] [Website]

• ESDF: "Sampling-free obstacle gradients and reactive planning in Neural Radiance Fields", arXiv. [Paper]

• CLIP-Fields: "Open-label semantic navigation with pre-trained VLMs and language models", arxiv. [Paper] [Pytorch Code and Tutorials] [Website]

• Voxfield: Non-Projective Signed Distance Fields for Online Planning and 3D Reconstruction", IROS, 2022. [Paper] [Pytorch Code]

• Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning, IROS, 2017. [Paper]

• NFOMP: Neural Field for Optimal Motion Planner of Differential Drive Robots With Nonholonomic Constraints", RA-L, 2022. [Paper] [Video]

• CATNIPS: Collision Avoidance Through Neural Implicit Probabilistic Scenes", arXiv. [Paper]

• MeSLAM: Memory Efficient SLAM based on Neural Fields, IEEE SMC, 2022. [Paper]

• NTFields: "Neural Time Fields for Physics-Informed Robot Motion Planning", ICLR, 2023. [Paper]

• "Real-time Semantic 3D Reconstruction for High-Touch Surface Recognition for Robotic Disinfection", IROS, 2022. [Paper]

• NeurAR: "Neural Uncertainty for Autonomous 3D Reconstruction", RA-L, 2023. [Paper]

• IR-MCL: "Implicit Representation-Based Online Global Localization", RA-L, 2023. [Paper] [Pytorch Code]

• 360Roam: "Real-Time Indoor Roaming Using Geometry-Aware 360◦ Radiance Fields", arXiv. [Paper] [Pytorch Code]

• "Learning Deep SDF Maps Online for Robot Navigation and Exploration", arXiv. [Paper]

• DroNeRF: Real-time Multi-agent Drone Pose Optimization for Computing Neural Radiance Fields. [Paper]

• "Enforcing safety for vision-based controllers via Control Barrier Functions and Neural Radiance Fields", arXiv. [Paper]

• "Full-Body Visual Self-Modeling of Robot Morphologies", arXiv. [Paper] [Website]

• "Efficient View Path Planning for Autonomous Implicit Reconstruction", arxiv. [Paper]

• "Multi-Object Navigation with dynamically learned neural implicit representations", arxiv. [Paper] [Website

Citation

If you find this repository useful, please consider citing this list:

@misc{irshad2022implicitnerfroboticsresources,
    title = {Awesome Implicit NeRF Robotics - A curated list of resources on implicit neural representations and nerf relating to robotics},
    author = {Muhammad Zubair Irshad},
    journal = {GitHub repository},
    url = {https://github.com/zubair-irshad/Awesome-Implicit-NeRF-Robotics},
    DOI= {10.5281/ZENODO.7552613}
    year = {2022},
}