Kaolin Wisp Versions Save

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp and VQAD).

v0.1.2

1 year ago

Highlights

  • Easier installation: pycuda deprecated in favor of cuda-python, support for WSL added, some dependencies removed.
  • Major refactor to the lib: cleanup of stale research code, added documentation and introduction of modular base classes such as BaseAS (acceleration structure, i.e. for raymarching), WispModule (any wisp module that can display properties in the renderer).
  • Optimized hashgrid kernel and pruning
  • Jupyter notebook support added

What's Changed

New Contributors

Full Changelog: https://github.com/NVIDIAGameWorks/kaolin-wisp/compare/v0.1.1...v0.1.2

v0.1.1

1 year ago

Highlights

The second release of Wisp features:

  • Support for installation with Docker
  • Tracers now handle extra channels properly
  • Object transformations allowing to scale, rotate and translate ray based neural fields
  • Support for running Wisp on Windows (external contribution by @3a1b2c3)
  • Integration with Weights&Biases, including 360 visualizations (external contribution by @soumik12345)
  • Two demos: Visualizing NeRF latent channels and Browsing Structured Point Clouds.
  • Various bug fixes and more..

What's Changed

New Contributors

Full Changelog: https://github.com/NVIDIAGameWorks/kaolin-wisp/compare/v0.1.0...v0.1.1

v0.1.0

1 year ago

Highlights

First kaolin-wisp release. Supports full optimization pipelines of neural radiance fields and signed distance functions. Also includes an interactive renderer to visualize the optimization progress.

Features List

Core

Acceleration Structures

An Octree allows for fast location based queries and raymarching operations. On NVIDIA RTX 3090, that brings NeRF training time down to anywhere between tens of seconds to single minutes for single objects like the bulldozer. (The exact performance is affected by other blocks in the neural pipeline: tracer, feature structure, etc..)

Feature Grids

The initial version of wisp is shipped with 4 types of feature grids:

Additional Building Blocks

Support for Positional Encoding (Fourier), torch MLP Decoders.

Neural Fields

  • Neural Fields can be composed and configured with the various building blocks: occupancy & feature structures, embedders, decoders and so forth.
  • Wisp is shipped with implementations for NeuralRadianceFields and NeuralSDFs.
  • New types of neural fields can be registered for research purposes.

Tracers

The tracers framework supports types of channels (i.e: rgba, sdf) 2 implementations of differentiable ray tracers are included.

  • Radiance Fields
  • Sphere tracer for SDFs Both tracers keep samples as packed tensors and use the octree to accelerated performance. Raymarching backed by the octree supports 2 modes of sampling along the ray:
  • 'voxel' - fixed amount of samples will be generated per "octree cell"
  • 'ray' - fixed amount of samples will be generated between the near / far planes of the camera.

Core Structures

  • RenderBuffer - smart pixel buffers used by tracers. Support custom types of Channels and various blending and channel normalization modes.
  • Ray - batched ray packs for ray generation

Ray Generators

  • Ray generation uses kaolin's Camera module.
  • Default ray generators for perspective and ortho cameras are available.

Ops

A toolkit for loading, processing and evaluation of various modalities is included in this version (including images, meshes, sdfs and kaolin's Structured Point Clouds.

Data Loaders

wisp ships with data loaders support for:

  • Multiview images: NeRF standard dataset format, including extensions to the format introduced by Instant Neural Graphics Primitives).
  • Multiview images: RTMV dataset .
  • SDFs: OBJs + online samples (usually done once in the beginning of optimization)

Training

2 trainers are currently supported (for NeRF and SDFs). Trainers can run with or without the interactive renderer.

Interactive Renderer

Wisp includes an interactive renderer with flexible support for new types of neural objects. The renderer was built in mind to support future neural pipelines which may not yet exist. Dependencies: pycuda, glumpy, glfw, imgui.

Scene Graph & Renderers

  • The renderer core fully supports neural fields and tracers currently available in wisp core (NeRF, SDFs traced via the packed tracers).
  • Neural fields are implemented as standalone renderers which produce a RenderBuffer. The renderer core supports multiple shape types by blending such buffers.
  • All objects are stored inside a scene graph. The scene graph can be manipulated to toggle objects visibility and adjust properties.

Debug Visualizations

  • Line primitives can be drawn on the canvas to display additional layers of information (i.e. Octree occupancy).
  • The optimization runs in parallel to the renderer, allowing for ad-hoc interaction with the optimized scene.

Canvas Modes

  • The canvas supports visualization of custom channels (rgb, depth..)
  • Camera can switch between perspective and ortho
  • Camera controllers: First person, Trackball, Turntable

Custom Apps & Gui

  • Wisp apps are highly customizable: the gui and canvas gizmos can be configured to display specific menus / canvas transients (such as the world grid).
  • The gui can be extended with new types of widgets.

Full Changelog: https://github.com/NVIDIAGameWorks/kaolin-wisp/commits/v0.1.0