A PyTorch Library for Accelerating 3D Deep Learning Research
Add methods and property to simplify camera transformation (#751) Allow camera width and height to be modified without modifying FOV (#751) Support .obj files with R G B defined (#760) Make arguments optional in PBRMaterials and add helper methods, and improve import / export of materials for USD (#762) Add GLTF loader (#763) Add translation movement in turntable visualizer (#767) Improve API of usd and obj loaders for triangulation (#774) Add optional canvas as argument to visualizers (#775) Add vertex tangents to the mesh class (#763)
Fix drifting bug in turntable visualizer (#767)
Added support for pytorch 2.1.0 and 2.1.1 (#765) Added support for cuda 12.1 (#765)
Sanja Fidler Clement Fuji Tsang Or Perel Tianchang (Frank) Shen Masha Shugrina Alexander Zook @mikerjacobi
Added mesh to spc conversions (#696) Added interactive visualizers (#704, #723, ##742) Added high-level SurfaceMesh class (#740)
Fixed trilinear interpolation gradient when the point is not on voxel (#684) Simplified and fix material importation from USD and OBJ (#708, #718, #724) Fixed UV bug in examples using nvdiffrast (#742)
Added support for pytorch 2.0.0 and 2.0.1 (#717, #741) Added support for python 3.10 Dropped support for python 3.7
Sanja Fidler Clement Fuji Tsang Charles Loop Or Perel Masha Shugrina Alexander Zook
Reformat dataset preprocessing with CachedDataset (#626) Added uniform spherical sampling and conversion between spherical and cartesian (#661)
Add lighting features: (#663)
Added support for internal transforms in USD mesh import (#667) Add gradients on coordinates for trilinear interpolation (#650)
Improved memory consumption of uniform_laplacian (#643) Improved ~10% speed on SPC ray tracing (#634)
Fix bug enabling raytracing with SPC from within the [-1, 1] range (#634) Fix bug on SPC scan_octrees with sometimes allocation issue (#653)
Added new recipe for Fast Mesh Sampling preprocessing (#626) Added new recipe for SPC 3D convolution (#621) Added Lighting tutorials for diffuse and specular (#663)
Kaolin is now pip installable.
Sanja Fidler Clement Fuji Tsang Jean-Francois Lafleche Charles Loop Or Perel Masha Shugrina Towaki Takikawa Alexander Zook @Mason-McGough
With the version 0.12.0 we have added a Camera API, allowing to use all our renderers and multiple coordinate systems.
Checkout our news tutorials:
Added Camera API
Fix bug with kaolin.ops.mesh.check_sign when the point is aligned with a vertice or an edge.
Fix bug on some ops when using 2nd GPU
Added a bunch of recipes for Camera API Added a tutorial to show how to use Camera API with nvdiffrast
Sanja Fidler Clement Fuji Tsang Or Perel Masha Shugrina Towaki Takikawa Jiehan Wang Alexander Zook
In Kaolin 0.11.0 we are focusing on improving performance for our main renderers, strongly improving SPC raytracing and trilinear interpolation, and integrating nvdiffrast as a backend for DIB-R rasterization. We are also adding a few features such as tetrahedral and triangle mesh subdivision, support for heterogeneous mesh in obj loader, as well as improving Dash3D usability.
Finally, several tutorials and recipes are implemented for new users to quickly get a grasp on Kaolin features.
Improved Dash3D usability (#538) Improved SPC raytracing memory usage / speed (~12x less memory used / +33% on a test model at level 11) (#548) Added tetrahedral mesh subdivision used in DMTet (#551) Added triangle mesh subdivision used in DMTet (#562) Allowed SPC unbatched_query to output parents nodes (#559) Splitted rasterization and dibr soft mask in two functions (#560) Integrated nvdiffrast for rasterization (measured up to x640 faster on large model from ShapeNet at 1024x1024 resolution) (#560) Added support for Heterogeneous mesh in obj loader (#563) Implemented a fused trilinear interpolation kernel with computation of dual of octree (#567)
Added recipe to convert pointcloud to SPC (#535) Added recipe for basic explanation of spc's octree (#535) Added example for fitting a 3D bounding box using differentiable renderer (#543) Added recipe to compute Occupancy using check_sign (#564) Showed backend keyword for rasterization in DIB-R tutorial (#569) Added recipe for the dual octree and trilinear_interpolation
Fixed DIB-R tutorial due to normalization changing in the Omniverse App (#532) Fixed issue with uint type issue on Windows (#537) Fixed f_score reduction bug (#546) Fixed indexing bug on understanding SPC tutorial (#553)
Sanja Fidler Clement Fuji Tsang Charles Loop Or Perel Frank Shen Masha Shugrina Gavriel State Towaki Takikawa Jiehan Wang Alexander Zook
In Kaolin 0.10.0 we are focusing on Volumetric rendering, adding new features for tetrahedral meshes, including DefTet volumetric renderer and losses, and Deep Marching Tetrahedrons, and adding new primitive operations for efficient volumetric rendering of Structured Point Clouds, we are also adding support for materials with USD importation.
Finally we are adding two new tutorials to show how to use the latest features from Kaolin:
How to use DMtet to reconstruct a mesh from point clouds generated by the Omniverse Kaolin App
An Introduction to Structured Point Clouds, with conversion from mesh and interactive visualization with raytracing.
Simplify kaolin.ops.spc.unbatched_query API (#442) Added point to vertice type of distance in kaolin.metrics.trianglemesh.point_to_mesh_distance, with a little speedup (#446, #458) Added new “thin” mode for kaolin.ops.voxelgrids.extract_surface (#448) Adding Marching Tetrahedra (#476) Extend SPC raytracing to output depth (#478) Refactor SPC raytracing API (#478) Adding Differentiable Volumetric rendering features for SPC (#485) Adding “squared” option for Chamfer distance (#486) Adding Deftet Volumetric Renderer (#488) Adding Deftet losses (#496) Adding interpolation of features for point sampling on mesh (#499) Adding DMtet Tutorial (#492) Adding SPC Tutorial (#500) Adding unbatched_pointcloud_to_spc wrapper (#498) Adding materials support for USD importers (#502)
Fix small bugs on USD importer / exporter (#441, #445) Fix trianglemesh_to_voxelgrids when sparse (#455) Fix bug where Kaolin were not building with submodule CUB (#457) Fix Preprocessing bug where “name” attributes contains “/” (#469)
Define a proper C++ Style Guide and fine-tune codebase accordingly (#470, #471, #472, #477)
DIB-R Deftet DMtet GradSim NeuralLOD Text2Mesh
Sanja Fidler Clement Fuji Tsang Jun Gao Jean-François Lafleche Michael Li Charles Loop Or Perel Frank Shen Masha Shugrina Gavriel State Towaki Takikawa Jiehan Wang Alexander Zook (github) Talmaj (github) le-Greg
The latest Kaolin release includes a new representation, structured point clouds, an octree-based acceleration data structure, with highly efficient convolution and ray tracing capabilities. This representation is useful for neural implicit representations, popular in 3D DL research today, and beyond, and powers the latest version of NeuralLOD training.
The release is also coming with extended support for 3d dataset like ModelNet / ShapeNet / SHREC, new utility functions to improve usability and speedups on import / export of USD used in checkpoints. In this version, we added a lightweight visualizer Dash3D for quickly visualizing from a low-end config such as a laptop.
index_by_face_vertices (#419)do not use
The Kaolin 0.9 release include a reformat of the API and various improvment of the performance and the ergonomy of Kaolin. A reformat was required to be able to have a maintainable, clean and reliable Kaolin in the long term.
Mesh class contained too many attributes and methods that were too specific or unused or redundant. Also given how quickly the field can shift to new methods, having a fixed class representation can be a constraint. We chose to focus on low-level functions with torch tensors as inputs / outputs, to favor reusability. High-level representation will be added later once the common use cases get more easy to define.
Maintainable and reliable Kaolin means a more compact library. We decided to move the model zoo out of Kaolin, this model zoo will have a dedicate repository, will rely on release of Kaolin, and so will be maintained separately.
Kaolin is now fully batched, by default with a fixed topology, but also (with limited support) representation for heterogenous structures using packed and padded approch, see documentation for more details. We intend to provide more primitive ops for heterogenous structures.
We've been mostly focusing on GPU efficiency. Among the optimizations, speedups are reported on:
kaolin.render.mesh.rasterization.dibr_rasterization(height, width, face_vertice_z, face_vertices_image, face_features, face_normals_z) (~1.35x faster).kaolin.ops.gcn.GraphConv(node_feat, adj, normalized_adj=False) (~1.85x faster).kaolin.ops.mesh.check_sign(vertices, faces, points, hash_resolution): (~2.75x faster).kaolin.ops.mesh.sample_points(vertices, faces, num_samples, areas): added a functionality of pre-computation of faces areas (~1.6x faster)kaolin.ops.conversions.voxelgrids_to_cubic_meshes(voxelgrids, is_trimesh) (~17x faster on cpu, >10000x faster on gpu)kaolin.ops.voxelgrid.downsample(voxelgrids, scale) (~6.2x faster on cpu, ~25x faster on gpu)kaolin.ops.voxelgrid.fill(voxelgrids) (~1.3x faster on cpu)kaolin.ops.voxelgrid.extract_surface(voxelgrids) (~6.9x faster on cpu, ~37x faster on gpu)kaolin.ops.voxelgrid.extract_odms(voxelgrids) (~250x faster on cpu, ~1276x faster on gpu)kaolin.ops.voxelgrid.project_odms(odms, voxelgrids, votes) (~125x faster on cpum ~882x faster on gpu)
We added a cuda implementation of lorensen's marching cube (used in kaolin.ops.conversions.voxelgrids_to_trianglemeshes(voxelgrids, iso_value))
We added backpropagation to the triangle distance (used in kaolin.metrics.trianglemesh.point_to_mesh_distance(pointcloud, vertices, faces)) and side distance (used in kaolin.metrics.pointcloud.sided_distance(p1, p2), kaolin.metrics.pointcloud.chamfer_distance(p1, p2, w1, w2), kaolin.metrics.pointcloud.f_score(gt_points, pred_points, radius, eps)).We now provide importer and exporter to Universal Scene Description files, see the documentation for more information. You can open those file using the Omniverse companion app, see Kaolin Devpage.
In alphabetical order:
Wenzheng Chen Sanja Fidler Clement Fuji Tsang Jason Gorski Jean-Francois Lafleche Rev Lebaredian Jianing Li Frank Shen Masha Shugrina Gavriel State Jiehan Wang Tommy Xiang
first version of Kaolin beta