An unofficial Unity port of the MobileNeRF viewer
This repository contains the source code for a Unity port of the web viewer from the paper MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures[^1]
Please note, that this is an unofficial port. I am not affiliated with the original authors or their institution.
Go to the releases section, download the Unity Package, and import it into any Unity project. This is a 'Hybrid Package' that will install into your project as a local package.
In Edit -> Project Settings -> Package Manager
, add a new scoped registry:
Name: Doji
URL: https://package.openupm.com
Scope(s): com.doji
In the Package Manager install 'com.doji.mobilenerf either by name or via Package Manager -> My Registries
In Package Manager -> Add package from git URL...
paste https://github.com/julienkay/MobileNeRF-Unity-Viewer.git
as described here
After succesful installation, you can use the menu MobileNeRF -> Asset Downloads
to download any of the sample scenes available.
In each scene folder there will be a convenient prefab, that you can then drag into the scene and you're good to go.
Since the initial release a small number of features have been added to the automatic shader generation code.
That means, that if you have already downloaded some scenes before, you'll have to regenerate the source files by going to MobileNeRF -> Asset Downloads
again.
(This will not actually redownload Assets unless necessary, so this shouldn't take too long)
If you have successfully trained your own MobileNeRF scenes using the official code release and want to render them in Unity, you can use the menu MobileNeRF -> Import from disk. This lets you choose a folder that should contain all the output files of your training process.
More specifically, the following assets are required:
The project was created with Unity 2021.3 LTS using the Built-in Render Pipeline.
The biggest deviation from the official viewer is, that this project doesn't use Deferred Rendering, but uses Forward Rendering instead. This has certain implications on performance. While the MobileNeRF representation itself greatly reduces the cost to render NeRFs, it still requires evaluating a small, view-dependent MLP (Multi Layer Perceptron) per fragment. Whenever the bottleneck is in the fragment shader, Deferred Rendering has obvious advantages, as each pixel only needs to run a single fragment shader.
Forward Rendering however gives us MSAA, which is important for VR use cases. Additionally, in VR the image has to be rendered twice, once for each eye, with a fairly large resolution. The larger the G-buffer, the smaller the benefit of Deferred Rendering. Still, MobileNeRFs mesh representations have a fairly large poly count which works against us using Forward Rendering here.
Some things to possibly look into:
Project Settings -> Quality -> Anti Aliasing
Thanks to
Other projects exploring NeRFs and related techniques in Unity: