Nerf Pl Versions Save
NeRF (Neural Radiance Fields) and NeRF in the Wild using pytorch-lightning
nerfw_branden
3 years agoUsed for nerfw branch.
Train command (trained on 8 time downscaled images, just for proof of implementation):
python prepare_phototourism.py --root_dir /home/ubuntu/data/IMC-PT/brandenburg_gate/ --img_downscale 8
python train.py \
--root_dir /home/ubuntu/data/IMC-PT/brandenburg_gate/ --dataset_name phototourism \
--img_downscale 8 --use_cache \
--N_importance 64 --N_samples 64 --encode_a --encode_t --beta_min 0.03 --N_vocab 1500 --N_emb_xyz 15 \
--num_epochs 20 --batch_size 1024 \
--optimizer adam --lr 5e-4 --lr_scheduler cosine \
--exp_name brandenburg_scale8_nerfw
Profiler Report
Action | Mean duration (s) |Num calls | Total time (s) | Percentage % |
-----------------------------------------------------------------------------------------------------------------------------
Total | - |_ | 2.5398e+04 | 100 % |
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch | 1269.8 |20 | 2.5396e+04 | 99.991 |
run_training_batch | 0.14633 |170760 | 2.4988e+04 | 98.384 |
optimizer_step_and_closure_0 | 0.12823 |170760 | 2.1896e+04 | 86.212 |
training_step_and_backward | 0.1241 |170760 | 2.1192e+04 | 83.438 |
model_backward | 0.099837 |170760 | 1.7048e+04 | 67.124 |
model_forward | 0.024055 |170760 | 4107.6 | 16.173 |
on_train_batch_end | 0.00052083 |170760 | 88.938 | 0.35018 |
get_train_batch | 0.00023393 |170760 | 39.946 | 0.15728 |
evaluation_step_and_end | 0.52576 |21 | 11.041 | 0.043472 |
cache_result | 1.2894e-05 |854050 | 11.012 | 0.043357 |
on_after_backward | 1.0743e-05 |170760 | 1.8345 | 0.007223 |
on_batch_start | 1.0535e-05 |170760 | 1.799 | 0.0070832 |
on_batch_end | 9.6894e-06 |170760 | 1.6546 | 0.0065145 |
on_before_zero_grad | 8.5198e-06 |170760 | 1.4548 | 0.0057282 |
training_step_end | 6.6891e-06 |170760 | 1.1422 | 0.0044974 |
on_train_batch_start | 5.9285e-06 |170760 | 1.0124 | 0.003986 |
on_validation_end | 0.027978 |21 | 0.58754 | 0.0023133 |
on_validation_batch_end | 0.00055518 |21 | 0.011659 | 4.5904e-05 |
on_epoch_start | 0.00054319 |20 | 0.010864 | 4.2774e-05 |
on_validation_start | 0.00024484 |21 | 0.0051417 | 2.0244e-05 |
on_validation_batch_start | 5.3095e-05 |21 | 0.001115 | 4.3901e-06 |
validation_step_end | 2.1799e-05 |21 | 0.00045779 | 1.8024e-06 |
on_train_epoch_start | 1.7319e-05 |20 | 0.00034637 | 1.3638e-06 |
on_epoch_end | 1.5776e-05 |20 | 0.00031551 | 1.2423e-06 |
on_train_end | 0.0002874 |1 | 0.0002874 | 1.1316e-06 |
on_validation_epoch_end | 1.1708e-05 |21 | 0.00024586 | 9.6803e-07 |
on_validation_epoch_start | 8.0324e-06 |21 | 0.00016868 | 6.6415e-07 |
on_train_start | 0.00015864 |1 | 0.00015864 | 6.2463e-07 |
on_train_epoch_end | 7.2367e-06 |20 | 0.00014473 | 5.6986e-07 |
on_fit_start | 1.4059e-05 |1 | 1.4059e-05 | 5.5355e-08 |
Eval command (used for scale2_epoch29 model):
python eval.py \
--root_dir /home/ubuntu/data/IMC-PT/brandenburg_gate/ \
--dataset_name phototourism --scene_name brandenburg_test \
--split test --N_samples 256 --N_importance 256 \
--N_vocab 1500 --encode_a --encode_t \
--ckpt_path ckpts/brandenburg/scale2/epoch\=29.ckpt \
--chunk 16384 --img_wh 320 240
You can change the test camera path in eval.py.
nerfw_all
3 years agoUsed for nerfw branch.
Train command:
python train.py \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
--dataset_name blender --img_wh 200 200 --data_perturb color occ \
--N_importance 64 --N_samples 64 --noise_std 0 --encode_a --encode_t --beta_min 0.1 \
--num_epochs 20 --batch_size 1024 \
--optimizer adam --lr 5e-4 --lr_scheduler cosine \
--exp_name lego_nerfw_all
Eval command:
python eval.py \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
--dataset_name blender --split test --img_wh 200 200 \
--N_importance 64 --encode_a --encode_t --beta_min 0.1 \
--ckpt_path ckpts/lego_nerfw_all/epoch\=19.ckpt \
--scene_name nerfw_all
Eval output: Mean PSNR : 24.86
Profiler Report
Action | Mean duration (s) |Num calls | Total time (s) | Percentage % |
-----------------------------------------------------------------------------------------------------------------------------
Total | - |_ | 1.1659e+04 | 100 % |
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch | 582.57 |20 | 1.1651e+04 | 99.931 |
run_training_batch | 0.14307 |78140 | 1.1179e+04 | 95.882 |
optimizer_step_and_closure_0 | 0.12437 |78140 | 9718.4 | 83.352 |
training_step_and_backward | 0.12006 |78140 | 9381.8 | 80.465 |
model_backward | 0.095661 |78140 | 7475.0 | 64.111 |
model_forward | 0.024116 |78140 | 1884.5 | 16.162 |
evaluation_step_and_end | 1.8998 |161 | 305.86 | 2.6233 |
on_train_batch_end | 0.00053565 |78140 | 41.856 | 0.35898 |
get_train_batch | 0.00026832 |78140 | 20.966 | 0.17982 |
cache_result | 1.6708e-05 |391370 | 6.5391 | 0.056084 |
on_after_backward | 1.3945e-05 |78140 | 1.0897 | 0.0093458 |
on_batch_start | 1.1257e-05 |78140 | 0.87959 | 0.007544 |
on_batch_end | 1.0574e-05 |78140 | 0.82626 | 0.0070866 |
on_before_zero_grad | 9.9755e-06 |78140 | 0.77948 | 0.0066854 |
training_step_end | 7.3524e-06 |78140 | 0.57452 | 0.0049275 |
on_train_batch_start | 7.0481e-06 |78140 | 0.55074 | 0.0047235 |
on_validation_end | 0.025579 |21 | 0.53715 | 0.004607 |
on_validation_batch_end | 0.00039767 |161 | 0.064025 | 0.00054912 |
on_epoch_start | 0.00074399 |20 | 0.01488 | 0.00012762 |
on_validation_start | 0.00024646 |21 | 0.0051757 | 4.439e-05 |
on_train_end | 0.0033677 |1 | 0.0033677 | 2.8884e-05 |
on_validation_batch_start | 1.301e-05 |161 | 0.0020947 | 1.7965e-05 |
validation_step_end | 9.2702e-06 |161 | 0.0014925 | 1.2801e-05 |
on_epoch_end | 1.6658e-05 |20 | 0.00033316 | 2.8575e-06 |
on_validation_epoch_end | 1.4696e-05 |21 | 0.00030862 | 2.6469e-06 |
on_train_start | 0.00020975 |1 | 0.00020975 | 1.799e-06 |
on_validation_epoch_start | 9.7831e-06 |21 | 0.00020545 | 1.7621e-06 |
on_train_epoch_start | 9.096e-06 |20 | 0.00018192 | 1.5603e-06 |
on_train_epoch_end | 8.8208e-06 |20 | 0.00017642 | 1.5131e-06 |
on_fit_start | 1.3749e-05 |1 | 1.3749e-05 | 1.1792e-07 |
nerfa_color
3 years agoUsed for nerfw branch.
Train command:
python train.py \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
--dataset_name blender --img_wh 200 200 --data_perturb color \
--N_importance 64 --N_samples 64 --noise_std 0 --encode_a \
--num_epochs 20 --batch_size 1024 \
--optimizer adam --lr 5e-4 --lr_scheduler cosine \
--exp_name lego_nerfa_color
Eval command:
python eval.py \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
--dataset_name blender --split test --img_wh 200 200 \
--N_importance 64 --encode_a \
--ckpt_path ckpts/lego_nerfa_color/epoch\=19.ckpt \
--scene_name nerfa_color
Eval output: Mean PSNR : 28.20
Profiler Report
Action | Mean duration (s) |Num calls | Total time (s) | Percentage % |
-----------------------------------------------------------------------------------------------------------------------------
Total | - |_ | 1.0174e+04 | 100 % |
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch | 508.31 |20 | 1.0166e+04 | 99.922 |
run_training_batch | 0.12504 |78140 | 9770.7 | 96.036 |
optimizer_step_and_closure_0 | 0.10593 |78140 | 8277.7 | 81.362 |
training_step_and_backward | 0.10272 |78140 | 8026.6 | 78.893 |
model_backward | 0.081418 |78140 | 6362.0 | 62.532 |
model_forward | 0.021105 |78140 | 1649.1 | 16.209 |
evaluation_step_and_end | 1.6237 |161 | 261.41 | 2.5694 |
on_train_batch_end | 0.00040171 |78140 | 31.39 | 0.30853 |
get_train_batch | 0.0002557 |78140 | 19.981 | 0.19639 |
cache_result | 1.4961e-05 |391370 | 5.8553 | 0.057551 |
on_after_backward | 1.136e-05 |78140 | 0.88768 | 0.008725 |
on_batch_start | 1.0067e-05 |78140 | 0.78663 | 0.0077318 |
on_batch_end | 9.6172e-06 |78140 | 0.75149 | 0.0073863 |
on_before_zero_grad | 9.0155e-06 |78140 | 0.70447 | 0.0069242 |
on_validation_end | 0.026961 |21 | 0.56618 | 0.005565 |
training_step_end | 6.6222e-06 |78140 | 0.51746 | 0.0050861 |
on_train_batch_start | 6.3198e-06 |78140 | 0.49383 | 0.0048539 |
on_validation_batch_end | 0.00036434 |161 | 0.058659 | 0.00057656 |
on_epoch_start | 0.00047801 |20 | 0.0095601 | 9.3966e-05 |
on_validation_start | 0.00024532 |21 | 0.0051518 | 5.0637e-05 |
on_validation_batch_start | 1.2674e-05 |161 | 0.0020406 | 2.0057e-05 |
validation_step_end | 8.6672e-06 |161 | 0.0013954 | 1.3716e-05 |
on_epoch_end | 1.7733e-05 |20 | 0.00035466 | 3.4859e-06 |
on_train_end | 0.00025723 |1 | 0.00025723 | 2.5283e-06 |
on_validation_epoch_end | 1.1715e-05 |21 | 0.00024602 | 2.4181e-06 |
on_train_epoch_start | 1.1723e-05 |20 | 0.00023446 | 2.3045e-06 |
on_train_start | 0.00021311 |1 | 0.00021311 | 2.0946e-06 |
on_validation_epoch_start | 8.2239e-06 |21 | 0.0001727 | 1.6975e-06 |
on_train_epoch_end | 8.1054e-06 |20 | 0.00016211 | 1.5934e-06 |
on_fit_start | 1.3379e-05 |1 | 1.3379e-05 | 1.315e-07 |
nerfu_occ
3 years agoUsed for nerfw branch.
Train command:
python train.py \
--dataset_name blender --img_wh 200 200 \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
--N_importance 64 --N_samples 64 --noise_std 0 \
--num_epochs 20 --batch_size 1024 \
--optimizer adam --lr 5e-4 --lr_scheduler cosine \
--exp_name lego_nerfu_occ --beta_min 0.1 --data_perturb occ --encode_t
Eval command:
python eval.py \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego \
--dataset_name blender --img_wh 200 200 --split test \
--N_importance 64 \
--ckpt_path ckpts/lego_nerfw_occ/epoch\=19.ckpt \
--encode_t --beta_min 0.1 \
--scene_name nerfu_occ
Eval output: Mean PSNR : 28.60
Note I use a very small image size (200x200) to speed up my experiments.
Profiler Report
Action | Mean duration (s) |Num calls | Total time (s) | Percentage % |
-----------------------------------------------------------------------------------------------------------------------------
Total | - |_ | 1.0901e+04 | 100 % |
-----------------------------------------------------------------------------------------------------------------------------
run_training_epoch | 544.74 |20 | 1.0895e+04 | 99.947 |
run_training_batch | 0.13381 |78140 | 1.0456e+04 | 95.921 |
optimizer_step_and_closure_0 | 0.11587 |78140 | 9053.8 | 83.057 |
training_step_and_backward | 0.11173 |78140 | 8730.4 | 80.09 |
model_backward | 0.088715 |78140 | 6932.2 | 63.595 |
model_forward | 0.02281 |78140 | 1782.4 | 16.351 |
evaluation_step_and_end | 1.7842 |161 | 287.25 | 2.6352 |
on_train_batch_end | 0.00042159 |78140 | 32.943 | 0.30221 |
get_train_batch | 0.00025085 |78140 | 19.602 | 0.17982 |
cache_result | 1.4803e-05 |391370 | 5.7934 | 0.053147 |
on_batch_start | 1.0333e-05 |78140 | 0.80743 | 0.0074072 |
on_after_backward | 9.5508e-06 |78140 | 0.7463 | 0.0068464 |
on_batch_end | 9.4638e-06 |78140 | 0.7395 | 0.006784 |
on_before_zero_grad | 8.3572e-06 |78140 | 0.65303 | 0.0059908 |
on_validation_end | 0.025442 |21 | 0.53429 | 0.0049014 |
training_step_end | 6.2163e-06 |78140 | 0.48574 | 0.0044561 |
on_train_batch_start | 5.99e-06 |78140 | 0.46806 | 0.0042939 |
on_validation_batch_end | 0.00042104 |161 | 0.067788 | 0.00062187 |
on_epoch_start | 0.00079988 |20 | 0.015998 | 0.00014676 |
on_validation_start | 0.00024023 |21 | 0.0050449 | 4.6281e-05 |
on_validation_batch_start | 1.391e-05 |161 | 0.0022395 | 2.0544e-05 |
validation_step_end | 8.4167e-06 |161 | 0.0013551 | 1.2431e-05 |
on_train_end | 0.0003507 |1 | 0.0003507 | 3.2172e-06 |
on_epoch_end | 1.611e-05 |20 | 0.0003222 | 2.9558e-06 |
on_train_epoch_start | 1.5704e-05 |20 | 0.00031408 | 2.8813e-06 |
on_validation_epoch_end | 1.4037e-05 |21 | 0.00029477 | 2.7041e-06 |
on_validation_epoch_start | 8.7303e-06 |21 | 0.00018334 | 1.6819e-06 |
on_train_start | 0.00016846 |1 | 0.00016846 | 1.5454e-06 |
on_train_epoch_end | 7.923e-06 |20 | 0.00015846 | 1.4537e-06 |
on_fit_start | 1.3867e-05 |1 | 1.3867e-05 | 1.2721e-07 |
v2.0.2
3 years agosucculent plant model by @SpongeGirl
image size 496x372, spheric pose
v2.0.1
4 years agorelease silica nerf model and reconstructed mesh (spheric poses) link to the data image size is 504x378 (original size 4032x3024)
Usage: place the poses_bounds.npy under a folder $DIR (anywhere you want), then you can run
python eval.py \
--root_dir $DIR \
--dataset_name llff --scene_name silica \
--img_wh 504 378 --N_importance 64 --spheric_poses --ckpt_path $CKPT_PATH
as usual. To extract the mesh, follow README_mesh.
v2.0
4 years agoCommand:
python train.py \
--dataset_name llff \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_llff_data/fern/ \
--N_importance 64 --img_wh 504 378 \
--batch_size 1024 --num_epochs 30 \
--optimizer adam --lr 5e-4 \
--lr_scheduler steplr --decay_step 10 20 --decay_gamma 0.5 \
--exp_name fern
Profile
Profiler Report
Action | Mean duration (s) | Total time (s)
-----------------------------------------------------------------
on_train_start | 0.00023312 | 0.00023312
on_epoch_start | 0.00029521 | 0.0088563
get_train_batch | 0.00023997 | 25.456
on_batch_start | 6.1591e-06 | 0.65317
model_forward | 0.019652 | 2084.1
model_backward | 0.069537 | 7374.4
on_after_backward | 1.5543e-06 | 0.16483
optimizer_step | 0.0037302 | 395.59
on_batch_end | 0.00030407 | 32.247
on_epoch_end | 9.9102e-06 | 0.00029731
on_train_end | 0.00036468 | 0.00036468
v1.0
4 years agolego model and reconstructed mesh. Command:
python train.py \
--dataset_name blender \
--root_dir /home/ubuntu/data/nerf_example_data/nerf_synthetic/lego/ \
--N_importance 64 --img_wh 400 400 --noise_std 0 \
--batch_size 1024 --num_epochs 16 \
--optimizer adam --lr 5e-4 \
--lr_scheduler steplr --decay_step 2 4 8 --decay_gamma 0.5 \
--exp_name exp3
Detailed profile:
Action | Mean duration (s) | Total time (s)
-----------------------------------------------------------------
on_train_start | 1.2281e-05 | 1.2281e-05
on_epoch_start | 6.1691e-06 | 9.8706e-05
get_train_batch | 0.00023678 | 59.198
on_batch_start | 4.4245e-06 | 1.1061
model_forward | 0.041729 | 1.0432e+04
model_backward | 0.046964 | 1.1741e+04
on_after_backward | 1.5339e-06 | 0.38347
optimizer_step | 0.0035952 | 898.81
on_batch_end | 4.1799e-06 | 1.045
on_epoch_end | 5.4906e-06 | 8.785e-05
on_train_end | 9.583e-06 | 9.583e-06