Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis

Project page | Paper

"Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis"
Jeong-gi Kwak, Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, Hanseok Ko
ECCV 2022

This repository includes the official Pytorch implementation of SURF-GAN.

SURF-GAN

SURF-GAN, which is a NeRF-based 3D-aware GAN, can discover disentangled semantic attributes in an unsupervised manner.


(Tranined on 64x64 CelebA and rendered with 256x256)

Get started

• Clone the repo.

git clone https://github.com/jgkwak95/SURF-GAN.git
cd SURF-GAN

• Create virtual environment

conda create -n surfgan python=3.7.1
conda activate surfgan
conda install -c pytorch-lts pytorch torchvision 
pip install --no-cache-dir -r requirements.txt

Train SURF-GAN

At first, look curriculum.py and specify dataset and training options.

# CelebA
python train_surf.py --output_dir your-exp-name \
--curriculum CelebA_single

Pretrained model

Or, you can use the pretrained model.

Semantic attribute discovery

Let's traverse each dimension with discovered semantics:

python discover_semantics.py  --experiment your-exp-name \
--image_size 256 \
--ray_step_multiplier 2 \
--num_id 9 \          
--traverse_range 3.0 \    
 --intermediate_points 9 \
--curriculum CelebA_single     

The default ckpt file to traverse is the latest file (generator.pth). If you want to check specific cpkt, add this in your command line, for example,

--specific_ckpt 140000_64_generator.pth

Control pose

In addition, you can control only camera paramters:

python control_pose.py --experiment your-exp-name \
--image_size 128 \
--ray_step_multiplier 2 \
--num_id 9 \
--intermediate_points 9 \
--mode yaw \
--curriculum CelebA_single \

Render video

• Moving camera

Set the mode: yaw, pitch, fov, etc. You can also make your trajectory.

python render_video.py  --experiment your-exp-name \
--image_size 128 \
--ray_step_multiplier 2 \
--num_frames 100 \
--curriculum CelebA_single \
--mode yaw

• Moving camera with a specific semantic

Choose an attribute that you want to control LiDj.

python render_video_semantic.py  --experiment your-exp-name \
--image_size 128 \
--ray_step_multiplier 2 \
--num_frames 100 \ 
--traverse_range 3.0 \
--intermediate_points \
--curriculum CelebA_single \
--mode circle
--L 2
--D 4

3D-Controllable StyleGAN

Injecting the prior of SURF-GAN into StyleGAN for controllable generation.
Also, it is compatible with many StyleGAN-based methods.

Video

Pose control	+ Style (Toonify)

It is capable of editing real images directly. (with HyperStyle)

Pose	+Illumination (using SURF-GAN samples)

+Hair color (using SURF-GAN samples)	+Smile(using InterFaceGAN)

Citation

@inproceedings{kwak2022injecting,
  title={Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis},
  author={Kwak, Jeong-gi and Li, Yuanming and Yoon, Dongsik and Kim, Donghyeon and Han, David and Ko, Hanseok},
  booktitle={European Conference on Computer Vision},
  pages={236--253},
  year={2022},
  organization={Springer}
}

Acknowledgments

• SURF-GAN is bulided upon the pi-GAN implementation and inspired by EigenGAN (EigenGAN-pytorch). Thanks to the authors for their excellent work!
• We used pSp encoder and StyleGAN2-pytorch to build 3D-controllable StyleGAN. For editing in-the-wild real images, we exploited e4e and HyperStyle with our 3D-controllable StyleGAN.