Resources and Implementations of Generative Adversarial Nets: GAN, DCGAN, WGAN, CGAN, InfoGAN
I organized this reposity mainly for learning GANs, so all codes about classical GANs were implemented with simple network structure and tested by MNIST dataset.
Just know about how mathmatical analysis (in network structure and loss function) works in actual codes, learn how others implement the GANs, and finally, enjoy the magic of GANs :-D
For more theoretical details and pratical codes about GANs, please go to GAN_Theories and GAN_Applications, thanks!
All have been tested with python2.7+ and tensorflow1.0+ in linux.
Note:
The final layer can be sigmoid(data: [0,1]) or tanh(data:[-1,1]), my codes all use sigmoid.
Using weights_initializer=tf.random_normal_initializer(0, 0.02) will converge faster.
DCGAN
Conditional GAN
D:G = 1:1
D:G=1:1 faster than mlp
D:G:C = 1:1:1 very fast
D:G:C = 5:1:1 fast
Note:
a. The step ratio of G and D is important and it takes some time to reach the balance. Condition+mlp with D:G = 1:1 works better than 2:1.
b. Adding a classfier to trained with conditions and constraint G works faster and better than appending conditions to images for D training.
Wasserstein GAN
D:G = 5: 1 not good, need to *modify*
D:G = 5:1 clip = 0.01
infoGAN
Q loss to update G(as feedback) not good
not good. lacking numbers
clear and have 10 number
the same with not share, not faster?
to be done
Results are shown in the end of this page.
The beginning.
The first paper.
Two main research directions:
[openai/generative-models] (Motivation, Game Theory)
[wiseodd/gan-tensorflow] (Introduction, Implementation)
stabilize the training with some architectural constraints.
GAN is hard to train.
Stabilize Generative Adversarial networks with some architectural constraints.
Popular used in cv. Most used architecture.
[Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks]
Architecture guidelines for stable Deep Convolutional GANs
[bamos/deep-completion] (Introduction, Implementation)
[carpedm20/DCGAN-tensorflow]( star 1.6k+, many files, easy to run, hard to read and modify)
G: fc-->reshape--> deconv bn relu (4) --> tanh
D: conv bn lrelu[leak=0.2] (4) --> reshape--> fc(opn=1)-->sigmoid
G Loss:
tf.reduce_mean(sigmoid_cross_entropy_with_logits(self.D_logits_,tf.ones_like(self.D_)))
D Loss:
tf.reduce_mean(sigmoid_cross_entropy_with_logits(self.D_logits, tf.ones_like(self.D))) + tf.reduce_mean(sigmoid_cross_entropy_with_logits(self.D_logits_, tf.zeros_like(self.D_)))
Solver: Adam lr=0.0002 stepG:stepD=1:1
Data: mnist celebA normalized to [-1,1] for tanh and [0,1] for sigmoid
[sugyan/tf-dcgan]( star 20, easy to read, just 1 dcgan file. dcgan, generator, discriminator all class. not tf.contrib.layers. )
G: fc-->reshape--> deconv bn relu (4) --> tanh
D: conv bn lrelu[leak=0.2] (4) --> reshape--> fc(opn=2 one-hot?)
Losses: softmax_cross_entropy_with_logits
Solver: Adam lr=0.0002 stepG:stepD=1:1
Apply GAN by adding condition(supervised)
Add conditions to GAN by feeding y to G.
G(z)-->G(z,y) D(X)-->D(X,y)
[Conditional Generative Adversarial Nets]
[wiseodd/conditional-gan-tensorflow] (Fomulation, Architecture, Implementation)
[wiseodd/conditional_gan](star 500+, very simple, 1 file, easy to read and run, not conv, inconvinient to extend)
G: concat(z,y)-->fc-->sigmoid
D: concat(z,y)-->fc-->sigmoid loss
Solver: Adam lr=0.001 stepG:stepD=1:1
Data: mnist [0,1]
[zhangqianhui/Conditional-Gans](star 16, easy to extend)
G: concat(z,y)-->fc-->conv-->sigmoid
D: conv_concat(x,y)-->conv-->fc-->sigmoid loss
Solver: Adam lr=0.0002 stepG:stepD=2:1
Data: mnist [0,1]
[fairytale0011/Conditional-WassersteinGAN] (star 6. use wgan to train GAN, use separate classifier to enforce the condition. very clear, easy to read and modify)
G: concat(z,y)-->fc-->conv-->tanh
D: X-->conv-->fc-->sigmoid loss
classifier: X-->conv-->fc-->softmax loss (real label to train classifier, fake label to train G)
clip D var
Solver: RMS lr=0.0002 stepG:stepD:stepC_real:stepC_fake=1:10:0.5:0.5
Data: mnist [-1,1]
stabilize the training by using Wasserstein-1 distance
GAN before using JS divergence has the problem of non-overlapping, leading to mode collapse and convergence difficulty.
Use EM distance or Wasserstein-1 distance, so GAN solve the two problems above without particular architecture (like dcgan).
Algorithm guidelines for stable GANs
[AidenN/WassersteinGAN] (Theory)
[wiseodd/wasserstein-gan] (Introduction, Implementation)
[zhihu/Wassertein GAN] (Introduction, Analysis)
[wiseodd/wgan_tensorflow](very simple, use mlp)
G: fc-->sigmoid
D: fc clip D
G Loss:
G_loss = -tf.reduce_mean(D_fake)
D Loss:
D_loss = tf.reduce_mean(D_fake) - tf.reduce_mean(D_real)
Solver: RMSProp lr=0.0001 stepG:stepD=1:5
Apply GAN by learning conditions(unsupervised)
Attempt to make conditional learned automatically. Find and control some useful information in the images.
Define: Q(c|x) to approximate P(c|x)(which is the conditional distribution)
[wiseodd/infogan] (Introduction Implementation)
[openai/infogan]( star 300+, hard to read and modify, too much files)
G: fc(1024 bn relu)-->fc (bn relu) reshape--> deconv bn relu --> deconv flatten--> activate
D and Q:
shared: reshape-->conv lrelu --> conv bn lrelu --> fc bn lrelu
D: fc(1) --> activate
Q: fc(128) bn lrelu --> fc (c_dim) --> activate
activate: softmax(Categorical) / mean stddev:sqrt(e^x) (Gaussian) / sigmoid(Bernoulli)
Losses:
D and G: sigmoid as DCGAN
Q: cross entropy loss (softmax for discrete) ** add Q loss to D and G loss **
Solver:
Adam beta1=0.5 stepG:stepD=1:1
my understanding:
Adding Q loss to D and G loss, then updating D var and G var by 1:1 equal to Q loss to update Q(D) var and G var by Q:D:G=2:1:1
[wiseodd/infogan-tensorflow](also simple, use mlp)
Q: fc --> softmax with c (not share with D) update vars: G and Q
Solver: Adam G:D:Q = 1:1:1
All images shown here are from ./Samples
, for more information about results, please go to the folder.
Then first number of image name is the trianing epoches.
For better observing, I keep generated images in the beginning, middle and final training process for each algorithm.
Conditional GAN
each pic has the same condition
each row has the same condition: [0 0 0 0; 1 1 1 1; 2 2 2 2; 3 3 3 3]
condition: [0 1 2 3; 4 5 6 7; 0 1 2 3; 4 5 6 7]
condition: [0 1 2 3; 4 5 6 7; 0 1 2 3; 4 5 6 7]
InfoGAN
The generated images with the same condition belong to the same category.
condition: [0 1 2 3; 4 5 6 7; 8 9 0 1; 2 3 4 5]
condition: [0 1 2 3; 4 5 6 7; 8 9 0 1; 2 3 4 5]
DCGAN
Tensorflow style: https://www.tensorflow.org/community/style_guide
tf.concat(1,[x,y]) in tf 0.12- ---> tf.concat([x,y],1) in tf 1.0+.
use tf.get_Variable or tf.contrib.layers to reuse variables.
A good website to convert latex equation to img(then insert into README): http://www.sciweavers.org/free-online-latex-equation-editor