MNIST classification using Multi-Layer Perceptron (MLP) with 2 hidden layers. Some weight-initializers and batch-normalization are implemented.
An implementation of weight/bias initializers and batch normalization in Tensorflow.
MNIST database is used to show performance-comparison
In order to examine the effect of initializers and batch normalization, a simple network architecture called multilayer perceptrons (MLP) is employed.
MLP has following architecture.
The following initializers for weights/biases of network are considered.
Sample results are following.
Index | Weight Initializer | Bias Initializer | Accuracy |
---|---|---|---|
normal_w_normal_b_0.9451 | normal | normal | 0.9451 |
normal_w_zero_b_0.9485 | normal | zero | 0.9485 |
truncated_normal_w_normal_b_0.9788 | truncated_normal | normal | 0.9788 |
truncated_normal_w_zero_b_0.9790 | truncated_normal | zero | 0.9790 |
xavier_w_normal_b_0.9800 | xavier | normal | 0.9800 |
xavier_w_zero_b_0.9806 | xavier | zero | 0.9806 |
he_w_normal_b_0.9798 | he | normal | 0.9798 |
he_w_zero_b_0.9811 | he | zero | 0.9811 |
Batch normalization improves performance of network in terms of final accuracy and convergence rate. In this simulation, bias initalizer was zero-constant initializer and weight initializers were xavier / he.
Sample results are following.
Index | Weight Initializer | Batch Normalization | Accuracy |
---|---|---|---|
xavier_woBN_0.9806 | xavier | Unused | 0.9806 |
xavier_withBN_0.9812 | xavier | Used | 0.9812 |
he_woBN_0.9811 | he | Unused | 0.9811 |
he_withBN_0.9837 | he | Used | 0.9837 |
python run_main.py --weight-init <weight initializer> --bias-init <bias initializer> --batch-norm <True or False>
<weight initializer>
must be selected in [normal, truncated_normal, xavier, he].
<bias initializer>
must be selected in [normal, zero].
You may command like python run_main.py --weight-init xavier --bias-init zero --batch-norm True
.
This implementation has been tested on Tensorflow r0.12.