Batch normalization

Normalize to zero mean and one variance each feature across the batch. It’s a method of adaptive reparametrization.

The normalzation operations are considered in the computational graph of the network! for each feature $x_i$ in the of size $m$:

$${\mu }_i=\frac{1}{m}\sum _{i=1}^m{x}_i^{(m)},{\sigma }_i=\sqrt{\delta +\sum _{i=1}^m(x_i^{(m)}-{\mu }_i{)}^2}$$

$\delta$ is a small value (e.g. $10^{-8}$) added for numerical stability (at zero $\sqrt{x}$ is non differentiable). So the rescaled features are

$${\hat{x}}_i=\frac{x_i-{\mu }_i}{{\sigma }_i}$$

During inference time, the mean and variance may be replaced by the values obtained one the whole training dataset.