Theroretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Rate. (arXiv:2209.07587v1 [stat.ML])

Batch normalization is widely used in deep learning to normalize intermediate
activations. Deep networks suffer from notoriously increased training
complexity, mandating careful initialization of weights, requiring lower
learning rates, etc. These issues have been addressed by Batch Normalization
(\textbf{BN}), by normalizing the inputs of activations to zero mean and unit
standard deviation. Making this batch normalization part of the training
process dramatically accelerates the training process of very deep networks. A
new field of research has been going on to …

arxiv data insight normalization rate regularization