[D] Pre-trained networks and batch normalization

When we take a pre-trained network, e.g., ResNet50 on ImageNet, and want to apply it to a new dataset, what we typically do is:

1. Freeze the backbone, but keep the classifier trainable
2. Train until convergence
3. Unfreeze the backbone and train with a low learning rate until convergence

However, I noticed that when we freeze a network with batch normalization layers, the following parameters are still being updated because the batch normalization layers are in training mode: `running_mean`, …

machinelearning networks normalization