Monitoring BERT Model Training with TensorBoard

Gradient flow and update ratios

In the previous article, we explained all the building components of the BERT model. Now we are going to train the model monitoring the training process in TensorBoard, looking at the gradient flow, updates-parameters ratios, loss and evaluation metrics.

Why would we like to monitor gradients flow and updates ratios instead of simply looking at the loss and evaluation metrics? When we start the model training on a big amount of data, we …

bert model-monitoring monitoring neural networks nlp tensorboard training