General Cyclical Training of Neural Networks. (arXiv:2202.08835v2 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2202.08835

This paper describes the principle of "General Cyclical Training" in machine
learning, where training starts and ends with "easy training" and the "hard
training" happens during the middle epochs. We propose several manifestations
for training neural networks, including algorithmic examples (via
hyper-parameters and loss functions), data-based examples, and model-based
examples. Specifically, we introduce several novel techniques: cyclical weight
decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature,
cyclical data augmentation, cyclical gradient clipping, and cyclical
semi-supervised learning. In addition, …

arxiv general lg networks neural neural networks training