Variational Learning is Effective for Large Deep Networks

arXiv:2402.17641v1 Announce Type: new
Abstract: We give extensive empirical evidence against the common belief that variational learning is ineffective for large neural networks. We show that an optimizer called Improved Variational Online Newton (IVON) consistently matches or outperforms Adam for training large networks such as GPT-2 and ResNets from scratch. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We show several new use cases of IVON where we improve fine-tuning and model merging in …

abstract adam arxiv belief computational cs.ai cs.cl cs.lg evidence gpt gpt-2 math.oc networks neural networks scratch show stat.ml training type