A Large-Scale Exploration of $\mu$-Transfer

arXiv:2404.05728v1 Announce Type: new
Abstract: Large neural network models have become a mainstay of natural language processing and computer vision, yet their initialization and learning rates are set in a largely heuristic fashion, potentially varying from paper to paper and one model size to the next. The $\mu$-Parameterization ($\mu$P) offers a potential solution to these challenges, yielding scaling rules for model initialization and learning rates, and reportedly enabling zero-shot hyperparameter transfer from small to large models in a variety of …

abstract arxiv become computer computer vision cs.lg exploration fashion language language processing natural natural language natural language processing network neural network next one model paper processing scale set solution transfer type vision