Mechanistic Design and Scaling of Hybrid Architectures

arXiv:2403.17844v1 Announce Type: new
Abstract: The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation. We set out to simplify this process by grounding it in an end-to-end mechanistic architecture design (MAD) pipeline, encompassing small-scale capability unit tests predictive of scaling laws. Through a suite of synthetic token manipulation tasks such as compression and recall, designed to probe capabilities, we …

abstract architecture architectures arxiv compute costs cs.lg deep learning design development evaluation hybrid pipeline process prototyping scale scaling set space training type vast