Merging Text Transformer Models from Different Initializations

arXiv:2403.00986v1 Announce Type: cross
Abstract: Recent work on one-shot permutation-based model merging has shown impressive low- or zero-barrier mode connectivity between models from completely different initializations. However, this line of work has not yet extended to the Transformer architecture, despite its dominant popularity in the language domain. Therefore, in this work, we investigate the extent to which separate Transformer minima learn similar features, and propose a model merging technique to investigate the relationship between these minima in the loss landscape. …

abstract architecture arxiv connectivity cs.ai cs.cl cs.lg domain language line low merging text transformer transformer architecture transformer models type work