Training-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion

In deep learning, stochastic gradient descent often yields functionally similar yet widely scattered solutions in the weight space even under the same initialization, causing barriers in the Linear Mode Connectivity (LMC) landscape. Overcoming these barriers is crucial for understanding deep learning dynamics and enhancing model-fusion algorithms. Previous studies highlight the role of permutation symmetry in reducing post-training barriers through network permutation. However, these post-hoc methods, demanding extra computations, are less effective for larger, complex models (e.g., ViT, LLM) due to …

algorithms alignment connectivity cs.lg deep learning dynamics fusion gradient landscape linear neuron solutions space stat.ml stochastic through training understanding