Understanding the Role of Layer Normalization in Label-Skewed Federated Learning

arXiv:2308.09565v2 Announce Type: replace
Abstract: Layer normalization (LN) is a widely adopted deep learning technique especially in the era of foundation models. Recently, LN has been shown to be surprisingly effective in federated learning (FL) with non-i.i.d. data. However, exactly why and how it works remains mysterious. In this work, we reveal the profound connection between layer normalization and the label shift problem in federated learning. To understand layer normalization better in FL, we identify the key contributing mechanism of …

arxiv cs.lg federated learning layer normalization role stat.ml type understanding