Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization

arXiv:2307.12851v2 Announce Type: replace
Abstract: This paper studies the problem of training a two-layer ReLU network for binary classification using gradient flow with small initialization. We consider a training dataset with well-separated input vectors: Any pair of input data with the same label are positively correlated, and any pair with different labels are negatively correlated. Our analysis shows that, during the early phase of training, neurons in the first layer try to align with either the positive data or the …

abstract alignment arxiv binary classification cs.lg data dataset flow gradient layer network networks neuron paper relu small studies training type vectors