Implicit Regularization of Gradient Flow on One-Layer Softmax Attention

arXiv:2403.08699v1 Announce Type: new
Abstract: We study gradient flow on the exponential loss for a classification problem with a one-layer softmax attention model, where the key and query weight matrices are trained separately. Under a separability assumption on the data, we show that when gradient flow achieves the minimal loss value, it further implicitly minimizes the nuclear norm of the product of the key and query weight matrices. Such implicit regularization can be described by a Support Vector Machine (SVM) …

abstract arxiv attention classification cs.ai cs.lg data flow gradient key layer loss math.oc query regularization show softmax stat.ml study the key type value