Transformers as Support Vector Machines

arXiv:2308.16898v3 Announce Type: replace
Abstract: Since its inception in "Attention Is All You Need", transformer architecture has led to revolutionary advancements in NLP. The attention layer within the transformer admits a sequence of input tokens $X$ and makes them interact through pairwise similarities computed as softmax$(XQK^\top X^\top)$, where $(K,Q)$ are the trainable key-query parameters. In this work, we establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem that separates optimal input tokens from non-optimal …

abstract architecture arxiv attention attention is all you need cs.ai cs.cl cs.lg key layer machines math.oc nlp parameters query softmax support support vector machines them through tokens transformer transformer architecture transformers type vector