Upper and lower memory capacity bounds of transformers for next-token prediction

arXiv:2405.13718v1 Announce Type: new
Abstract: Given a sequence of tokens, such as words, the task of next-token prediction is to predict the next-token conditional probability distribution. Decoder-only transformers have become effective models for this task, but their properties are still not fully understood. In particular, the largest number of distinct context sequences that a decoder-only transformer can interpolate next-token distributions for has not been established. To fill this gap, we prove upper and lower bounds on this number, which are …

abstract arxiv become capacity cs.lg decoder distribution math.oc memory next prediction probability token tokens transformers type words