Tree Cross Attention

arXiv:2309.17388v2 Announce Type: replace
Abstract: Cross Attention is a popular method for retrieving information from a set of context tokens for making predictions. At inference time, for each prediction, Cross Attention scans the full set of $\mathcal{O}(N)$ tokens. In practice, however, often only a small subset of tokens are required for good performance. Methods such as Perceiver IO are cheap at inference as they distill the information to a smaller-sized set of latent tokens $L < N$ on which cross …

abstract arxiv attention context cs.lg good inference information making perceiver performance popular practice prediction predictions scans set small tokens tree type