From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers

arXiv:2310.11984v2 Announce Type: replace
Abstract: In this paper, we investigate the inherent capabilities of transformer models in learning arithmetic algorithms, such as addition and parity. Through experiments and attention analysis, we identify a number of crucial factors for achieving optimal length generalization. We show that transformer models are able to generalize to long lengths with the help of targeted attention biasing. In particular, our solution solves the Parity task, a well-known and theoretically proven failure mode for Transformers. We then …

abstract algorithms analysis arxiv attention capabilities cs.cl cs.lg identify paper show through transformer transformer models transformers type