A Transformer with Stack Attention

arXiv:2405.04515v1 Announce Type: new
Abstract: Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition …

abstract arxiv attention context cs.cl differentiable free language language models languages large language large language models modeling natural power stack tasks them transformer transformers type