StableMask: Refining Causal Masking in Decoder-only Transformer

The decoder-only Transformer architecture with causal masking and relative position encoding (RPE) has become the de facto choice in language modeling. Despite its exceptional performance across various tasks, we have identified two limitations: First, it requires all attention scores to be non-zero and sum up to 1, even if the current embedding has sufficient self-contained information. This compels the model to assign disproportional excessive attention to specific tokens. Second, RPE-based Transformers are not universal approximators due to their limited capacity …

architecture attention become cs.ai cs.cl current decoder embedding encoding language limitations masking modeling performance tasks the decoder transformer transformer architecture