Google AI Researchers Propose N-Grammer for Augmenting the Transformer Architecture with Latent n-grams

Web: https://www.reddit.com/r/LanguageTechnology/comments/wg83bb/google_ai_researchers_propose_ngrammer_for/

Introducing self-attention to neural networks has propelled rapid and outstanding progress in generative text modeling. Transformer models are now recognized as the fundamental approach to natural language processing. Hence, scaling these models has attracted several recent interests and investments. This paper suggests a simple alteration to the Transformer architecture, called the N-Grammer. During training and inference, the N-Grammer layer uses sparse operations only. This work discovers that while being substantially faster at inference, a Transformer framework integrated with the latent …

ai architecture google languagetechnology researchers transformer transformer architecture