DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging

The transformer architecture from Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size -- adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past …

application architecture cs.cl cs.lg domains flow image information language language processing natural natural language natural language processing perplexity processing simple speech speech processing standard transformer transformer architecture transformers understanding via