Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP. (arXiv:2207.07284v2 [cs.CV] UPDATED)

Vision multi-layer perceptrons (MLPs) have shown promising performance in
computer vision tasks, and become the main competitor of CNNs and vision
Transformers. They use token-mixing layers to capture cross-token interactions,
as opposed to the multi-head self-attention mechanism used by Transformers.
However, the heavily parameterized token-mixing layers naturally lack
mechanisms to capture local information and multi-granular non-local relations,
thus their discriminative power is restrained. To tackle this issue, we propose
a new positional spacial gating unit (PoSGU). It exploits the attention …

arxiv encoding mlp positional encoding relations vision