How Powerful Potential of Attention on Image Restoration?

arXiv:2403.10336v1 Announce Type: new
Abstract: Transformers have demonstrated their effectiveness in image restoration tasks. Existing Transformer architectures typically comprise two essential components: multi-head self-attention and feed-forward network (FFN). The former captures long-range pixel dependencies, while the latter enables the model to learn complex patterns and relationships in the data. Previous studies have demonstrated that FFNs are key-value memories \cite{geva2020transformer}, which are vital in modern Transformer architectures. In this paper, we conduct an empirical study to explore the potential of attention …

abstract architectures arxiv attention components cs.cv data dependencies head image image restoration learn multi-head network patterns pixel relationships self-attention studies tasks transformer transformers type