NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function

arXiv:2403.02411v1 Announce Type: cross
Abstract: The Attention mechanism is the main component of the Transformer architecture, and since its introduction, it has led to significant advancements in Deep Learning that span many domains and multiple tasks. The Attention Mechanism was utilized in Computer Vision as the Vision Transformer ViT, and its usage has expanded into many tasks in the vision domain, such as classification, segmentation, object detection, and image generation. While this mechanism is very expressive and capable, it comes …

abstract architecture arxiv attention computer computer vision cs.cv cs.lg deep learning domains function generated introduction multiple network tasks token transformer transformer architecture type vision