SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision

arXiv:2404.15721v1 Announce Type: new
Abstract: Selective attention helps us focus on task-relevant aspects in the constant flood of our sensory input. This constraint in our perception allows us to robustly generalize under distractions and to new compositions of perceivable concepts. Transformers employ a similar notion of attention in their architecture, but representation learning models with transformer backbones like CLIP and DINO often fail to demonstrate robustness and compositionality. We highlight a missing architectural prior: unlike human perception, transformer encodings do …

abstract arxiv attention concepts cs.ai cs.cv distractions flood focus notion perception robust sensory transformer transformers type vision