Vision Transformers with Natural Language Semantics

arXiv:2402.17863v1 Announce Type: new
Abstract: Tokens or patches within Vision Transformers (ViT) lack essential semantic information, unlike their counterparts in natural language processing (NLP). Typically, ViT tokens are associated with rectangular image patches that lack specific semantic context, making interpretation difficult and failing to effectively encapsulate information. We introduce a novel transformer model, Semantic Vision Transformers (sViT), which leverages recent progress on segmentation models to design novel tokenizer strategies. sViT effectively harnesses semantic information, creating an inductive bias reminiscent of …

abstract arxiv context cs.cv image information interpretation language language processing making natural natural language natural language processing nlp novel processing semantic semantics tokens transformer transformer model transformers type vision vision transformers vit