Adapting LLaMA Decoder to Vision Transformer

arXiv:2404.06773v1 Announce Type: new
Abstract: This work examines whether decoder-only Transformers such as LLaMA, which were originally designed for large language models (LLMs), can be adapted to the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to align with LLaMA's architecture, and find that directly applying a casual mask to the self-attention brings an attention collapse issue, resulting in the failure to the network training. We suggest to reposition the class token behind the image tokens with a …

abstract architecture arxiv attention computer computer vision cs.cv decoder language language models large language large language models llama llms self-attention standard step-by-step transformer transformers type vision vit work