Win-Win: Training High-Resolution Vision Transformers from Two Windows

arXiv:2310.00632v2 Announce Type: replace
Abstract: Transformers have become the standard in state-of-the-art vision architectures, achieving impressive performance on both image-level and dense pixelwise tasks. However, training vision transformers for high-resolution pixelwise tasks has a prohibitive cost. Typical solutions boil down to hierarchical architectures, fast and approximate attention, or training on low-resolution crops. This latter solution does not constrain architectural choices, but it leads to a clear performance drop when testing at resolutions significantly higher than that used for training, thus …

abstract architectures art arxiv attention become cost crops cs.cv hierarchical however image low performance resolution solutions standard state tasks training transformers type vision vision transformers windows