Residual Connections Harm Self-Supervised Abstract Feature Learning

arXiv:2404.10947v1 Announce Type: new
Abstract: We demonstrate that adding a weighting factor to decay the strength of identity shortcuts within residual networks substantially improves semantic feature learning in the state-of-the-art self-supervised masked autoencoding (MAE) paradigm. Our modification to the identity shortcuts within a VIT-B/16 backbone of an MAE boosts linear probing accuracy on ImageNet from 67.3% to 72.3%. This significant gap suggests that, while residual connection structure serves an essential role in facilitating gradient propagation, it may have a harmful …

abstract accuracy art arxiv cs.cv feature harm identity linear networks paradigm residual semantic state type vit