all AI news
Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer. (arXiv:2210.16365v1 [cs.LG])
cs.LG updates on arXiv.org arxiv.org
Self-supervised representation learning (SSL) methods provide an effective
label-free initial condition for fine-tuning downstream tasks. However, in
numerous realistic scenarios, the downstream task might be biased with respect
to the target label distribution. This in turn moves the learned fine-tuned
model posterior away from the initial (label) bias-free self-supervised model
posterior. In this work, we re-interpret SSL fine-tuning under the lens of
Bayesian continual learning and consider regularization through the Elastic
Weight Consolidation (EWC) framework. We demonstrate that self-regularization
against …
arxiv consolidation robustness self-supervised learning supervised learning transfer