Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors. (arXiv:2205.10279v1 [cs.LG])

Deep learning is increasingly moving towards a transfer learning paradigm
whereby large foundation models are fine-tuned on downstream tasks, starting
from an initialization learned on the source task. But an initialization
contains relatively little information about the source task. Instead, we show
that we can learn highly informative posteriors from the source task, through
supervised or self-supervised approaches, which then serve as the basis for
priors that modify the whole loss surface on the downstream task. This simple
modular approach …

arxiv bayesian easy learning loss transfer transfer learning