Pre-train or Annotate? Domain Adaptation with a Constrained Budget. (arXiv:2109.04711v3 [cs.CL] UPDATED)

Recent work has demonstrated that pre-training in-domain language models can
boost performance when adapting to a new domain. However, the costs associated
with pre-training raise an important question: given a fixed budget, what steps
should an NLP practitioner take to maximize performance? In this paper, we view
domain adaptation with a constrained budget as a consumer choice problem, where
the goal is to select an optimal combination of data annotation and
pre-training. We measure annotation costs of three procedural text …

arxiv budget domain adaptation