Promises and Pitfalls of Threshold-based Auto-labeling

arXiv:2211.12620v2 Announce Type: replace
Abstract: Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Threshold-based auto-labeling (TBAL), where validation data obtained from humans is used to find a confidence threshold above which the data is machine-labeled, reduces reliance on manual annotation. TBAL is emerging as a widely-used solution in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is …

abstract annotation arxiv auto confidence cs.ai cs.lg data datasets humans labeling machine machine learning machine learning workflows major quality reliance scale solution stat.ml supervised machine learning threshold type validation workflows