Show Your Work with Confidence: Confidence Bands for Tuning Curves

arXiv:2311.09480v2 Announce Type: replace-cross
Abstract: The choice of hyperparameters greatly impacts performance in natural language processing. Often, it is hard to tell if a method is better than another or just better tuned. Tuning curves fix this ambiguity by accounting for tuning effort. Specifically, they plot validation performance as a function of the number of hyperparameter choices tried so far. While several estimators exist for these curves, it is common to use point estimates, which we show fail silently and …

abstract accounting arxiv confidence cs.cl cs.lg impacts language language processing natural natural language natural language processing performance plot processing show stat.ml type validation work