Show Your Work with Confidence: Confidence Bands for Tuning Curves

Paper: [https://arxiv.org/abs/2311.09480](https://arxiv.org/abs/2311.09480)

Tweet: [https://x.com/NickLourie/status/1770077925779337563](https://x.com/NickLourie/status/1770077925779337563)

Code: [https://github.com/nicholaslourie/opda](https://github.com/nicholaslourie/opda)

Abstract:

>The choice of hyperparameters greatly impacts performance in natural language processing. Often, it is hard to tell if a method is better than another or just better tuned. Tuning curves fix this ambiguity by accounting for tuning effort. Specifically, they plot validation performance as a function of the number of hyperparameter choices tried so far. While several estimators exist for these curves, it is common to use point estimates, which we show fail …

abstract accounting function hyperparameter impacts language language processing languagetechnology natural natural language natural language processing performance plot processing validation