Beware of the hidden error in your test score.

Beware of the Hidden Error in Your Test Score

Why you should report confidence intervals on your test set

Fig. 1: The test score of your machine learning model is subject to statistical fluctuations. Image by author.

In experimental sciences, we are used to reporting estimates with error bars and significant digits. For example, when you weigh your sample in the lab, you can read off its mass up to, say, three digits. In machine learning this is different. When …

confidence-interval error machine learning reliability small data statistics test