Calibration-then-Calculation: A Variance Reduced Metric Framework in Deep Click-Through Rate Prediction Models. (arXiv:2401.16692v1 [cs.LG])

Deep learning has been widely adopted across various fields, but there has
been little focus on evaluating the performance of deep learning pipelines.
With the increased use of large datasets and complex models, it has become
common to run the training process only once and compare the result to previous
benchmarks. However, this procedure can lead to imprecise comparisons due to
the variance in neural network evaluation metrics. The metric variance comes
from the randomness inherent in the training process …

arxiv become click cs.lg datasets deep learning fields focus framework large datasets performance pipelines prediction prediction models process rate through training variance