Metrics for uncertainty evaluation in regression problems

How to evaluate uncertainty with Validity, Sharpness, Negative Log-Likelihood, and Continuous Ranked Probability Score (CRPS) metrics

Many real-world problems require uncertainty estimation for future outcomes for better decision-making. However, most state-of-the-art machine learning algorithms are capable of estimating only a single-valued prediction which is usually a mean or a median of the conditional distribution which suppose to match the real outcome well. What the single-valued prediction cannot suggest is how confident the prediction is. …

editors pick evaluation machine learning metrics probability-distributions quantile-regression regression uncertainty