Risk Implications of Excessive Multiple Local Minima during Hyperparameter Tuning

Our Epistemological Limitation and Illusion of Knowledge

3D visualization with Matplotlib’s plot_trisurf: Produced by Michio Suginoo

Excessive multiple local minima during hyperparameter tuning is a symptom of a highly sensitive model performance to small changes in the value of hyperparameter, as displayed in the chart above.

I encountered this very rugged performance landscape with multiple dips and bumps when I was performing the grid-search tuning on the hyperparameter pair, reg_alpha and reg_lambda, of the native XGBoost API. It was just …

bias-variance-tradeoff hyperparameter hyperparameter-tuning machine learning model optimization risk xgboost