Better Metrics ⇏ Happier Users

Designing a machine learning product to close the user-feedback loop

Imagine this “hypothetical” scenario: a new model was developed to replace the existing production model, due to detected low accuracies for some “important” classes. The new model’s metrics were much better, hence it was deployed to replace the current model.

Turns out, the new model actually made the user experience worse. Even though it was better in metrics, users didn’t feel like it …

evaluation machine learning metrics user-centered-design user-experience