Data Distribution Shifts and Monitoring

Note: This note is a work-in-progress, created for the course CS 329S: Machine Learning Systems Design (Stanford, 2022). For the fully developed text, see the book Designing Machine Learning Systems (Chip Huyen, O’Reilly 2022). Slides (much shorter 😁). Original Google Docs version.

Let’s start the note with a story I was told by an executive that many readers might be able to relate to. About two years ago, his company hired a consulting firm to develop an ML …

data distribution monitoring