Data Distribution Shifts and Monitoring

Note: This note is a work-in-progress, created for the course CS 329S: Machine Learning Systems Design (Stanford, 2022). For the fully developed text, see the book Designing Machine Learning Systems (Chip Huyen, O’Reilly 2022). Slides (much shorter 😁). Original Google Docs version.

Note : This note is a work-in-progress, created for the course CS 329S: Machine Learning Systems Design (Stanford, 2022). For the fully developed text, see the book Designing Machine Learning Systems (Chip Huyen, O’Reilly 2022). Slides (much shorter 😁). Original Google Docs version . Let’s start the note with a story I was told by an executive that many readers might be able to relate to. About two years ago, his company hired a consulting firm to develop an ML model to help them predict how many of each grocery item they’d need next week, so they could restock the items accordingly. The cons

related reading

• Real-time machine learning: challenges and solutionshuyenchip.com
• Scaling is subtler than it seemsberen.io
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• The Little Book of Deep Learningfleuret.org
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• A Course in Machine Learningciml.info
• GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.github.com
• CS231n Deep Learning for Computer Visioncs231n.github.io
• Stanford CS349Ecs349e.github.io
• Why are Machine Learning Projects so Hard to Manage? | by Lukas Biewald | Mediummedium.com
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
• [2303.14151] Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlearxiv.org