ML models fail in production constantly. A review of hundreds of Covid-19 prediction models found they simply did not work on real-world data. A water quality system in Toronto repeatedly cleared dangerous water as safe to swim in. These are not edge cases. They are documented in the AIAAIC repository, and researchers argue the pattern is broad enough to constitute a reproducibility crisis across science itself.
The failure modes are specific and repeatable. Hidden variables fool models into learning posture instead of disease. Spurious correlations let classifiers identify tanks by sky color rather than tank shape. MNIST and CIFAR, the benchmarks everyone uses, carry mislabeling rates around 2 percent, enough to obscure whether a marginal accuracy gain reflects a real improvement or noise. Data leakage further upstream corrupts results before training even begins. The REFORMS checklist from Princeton was designed to catch these errors at the methodology level, not after deployment.
Read the full piece for the granular breakdown of where each failure enters the pipeline, from labeling bias and hidden variables through leakage and evaluation design. The author has 20 years in ML and published the companion paper 'How to Avoid Machine Learning Pitfalls' on arXiv. The value here is not the conclusion. It is the itemized map of exactly where competent practitioners go wrong without knowing it.
[READ ORIGINAL →]