ML models that ace benchmarks fail in production. This is not rare. Hundreds of COVID-19 prediction models, reviewed by Roberts et al. in Nature Machine Intelligence, were found to be clinically useless. A water quality system in Toronto repeatedly cleared contaminated water as safe to swim in. The AIAAIC repository documents dozens more. Researcher Michael Midgley, drawing on 20 years in the field, argues in his arxiv guide (2108.02497) and this article that these failures share common, preventable causes.
The failure modes are specific. Hidden variables corrupt training data when a feature correlates with labels for the wrong reason: COVID chest scan models learned patient posture, not pathology, because sick patients were scanned lying down. Spurious correlations do the same without any causal link at all, the canonical case being a military neural net that identified weather instead of tanks because all tank photos were taken at the same time of day. Mislabelling in standard benchmarks like MNIST and CIFAR runs at roughly 2 percent, enough to obscure whether a marginal accuracy gain is real or an artifact of label noise. The article also details data leakage, where information bleeds from test sets into training, and metric gaming, where optimizing reported numbers diverges from optimizing real-world utility.
The REFORMS checklist from Princeton, introduced recently for ML-based science, is presented here as a concrete intervention. The article is worth reading in full not for the conclusions but for the diagnostic framework: how to use saliency maps to catch spurious correlations, how overlapping records corrupted COVID datasets before anyone noticed, and why adversarial training, while effective, costs more than simply looking at what your model is actually attending to.
[READ ORIGINAL →]