One AI output is not an evaluation. Nielsen Norman Group makes this the foundation of their latest piece: a single response from an AI system tells you only that the system *can* produce a result, not how often or how reliably it will. Ask an AI customer-service bot the same return-policy question three times and you may get three different answers, one accurate, one incomplete, one flatly wrong.
The core problem is nondeterminism. AI systems are not like deterministic software, where a feature that works once works every time. Teams trained on decades of traditional software testing apply that same one-run logic to AI and draw conclusions that do not hold. The article argues that meaningful evaluation requires multiple representative inputs, repeated runs, and confidence intervals, not a single inspection of output.
The full article is worth reading for its framework on how to structure those evaluations properly, including what counts as a representative input set and how to interpret variance across runs. If your team is currently signing off on AI features after one or two test prompts, this piece is a direct challenge to that process.
[READ ORIGINAL →]