Alibaba's Qwen 3.8-Max benchmark claims collapse under scrutiny of a single variable: time budget. Alibaba's own numbers give coding tasks a five-hour timeout and up to 12 hours on PaperBench. The independent VulcanBench harness allowed 45 to 60 minutes. That 5x to 16x difference in wall-clock time explains why Alibaba shows near-top performance while VulcanBench places the same model dead last at default settings. Both results are correct. They are measuring different things.
The real cost problem is not price per token. It is cost per successful task: total spend including failed and budget-exhausted runs, divided by tasks that actually passed your acceptance criteria. Long-Horizon-Terminal-Bench, published July 2025, ran 17 frontier models across 46 tasks and found that 79% of unresolved runs were timeouts, not wrong answers. Those are different failure modes with different fixes. VulcanBench's July 26 report shows Claude Opus 5 at lowest effort solving 20 of 23 tasks, versus 18 at high effort, where additional reasoning consumed the clock and scored zero. High effort cost 3.1 times more per solved task. DeepSeek-V4-Flash's Intelligence Index run at maximum effort burned 210 million output tokens against a class median of 100 million. Verbosity is a latency problem, not just a billing problem.
The article is worth reading in full because the operational consequences go beyond benchmarks. Qwen 3.8-Max defaults to its highest reasoning setting when the effort field is unset, and that setting was its worst performer in independent testing. Any team that has not explicitly set that parameter is paying maximum cost for the configuration with the lowest measured success rate. The piece also walks through how to instrument agent runs to separate timeout failures from verifier failures from harness errors, a distinction almost no current leaderboard makes, and why routing ladders that escalate on failure may be escalating directly into a token cap.
[READ ORIGINAL →]