Runway ML could not fix a drifting avatar bug in its real-time video model, so it shipped the bug as a product feature instead. Characters swayed off-center during generation when input images were misaligned. The fix: a frontend tool called 'Optimize for Image Quality' that auto-centers the user's image before generation starts. Users saw a helpful feature. The model limitation stayed hidden. Ryan Phillips, head of enterprise product at Runway ML, presented this case at VB Transform 2026 alongside a full breakdown of how the company builds, evaluates, and ships generative video models.
The evaluation process is more rigorous and more manual than most teams expect. Runway tracks daily test results in an Excel spreadsheet, logs outputs as 'minor' or 'major' failures, and sets a pass rate threshold before any model ships. To stress-test edge cases, they used a non-human character named 'Tooth', no nose, unusual teeth, to ensure the model behaves predictably outside standard human facial structures. A morphing background net in an otherwise clean video was graded a hard failure. For scale, they use LLMs as visual judges. On the infrastructure side, 8% of API calls were dropping to 16 fps after launch. The root cause was traced via a Claude-powered agent, Datadog, and Sentry to a single us-east-1 data center. The fix was physically replacing the GPUs.
The full presentation is worth reading for the distillation and adversarial post-training pipeline details, including how Runway cuts generation time by 80 to 90% while using APT to recover visual sharpness lost in the process. Phillips also frames the engineering grind honestly, calling prolonged stalls 'failure hell' and arguing teams must stay in that phase long enough to find the real breakthrough. The lessons apply whether or not you are building foundation models.
[READ ORIGINAL →]