LLMs have stalled on long-form non-fiction writing, and the author of a new post-training textbook, 'Reinforcement Learning from Human Feedback,' has the receipts. Nathan Lambert used current frontier models throughout the writing process and found a consistent ceiling: models can catch sentence-level typos and fix isolated sections, but collapse when asked to organize and sustain an argument across a full chapter. GPT 4.5 and Kimi K2 remain benchmarks for writing quality even as newer models blow past them in coding and math.
The failure mode is specific. Models increase entropy in long-form writing, compounding small errors as they stack additions on top of each other. Lambert draws a direct line from this to AI science ambitions: if a model cannot compellingly organize established knowledge, it cannot generate novel insight from open-ended problems. RLVR fixed analogous compounding errors in math and code. No equivalent intervention exists yet for prose. The gap between GPT 5.5 Pro finding deep typos in a 200-300 page manuscript and Claude offering substantive editorial suggestions points to something real: different models have carved out narrow, non-overlapping writing competencies.
The piece is worth reading in full because Lambert is optimistic about AI and still calls this alarming. He is not arguing writing is impossible for models, he is arguing the prerequisite for autonomous scientific contribution is unmet, and the path forward is unclear. The specific breakdown of where each model succeeded and failed during an actual book-writing project makes this concrete, not theoretical.
[READ ORIGINAL →]