Researchers at Peking University, Zhongguancun Academy, and Shanghai's Institute for Advanced Algorithms Research have released DataFlow-Harness, an open-source framework that forces LLM agents to build structured, auditable data pipelines instead of disposable scripts. The core finding is blunt: when Claude was constrained to generate native, governable pipeline graphs instead of free-form code, its success rate dropped from 94.2% to 83.3%. That 10.9-point gap is what the paper calls the 'NL2Pipeline gap,' and closing it is the entire point of the system.
DataFlow-Harness works through four interlocking components. A directed acyclic graph backend serves as the single source of truth. An MCP tools layer gives the agent live access to the operator registry and current pipeline state, so it cannot hallucinate unavailable dependencies. Markdown-based 'DataFlow-Skills' files inject domain knowledge directly into the model's context, teaching it operator compatibility rules without requiring it to ingest the entire codebase. A visual DAG editor lets engineers inspect and revise every AI-proposed change. The result is that the agent makes typed, incremental mutations to a persistent structure rather than emitting raw Python. On a 12-task benchmark covering QA generation, schema normalization, and review governance using Claude Opus 4.7, the framework hit a 93.3% end-to-end pass rate while cutting API costs 72.5% versus standard Claude Code and reducing latency by 49.9%.
The paper is worth reading in full because the benchmark methodology and the baseline comparisons reveal exactly where and why unconstrained agents fail on production data engineering tasks. The gap between disposable code performance and governed pipeline performance is documented with specifics, not asserted. DataFlow-Harness is open-source, and the next question the researchers leave open is whether the static validation checks can be extended to runtime schema enforcement, which is where production pipelines actually break.
[READ ORIGINAL →]