LLMs do not distinguish privileged system text from untrusted user input by position or tag. They distinguish it by style. That is the core finding from Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell in 'Prompt Injection as Role Confusion,' a paper targeting how models process role-tagged text like <system>, <think>, and <assistant> versus <user> blocks.
The attack vector is blunt and it works. Append text mimicking the style of a model's internal thinking block to a malicious user request, and models including gpt-oss-20b override their safety training. The researchers tested this directly: styled injection attempts succeeded 61% of the time. After 'destyling', rewriting the same content so it no longer matched the expected role-tag format, success dropped to 10%. Same words, different style, opposite outcome.
The researchers name this 'role confusion' and argue it makes prompt injection defense structurally unsolvable until models develop genuine role perception. The paper is worth reading in full because the mechanism, not just the jailbreak result, is what matters. The threat they outline at the end is the slow one: injections that shift model behavior incrementally through innocuous-looking text, deployed legally and at scale.
[READ ORIGINAL →]