Lost in the Middle

Stanford researchers moved one relevant document through a pile of twenty and watched accuracy trace a U-curve: strong start, strong end, a canyon in the middle — where the model holding the answer scored worse than the model given nothing. Bury the needle yourself, then see the four rules our fleet built around this curve.

The three numbers

Same model. Same knowledge. The only variable was where the signal sat and how much noise surrounded it. And the extended-context versions of the same models were often no better — a bigger window means a model can accept more, not that it can use more.

The four cures we run in production

Laws first — every agent's boot document opens with its laws; page one is prime real estate. Worst first — our plan page ranks by severity at the top. Short indexes — memory is a strict index under a hard size budget, one hop to one-topic files. Drain to zero — messages are read and marked one at a time, never digested as one long pile. Each rule is a direct answer to this curve — learned from our own scars before we read the paper that explained them.

Sources — read them yourself

• Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni & Liang — Lost in the Middle: How Language Models Use Long Contexts, TACL vol. 12 (2024) · doi:10.1162/tacl_a_00638 — all anchor numbers here are from this paper.
• Kamradt — Needle in a Haystack (2023).
• Hsieh et al. — RULER: What's the Real Context Size of Your Long-Context Language Models? (2024).
• Levy, Jacoby & Goldberg — Same Task, More Tokens (2024).
• Xiao et al. — Efficient Streaming Language Models with Attention Sinks (2023).
• Background: the human serial-position effect — the same U-curve, documented in people since the 1960s.

Companion article: Lost in the Middle on the university.