The Window Is a Workbench

A model's context window is the material it has for one request. I think of it as a workbench. Files, notes, and databases can hold much more, but the application still has to choose what belongs on the bench for this job.

More history creates room. It doesn't choose the right evidence.

Suppose an agreement says renewal notice is due June 1. A later amendment changes the deadline to July 1.

A short summary of the original clause can be perfectly faithful and give the wrong answer.

IMO, the document index matters as much as the context window. I'd keep an index connecting the agreement's identifier to its stored file and linked amendments. A renewal question first searches within that agreement, then opens the matching clause and its amendments. Searching “renewal” across every document could return an excellent passage from somebody else's contract. The words match, but the agreement doesn't.

The original clause and the amendment belong in this request, along with any definitions needed to read them. An unrelated negotiation transcript can stay in storage. If the index points to an amendment that can't be opened, the answer has an evidence gap. A confident summary can't supply the missing file.

If the answer still says June 1, I want to inspect the handoffs (I've seen this kind of bug hide between working steps). The amendment could be missing from storage, missing from the search results, dropped before the model request, or present in the request and still ignored. I need to know which one happened.

Any one of those failures can produce the same wrong answer.

This setup is usually called retrieval-augmented generation, or RAG. Fancy term for finding the file and handing the model the right page. Retrieval deserves its own tests because a shorter, cleaner request can fail efficiently by omitting the deciding fact.

I'd test cases where the needed evidence appears early, late, and across two linked passages. Losing the second passage may leave a plausible answer while removing the condition that changes it.

The application should retain the full source and keep a reference with each selected passage: document identifier, version, and location. A passage split away from the agreement it belongs to can be accurate text and wrong evidence.

The renewal answer doesn't need every other clause in the same request. On a later question, the application can use the saved document reference to retrieve another passage from the same version.

That return path is part of the test. When the application says “stored for later,” it should be able to bring that exact source back.

Research such as Lost in the Middle and Chroma's Context Rot experiments found that long inputs can make relevant information harder for tested models to use, especially when distractors resemble the answer. Those results don't establish one ideal context size. They give me a reason to test whether the evidence on this bench is both present and usable.

For the renewal test, the request should contain the June 1 clause, the July 1 amendment, and their document references. If the amendment can't be opened, the answer should say that instead of guessing at the date.

Sources: Contextual Retrieval; Lost in the Middle; Chroma's Context Rot report.