Same Draft, Two Grades: D and B-

In one review of PenForge, my writing pipeline, identical text received D/62 and B-/81.

The draft hadn't changed between those grades.

I dug in and found the swing came from the model-judged part of the score. I had trusted the number enough to rank drafts by a point or two. The same bytes moved nineteen points.

So I stopped trusting the score by itself.

When a model judges writing, the result belongs with the exact draft, rubric, judge model, instructions, and settings that produced it. Without those, the number is difficult to reproduce and easy to overinterpret.

Repeated runs tell me whether a draft's score is stable enough for the decision I want to make. A wide spread doesn't mean the grader is useless. It means one draw shouldn't decide which article survives.

I can use a model review to surface sentences, explain a reaction, or find patterns worth inspecting. I won't let a single score decide the article, especially when the process treats 79 and 80 as different editorial states while identical text can move nineteen points.

Some parts of a drafting pipeline have answers that don't require literary judgment.

I can check whether the saved file matches the generated draft and whether a restart preserves the newer version. Failed attempts should leave a record. If I stop the agent, the logs should show that it made no later paid call.

PenForge records failed generation attempts alongside successful and fallback attempts. A failure may leave no readable draft, but it still used an input and made an API call. Losing that record makes the next investigation harder.

Consider a restart where article.md contains version 3 and the checkpoint still names version 2. The correct response isn't automatically “start again” or “continue.” The application should preserve the newer file, report the disagreement, and let the recovery step establish what finished. A checkpoint containing only “done” would supply almost none of that evidence.

These checks tell me whether the system preserved the work. I still have to read the paragraph and decide whether it says what I mean.

IMO, writing quality needs representative tasks, known failures, and comments tied to exact passages. A model can help compare the draft with those examples. I still make the editorial decision.

After the D/62 and B-/81 results, I stopped looking for the true number. I went back to the flagged passages and read the draft myself.

Sources: Anthropic on agent evaluations; OpenAI's evaluation guidance.