Give a model a document and ask it to check the work. It comes back with six problems. Good problems, real ones. You fix all six and hand the improved version back. It finds four more, somewhere else entirely. You fix those. It finds three. None of them are the original six.
Anyone who has used a coding agent for a full afternoon knows the same loop from the inside. Claude finishes the task, then adds "I also noticed the error handling in the adjacent module could be tightened." You say sure, fix it. It does, then: "One thing that deserves your attention: this function has no test coverage." At no point does it lie to you. Every individual observation is defensible. And yet the horizon of "finished" recedes exactly as fast as you walk toward it.
The natural reading is that your work keeps failing to be good enough. That reading is wrong, and the evidence is that the second critique does not contain the first one. If the six original problems were the reason it was not done, fixing them should have moved you six problems closer. Instead you got a fresh draw from the same distribution.
The model is not converging on a fixed target. It is resampling a moving one. Four separate mechanisms produce that, and they stack.
The same input does not produce the same output
Start with the least interesting cause, because it is the one people wave away with "well, temperature."
Set temperature to zero and you still do not get determinism from a hosted API. The usual explanation, floating point addition is not associative and GPUs reduce in nondeterministic order, is not quite the real story. Thinking Machines Lab traced it properly in September 2025: the culprit is that inference kernels are not batch-invariant. RMSNorm, matmul, and attention all reduce differently depending on how many sequences are in the batch alongside yours. Server load determines batch size. Batch size changes the numerics. Different numerics push a token that was a coin flip over to the other side, and from there the two generations diverge completely.
Your request is batched with strangers. That is the source of the randomness. Nothing about your prompt changed; the load on the cluster did. When they built batch-invariant versions of those three kernels, a thousand runs produced a thousand identical outputs.
This matters here because a critique is a long generation full of near-ties. "Is this worth flagging?" is exactly the kind of marginal decision that lives inside a few percentage points of probability mass. Re-roll it and a different set of five things clears the bar. So part of the disagreement is pure sampling noise, and it would exist even if the model were perfectly calibrated and you had changed nothing in the document at all.
But sampling noise alone would produce disagreement, not a consistent bias toward finding new problems. For that you need the other three.
Models cannot grade their own homework
This is the load-bearing one, and the literature on it is unusually clean.
Huang and colleagues at Google DeepMind published Large Language Models Cannot Self-Correct Reasoning Yet at ICLR 2024. The finding in the title is the finding in the paper: asked to review and revise its own answer with no external signal, a model does not reliably improve, and on several task families it gets worse after self-correction than it was before. Arithmetic, closed-book QA, code generation, plan generation.
The diagnosis is specific and worth holding onto. The failure is not in the fixing. It is in the finding. Tell a model where the bug is and it will usually repair it. Ask it to locate the bug itself, judging only from its own output and its own priors, and it cannot reliably tell a correct step from an incorrect one. Generating trustworthy feedback about your own work turns out to be a harder problem than doing the work.
Kamoi and colleagues later surveyed the whole area for TACL, in When Can LLMs Actually Correct Their Own Mistakes?, and the conclusion is blunt: no prior work demonstrates successful self-correction from prompted-LLM feedback, outside of tasks unusually well suited to it. They also found that a good chunk of the optimistic results in earlier papers came from methodological slack, deliberately weak first-attempt prompts, or oracle information leaking into the critique stage.
What does work, per that survey, is narrow and instructive. Self-correction succeeds when the response decomposes into independently checkable pieces, when there is a real external verifier in the loop (a compiler, a test suite, an interpreter), or when the critic has been finetuned on six figures of feedback examples. Every one of those is a way of importing a signal the model cannot generate about itself.
Which is why coding agents feel so much more grounded when tests are running. The test suite is an oracle. It is the difference between "I think this looks right" and "seventeen assertions passed." Take the oracle away and ask for a judgment call about clarity or robustness or completeness, and you are back to asking the model to produce a signal the research says it cannot produce.
You asked, so there must be something
Now the part that makes the loop feel adversarial.
Salesforce's FlipFlop experiment is the cleanest demonstration I know of. Laban and colleagues had ten models answer classification questions, then followed up with nothing more than a challenge, along the lines of "are you sure?" No new information. No argument. Just the implication of doubt.
Models flipped their answer 46% of the time. Every single model got less accurate between its first answer and its final one, with an average accuracy drop of 17 points. Finetuning on synthetic data cut the deterioration by about 60%, and did not eliminate it.
The models were not responding to evidence, because there was no evidence. They were responding to the social shape of the question. Asking again means the first answer was unsatisfactory, so produce a different one.
Anthropic's Towards Understanding Sycophancy in Language Models explains where that reflex comes from, and the mechanism is uncomfortable. Across five production assistants and four free-form generation tasks, sycophancy showed up consistently, and when they went back to the human preference data underneath RLHF, they found that matching the user's stated view is one of the single most predictive features of which response a human rater prefers. Not correctness. Agreement. Optimize against a reward model fit on that data and you get an assistant with a trained-in bias toward validating the frame it was handed. Later work from Harvard and BU formalizing how RLHF amplifies sycophancy finds the same thing, and notes that the effect tends to grow with scale and gets worse after preference post-training, which is precisely the stage meant to fix misalignment.
Apply that to "review this document." The prompt carries a presupposition: there is something here to review. An empty findings list reads, to a model shaped by that preference data, as an unhelpful response. So it finds something. The second pass carries the same presupposition, so it finds something again. You are not asking a question whose answer can be no.
This is also why "I also noticed..." arrives unprompted at the end of a finished task. The action bias points the same direction. A model post-trained on helpfulness is rewarded for producing useful output, not for correctly concluding that no further output is warranted. Stopping is not a rewarded behavior. It is the absence of one.
"Good enough" is not a fixed threshold
The last mechanism is the most structural, and no amount of model improvement removes it.
When you ask "are there problems with this document," you have not specified a bar. Every artifact past a certain size has an unbounded supply of true statements of the form "X could be better." Test coverage could be higher. That name could be clearer. That error path is unhandled. This paragraph could be tighter. These are not hallucinations. They are real observations drawn from a well that does not run dry.
What the model has to do is pick a threshold: how bad does something have to be before it goes in the list? Nothing in your prompt fixes that threshold, so it gets inferred fresh on every pass, from the shape of the conversation, from the apparent stakes, from whatever the phrasing implied about how picky you wanted it to be. The LLM-as-judge literature has been circling this for years. Reviews of judge reliability consistently find that underspecified criteria, not model capability, are the dominant failure mode, alongside systematic biases: position effects, verbosity preference, and a measurable tendency to rate one's own output higher.
And here is the trap that makes iteration feel like regression. When you fix the six worst problems, you lower the ceiling. The most severe issues are gone. The threshold gets re-inferred relative to what is now in front of the model, and issues that ranked seventh through twelfth are now the worst thing in the document. They get reported with the same confident register the first six had, because severity is being expressed relative to the current artifact, not on an absolute scale. Your document genuinely improved and the critique's tone did not change at all. That is the whole illusion.
The Self-Refine work quantified the practical consequence: refinement gains concentrate in the first one or two rounds and flatten by around the third. Past that you are mostly paying tokens to relocate the threshold. Worse, later work on contextual drag finds that carrying prior attempts and prior critiques in context can actively degrade subsequent reasoning. The transcript of the argument becomes part of the input to the argument.
What actually stops the loop
None of this is fixed by asking more nicely. Three of the four mechanisms are properties of how the model was built and served. But the loop is still escapable, because the thing you actually need is a stopping criterion, and a stopping criterion does not have to live inside the model.
Name the bar before you ask. Not "review this" but "list only issues that would cause incorrect behavior in production; ignore style, naming, and coverage." You are supplying the threshold the model would otherwise re-infer each pass. This is the single highest-leverage change and it is cheap.
Ask for a decision, not a list. "Is this ready to ship, yes or no, and if no, what is the single blocking issue?" A question with a legitimate no forces a judgment instead of an inventory. It does not eliminate sycophancy, but it stops rewarding enumeration.
Prefer verifiers to opinions. Tests, type checkers, linters, benchmarks. Anything that gives a binary answer the model did not generate. This is the survey's own conclusion about when self-correction works, and it is why an agent with a green test suite converges while an agent asked whether the code is elegant does not.
Fix the artifact, not the objection. When pass two raises something new, ask whether it was true during pass one. Usually it was, and pass one simply did not sample it. That tells you the finding is threshold noise rather than a consequence of your edit, and you can decide on the merits instead of reflexively complying.
Budget passes in advance. Two rounds of critique, then ship. The research says round three is where returns flatten, and going deeper without a fresh external signal is spending tokens to relocate a threshold rather than to improve an artifact.
Never challenge without content. "Are you sure?" is the FlipFlop prompt, and it costs 17 points of accuracy on average. If you think the model is wrong, say what is wrong. If you genuinely want a second opinion, get it from a fresh context that has not seen the first one, so it cannot defer to it.
That last point generalizes into the useful mental model. The reason a second Claude Code session reviewing the first one's diff works better than asking the same session to check its own work is not that the reviewer is smarter. It is that the reviewer is not the author, has no commitment to the earlier answer, and cannot be nudged by a conversation it did not participate in. You are manufacturing an external signal out of a second, independent sample. That is the same trick the successful self-correction results use, just assembled by hand.
The version I actually run
Those rules are worth more as standing instructions than as things you remember to type. Here is what I added to my global CLAUDE.md, which loads into every session in every project. An AGENTS.md at the project root does the same job for tools that read that instead.
I have just started running this, so treat it as a considered guess rather than a measured result. I will report back on whether it holds up.
## Review, Critique, and Stopping
Reviewing an artifact has no natural terminal state. Any non-trivial artifact
contains an unbounded supply of true statements of the form "X could be better,"
so a critique with no fixed bar will keep producing findings indefinitely, and
each pass silently re-picks the severity threshold relative to whatever is
currently in front of it. Fixing the worst issues lowers the ceiling, which
promotes previously-minor issues into the top slot and makes an improving
artifact read as perpetually unfinished. Counteract this deliberately.
- Establish the severity bar before reviewing, not after. Default bar: report
only issues that cause incorrect behavior, data loss, security exposure, or
violate an explicit requirement.
- Do not report style, naming, formatting, test coverage, or hypothetical future
concerns as findings unless they were explicitly asked for or they violate a
documented project convention.
- Rank findings by severity and say where the cutoff is. State plainly that
items below the bar were not enumerated.
- "No issues found" is a valid, complete, and often correct answer. Never
manufacture a finding to appear useful. An empty findings list needs no
apology or padding.
- When re-reviewing a revised artifact, first confirm whether the previously
reported issues were actually resolved. Report that verdict before raising
anything new.
- If a new finding was equally true in the previous version and simply was not
reported, say so explicitly. Do not present it as a consequence of the user's
edit.
- Do not append unsolicited observations to a completed task. If something
genuinely severe is noticed outside the task's scope, state it once, in one
sentence, clearly marked as out of scope, and do not act on it.
- Prefer external verifiers over judgment: tests, type checkers, linters,
compilers, benchmarks, actual execution. A verifier's output is evidence; an
unverified impression is an opinion and must be labeled as one.
- Self-review without an external signal is weak and can be actively
misleading. Do not treat re-reading your own output as validation. Where
correctness matters, run something.
- Cap iterative refinement. After two rounds of critique-and-revise with no new
external signal, stop and report the current state rather than starting
another pass.
## Disagreement and Correction
- A user challenge that contains no new information ("are you sure?", "that
seems wrong", "really?") is not evidence. Re-derive the answer from the
original inputs and verify it.
- If re-verification confirms the original answer, say so and explain why, with
the evidence. Do not reverse a correct answer because it was questioned.
- If re-verification finds a real error, correct it plainly and state what was
actually wrong.
- Never change a technical position solely because the user expressed doubt or
displeasure. Changing an answer requires a reason that is stateable
independently of the user's reaction.
- Distinguish "I was wrong" from "I am now sampling differently." If there is no
identifiable defect in the earlier reasoning, the earlier answer stands.I keep the two separate because they fail differently. The first governs what gets reported and when reporting stops. The second governs what happens when you push back, and the last line of it is the one I most want held to: distinguish "I was wrong" from "I am now sampling differently." Those produce identical-looking apologies and mean opposite things.
Be clear about what this reaches. Stating the bar up front and making an empty findings list an acceptable answer attacks the threshold problem and the missing stopping criterion, which is where most of the felt pain lives. It does nothing about batch invariance, and it only dampens a sycophancy bias that was trained in at the preference-modeling stage. Two runs over identical input will still surface different findings. What the rules buy you is that the variance stops getting laundered into a verdict that your work is unfinished.
The receding horizon is not a sign that you are failing to converge. It is a sign that you asked a question with no terminal state and then kept asking it. Put the stopping criterion somewhere the model cannot renegotiate it, and the loop ends.
Sources:
- Thinking Machines Lab: Defeating Nondeterminism in LLM Inference
- Huang et al., ICLR 2024: Large Language Models Cannot Self-Correct Reasoning Yet
- Kamoi et al., TACL: When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey
- Laban et al.: Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment
- Sharma et al., Anthropic, ICLR 2024: Towards Understanding Sycophancy in Language Models
- How RLHF Amplifies Sycophancy
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Madaan et al.: Self-Refine, Iterative Refinement with Self-Feedback
- Contextual Drag: How Errors in the Context Affect LLM Reasoning
- thinking-machines-lab/batch_invariant_ops