Inference contamination in docs, and cleaning it up with an LLM

What it looks like

Inference-contaminated documentation reads plausible in isolation and misleads only when checked against source. A few recognisable patterns:

• Verdicts framed as facts. "This handler is idempotent." Was it, in the version reviewed? Is it now? The sentence is a conclusion, not a citation.
• Motivations that survive the code. "We use a linked list here because random-access is never needed." Two refactors later the same file iterates with an index and slices by range - the "because" claim never got updated because nobody thought to remove it.
• Performance and complexity claims with no anchor. "O(log n) lookups", "sub-millisecond in the hot path", "safe for 10k concurrent connections". True at some point on some machine, quoted forever.
• Cross-referenced assumptions. Doc A says "state is owned by module X". Doc B, written later, says "since X owns state, we can skip locking here". The chain of trust in the second claim rests entirely on the first - and if the first was already stale, the second is compounding it.
• Architecture-diagram fiction. The diagram shows a queue between two services. The queue was removed in a refactor; the diagram wasn't. Anyone reading the diagram (including an LLM you paste it into) reasons about a system that no longer exists.
• Comments that explain the wrong invariant. // x is always non-null here next to a line where x can now be null in three call paths added after the comment was written.

The common thread: none of these are lies. Every one was a valid inference at some point. The problem is that documentation preserves inferences with the same fidelity as facts, and inferences decay faster than facts do.

Why it compounds

Once a claim is written down, downstream readers do not usually re-derive it. They treat it as a fact and build further inferences on top. Six months later:

• A design doc cites the README's now-stale claim as a constraint.
• A test is written to enforce the "invariant" from the outdated comment.
• An LLM, given the repo as context, hallucinates entirely coherent code around the wrong assumption because the docs said so and the docs are authoritative.

The last one is the modern accelerant. LLMs are excellent at treating your documentation as ground truth. If the docs are contaminated, the model's output is confidently wrong in the same direction. And when a human then writes down what the model said, that becomes new documentation - contaminated at layer n+1. The loop is fast and quiet.

Facts vs. inferences: a working distinction

The cleanup only works if you can name what stays and what goes. A rough taxonomy:

The point of the table is not to be rigid; it is to give the LLM something concrete to test each sentence against.

The cleanup workflow

The naive approach - paste a doc into a chat and say "clean this up" - produces the opposite of what you want. The model rewrites the prose in its own voice, keeps the confident tone, and often strengthens unverified claims because that reads better. What follows is a structured workflow that avoids this failure mode.

Step 1 - Establish the ground truth first

Never hand the model documentation and code together and ask it to reconcile them. It will average the two - and the docs get equal weight even when they are the thing you distrust. Instead, hand it code first, alone, and ask it to characterise the code with no reference to any documentation:

You are a code analyst. Read the following source files and produce a
plain factual description of what the code does. Rules:

1. Every claim must be directly supported by a line in the code you were
   given. Cite the file and line number for each claim.
2. Do NOT infer intent, motivation, or design rationale. Describe
   behaviour, not "why".
3. Do NOT describe how the code "should" work or how it is "typically"
   used. Describe what it does.
4. Where behaviour depends on external inputs or environment, say so
   explicitly and stop — do not guess the values.
5. If something is ambiguous, list it as an open question rather than
   resolving it.

Files:
<paste source>

The output is a citation-anchored description of the code as-is. This becomes your reference for the next step.

Step 2 - Classify each claim in the existing doc

Now hand the model the existing documentation and the ground-truth description from Step 1. Ask it to classify, not rewrite:

You will receive two inputs:
  A) Existing documentation for a component.
  B) A citation-anchored description of the same component's source
     code, produced from the code alone.

For every claim in (A), classify it into exactly one of:

  SUPPORTED     — the claim is directly evidenced by (B). Quote the
                  supporting line(s) from (B).
  UNSUPPORTED   — the claim is not in (B) and cannot be derived
                  purely from what (B) says.
  CONTRADICTED  — (B) contains evidence against the claim.
  RATIONALE     — the claim is about intent or motivation, which (B)
                  cannot confirm or deny. Neither support nor refute.

Rules:
- Do not merge or paraphrase claims. Keep each sentence-sized claim
  separately classified.
- Do not use general knowledge to bridge (A) and (B). If (B) does not
  say it, you do not know it.
- Output a table: claim, classification, evidence-quote (or "none").

Input A:
<paste docs>

Input B:
<paste Step 1 output>

The output is a decision matrix. Now you know exactly which sentences in your docs are contaminated and which are safe.

Step 3 - Rewrite with an explicit deletion budget

The trap here is instructing the model to "rewrite the contaminated parts". It will replace them with equally-confident new sentences. Instead, force it to either delete, downgrade, or anchor:

Rewrite input (A) using the classification table. For each claim:

  SUPPORTED    → keep, and add a citation from (B) inline.
  UNSUPPORTED  → delete the sentence. Do NOT replace it with a
                 hedged version. Deletion is the default.
  CONTRADICTED → delete, then add one sentence describing the
                 actual behaviour from (B) with a citation.
  RATIONALE    → keep only if it is explicitly framed as
                 authored rationale ("Original design intent:
                 <YYYY-MM>"). Otherwise delete.

Hard constraints:
- The output must be shorter than the input. Deleting is a feature.
- Do not smooth over deletions with new prose. Leave the structure
  choppy if that is what strict rules produce.
- Do not add any new claim that is not in (B).
- Do not "improve" tone, register, or wording of SUPPORTED claims
  beyond adding the citation.

Input A: <paste docs>
Input B: <paste Step 1 output>
Classification: <paste Step 2 output>

The shorter than the input and deletion is a feature instructions are load-bearing. Without them the model will inflate the doc back to its original length by paraphrasing.

Step 4 - Adversarial pass

Run one more round on the cleaned doc. Change models if possible - a different family will not share the first model's blind spots.

You are auditing documentation for inference contamination.

Given:
  A) A cleaned documentation draft.
  B) A citation-anchored source description.

Find every remaining sentence in (A) that asserts something about the
code without being directly supported by (B), and every sentence
where the supporting citation does not in fact support the claim
made. Be adversarial: assume the draft still contains inferences.

Output:
  - List each suspect sentence.
  - Explain the mismatch in one line.
  - Do not propose fixes. Only diagnose.

The "do not propose fixes" rule keeps the model in critic mode. If you let it fix things it will inference-contaminate the fix.

Step 5 - Human triage of the diagnosis

The final step does not use the LLM. Read the diagnosis from Step 4 and decide, per finding, whether to delete the offending sentence or to run the code / write a test that promotes the claim to SUPPORTED. This is the point at which claims can be legitimately reintroduced - but only with new evidence, not new prose.

Prompt patterns that reduce recontamination

A few patterns are worth reusing across the whole workflow:

• Cite or delete. Any prompt that produces documentation should require an inline citation (file:line, test name, or command output) for every substantive claim. Sentences without a citation are removed at the end.
• Ban weasel adverbs. "Typically", "generally", "usually", "should", "designed to". These are inference-shaped words. Instruct the model not to use them; they always precede an unverified claim.
• Separate is from should. Docs mix these silently. In prompts, ask for two sections: Observed behaviour (what the code does) and Intent as of <date> (what the author says it is for). Never let intent claims sit unmarked in an "is" paragraph.
• Fresh context, per section. Long-running chats accumulate the model's own earlier inferences and start treating them as facts. For each doc section, start a new context with only the source files and the ground-truth output from Step 1.
• Refuse "sounds right". If the model is uncertain, "unknown" or "not evidenced by source" are correct answers. Say so in the system prompt; models default to plausibility unless prevented.

What to look at first (triage)

You cannot clean every doc at once. Prioritise:

1. Docs an LLM is currently reading. Any file in a repo that is fed as agent context (READMEs, CONTRIBUTING, ARCHITECTURE.md, CLAUDE.md, agent instructions, memory files, project rules) - contamination here amplifies fastest.
2. Docs cited elsewhere. Follow the citation graph. If doc B cites doc A, clean A first - otherwise B's cleanup will re-inherit A's staleness.
3. Docs older than the last major refactor of the code they describe. Use git log to find the last significant change to the described module, and check if the doc has been touched since.
4. Onboarding docs. These get read the most times and updated the least. High blast radius per stale sentence.

Prevention: keep the drift small at the source

Cleaning up after the fact is the recovery path. To keep the surface small:

• Prefer citations to prose. A line that says "See src/cache.go:42 for the eviction policy" ages better than a paragraph paraphrasing what happens at line 42.
• Mark opinions as opinions. "Rationale (2026-06, author):" is a signal to future readers - and future LLMs - that the following is authored intent, not code-derived fact.
• Delete when in doubt. An empty section is a smaller lie than a stale one.
• When an LLM produces documentation, require it to cite for every claim. Reject output that reads well but does not cite.
• Bar the model from reading the existing doc while writing a new one for the same code. Otherwise the old doc's inferences show up in the new doc, laundered.

When automation is not enough

The workflow above catches sentence-level contamination well. It is weaker against:

• Structural contamination - a whole section devoted to a subsystem that no longer exists. The LLM will happily rewrite the section rather than notice it should be deleted. A human should check the section list against the actual module list once per pass.
• Diagram fiction - boxes-and-arrows drawings. LLMs are poor at reasoning about diagrams as ground truth; either regenerate the diagram from the code (many tools can), or delete it.
• Cross-repo assumptions - docs that assert something about another repository. The ground-truth step only has this repo's source; the LLM cannot verify cross-repo claims. Flag them for manual review.

Treat the LLM as a fast, tireless, slightly credulous editor. It will remove more contamination in an afternoon than a human can in a week, provided the prompts refuse to let it invent replacements.