Case study · LLM engineering
Making invented numbers detectable
Language models invent figures with exactly the confidence they use for real ones. This library makes two specific failures detectable by code: citing a source that was never provided, and stating a number that appears nowhere in the input. Extracted from a personal health platform, where a fabricated trend changes what you eat.
How it works
The model is handed a context document whose facts carry labels, and must answer in a structure where every number is accompanied by the label it came from. Ordinary code — not another model — then checks that the label existed and that the number appears in what was actually sent.
REJECTED — cites a source that does not exist
[error] unknown_citation: citations to sources absent from the context: M:hrv
REJECTED — states a number from nowhere
[error] invented_number: numbers in neither the claims nor the context: 102
ACCEPTED — uses a context number without a claim
[warning] unclaimed_number: context numbers used in prose without a claim: 77
Four layers, cheapest first
- A schema that is awkward to lie in. A numeric claim without a source does not fail a check — it fails to parse. This is the only non-probabilistic layer, which is why it goes first: no prompt wording talks its way past a parser.
- Deterministic validation. Citations and numbers are checked against the rendered context. No model is involved, so the checker cannot hallucinate its own verdict.
- Retry with the complaints. Rejecting without retrying spends the request and returns nothing. Most rejections are mechanical and clear up once the model is told precisely what was wrong.
- An isolated critic. A second model reviews the advice against recorded constraints and never sees the conversation — so it does not know the user was pleasant, insisted, or sounded disappointed. Isolation is the mechanism; "be objective" in a prompt is not.
The largest lever is not in the library at all: compute metrics in SQL and hand the model finished numbers. A model asked to average four thousand rows produces a plausible figure rather than a correct one. Nothing here fixes that; not asking it to is what fixes it.
Two mistakes worth the write-up
A validator that rejects the truth is worse than none. The number check originally compared prose against claims alone, so anything mentioned in passing without being duplicated as a claim — a date, a set count, a target the model had been given — was declared invented. A routine question was rejected over fourteen such numbers, none of them fabricated. The fix distinguishes invented (in neither claims nor context) from unclaimed (present in the context, just not broken out): the first is an error, the second is untidy bookkeeping the answer survives.
The general lesson is the reusable part: an honesty check that fires on honest output trains people to ignore it, and then it protects nothing.
A schema must not forbid what the prompt permits. The prompt allowed common-knowledge estimates — "a McChicken is around 400 kcal" has nothing to cite, because it is not a claim about the user — while the schema still demanded a source for every number. The model did exactly as instructed and failed to parse. The fix marks such figures explicitly, which weakens nothing for numbers about the user and makes the estimate visible so a rough figure is never rendered as if it were measured.
Decisions & lessons
- Citable labels are derived from the rendered document, never tracked beside it. Drop a section to fit a token budget and its labels stop being citable in the same instant — a parallel registry would still list them, and the validator would start accepting citations to text that was never sent.
- Know what a second model is bad at. It comes from the same family as the first, so its errors correlate and it tends towards agreement. Facts belong to the deterministic layer; the critic gets judgement, which code cannot reach.
- The library states what it does not solve. A model can still be confidently wrong about claims carrying no numbers, and no layer here notices.
- Extracted, not copied. Fresh history, English throughout, plus a privacy guard: what leaks out of a personal repository is rarely a key — it is a hostname in a comment or a body weight in a fixture, neither of which looks like a secret to a scanner.
Where it comes from
The parent project is a private personal platform: Postgres with a bitemporal fact store, metrics computed as SQL views, ingestion from two external APIs, Terraform and Ansible for a single-server deployment, Prometheus and Grafana with alerting, and nightly backups that verify themselves by restoring into a scratch database before they are trusted. This library is the part of it that belongs to nobody personally.