An AI researcher that gets rejected when it invents a number.
The interesting work is not the model call. It is making honesty a property of the shape of the output rather than of the prose.
A decision memo, not a report.
You describe an idea and the decision actually in front of you. A panel of synthetic personas answers a generated survey. What comes back is a verdict, the evidence behind it, the strongest objection to it, and the one test worth running next.
The claim being tested is not that the system is safe or accurate. It is that it is auditable: a reader who assumes the model lies should still be able to establish which sentences could not have been invented.
- Relationship to the work
- Executed, as founder and sole engineer
- Read from
- Repository at 9e3afab, 2 Oct 2026
- History
- 96 commits from 9 Jul 2026
- Status
- Live; payments in test mode by its own record
research modules
automated checks on one command
unrelated ideas in the stress harness
frameworks and build steps
The promise everybody makes, and nobody can be held to.
One block of generated prose
- Findings, reasoning, recommendation and forecast in one paragraph, one voice, one confidence.
- A reader cannot tell which sentence came from a measurement. Neither can the author.
The failure is a plausible number
- Not a wrong figure — one that reads exactly like the measured figures beside it.
- A system that cannot be checked is not trusted because it is honest. It is trusted because checking is impossible.
So: separate the kinds of statement, then put a deterministic gate between the model and the reader — one that rejects rather than corrects. A corrected memo is a memo whose author is the corrector.
Four kinds of statement, kept apart on purpose.
Written at the top of the strategist module as the thing that module exists to preserve. Each row carries a different burden of proof, so blurring two of them is a visible error rather than a stylistic choice.
The four-way split
Evidence
Comes from the panel. Computed, not written or summarised.
Reasoning
Comes from the model, and is labelled as such.
Recommendation
Synthesises evidence and must cite it. A claim citing nothing is rejected: judgement must be traceable.
Prediction
Only where something was measured. The module's own note: nowhere, until a variant runs.
The boundary that makes verification possible
The strategist consumes only the evidence bundle
It never touches the graph's raw panel data, so what it can see is itself a constraint — and the verifier checks against the same object.
Everything the reader sees is a view of one typed graph
Memo, summary, report and conversation hold no fact of their own, which is why a citation can be a node id.
What verify() actually enforces.
Eight mechanical rules, each of which rejects the whole output rather than editing it. An LLM strategist is a drop-in replacement for the rules baseline and must pass exactly the same gate.
| Rule | What is rejected |
|---|---|
| Everything cites something | An empty citation list, or a citation to a node id that is not in the graph. |
| No fabricated precision | A quantity asserted in prose that is absent from the whitelist of quotable figures. Anything carrying % or pp is checked at any magnitude. |
| Confidence is not self-granted | A confidence that differs from the evidence engine's — the strategist cannot grant itself more certainty than the panel supports. |
| The case against must exist | A recommendation with an empty against list. A recommendation with no case against it is advocacy, not judgement. |
| Branches must be measured | A recommended branch no persona evaluated. A recommendation requires a measured branch, not a plausible one. |
| A ceiling is labelled a ceiling | A mass quoted without being marked a ceiling, or one that does not match the evidence exactly. |
| Kinds and horizons are closed sets | An unknown move kind, or a horizon outside now / 30d / 90d. |
| The contract is versioned | Output whose strategist version does not match the verifier's. |
Every row is a branch of verify() in js/research-strategist.js; the italicised phrases are that file's own error strings.
The verifier quietly stopped verifying.
- Before
- The first version allowed any number found anywhere in the evidence bundle. Reasonable while the bundle was small, with rounding tolerance built in deliberately, because that is how a memo quotes a share.
- What forced it
- The bundle grew. Adding response segments put every segment's category, branch and substitute mix into scope, and the allowed set became large enough that almost any plausible figure matched something. A fabricated claim that something would increase signups by 40% found its match in one cluster's third-ranked objection share, and passed.
- The decision
- What may be quoted is now declared field by field rather than inferred from whatever is in scope: the headline statistics, sampling, confidence, polarisation, and a named handful of fields per segment and per objection.
- What it changed
- A new headline figure has to be added on purpose. A new internal or diagnostic field cannot widen the set by accident. The lesson the code records in its own comment: nothing about the broken verifier looked broken.
Where the gate is not needed, because there is nowhere to put a lie.
Messaging carries no quantities, by construction
Positioning directions are nine fixed promises, one per objection type. The file's reason: a messaging section is the easiest place in a memo to smuggle in invented numbers, so the shape has nowhere to put one.
Segments are computed, not stored
Clustering discovers them. Nothing writes a segment down, so nothing can invent one, and a reference to a cluster that no longer exists is a dangling pointer the verifiers catch — which they did, in August 2026.
The fallback announces itself
When the live pipeline is unavailable the study still renders and the output metadata records the engine as live or mock. A silent fallback is the thing to catch.
Rules that cost money live in the database
The house rule: if a rule can be bypassed by editing JavaScript in devtools, it is in the wrong place. Orders trigger server-side job and invoice creation; coupons and credits re-validate in Postgres; the arithmetic on screen is display only.
A system this strict still shipped broken.
- Research was down in production for a period — three tables answering not-found, because an idempotent migration had not been re-run. The schema file is the whole schema and there is no migration tool, so a change that is not re-applied is simply absent.
- A silent sign-in race: the session check sometimes beat the code exchange and bounced valid sign-ins. Fixed by waiting only when the URL shows a provider return, so an anonymous visit stays fast.
- The domain served an unrelated project's placeholder after being claimed on a different hosting account. Recovered by proving ownership, and written into the handover as the first thing to check if wrong content ever appears.
- A plain read through the wrong shell destroyed the rupee symbols in the migration file. The handover now records the exact safe command and the file's checksums.
- None of the four was caught by a test, because none was a logic error — a missing re-run, a timing assumption, an ownership record and a text encoding
- A verifier cannot see any of those, which is the honest limit of the approach
- Payments run in test mode until the gateway account completes verification; no real money has moved
Judgement that must trace to evidence, or be discarded.
What verify() enforces, in one line
Read the full study.
Twelve panels, ending on what could not be verified.