AgentOps: AI agents you can audit

Architecture Decision Records are written once and trusted for years. The code moves on. Nobody goes back to check, so the records drift from what is actually running.

A wrong ADR is worse than none. Engineers and coding agents read it as truth and build on it. I wanted a tool that tells me, per decision, whether the record still holds, and shows its working.

It is not an agent framework. Gather context deterministically, make one model call with a JSON schema, write the artefacts. Only one step guesses.

Models are frontier OpenAI and Gemini models with tiered escalation: a first call, bounded repair attempts, then a stronger tier.

Models will cite things that aren't there. So the model never decides what counts. The validator does.

• —Each quote is checked segment by segment against the evidence corpus the model was shown. The ADR text and the instructions are excluded, so the model can't quote the claim back as proof of itself.
• —Segments have a minimum length and must appear in order within a line. Passages stitched together from too many fragments are rejected.
• —A claim that something is "planned" needs a past or future marker that governs the artefact and names it. Otherwise it is rejected as an unsupported excuse for drift.

I reviewed a sample afterwards. Most scope rejections were genuine grounding errors by the model that the rules correctly stopped. One rule, in-order quoting across lines, was too strict. I narrowed it under review and replayed: 5 of the 6 affected ADRs cleared.

An audit nobody can reproduce is an opinion. So the harness runs like a production system, with the same discipline I'd want from a payments service.

• —Frozen manifests. The plan freezes the release directory, component hashes, models, effort and pinned commit. The run re-verifies all of it before any dispatch.
• —Budgets. Model invocations, prompt and output tokens, attempts per ADR, wall time and HTTP retries are enforced before each dispatch and across restarts. Unknown usage is charged at the worst case. Quota and billing errors stop the run at once; rate limits are honoured and paced.
• —Crash safety. Each attempt gets its own directory and an OS-released lease. Timeouts kill the whole process session. Resume refuses to start beside a live orphan, and cleanup holds survive a reboot.
• —Releases and rollback. Releases are byte-identical rebuilds of a pinned commit, checked on the host against a 4,574-file manifest. Deploy is an atomic symlink switch under a host lock. The three newest releases are kept, so rollback is one command.

The first version looked like it worked. An early review showed the deterministic layer, the part I trusted most, was wrong.

• —The gatherer assumed the Status line sat on line 2 of every ADR. That one assumption fabricated a finding on 165 of 195 ADRs.
• —A suppressed cloud auth failure was reported as "0 Kubernetes clusters". The tool stated a falsehood about production with full confidence.
• —Batch selection ran before ledger filtering, so some batches audited zero ADRs and still reported success.

I don't publish an accuracy number until it has been independently judged against a pre-registered sample. That study is in progress. Until it is scored, this page has no accuracy figure.

Read that precisely. The selector sent 66.7% fewer source bytes at equal measured coverage. I did not measure whether the audit verdicts were as good with the smaller context. The same evaluation found the selector was not fit for implementation judgement (2 of 9), and I recorded that as a NO_GO.

• —Pre-registered protocols. The sample, labels and scoring rules are frozen before results exist.
• —Samples chosen by hash, not by me.
• —Independent judging. Labels come from a reviewer who is not the author.
• —Blind sets, mutation benchmarks, repeatability sets and Wilson intervals.

Under two calls per ADR on average. That is what one schema-bound call per decision buys you, compared with an agent loop that wanders the repo.

The ADR audit is one use. The pattern underneath, deterministic evidence, one bounded model call, a validator that decides and receipts for everything, fits most places a model's output needs to be trusted.

• —Architecture drift audit. I run this approach on your repository and hand back DRIFTED and STALE decisions, each with traced evidence.
• —Agent reliability harness. Frozen evidence, receipts, validation, budgets and rollback around the agents you already run.
• —Evaluation design. Pre-registered protocols, independent labels and honest intervals, so the number you report survives scrutiny.

Get in touch if your agents need to be auditable.