Enterprise analytics: preventing silent data errors

A media client needed repeatable, client-ready reports from a large licensed demographic and behavioural dataset. Analysts were spending most of their time wrangling data, and a custom report took weeks.

I joined a small team as the security and infrastructure lead. In practice I also did much of the data engineering: restoring the source archives, loading a dataset dictionary of several thousand variables, building the analytics tables and mirroring them with count checks. The client is under NDA, so no names, data or screens here.

Most of the code would be written by AI coding agents. That's fast, and the failure mode isn't a crash. It's a number that looks right and isn't. In a report, a wrong number nobody notices is worse than an outage.

So the question wasn't "can agents build it". It was how do we know the numbers are right when agents built it.

• —Docs-as-code governance. Decisions and specs live in the repo as a structured knowledge bundle (about 39 ADRs and 18 specs) that CI validates. Agents work from it, and the build fails if the bundle doesn't conform.
• —A contributor contract for agents. A short, copy-paste set of rules every AI session starts with. Generated files can't be hand-edited; they're rebuilt from source.
• —One gate. A single check command runs typecheck, boundary guards, knowledge-bundle conformance, lint, registry checks, and app, sums and pipeline tests.
• —Sum and benchmark tests. Totals are reconciled against independent counts, and a census benchmark accounts for every variable it reports on.
• —A data-source seam. The engine talks to an adapter, so the same code runs against the client's on-prem SQL Server in production and a Postgres mirror in rehearsal.

None of these threw an error. All three produced wrong numbers silently.

1. 01Mean of averages. Average household income was computed by averaging pre-averaged values, overstating it by 8.6%. The fix stores the aggregate and its weight as separate rows, so every rollup is a true weighted mean.
2. 02Unordered pagination. Paged reads without a stable sort order silently dropped or duplicated about 37% of rows, with 17% variance from run to run. Every paged query now has a deterministic order.
3. 03A capture rate of zero. A capture-rate metric silently came out as 0. It was caught and fixed.

The pattern is the point. Agents are good at code that runs. They aren't good at noticing when a number is quietly wrong. That has to be engineered in.

Put sum and row-count reconciliation in the first commit, not next to the first report. Checks against an independent count are cheap, and they catch exactly the bugs agents make.

• —Engine: Node 22 type-stripped TypeScript, no build step, zero runtime dependencies
• —Data: SQL Server in production behind an adapter, Postgres mirror, SQL Server archives restored via Docker
• —App: Next.js
• —Quality: node --test, coverage floors, CI-validated knowledge bundle
• —Infra: on-prem production, IPsec VPN to the client network, cloud rehearsal deploys