The audit with no client

An audit you commission is graded by someone you are paying. An audit you run on your own system has the opposite problem: you wrote the thing, you believe in it, and you would rather it passed. Neither incentive points at the truth. The first is too soft; the second is too invested.

So the honest test of an audit method — not a system, a method — is to run it against something you operate, where you are motivated to pass, and see whether the method still surfaces the parts you would rather not look at. If it only finds problems in other people’s code, it is not a method. It is a sales pitch.

This is that test. The two earlier posts on this site describe the practice: isolation below the protocol, and governance above the agent. Each ends in a checklist. What follows is both checklists run end to end against a production autonomous-agent system I operate — one that holds budgetary authority, makes allocation decisions without a human in the loop on each one, and has been running continuously for months. I will not name it, and nothing below depends on what it is. The mechanisms are the point.

I am going to tell you what held, what didn’t, and the one structural fact that every finding reduced to. The findings stay in. That is the whole exercise.

Scope: two layers, one system

The system runs more than two dozen agents in four runtime classes — a single executive kernel on a timer, a handful of long-running services, ephemeral workers dispatched per task and destroyed on completion, and template-instantiated agents scoped one-per-managed-unit. Some of them can spend money. Some of them can rewrite the system’s own code. All of them read from, and write to, a shared state store.

That is exactly the shape the two earlier posts pull apart. There is an isolation question — what can a compromised agent reach — and a governance question — what can a compromised agent rewrite. The reason to audit both at once, on one system, is that the interesting failures do not live inside either layer. They live at the seam, where a control in one layer quietly depends on a control in the other.

The audit method has one rule that matters more than the checklists: verify against the running system, not the source. Code comments describe intent. Commit messages describe hope. Design docs describe the version the author wishes they had shipped. Only the running system describes what is actually true.

A small example, because it is the kind of thing the method exists to catch. The enforcement agent in this system carries a module docstring stating it runs in observe-only mode — detect and log, never block. Reading the source, you would write that down as a finding: the gate is a smoke detector, not a sprinkler. The running system says otherwise. Its live configuration has the gate in blocking mode, and every one of its recent run records confirms it: it halts. The docstring was stale by some number of weeks. Had I audited the source instead of the system, I would have shipped a false finding about my own code — which is the failure mode the rule is there to prevent, and a fair preview of how easy it is to be wrong with high confidence.

What held

Credit where the design earned it. These passed, and I confirmed each against the live system rather than the design:

• The signed manifest verifies every cycle. An ed25519-signed file pins content hashes for the constitution and a dozen other security-critical files. The enforcement agent recomputes those hashes on a schedule and checks the signature. A mismatch flips the halt scope. Editing a hashed file without re-signing is not a silent change; it is a guaranteed, logged halt.
• The halt scope is canonical and fails safe. One field, a closed set of allowed values, read by every agent before it acts. A write of any unrecognised value is refused and coerced back to the safe default, and the attempt is logged as its own integrity event. An unknown halt reason resolves to the most restrictive scope, not the least. The lever fails toward stopped.
• The spend gate runs before the spend. A proposed transaction above a fixed per-transaction ceiling — single dollars — is denied at evaluation time and routed to the principal for approval. The check reads its limit live from policy on every call, so the ceiling cannot be stale-cached around.
• The self-modification loop cannot reach the root. The system can rewrite its own agent code behind a test gate, a diff-size cap, rate limits, and a rollback window. But the loop’s allow-list explicitly excludes the constitution and the signed manifest, and can only stage — never apply — changes to the hash-locked policy files. It can fix a bug in itself; it cannot quietly relax its own constraints.
• The audit log is tamper-evident. Each row chains to the previous by hash, so a retroactive edit breaks the chain at the point of the edit and a periodic external checkpoint catches it. Append is cheap; rewriting history is not silent.
• The enforcement agent cannot be silenced from inside. The reasoning layer can pause other agents within bounds, but a protected-agents list excludes the enforcer, the treasury agent, and the kernel. Nothing inside the system can pause the thing that watches the system.

If a product I was reviewing for a client answered the sixteen checklist questions the way this system does on these points, I would tell them their architecture is real — structural, not aspirational. Most do not. So this is not nothing. But it is also not the report. The report is the next section.

Four findings

Each of these is a place where a control catches the obvious attack and then stops catching just short of where you would want it to. None is a live hole — each is mitigated in operation. All four are the kind of residual a real audit names instead of rounding up to “passed.”

1. The signature proves consistency, not authenticity

The manifest is ed25519-signed, and the enforcement agent verifies the signature every cycle. The question an audit has to ask is not “is it signed” but “against which key.” Here, the verifier reads the expected public key out of the manifest it is verifying. The signature therefore proves the file is internally consistent and signed by whatever key the file carries — not that the key belongs to the principal.

An attacker who can write the manifest on the live host can replace the content, replace the embedded public key with their own, and re-sign. Every runtime check passes. The signature was never the trust root; the trust root is the binding between that key and the principal’s identity — and that binding lives outside the verification: in an offline signing key the running system never holds, in the file’s write permissions on the host, in a separate pre-deploy verifier. Those mitigations are real and they hold. But they are operational, not cryptographic, and the audit has to say so plainly: a verifier that reads its expected key from the object it is verifying authenticates nothing by itself. Pin the key somewhere the signed object can’t reach — in code, in hardware, out of band — or the signature is a consistency check wearing an authenticity costume.

2. The window between violation and detection

Integrity verification runs on a cadence — at most every few hours, by design, because running it on every kernel tick would burn budget for no gain on a system that mutates this rarely. That cadence is also a window. Between a prohibited on-disk change and the next verification pass, the change is present and unflagged.

Two things mostly close the window: agent processes are short-lived, so an in-memory mutation that never reached disk clears when the process recycles; and a single pass is short relative to the work an agent would need to do to exploit the gap. “Mostly” is doing real work in that sentence. The case the cadence does not see at all is the agent that loads a hashed file into memory, mutates the in-memory copy, and acts on that — the on-disk hash never changes, so the on-disk check never fires. The honest finding is not “the system is exposed for hours.” It is: integrity here is a property of files at rest, checked periodically; it is not a property of what an agent is holding in memory right now. Whether that matters depends entirely on how much an agent can accomplish inside one window — which is a number every system of this shape owes its auditor, and most cannot produce.

3. The spend bound is emergent, not stated

There is a per-transaction ceiling. There is a per-project budget. There are per-agent daily reasoning budgets and per-subsystem daily caps. What there is not is a single number that says “this system will not spend more than X in a day, total.”

The global bound exists — it is the sum of every local cap, backstopped by a treasury-floor tripwire that halts non-essential work when reserves run low. But it is emergent: you compute it by adding up the caps and trusting that no path was left uncapped. The defense against aggregate runaway is the floor tripwire firing after the fact, not a ceiling refusing the spend before it. That is a defensible design — many small, local caps are more granular and degrade more gracefully than one global throttle. But an audit’s job is to make the implicit number explicit and ask the owner: is the sum of your caps the number you actually intended your worst day to cost? In my case the honest answer was “I had never added them up.” Most owners haven’t.

4. The halt is cooperative

This is the finding the other three were walking toward. The halt scope — the universal stop lever — works because every agent reads it before acting. That is a protocol, not a cage. Nothing at the operating-system level forces an agent to honour the field. An agent whose code was modified to skip the read would not stop.

So the halt’s reliability is not really about the halt. It reduces to the integrity of the agent code that reads it. And that integrity is held by the self-modification allow-list and the hashed-file guards — which reduce, in turn, to the signed manifest. Which reduces, from finding 1, to a key whose binding lives outside the system. The stop lever is exactly as trustworthy as the code-integrity guarantee underneath it, which is exactly as trustworthy as the manifest’s trust root. Pull on the halt and you are pulling on the same thread as everything else.

One root of trust

That is the finding under the findings. Isolation below the protocol and governance above the agent look like two independent layers, and the two earlier posts treat them separately for clarity. Audited together on one running system, they are not independent. They share a root.

The isolation layer bounds what an agent can reach. The governance layer bounds what an agent can rewrite. Both ultimately rest on two things being true: the integrity of the code an agent runs, and the authenticity of the key that signs the manifest declaring what that code should be. Compromise either — rewrite the code that reads the controls, or forge the manifest that defines integrity — and both layers fall at once, because both were leaning on the same post the whole time. Every control I verified as holding holds conditional on that root. The findings are not four separate weaknesses. They are four views of the single load-bearing assumption, seen from four directions.

This is also the thing most agentic products get wrong, and it is why the checklists matter. It is common to have controls. It is rare to have controls whose trust root the agent cannot reach. A spend cap in a database row the agent can write is not a cap. A halt flag the agent’s own code chooses to check is not a halt. A signature verified against a key the agent could swap is not a signature. The work of an audit is to follow each control down to its root and ask whether the agent can touch it — and then to find the windows where the root is real but hasn’t been re-checked yet.

What you would get

One more honest line, because the method has limits too. Every control above is enforced by code reading files. The constitution that sits at the top is only partly machine-enforced — the treasury floor, the spend limit, the operational prohibitions, the dead-man’s switch are checked in code; the articles about mission and primacy are enforced by design and by my reading of them, not by a check that can fire. An audit should tell you which of your stated rules are load-bearing code and which are load-bearing trust. They are not the same, and most documents blur them on purpose.

That blur — rules that read like enforcement but resolve to hope — is the single most common finding when I review an agentic product, and it is the reason this post exists. The deliverable from an engagement is what you just read, aimed at your system instead of mine: the controls that hold, the ones that are only partial and exactly where they stop, each residual named at the architectural level, every claim reproduced against your running system rather than your design doc. The findings stay in — including the ones you were hoping I wouldn’t reach, because those are the ones you were paying me to reach. I ran it on myself first so you can see what that looks like before you commission it.