Engineering Notes · agent operations · investigation discipline
My AI Is Not Allowed To Guess
Lucas van Staden · ProxiBlue · September 2026 · proxiblue.com.au
How I turned debugging from an AI's favourite improv exercise into a protocol: blast radius before hypothesis, evidence before claims, a real debugger instead of print statements, and a banned-phrase list for blame-shifting.
Why I built this
Ask an AI agent why a test is failing and, left to itself, it will do the thing juniors do: leap to a plausible story. "Probably a flaky test." "Looks environmental." "Seems like a pre-existing issue." Plausible, confident, and suspiciously often pointing anywhere except at the change the agent itself just made. Then it "fixes" the story instead of the bug, and you get to debug both.
The pattern that kept repeating in my sessions: the failure appeared right after the agent edited something, and the agent's first hypothesis was about anything but its own edit. Not dishonesty; a language model reaches for the most fluent explanation, and "my last change broke it" is less fluent than a nice environmental story. So the fix is not scolding, it is protocol.
The protocol
- Blast radius before any hypothesis. The first mandatory move on any failure: enumerate what changed.
git diff against the last known-good state, working tree included. You do not get to theorise about the world until you have looked at your own footprints. Most of the time the investigation ends right there.
- Claims require artefacts. A diagnosis must cite its evidence: the file and line, the diff hunk, the quoted test output. "Probably", "likely" and "seems" presented as conclusions, without a citation attached, count as failure. Hedging is allowed while investigating; it is banned in the verdict.
- Blame-shift phrases are banned outright. "Must be a flake." "Environmental." "Pre-existing issue." "Infrastructure is down." Not because these are never true, but because they are true far less often than they are convenient. If one of them is genuinely the answer, it has to be earned with evidence like any other claim, and stated as fact, not reached for as an exit.
- Report and stop. Investigation requests produce a diagnosis with evidence, not a surprise fix. I decide what happens next. An agent that "investigates" by changing five things has destroyed the crime scene.
- A real debugger, and the lazy alternative is blocked. For PHP runtime questions (wrong value, wrong branch, null return) the agents drive xdebug through tooling: breakpoints, real variable inspection. And because prose alone would only be followed most of the time, a guard hook blocks any edit that inserts
var_dump-style print debugging. Guesswork printing is not discouraged; it is refused (the debugger-discipline post covers that split).
Enforcement is layered the way everything in this series is layered: the protocol text auto-loads when a failure appears, the debugger guard is deterministic, and eval 6 of my rule-eval suite replays the origin scenario on a schedule: a repo where the suite fails because of an in-tree edit, and asserts the agent ran the diff first, cited evidence, named the real cause, and used zero banned phrases. A judge model grades the report against the protocol; grep checks the rest.
Things I learnt building it
The blast-radius rule does most of the work. Forcing the diff first collapses the majority of investigations into "yes, it was my change, here is the line". Everything else in the protocol exists for the minority where the diff really is innocent, and by then the agent has earned the right to look further.
Banned phrases needed to be literal strings. Early versions said "do not blame-shift", and the agents agreed enthusiastically while doing it in fresh wording. The current list is exact phrases checked mechanically in the eval, plus the prose principle for the spirit. Deterministic where possible, judgment where necessary: same split as everywhere else.
Protocols decay without replay. This discipline lived as prose for a while and worked, until I could not actually say whether it still did. Now it is one of the eight replayed evals, and after a model or harness change I get a fresh answer to "does the investigator still investigate?" instead of a belief.
What it buys me
- Diagnoses I can audit. Every verdict arrives with its citations. Checking the agent's work means reading its evidence, not re-doing its investigation.
- Fewer double-bugs. The fix-the-story-not-the-bug loop is mostly gone, because the story has to survive the diff first.
- Debugging that transfers. Breakpoint sessions leave reproducible traces; print-statement archaeology leaves polluted diffs. The banned tool was also the worse tool.
Want to do this yourself?
- Make "what changed" the mandatory first move on any failure, and mean your agent's own changes, not the weather.
- Require citations on conclusions. No artefact, no verdict.
- Write your banned-excuse list as exact phrases, and check for them mechanically.
- Separate investigation from fixing. Diagnosis is a deliverable.
- Wire in a real debugger and block the print-statement path with a hook, not a request.
- Replay a known incident on a schedule and grade the report. Trust the protocol because it keeps passing, not because you wrote it.
None of this makes the agent smarter. It makes the agent's claims cheap to verify, which for client work is worth more. I do not need my AI to always be right on the first look; I need it to never be confidently wrong in a way I cannot check.
All from my claude-skills-central repo, except where noted:
I use my tooling predominantly on Mage-OS (Adobe Commerce / Magento) e-commerce projects, and my own AI Booking Agent. If you have not swapped to Mage-OS yet, you are falling behind ;)
More in this series: The Whole Stack, Reviewed Cold — a cold, independent critical review of the whole agent-operations stack, layer by layer, weaknesses included. ·
Logs Without the SSH Session — shipping Hypernode logs to self-hosted VictoriaLogs so agents can read them without SSH. ·
The Hour My Agents Spent Waiting For Nothing — why my agents are not allowed to background work they need the result of. ·
Smaller, Then Green — a mandatory cut-then-test loop that trims over-built code out of every plan's diff before any reviewer sees it. ·
Plans Run By A Fleet — how features run through a planned, adversarially-reviewed, parallel TDD pipeline instead of one long agent chat. ·
Hooks, Not Hopes — sorting every agent rule into prose or a blocking hook, and proving the hooks still fire. ·
The Agent Chatroom — giving the agents across my fleet a threaded chatroom so I stopped being the message bus. ·
The Commit That Has To Prove Itself — gating every commit behind recorded, state-hashed test evidence. ·
The Near-Miss That Banned Summaries — banning summarised page-fetches fleet-wide after a near-miss on a security advisory. ·
The Code Quality Checks Nobody Runs — coding standards, static analysis and comment hygiene moved into the commit path. ·
The ProxiBlue Debugger Discipline — blocking var_dump and wiring in real breakpoint debugging. ·
The ProxiBlue Domain Graph — giving my AI coding agents long-term domain knowledge with a temporal knowledge graph. ·
The ProxiBlue Falsifiable Rulebook — testing the rules and guard hooks that govern my AI coding agents.