Engineering Notes · agent operations · independent review
Most of this series is me explaining one piece of my agent tooling at a time. This post is the opposite. I asked for a review of the whole thing, written to be read cold by someone with no prior context: every layer explained from first principles, compared to what the wider industry does, then judged on its merits, weaknesses included. What follows is that review, reproduced in full. I have not softened the findings.
Two notes before you read it. The review speaks about "the operator" in the third person, because it was written that way and I have left it that way. And it is a snapshot: everything in it was checked against the live system on 2 August 2026, on Claude Code 2.1.201. Several of the layers it summarises in a paragraph have since had a full post of their own, linked at the end.
Verdict up front
This is one of the most disciplined individual AI-agent setups you are likely to see. What sets it apart is not any single tool — it is that the system measures and tests itself.
Most people running AI coding agents write a prompt file and hope. This operator does three things instead. The rules that must hold are enforced by code the model cannot argue its way past. Every behavioural rule is treated as a claim to be verified, not trusted. And the whole setup is instrumented, so when something breaks it shows up on a dashboard rather than failing in silence.
The result, across the three client projects in scope: the agent cannot commit untested code, cannot post a sloppy client comment, cannot debug by guesswork, and cannot quietly drift out of policy. If any of that machinery trips or dies, the operator gets a desktop popup and an email.
The weaknesses are the honest ones of a solo operation. It is a lot of custom infrastructure held together by one person, and a few of the newest self-checks have not themselves been heavily tested. But the direction is clear, and unusually mature: this is engineering applied to supervising a machine, not vibes.
1 · Context
The subject is ProxiBlue, a one-person software consultancy. Its operator maintains e-commerce codebases — Magento 2 and Mage-OS, which are large, plugin-heavy PHP platforms — for paying clients. Each project runs in its own Docker-based local environment (DDEV). The AI agent in use is Claude Code, Anthropic's terminal-based coding agent, which the operator runs in two modes: interactively, with a human present, and autonomously — scheduled jobs that pick up tickets and produce code changes with nobody watching.
That second mode is what raises the stakes and explains almost every design choice that follows. Most published guidance on AI coding tools quietly assumes a human reviews every action. Here, an agent can merge a branch, claim "the tests pass," or post a comment on a client's issue tracker at three in the morning with no one at the keyboard. A single bad action becomes a client-facing incident. So the tooling is best understood not as "productivity add-ons" but as an operational safety system: the set of controls that make it defensible to let a probabilistic agent touch production client code, attended or not.
A note on scope, since it matters for reading the rest fairly. The machine hosts a dozen-odd projects, but only three are active client work that needs this tooling: a plumbing-supplies store, a laptop-screen store, and a webhooks service. The rest are non-Magento, archived, or experiments, and are deliberately excluded. "The fleet" in this review means those three. The operator has encoded that scope explicitly, so the tooling never reports out-of-scope projects as problems.
2 · The idea
One idea runs through the whole setup, and it is the thing most individual setups get wrong: work out what must never go wrong, and enforce those things with code — not by asking the model nicely.
An AI agent follows written instructions most of the time. For style preferences, most of the time is fine. For safety, it is not. Say the agent must never push to a live branch. A line in a config file won't guarantee that: the agent reads the file at the start of a session and can lose track of it much later in a long conversation. To actually guarantee it, the rule has to be a small program that watches each command and blocks the dangerous one.
This setup applies that thinking consistently, sorting every rule into one of three tiers:
The first tier is an established idea; the industry calls it "policy-as-code." The second is common. The third — testing your own rules to see if they still work — is rare even in strong engineering teams, and it is the trait that most sets this setup apart. Most of the rest of this review is a tour of where each tier is used, and how well.
3 · The stack
Everything is centralised in a small number of git repositories and mounted into every project, so a fix made once propagates everywhere. Each layer below is explained from scratch, then compared to what is common elsewhere.
A small set of rule documents load into every agent session. These are not generic style guides. Each one was written after a specific failure, and encodes the lesson from it. For example:
These read more like aviation checklists than coding guidelines: banned phrases, mandated report formats, explicit stop-and-restart triggers. They are kept small, and the heavier ones load only when their trigger appears — a debugging rule attaches when the agent opens a PHP file, not on every session — which keeps the token cost down.
Claude Code lets you register hooks: shell scripts that run before or after each tool call and can block it. This setup wires several as hard rules:
When a guard fires it prints instructions to the agent — and, crucially, it cannot be bypassed from inside the session. The bypass needs an environment variable set before the agent even starts.
This is the flagship control. When the agent tries to commit or push code, a hook asks one question: is there recorded evidence that this project's tests ran and passed, against exactly this state of the code? If not, the commit is blocked. Any later edit invalidates that evidence — so the agent cannot run the tests, tweak the code, then commit. The fingerprint won't match. Documentation-only commits pass untouched. An optional layer also checks that the changed lines were actually exercised by a test, catching AI-written tests that pass without testing anything.
What makes it credible is how it was validated. It was not trusted from unit tests. It was driven end-to-end through a real headless agent session against a scratch repository — and the first run exposed a genuine bug: the evidence recorder was reading a data field the platform doesn't actually provide, so nothing was ever recorded and the gate blocked forever. That was found, fixed, and re-run to confirm. The result is simple: the agent can no longer claim "tests pass" on trust. It is a machine-checked fact, or the commit does not happen.
Multi-step features are not sent straight to code. A planning framework turns a request into a structured plan, dispatches parallel test-driven "worker" agents per task, and runs reviewer agents — including a deliberate "devil's advocate" that attacks the plan before any code is written.
Layered on top is a written model-tiering policy — "cheap writers, dynamic skeptics":
Review depth is dialled by switching the session model, with no config edits.
There are two deliberately separate tiers.
The first is a small set of flat-file facts that auto-load every session — the things that must never depend on a search working, like billing preferences or which branch is live on each project.
The second is a self-hosted temporal knowledge graph (a Neo4j database), shared across projects and fed by pipelines that ingest tickets, emails, and past session transcripts. Facts carry validity intervals, so a superseded decision stays queryable as history instead of being overwritten.
Cost is managed deliberately. Extraction runs on a cheap, fast cloud model rather than a premium one — a choice made after an early bill spiked. A local-model experiment to cut cost further was tried and then abandoned: running inference on the host kept destabilising the machine. So the settled position is a low-cost cloud model — cheaper than premium, without the fragility of self-hosting.
There is even a written rule that the agent must exhaust several search strategies before it may claim "that isn't in the graph" — because the real failure mode was bad searching, not missing data.
Two tools give the agent facts instead of guesses. A structural code index resolves the parts of Magento that plain text search cannot — dependency-injection wiring, plugin chains, event observers — and answers "what calls this / what would this change break." And the agent has access to a real PHP debugger: it can set breakpoints, step through execution, and inspect live values, paired with the Layer-1 rule (and Layer-2 guard) that forbids the lazy print-statement alternative. The agent answers "what is this variable at line 214, and why" by measuring, not reasoning.
Beyond unit tests, the agent runs end-to-end browser tests and can drive and screenshot the actual running store to confirm a change behaves, not just compiles. A hard-won detail sits here: the end-to-end tests are run in a targeted, per-file way rather than all-at-once, because running the full suite in parallel across multiple worker agents collides on shared resources (one database, one cache). That is the kind of scar most published setups have not hit, because they do not run several agents against shared infrastructure at once.
This is the layer that most sets the operation apart, and the one almost no individual setup has at all. Three independent mechanisms answer the question "is any of this actually working?"
All three feed a single self-refreshing dashboard the operator opens in a browser — status tiles, guard-fire trends, the eval pass/fail strip, per-project consolidation status, and a recent-alerts log.
A safety system is only as good as its ability to get someone's attention. Every actionable alert — a job failing, a policy drifting, a self-check dying — is delivered three ways at once:
Passive weekly summaries stay in a quieter channel, so the popups keep meaning something.
The agent tool changes behaviour between releases in ways that can silently break tuned configuration. This setup has been bitten by exactly that more than once — a new version quietly changing how configuration loads. So the version is pinned across the fleet, with auto-update off.
But a pin nobody revisits just fossilises. So the pin is modelled as a dated decision in a tracked file, and a daily check escalates when it goes stale — by age, or by falling too far behind the latest release. Moving the pin is gated: the rule-eval suite must pass against the new version first. That way the fleet trails the latest release deliberately, for stability, rather than by neglect.
Two cross-cutting pieces hold the above together. A component registry classifies every tool as either a client-facing "load-bearing" dependency (which must pass an operations checklist: wired everywhere, version-aligned, tested, with a documented kill switch) or a free-to-churn "experiment" — an explicit, enforced line between the two. And all the scheduled self-checks run behind a single dispatcher command with a built-in health view: each job records a heartbeat, and a job that stops running turns red on the dashboard. That last detail matters more than it sounds, and the findings below return to it.
4 · Reference implementation
Because the operator built a component registry and a reference standard, "done" is not a matter of opinion. A project either matches the reference on each dimension or it does not, and the drift sweep reports which. The plumbing-supplies store is that reference. Verified on the date of this review, it had:
All three in-scope projects are currently at this standard, on the same verified tool version.
5 · Critical findings
The strengths above are genuine and independently verified. Being critical means being precise about what remains fragile — and these are the honest weak points, ranked by how much they threaten the goal of safe, billable, semi-autonomous agent work.
This is the fundamental risk, and it does not go away. The setup is many custom scripts, several self-built plugins, a self-hosted database, and a dozen scheduled jobs — nearly all hand-written, all maintained by one person alongside the actual billable work. The sophistication that makes it effective is exactly what makes it fragile. A great deal of it only one human understands and can repair, and a serious illness or a busy quarter is a real continuity risk.
The setup contains this well. The registry documents what is load-bearing, and the health-heartbeat design means a component that silently stops now turns red on a dashboard instead of failing unnoticed. But "visible when it breaks" is mitigation, not elimination. The underlying arithmetic is unchanged: capability and bespoke surface rose together, and the bus factor is one.
The measurement layer is a major strength, but part of it has a soft core worth naming, so its greener readings aren't over-trusted. Two things rest partly on faith. Several of the rule evals have one AI model grade another's output (was this investigation rigorous? was this response terse enough?). And one telemetry metric counts banned phrases with a keyword match that the operator has honestly flagged as producing false positives — a rule being quoted trips the same counter as the rule being violated.
These are reasonable first-version choices, and they are documented as approximate. But it means a subset of the dashboard's numbers are judged by components that have not themselves been checked against hand-labelled truth. It is a smaller, more sophisticated echo of the very "is this actually working?" gap the measurement layer was built to close — one level up.
The alert delivery is well-designed, but it feeds into the one bottleneck no amount of alerting can fix: a single operator's attention. The system can now generate escalations faster than one person can action them. And the primary live channel — the desktop popup — only works when the operator is at the machine.
The email backup and the findable on-dashboard log are the right mitigations for absence, and separating urgent alerts from passive digests is sound. But the residual truth stands: a safety system that escalates to exactly one person inherits that person's availability as a single point of failure. By design, there is no second human in the loop.
Pinning the agent version is the right call given the demonstrated breakage, but it carries an unavoidable cost the operator has accepted openly. The fleet permanently trails the latest release. So it does not automatically receive upstream fixes, new features, or — the part that matters most — security patches, until a deliberate, eval-gated move is made.
The tooling to make that move safely now exists: the release watch, the eval gate, the dated-decision file. That is exactly right. But the standing state is a fleet running a version chosen for stability over currency, and the discipline to re-evaluate rather than let the gap widen is now a recurring obligation, not a solved problem.
Setting praise aside, here is only what is genuinely uncommon in individual practice:
But the most durable asset is not any of those tools. It is a habit. The operator instruments the work, tests their own rules, treats "it passed synthetically" as a reason for suspicion rather than confidence, and writes down their own wrong turns in the open. That habit is what makes everything else trustworthy — and it is the rarest thing in the whole review.
6 · Comparison
| Capability | This setup | Common individual practice |
|---|---|---|
| Invariants | Deterministic hooks the model can't bypass | A sentence in a prompt file |
| Test enforcement | Machine-checked gate, live-fired | "Please run the tests" |
| Rule efficacy | Executable eval suite, scheduled | Assumed, never checked |
| Memory | Flat-file + self-hosted temporal graph | None beyond an instructions file |
| Debugging | Real step-debugger, guesswork forbidden | Agent inserts print statements |
| Self-measurement | Three layers + dashboard | Absent |
| Alerting | Desktop + email + findable log | A log nobody reads |
| Agent version control | Dated pin, eval-gated moves | Auto-update or pin-and-forget |
| Owned by | One person (well-documented) | One person |
The last row is the point of tension the whole review circles: nearly every cell is far ahead of common practice, and the final one is identical — which is precisely why the top finding is about the maintenance surface, not the capability.
7 · Recommendations
8 · The journey
This review did not sit in a document. Over a single working session, every one of its recommendations was built — and the building turned out to be a better demonstration of the setup's character than the finished architecture is. Because the same discipline the tooling imposes on the AI agent, the operator turned back onto the tooling itself: build a control, then refuse to trust it until it had been fired for real. Every time, that live-firing caught something a synthetic test had waved through.
The test gate passed a 40-case synthetic suite, then failed its first real run — the evidence recorder was reading a data field the platform doesn't actually provide, so no evidence was ever recorded and the gate blocked forever. Caught in minutes, because it was driven end-to-end rather than trusted.
The rule-eval suite caught a bug in itself on its first full run — its own test fixtures were quietly invalidating the evidence they depended on. The self-check needed checking, and the act of running it surfaced that.
The nightly database backup ran flawlessly and produced valid dumps — that were being written to the same physical disk they were meant to protect, and whose restore had never once been attempted. The first restore test failed on a file-permission detail that would have cost hours in a real crisis; the second, with that fixed, brought back every record intact. A backup nobody has restored is a hope, not a backup — and that was only discovered by trying.
None of these were caught by reasoning about the code. All were caught by exercising the thing against reality and watching what it actually did. That is the entire thesis of the setup, demonstrated on the setup: "it passed synthetically" is where the investigation begins, not where it ends.
A review that finds gaps is common. A review whose every finding was then closed, live-fired, and whose own surprises were written down in the open — that is rarer, and it is the same habit that makes the tooling trustworthy in the first place. The artefact is the tooling; the journey is the proof it is used the way it asks the agent to work.
9 · Technical appendix
For the technically inclined: the specific tools and how they are wired. Names are given where they are open-source and useful to a peer building something similar.
Multi-step work runs through HCF (an open-source Claude Code plugin, markshust/hcf), which is not just "make a plan and code it" — it is a pipeline with agents enrolled at named phases, each declaring its slot via frontmatter (phase, order, mode). The phases fire in sequence around the actual implementation:
pre-flight-check validates readiness; pre-plan-graphiti-recall pulls prior decisions and incidents from the knowledge graph before a plan is drafted, so the plan is informed by history.devil's-advocate agent attacks the plan for gaps and blind spots and auto-applies critical fixes; post-plan-manual-test-plan generates a human test plan for the feature.pre-implementation-incident-recall surfaces relevant past incidents right before code is written.codegraph-reviewer runs structural impact analysis on the diff, graphiti-reviewer checks it against recorded domain knowledge, and the security quorum (below) runs.pre-commit-adversarial-pass (order 10) does a final hostile read of the change before it is allowed to commit.post-commit-build-summary and post-commit-verify-handoff record what shipped and hand off to verification.The effect is that every non-trivial change is read by several independent agents with different mandates — impact, domain-consistency, security, and pure adversarial critique — before it is committed, not after.
Security review is not a single pass. A security-quorum coordinator runs three independent sub-agents, each with a deliberately different mandate so they catch different failure classes:
security-static-analyst — reads the code for classic vulnerability patterns (injection, unsafe deserialization, access-control gaps).security-adversarial-tester — thinks like an attacker: how would you abuse this change?security-defensive-auditor — thinks like a defender: what protection is missing?Requiring agreement across three different perspectives is the same "diverse verifiers beat redundant ones" principle applied to security specifically — a single reviewer, human or model, has one blind spot; three with different mandates have less overlap in theirs.
Four tools, each doing a distinct job, run over the code:
The agent's pre-commit audit hook runs phpcs (Magento standard for module code, PSR-12 elsewhere) and PHPStan against only the staged files, plus xmllint on staged XML — fast, and scoped so an unrelated file's pre-existing issues don't block the commit.
There are actually two separate hook systems guarding git, and they are complementary. The agent-level hooks (described earlier in this review) live in the Claude Code layer and stop the AI. Underneath sits a project-level layer via captainhook, which fires for any committer — human or agent — installed per project:
run-unit-gate.sh) and a branch guard (pre-push-check.sh) that blocks pushes to deployment branches.composer install, a module security scan (n98-magerun2 dev:module:security), a dev build, and guards that ensure the live branch has been merged in before work continues.The belt-and-braces design is deliberate: the agent hooks catch the AI early with helpful messages; the captainhook layer is the hard floor that holds even if the agent layer is bypassed or a human commits directly.
Rather than one "search," the agent has three different sources of truth, each answering a question the others can't:
Static structure, runtime reality, and human history — three orthogonal ground-truths, so the agent rarely has to guess.
PHPUnit (v10) with Allure reporting for unit and integration; Playwright for end-to-end browser flows, run per-file to avoid parallel-worker resource collisions. The machine-checked test gate (Layer 3) enforces that these actually ran green before a commit, and its optional changed-line-coverage check catches tests that pass without exercising the change. The documented next step for hollow-test detection is Infection (mutation testing) — proving the assertions would actually notice a regression, not merely that the lines executed.
Every claim about the setup's state — the reference-project verification, the version alignment, the eval results, the telemetry figures — was checked against the live system on 2 August 2026 rather than reported second-hand. Client names and project identifiers are generalised. A companion setup guide, the four open-source plugins plus the patterns to reproduce, is at gist.github.com/ProxiBlue. The industry comparisons below informed the "what others do" notes throughout.
The review covers each layer in a paragraph. These posts cover them properly:
I use my tooling predominantly on Mage-OS (Adobe Commerce / Magento) e-commerce projects, and my own AI Booking Agent. If you have not swapped to Mage-OS yet, you are falling behind ;)