Engineering Notes · agent operations · security discipline

The Near-Miss That Banned Summaries

Lucas van Staden · ProxiBlue · September 2026 · proxiblue.com.au

How I nearly shipped past a critical security advisory because my AI agent read a summary of a page instead of the page, and what got banned fleet-wide the same day.

The incident

Routine job: a vendor publishes a security advisory page covering dozens of their Magento modules, and I ask the agent on a client project to scan the page and check whether any of the client's installed modules are affected. The agent fetched the page, searched it, reported back: nothing critical for us.

Wrong. Two High-severity lines for a module we ran, plus a related url-rewrite issue, sat in a later chunk of the advisory. The agent had not read the page. It had read preview windows: the summarised, chunked view its page-fetch tooling returns by default, and searched inside those. The chunks it was shown did not include the lines that mattered. When I pushed back, it fetched more, and there they were. High-severity, installed, missed.

Task failed. Caught only because it didn't survive contact with human review: I knew this client's stack and what that module was actually doing in production, and a clean report didn't square with that domain knowledge. That mismatch — what I knew about the client's own install versus what the agent confidently reported — is what superseded the finding and sent me back in to investigate why the scan had gone so wrong in the first place. The uncomfortable generalisation landed immediately: this was not a security-scan problem, this was every fetch. Research, vendor comparisons, changelog reviews: any task where the agent "reads a page" was actually the agent reading an excerpt it did not choose, with silent gaps it could not see.

Under the hood: what a "fetch" actually returns

This is the part to carry across, because once you see the mechanism you can fix it in your own setup. When a Claude Code agent uses its built-in fetch tool (WebFetch), the main agent never receives the page. This is not reverse-engineering on my part; it is documented behaviour. From the official Claude Code tools documentation:

"It fetches the page, converts the response to Markdown when the server returns HTML, and runs the prompt against the content using a small, fast model. For most fetches, Claude receives that model's answer, not the raw page."
"This makes WebFetch lossy by design. The extraction prompt determines what reaches Claude, so a result that says a page doesn't mention something may only mean the prompt didn't ask about it."

Read that second line again. "The page doesn't mention it" can officially mean "the extraction prompt didn't ask". So the agent's "I read the page and found nothing" decomposes into: a cheap model read a markdown conversion, answered a prompt, and did not mention the thing, and the main agent trusted that. Lossy steps, each invisible in the final answer. Related tooling has the same shape: search utilities that index a page into chunks and return preview windows around matches will happily let the agent grep the previews and mistake them for the document, which is the exact variant that bit me.

Credit where due: the same doc section suggests using curl via Bash for the unprocessed page, which is exactly the approved path my guard enforces. The design is sensible for token-cheap casual reading. The problem is that nothing about the answer tells you which pipeline produced it: a summarised read and a full read produce the same confident sentence, and on a security advisory that difference is the whole job.

What changed, same day

The summarising fetcher is banned at the tool level. A guard hook blocks the built-in page-fetch tool outright: not discouraged in prose, blocked. Blanket, fleet-wide, for every fetch, not just security pages. The block message tells the agent the approved path.
The approved path is raw and whole. Fetch with curl, read the full output, like a user with the page open. JavaScript-heavy or authenticated pages go through raw browser-tool text extraction instead. Either way: the whole page, never a preview.
Matching is deterministic where it counts. The re-done scan grepped the raw HTML for every installed vendor package name, mechanically. "Search what the tool surfaced" became "enumerate what we own against everything on the page". A completeness question deserves a completeness method, not semantic search.
The incident went into the knowledge graph, fleet-wide. Every project's planning agent now knows summarised fetches are a banned failure mode and why. One project's near-miss became every project's rule (the domain-graph article covers that machinery).

Why this failure mode is nasty

The failure is invisible from inside. The agent was not lying and not lazy: it searched what it was shown and reported honestly. The tool decided which chunks existed. Neither the agent nor I could see the gap in the answer, because the gap was in the input.

Clean answers get less scrutiny than messy ones. "Nothing critical for us" invites a nod and a merge. The failure mode optimises for exactly the answer you will not double-check. This one got caught on instinct; instinct is not a control.

Convenience defaults are calibrated for chat, not for stakes. Summarised fetching saves tokens and is perfectly fine for "what is this blog post about". Security advisories, licence terms, upgrade notes and changelogs are completeness-critical: every line skipped is a decision made blind. The default does not know the difference, so the tooling has to.

What it buys me

Full-page reads are structural, not requested. I do not have to remember to say "read the whole thing" on the tasks where it matters, which is precisely when I would forget.
Security scans became mechanical. Installed-package list against raw page, every name, every time. Boring, repeatable, auditable.
The lesson propagates. Guard plus graph fact means a new project inherits the discipline on day one, and the eval habit from the rulebook article means the guard's firing is provable, not assumed.

Want to do this yourself?

Find out what your agent's page-fetch actually returns, from its documentation, not its vibes. For Claude Code the answer is written down: fetch, markdown-convert, then a small fast model answers the prompt, "lossy by design". If your tool does anything similar, assume every "I read the page" is really "I read an extraction".
Blocking it is a few lines. In Claude Code, a PreToolUse hook matching the WebFetch tool that exits with a block code kills it deterministically, with your explanation as the block message pointing at the approved path. A permissions deny rule on the tool works too; I prefer the hook because it teaches the recovery path in-session.
Classify completeness-critical fetches: advisories, licences, upgrade notes, anything where a missed line is a wrong decision. For those, force raw-and-whole. I chose blanket enforcement because I did not trust myself to classify correctly under time pressure.
For any "are we affected" question, flip the direction: enumerate what you own and check each item against the raw source, deterministically.
Enforce with a hook, not a sentence, and record the incident where every future session will find it.
Treat a suspiciously clean answer on a high-stakes scan as a trigger to check the input, not the reasoning.

The near-miss cost nothing in the end: caught before anything shipped. What it bought was expensive-looking and cheap: one guard hook, one graph fact, one changed default, and a permanent downgrade of my trust in any answer whose input I have not seen whole.

hooks/webfetch-completeness-guard.sh — the PreToolUse hook that blocks WebFetch outright, fleet-wide
rules/reference/completeness-critical-fetch.md — the rule explaining when and why to force a raw, whole read
Claude Code tools documentation — WebFetch behaviour — the vendor's own description of the summarise-then-answer pipeline this post is about

I use my tooling predominantly on Mage-OS (Adobe Commerce / Magento) e-commerce projects, and my own AI Booking Agent. If you have not swapped to Mage-OS yet, you are falling behind ;)

More in this series: The Code Quality Checks Nobody Runs — coding standards, static analysis and comment hygiene moved into the commit path. · The ProxiBlue Debugger Discipline — blocking var_dump and wiring in real breakpoint debugging. · The ProxiBlue Domain Graph — giving my AI coding agents long-term domain knowledge with a temporal knowledge graph. · The ProxiBlue Falsifiable Rulebook — testing the rules and guard hooks that govern my AI coding agents.