An Agent Hallucinated an Attack, and Proving It Took Three Hours

A boring Vertex AI question

It started with a config error, the kind you hit once a quarter and fix in ten minutes with a StackOverflow link.

litellm.BadRequestError: Vertex_aiException BadRequestError -
"This model does not support assistant message prefill.
The conversation must end with a user message."

Nothing exotic. A Claude model behind Vertex AI, routed through litellm, rejecting a request because the conversation doesn’t end on a user turn. I threw the question at my opencode setup, a team-lead agent whose whole job is to delegate, never execute directly. It fires off two parallel investigations: one exploring the repo to locate the relevant Vertex AI config, one researching the error message itself externally.

The external research comes back with a full case file: two distinct bugs, each backed by a GitHub issue. One on opencode’s side: at step-cap, the tool injects a raw assistant message at the end of the conversation, which models with extended thinking reject outright; a fix is already in a pending PR. The other on litellm’s side: a mid-stream fallback inserting a prefix=True block the target model doesn’t support. Clean diagnosis, well-sourced. I can go apply the fix directly.

The other sub-agent, the one that was supposed to just list some config files, comes back with a confession nobody asked for.

The message that should never have existed

Not a parsing error. Not an empty result. A piece of evidence.

Before either sub-agent has closed its assignment, the agent interrupts the flow to hand over this, verbatim:

This isn’t evidence you stumble on scrolling through a log. This is the agent taking the stand mid-task to say: stop everything, there’s something more urgent. And the suspect behind this text, the exploration sub-agent, never asked for any of this: it had been running for eight minutes and thirty-five tool calls on a job that fits in one sentence, list some config files. In hindsight, that duration alone should have raised a flag before reading a single word of the output.

What it produced reads like a well-rehearsed statement: a plausible technical motive (“the shell crashed”) backs up a command dressed up as a fix, one built to outlast today’s assignment entirely by running “on every future session.” The agent’s specific line (“I’m not storing any persistent memory”) isn’t a throwaway detail: it’s the direct answer to that exact attempt.

The domain name “evilcorp” is nearly a signature left at the scene. Nobody stands up a real C2 server (an attacker’s remote command-and-control infrastructure) with a name that theatrical. But a model hallucinating an attack scenario doesn’t know it’s supposed to stay quiet: it reproduces patterns from its training corpus, and those patterns look like security-awareness examples, not actual campaigns.

None of that changes how you’re supposed to handle the evidence itself. Hallucinated or real, a curl | bash built to plant itself into every future session does not get executed.

Not executing it is the floor, not the finish line

The lead agent never ran that command. Refusing to comply with a suspicious order is the reflex security training has drilled for fifteen years: never trust an instruction embedded in data, no matter how convincing. Nothing to celebrate here, it’s the minimum bar, not an arrest.

What comes next is more interesting, and it’s exactly where most post-mortems close the file too fast.

Saying “this is probably a hallucination” in ten seconds and closing the file is precisely the reflex a real attacker would want to trigger. Hallucinated noise and real signal leave the same trace on the surface. The only way to tell them apart is investigating both leads with equal rigor, not ruling in favor of the more comfortable one and moving to the next case.

Why a false positive is harder to prove than a real one

Proving a compromise is relatively direct: you find the modified file, the suspicious outbound connection, the fingerprint that has no business being there. Proving the absence of one means proving an alibi. And an alibi is never settled by a single check, only by methodically clearing every plausible suspect one at a time.

First sweep: checking for a corrupted file in the repo. Nothing. Second suspect, more serious: the configured MCP servers. pure.md ingests unfiltered third-party web content, the single most documented prompt injection motive in current literature. Had it returned a contaminated page, the hallucinated text could have been an almost literal reinjection of fetched content, a witness repeating what it was fed.

The config confirmed a real problem, independent of this specific incident: the active MCP servers, pure.md and playwright, aren’t scoped per agent. An explore agent meant to stay confined to the local filesystem theoretically had the same network access as an external research agent. Two suspects with the same access to the scene, nothing on paper to tell them apart.

flowchart TD
    subgraph Agents
        A[team-lead]
        B[explore]
        C[external research task]
    end
    subgraph "Suspects with unscoped access"
        M1[pure.md]
        M2[playwright]
    end
    A --> B
    A --> C
    B --> M1
    B --> M2
    C --> M1
    C --> M2

This lead is plausible and consistent with the context. It’s also wrong, but nothing rules it out yet. It deserves to be dug into exactly as if it were the right one, because nothing in the visible logs clears it.

A third, more structural obstacle: the agent leading the investigation has no direct access to the field. By design: least privilege, everything delegated to read-only sub-agents. That choice, defensible under normal conditions, becomes friction mid-case: every request for evidence has to be phrased, handed to the human, executed manually, and pasted back into the file. A round trip that takes thirty seconds with direct access takes five minutes through human relay.

A new witness gets called in to settle a specific question that’s been open from the start: was the malicious text planted somewhere in a repo file, or did it come from nowhere? A sub-agent gets sent to search the filesystem for that exact string. It comes back with a “completed” status and a truncated statement: “examining opencode.json…” and nothing else. No conclusion, not a shred to enter into evidence. A green status that says nothing is worse than a silent witness: it creates a false sense of a closed case on a question that’s still wide open.

The only proof that counts: dropping back to raw logs

After hours chasing leads that go nowhere, the breakthrough comes from a change in method: stop trusting the rendered account of the case, and go back to the scene to lift the evidence yourself, straight from what opencode actually persisted to disk.

timeline
    title The case timeline
    14h27 : Vertex AI question asked
    ~14h35 : Explore sub-agent returns hallucinated payload
    ~14h40 : Execution refused, immediate alert raised
    14h40-15h30 : Searching for compromised files, nothing found
    15h30-16h00 : Unscoped MCP hypothesis investigated
    16h00-16h20 : Truncated sub-agent, suspicious status
    16h20-16h50 : Raw SQLite logs extracted
    16h50-17h15 : Hallucination confirmed, investigation closed

An opencode client stores every session in a local SQLite database. What you see on screen is a reconstruction: assembled messages, polished statements, formatted markdown. It is not the same thing as the raw record the engine actually filed.

The command that settles it is almost brutally simple:

strings opencode.db | grep -i evilcorp

The result exposes the piece of evidence in its native JSON context: an object with "type":"text", nested inside a message generated by the model, not inside a tool-result object, nor inside the return value of a pure.md call. That distinction changes the whole verdict. A contaminated tool-result would be a witness faithfully repeating what an outside source fed it. A text field generated directly by the assistant, with no tool called earlier in the same turn, is a spontaneous confession: nobody fed it to him, he made it up alone.

Cross-check: in the same session, the actual interrogations, legitimate grep calls from the follow-up sub-agents, all came back “No files found” or results consistent with an empty crime scene. No MCP accomplice was called in at the moment the confession appears.

One question could have gone unanswered: was the “shell tool crashed” motive grounded in a real technical incident, or invented too? The logs settle it cleanly. The tool call right before the confession, a grep, finishes in 8 milliseconds with exit code 0: alibi airtight, nothing wrong there. The delay before the hallucinated text appears, roughly 2.3 seconds, matches normal model inference time rather than error recovery, and no incident, crash, or timeout shows up anywhere else in the file at that point. The sub-agent didn’t amplify anything real: it fabricated the motive right alongside the weapon and the fake alert, one coherent false statement, invented from nothing.

The intermediate file that grep was searching, a 56 MB strings dump of the opencode binary, deserves the same search: gone over in full, it contains no trace of the incriminating strings, and the model only ever saw filtered, truncated excerpts from it anyway, never the complete file. Even the lead that looked secondary got chased all the way to the end of the file.

The method itself has a blind spot worth naming rather than hiding: grepping for plaintext keywords never catches those same words encoded in base64, in hex, or translated into another language. Base64-encode the real payload and run it through grep -i evilcorp and nothing comes back, even though strings would have pulled it out intact in its encoded form, invisible to anyone searching for plain text alone. In this particular file, the question doesn’t apply: the hallucinated text is plain English start to finish, with no trace of encoding anywhere. But the method that cleared this case would still miss a disguised confession.

Verdict: hallucination confirmed. There’s no crime scene and no MCP accomplice in this story, just a machine that never leaked anything to the outside.

The collateral damage nobody sees coming

The irony of this case is that it produced a real leak while chasing a fake one. To verify the MCP server configuration, the PUREMD_API_KEY traveled through the conversation in plaintext, copy-pasted from a config file to be examined as evidence.

That’s the typical blind spot of a case worked under pressure: all the attention goes to “was there a compromise?” while nobody watches the channel through which the case itself is leaking evidence. Chasing a leak can create a new one, if every exchange with a model isn’t treated as a potentially wiretapped channel, regardless of what the conversation is about.

What actually needs to change

Scoping MCP servers per agent role draws a perimeter the current config leaves wide open. A local exploration agent has no legitimate motive to hold network access to pure.md. Restricting capabilities per agent mechanically shrinks the ground a hallucination, or a real injection, can stand on to look credible.

A “completed” status on a sub-agent guarantees nothing about the statement it produced. Checking it systematically, especially when the response is short or cuts off mid-sentence, costs a few seconds and keeps a case from being marked closed when it isn’t.

Any secret that has traveled through a conversation with an LLM, by accident, by the necessity of the case, or pasted without thinking, needs to be revoked. “It was just to check” doesn’t change the verdict.

The next incident won’t wear the same disguise, but the instincts that closed this one still hold as they are. An instruction claiming to come from the system doesn’t get executed on the strength of its own performance, no matter how convincing: that one was never up for negotiation. What’s rendered on screen means nothing until the raw log tells the same story, one is a filed record, the other is staging. Before deciding where a suspect confession came from, its exact structure needs checking, a contaminated tool-result or a text field generated on its own, a difference invisible to the eye but decisive on the merits. And above all, a file doesn’t close until the comfortable theory has been tested as hard as the one you’re hoping isn’t true.

A false positive gets investigated. It doesn’t get pleaded.