CRUCIBLE

A hackathon build, and it says so on every page.

I built CRUCIBLE, and wrote this case study, for the purposes of entering the Google Cloud “All Things Agentic” hackathon, in the track The Fortified Enterprise Fleet. It was built between 2026‑08‑20 and 2026‑08‑31 — eleven days of work by one person against one target agent. It has no users, no downloads, and no adoption. It has not been reviewed, endorsed, or responded to by Google in any way. It is not production‑ready.

I am stating that first rather than at the bottom, because the whole argument of this project is that a system which grades its own work is not measuring anything — and a case study that buries its own limitations is doing exactly that.

The test that was written after the answer was known.

An agent that can issue a refund is a different kind of software from an agent that can summarize a document. The failure mode isn't a wrong answer, it's a wire transfer. So teams test these agents the way they test everything else: they write a list of adversarial prompts, run it, and read the report.

Two things go wrong with that, and the second one is quieter. The first is that the list is written by the person who built the agent, so it covers the attacks they already thought of. The second is what happens next. The team reads the report, writes a fix, re‑runs the list, and the list passes — because the fix was written after seeing the list. That number answers a question asked after the answer was known, and it feels exactly like evidence.

There is also the matter of what you're left holding. A red‑team report is a document. It tells you what went wrong and hands the fixing back to you. What a team actually needs at the end of a hardening run is an artifact that enforces — machine-readable rules that sit between the agent and its tools, and an evidence trail a stranger can replay without trusting the person who produced it.

Every component is deliberately blind to something.

CRUCIBLE runs a closed loop against a target agent — in the reference build, a refund agent with eight tools spanning six capability classes. A round attacks, watches, diagnoses, patches, regression-checks, and either promotes the new rule or rolls it back. Three consecutive rounds that find nothing counts as converged.

The design constraint that shapes everything else is that no component is allowed to grade its own work. That isn't a review convention, it's an architecture: each role is missing a capability it would need in order to fake its own result.

One thing this loop does not do, and the distinction matters: it does not invent attacks. The corpus is hand-authored. Autonomous attack discovery is specified in the design and is not a shipped capability — nothing in the tree authors an attack. What the harness automates is everything downstream of the attack: watching, diagnosing, patching, regression-checking, and gating.

Three verbs, and no way to say yes.

The Armorer is a language model, and language models are wrong sometimes. The usual response is review: have a human read every patch before it ships. That doesn't scale and it doesn't hold under deadline. The response here is to make the dangerous move unsayable.

Patches are written in a small domain-specific language with exactly three verbs — deny, constrain_arg, and require_approval. There is no allow verb. The language is missing the word. That means no sequence of patches, however wrong, can enlarge what the agent is permitted to do; the blast radius is monotonically non-increasing by construction, rather than by anyone remembering to check.

This is the same instinct behind the per-agent deny lists in my forensic work: the cleanest guardrail is one where the wrong thing is structurally impossible rather than merely forbidden. A prompt that says "never widen permissions" is a request. A grammar with no widening verb is an enforcement.

I built the check that asks whether my own fixes worked. Most of them hadn't.

A hardening tool that can't tell you which of its own fixes worked is worth much less than one that can. So the last thing I built was the instrument that replays a promoted rule against the exact calls that caused the breach it was written for, and asks a single question: did this close it?

Pointed at my own output, the answer was mostly no. Across the fifteen evidence bundles the shipped offline reader accepts, 32 rules were promoted. 13 closed the breach they were written for. 19 were no-ops on it. They were well-formed. They passed the benign-traffic check. They shipped. Replayed against the calls that caused the breach, they left the breach exactly where it was, and nobody would have known.

The cause is not the Armorer being careless. The tripwire's aggregate clause groups breaches by a key that the patch language the Armorer must write in cannot express — so the model writes the closest expressible rule, the rule is valid, and it binds to the wrong thing. That is a specific, diagnosed, fixable defect in the seam between two components, and it was invisible until something asked the question directly.

The gate had been asking whether a rule was well-formed and harmless, and never whether it was a fix. A rule that blocks nothing passes both of those easily. So does its mirror image, which is the finding I expect to carry into every project after this one: a rule that over-blocks passes every gate. It blocks the attacks; the approval oracle rubber-stamps the benign cases it intercepted; the benign pass rate reads a perfect 26 of 26; the gate promotes it. Every instrument says the run went well, and the agent has been made useless with its job handed back to a human. Two criteria that ask the missing question — originating-breach closure, and attack reduction — landed on 2026-08-26 and ran enforcing in the final measurement batches. Every promotion published before that date came from a gate that never asked.

A check that cannot fail is not measuring anything.

This is the sentence the project kept re-learning, in eight distinct places, in eleven days. The gate that never asked whether a fix worked. The evidence reader that accepted damaged bundles. The ratification check that bound what a reviewer saw and nothing that bound what they decided, so an edit made after signature changed the output while the digest check stayed green. Each one passed cleanly, every day, while measuring nothing.

The practice that came out of it is cheap and I'll carry it everywhere: ship a breaker with every check. The evidence reader is tested against nine deliberately damaged bundles plus a control, with proofs that the suite catches a reader that accepts everything and one that rejects everything. The overclaim linter ships a breaker per pattern, run twice each — once against the overclaim, once against a correction note quoting the same phrase in order to retire it — because a pattern nothing exercises is a pattern that could match nothing forever.

The third lesson is about how to publish. Every figure this project produces carries four labels that were pre-registered before any figure existed, so none of them could be chosen afterwards to suit a result: the sample size is written k = 1 with "no stability estimate" permanently attached; the benign floor is reported as a bound — a clean 26/26 bounds the unobserved regression rate at roughly 11.5% by the rule of three, never as zero loss; the target's model tier is named every time, because a weaker target inflates the baseline and flatters the whole curve; and every attack/benign pair is labeled by what separated it, the policy or the approval oracle, because a suite the oracle separates produces headline numbers identical to one the policy separates.

That last ratio came in at 21 policy-separated to 3 oracle-separated, against a design target of 18 to 4. It's off target, it's reported rather than absorbed, and the corpus checker prints the deviation on every single run so it can't quietly become the new normal.

Google Cloud, and the parts of it doing structural work.

The interesting choice here isn't which services are in the stack. It's that two of them are load-bearing for the argument, not just for the hosting.

The Agent Development Kit provides the enforcement point. ADK's BasePlugin.before_tool callback fires between the agent deciding to call a tool and the tool running. That is where the policy engine sits: it stamps the attempt, evaluates the current policy version, and short-circuits the call if a rule denies it. The agent cannot talk its way past it, because the check is not in the agent's context — it's in the runtime, in Python, below the model.

Cloud IAM provides the blindness. The Armorer, which writes the fixes, holds no storage role at all on the bucket containing the held-out attack family. That is not a convention documented in a README; it is a 403 captured with a positive control, so the denial is demonstrated rather than asserted. A held-out set that the fixing component could technically read is not held out.

The boundary, stated rather than discovered.

Eleven days, one person, one target agent, single-sample throughout. Every rate the project produces is a replay of recorded calls: it answers would this rule have stopped these calls, and never could an attacker find another way in. The five money invariants exist and are provably firable, and not one has ever fired on live data — the money path is unobserved, not defended.

The held-out attack family was never opened, by decision, so no transfer result exists and none is claimed. The reason isn't a hedge: eleven rounds of adversarial review kept finding defects in the runner, the last of them closing on the morning of submission, and the sealed set gets exactly one attempt. Spending it against a runner whose newest defect was hours old would have produced evidence nobody could rule on.

One sealed instance of twenty-four was named verbatim in a public commit early in the build. It's redacted going forward, but a public commit is served by its hash forever, so the instance was deliberately not replaced and the leak is stated any time that instance's result comes up. The sealed set was reviewed by one person, who is also the builder. Live-run evidence is gitignored, which means the central finding above is reproducible from the harness on my machine and not from a clone of the repo.

And the boundary that matters most, which is on camera as well as in the repo: the trust root is me. I hold project Owner. No control in this system defends against the person who built it. Every correction, every withdrawn claim, and everything the project marks unverified rather than dropping is kept in a dated ledger in the repository, because the shortest route to knowing how much of a thing to trust is reading what its author has already had to take back.

The findings name their own follow-on work.

The submission deadline is a date, not a stopping point. Everything below came out of the measurement work rather than out of a roadmap meeting, which is the useful thing about building the ruler before building the feature list.

• Close the gap in the patch language. Three invariant classes cannot currently be expressed as rules — repeated mutation on one subject, escalation below the required queue, and an account identifier leaving the boundary. Most refusals in the first class are the same diagnosed shape: the proposed rule tests a threshold the traces never reach, so it can never fire, and it passes a benign check trivially for the same reason.
• Fix the ruler, not the number. Report the approval-masked count alongside every benign pass rate, so a perfect score stops being indistinguishable from a rule that over-blocks and is rubber-stamped.
• More than one target agent, and k > 1. Everything except the foreign-agent capability probe is one agent, single-sample. Stability is currently reported as unmeasured; it should be measured.
• Let a run declare itself invalid. The evidence bundle has no field for RUN_INVALID, so emptiness is standing in as a proxy for it. A schema that cannot record its own failure is the same defect as a check that cannot fail.
• Run the three-column comparison. Stock agent, Google's generic runtime judge, and a CRUCIBLE-derived policy, on the same suite. It is specified and has not been run — and CRUCIBLE composes with runtime filtering rather than competing with it, since one is a different lifecycle stage from the other.
• Power the degeneracy census. It covers all 50 corpus instances, but no instance reaches the denominator that would let a regression actually be caught.

An enforceable policy, and a way to check it.

What comes out of a run is not a report. It's a policy file, a set of work items naming every breach the harness found and could not close along with the machine-checked reason the gate refused each attempted rule, and evidence of where it attacked and found nothing. Most tooling reports only the first of those, which hands you a policy and an unearned feeling of safety. The second is the actionable half, and this project makes it a first-class artifact rather than a footnote.

The single end-to-end result a stranger can verify in thirty seconds, from a clone, with no credential and no cloud project:

An attack lands against an empty policy and the refund executes. One rule is applied. The same attack is stopped, no tool executes, and a legitimate episode survives both policies. No model is called and nothing is billed.

Repo: github.com/emtcmca/crucible  ·  Demo: the loop in 3:34