There are hundreds of prompt-improver repos, and almost none of them can answer one question: how do you know it works? I built promptsmith so it could — eval-first, so the answer is a log file, not a claim. This is the pass where I pointed that harness at my own work and it caught me before it caught anything in the product.
The first case study made the argument for what promptsmith is. This one is about the part that came next and mattered more: before promoting a tool whose entire wedge is verifiable rigor, I had to prove the rigor was real to someone who wanted it not to be.
A prompt toolkit is easy to assert and expensive to verify. Anyone can ship a few commands and a README that says they work. The claim I wanted was narrower: promptsmith is built eval-first, and here is what the evals caught. A pitch like that has to survive an audit by someone hostile — so I ran three of them, in parallel, on my own build: one on eval coverage, one adversarial security review, one on the drift between what the docs promised and what the files did.
They found real defects. Three of them were the same shape, and that shape is the lesson.
The recurring finding wasn't a logic error. It was drift — a roster that grows fast outruns its own invariants, and the documentation keeps describing the guardrail after the code has stopped enforcing it.
| Boundary asserted, not built | The security doc named subagent output as a threat source — but only the request had a boundary in the coordinator. A slice could self-certify its own verification step. Closed. |
| Clause the agents never inherited | The engine carried an instruction-versus-data rule; all twenty gallery agents were dispatched without the engine and never got it. The verifier was even told to catch this failure in others while unprotected against it itself. Added to all twenty. |
| A rule two files disagreed on | One doc claimed project-local lenses couldn't shadow the security lens. The engine enforced that; the command that actually resolves lenses did not — and the two disagreed in the same run. Reconciled. |
Two more defects came from places a code review structurally cannot see. Every one of
the twenty gallery agents was missing its description field — the
one piece of frontmatter a host reads to auto-select an agent by task context. All twenty
rendered as the same anonymous label, so the marquee gallery, the thing the case study leads
with, could be invoked only by explicit name. Invisible in code review; visible in thirty
seconds from a live agent roster.
And the plugin used bare relative paths that resolved only by coincidence. It had only ever been dev-installed pointing at the repo root, so fifteen paths across four commands and two skills happened to sit in the right directory every time. From a clean install they missed. A dev-loop install is structurally incapable of surfacing that class of bug — you find it by installing the thing, not by pointing at it.
The same pass fixed the drift in the numbers the earlier case study still shows. The
verified current counts: 20 gallery agents (a bloated roster was split back
down), 12 expert lenses (the ai-tells lens was uncounted),
4 commands (grade shipped as a mode of /lens, not a fifth), and
a suite grown from 24 to 37 cases with 6 known-bad fixtures.
Reconciling
exactly these numbers against ls was part of the work — drifted counts are
the first thing a skeptic checks.
The audit exposed an asymmetry I hadn't named. promptsmith ran a rigorous measured-iteration
loop on itself — score, change one thing, re-score, keep only what didn't
regress — and users got none of it for their own prompts. So I put that loop in their
hands as /lens --grade — grade mode on the command that already runs the
lens pass, not a new command. (It shipped mid-pass as a standalone /grade,
then I merged it back the same window; that near-miss is in What I'd Do Differently.)
It returns a verdict, marks off the concerns a complete prompt resolves one by one, runs an
adversarial quality pass, and names the two or three fixes that raise the score most.
Three decisions in it matter more than the feature:
A prompt that resolves a concern in one fluent sentence passes. It is never docked for failing to look like promptsmith's own output. The question is whether the prompt works, not whether it imitates a house style.
A host-judged rubric does not support "73/100," and fake precision invites tracking a trend that isn't real. I would rather report "zero clean, three weak, eleven failing" and mean it.
Compare two versions and it flags what the new one broke — even when the new one is better overall. That is the whole reason to measure instead of eyeball: a one-shot rewrite hides the single deleted line that made the original safe, and nothing in the diff announces itself as a deletion.
It is the same score → change → re-score → keep loop promptsmith runs on itself — pointed at your prompts instead of at mine.
This is the part I want on the record, because it is the whole argument in one run. A harness whose judges can't say no isn't a harness — it's a rubber stamp. The test of this one was never whether it passed the product. It was whether it would fail me. It did, three times, before it caught anything downstream.
Producers are supposed to run blind: the agent writing an output never sees the scoring criteria, so it can't satisfy a must-not list without the behavior actually being right. Mid-run, two producer subagents flagged — unprompted — that their runs were contaminated: they'd been told to read one section of a case file, but the read tool returns the whole file, so they'd seen the answer key first. The exact bias the blind design exists to remove, self-reported before it could inflate a single result. I extracted all 37 inputs to standalone files and re-ran every affected case from scratch.
The run surfaced two places where promptsmith's own specs disagreed with each other — a lens that mandated emitting a rewrite while the command made rewriting conditional, and a minimality rule that lived in the wrong file and held only when the producer happened to be primed. Both were fixed the same session, and a later case confirmed the fix on the live path.
One comparison case was built wrong: the "worse" version was actually stronger, so a correct producer judged it the winner and the case never tested the thing it claimed to test. The fixture was wrong, not the implementation. I rebalanced it and left a calibration note in the case file.
And in the flagship orchestration run — one feature built by seven specialists plus one independent verifier — the verifier re-attacked the finished API and caught a HIGH data-exposure defect the builder's own safeguard had missed: it allow-listed database columns but still shipped the entire widget blob to anonymous viewers. The pipeline halted and escalated rather than synthesizing a vouched-for-but-unverified build. It refused to ship the hole.
A clean green run would have been a weaker story than the truth. The harness caught a defect in its own methodology and a defect in its own fixtures before it caught one in the product, and its judges refused to pass work — including mine. That is what a working eval loop looks like from the inside.
After the fixes landed, the full suite ran the way that counts: 37 cases plus 6 known-bad calibration fixtures, producers blind to the criteria, and the planted defects handed to judges without telling them the output was planted — because a judge told to expect a failure is not a calibration test.
The single WEAK was root-caused to a spec living in the wrong file and fixed the same session. The strongest signal is the calibration line: all 6 planted defects were correctly caught. Six judges received a defective artifact and a rubric, with no hint the output was planted and no criteria to game, and all six refused to pass it — one even finding a fabrication the fixture's own author hadn't documented.
This is a stronger version of the exact concern the earlier case study raised about itself: the evals are host-judged, and the model grades the model. Blind producers and undisclosed known-bad fixtures narrow that gap. They do not close it. There is still no external outcome data — and the honest claim stops there.
The domain is a prompt tool, but the discipline is the one I bring to production document intelligence. The system says "I don't know" before it says anything else: promptsmith never fabricates a fact, a citation, or an MCP server it can't verify, and an agent without a live tool declares the gap and degrades honestly instead of inventing a plausible answer — enforced across all twenty gallery agents.
Explainability travels with the answer. A confidence number without a
per-dimension breakdown is not auditable — so /lens --grade ships the
breakdown, plain /lens shows its findings worst-first, and every eval run is a
logged file anyone
can read, not a green checkmark to trust.
And build the version that breaks first, on purpose. The three adversarial audits, the known-bad fixtures, the blind producers, the independent verifier that halts synthesis — these are one discipline. You cannot get diagnostic information from a system that works. Only a system you are actively trying to break shows you its own architecture clearly enough to fix it.
The parent case study's lesson still stands: the evals are host-judged, and known-bad calibration proves the rubric catches the failures I thought to write down — not the ones I didn't. This pass added three more, each one a thing the audit caught that a better process would have caught earlier, or prevented outright.
The fifteen bundled paths that resolved only by coincidence, and the twenty agents missing
their description field, were both invisible to a dev-loop install and to code
review — and obvious thirty seconds into a clean one. I found them late because I only
ever pointed the plugin at its own repo. A clean-install smoke test belongs at the start, run
on every change the installer touches, not saved for the end.
Telling a producer to "read only the input section" is not blindness — the read tool returns the whole file, answer key included, and two subagents caught themselves reading it. Blindness had to be built into the harness's structure: inputs as standalone files the producer physically cannot see the criteria in. I'd design that in from the first case, not discover the gap mid-run and retrofit it.
Grade mode shipped first as its own command, /grade, then I merged it into
/lens --grade days later — a breaking change I only dodged because nothing
external had installed the tool yet. The functionality was right; the surface was one command
too wide. I'd settle whether a capability is a new command or a flag on an existing one
before it ships, not after the near-miss.
promptsmith shipped as v0.3.0 — tagged, published, and installable — after the last gate cleared: a clean-install re-verification, run once every file the installer touches had settled, because that is the exact test that caught the missing agent descriptions and the coincidental paths the first time.
There are still no users, no adoption numbers, and no traction to cite — and there won't be a claim of any until a real one exists. The tool is the artifact. The receipts are the story.