The question a board member always asks isn't "what does the document say." It's "how do I know you're right — and which document wins if they disagree?" Everything below is the answer to that question, built into the system instead of promised in a prompt.
HOA and condominium governing documents are not written to be understood. They are written to be legally defensible — dense, cross-referential, layered across multiple instrument types, amended across decades, and routinely contradictory once a newer rule brushes against an older declaration.
Every time a board member asks "can we do this?", the real question underneath is a chain of harder ones. What does the declaration say. What do the bylaws say. Do they agree. Are there amendments. Which version applies. And if two documents disagree, which one actually controls.
I managed this problem by hand for fourteen years across thirty-four associations. I know exactly how many board meetings go sideways because someone challenges an interpretation and nobody in the room can cite the section that settles it. The friction is expensive, it erodes board confidence, and it slows every governance decision it touches.
There is a second, quieter problem underneath the first. Every board runs on institutional memory that lives in the heads of whoever happens to be at the table. A manager leaves. A long-serving president terms out. The person who remembers why the short-term rental policy was amended in 2019, and what the board actually intended, is gone. The documents remain. The context doesn't.
What follows is not a feature tour. It is the set of architectural decisions that had to be right for any of it to be trustworthy, including the ones I got wrong first and the ones I measured, disproved, and reversed.
The main thread is written to be read straight through without a technical background. Every section carries an optional block underneath it holding the configuration values, the shape of the code, and the specific rules — there for anyone who wants to check the work rather than take it on faith.
A corpus — the full set of documents governing one community — includes scans from before the association was incorporated, handwritten amendments, faxed notices, sideways-scanned survey plats, and pages that have been photocopied enough times that the text is a suggestion. Extraction is where trust is won or lost, because everything downstream inherits whatever the parser decided a document said.
An earlier version of this page described a multi-stage OCR cascade. That was true once and
is not true now. MistralAI is the sole active extraction path. LlamaParse is
retired — every call returns HTTP 410, confirmed when 100% of a real-OCR corpus run fell
through it. Google Vision is an unimplemented TODO that was never wired. The orchestrator
function is still named extractTextWithFallback and its return type is literally
source: 'mistral' | 'failed', which is the honest shape of the thing: one path,
and a documented failure state that routes to manual review rather than guessing.
Extraction is scored before it is accepted. scoreExtractedText grades output
against a legal-text heuristic and gates at 0.6; real-corpus scores run
around 0.90. A failure writes parse_status: 'failed' and an ingestion log
recording exactly what was attempted, because a document that silently half-imported is worse
than one that visibly didn't.
Mistral occasionally classifies a page as a pure image and returns nothing for it — most often a survey plat scanned sideways. That failure is invisible to every other check in the system, because a skipped page has no heading number to go missing.
When a page comes back image-only, the pipeline extracts that single page, rotates it 90°, 180°, and 270°, and re-submits each through the same proven Mistral call rather than reaching for a second vendor. Bounded at ten pages per document, and any page that still fails is reported rather than dropped, so the completeness gate downstream can see it. The fix reuses the extraction path that already works instead of adding a dependency to handle an orientation problem.
Splitting a governing document into sections is where the domain gets unforgiving. The parser detects a document's heading convention from its first 20,000 characters — structured, chapter, numbered, or unknown — then builds a precise splitting pattern from the confirmed hierarchy level rather than guessing at one. Roman numerals normalize to Arabic. Section numbering is checked for sequential integrity so a gap is visible instead of silent.
Two failures in that path taught me more than the design did. In the first, a document's CRLF
line endings caused a $-anchored heading regex to fail on all eight of its ARTICLE
headings, collapsing an entire set of bylaws into one undifferentiated chunk — a document that
imported "successfully" and was structurally useless. In the second, a naive
.replace(/---/g, '') intended to strip horizontal rules was quietly eating the
separator rows out of markdown tables, destroying exhibit data. The fix wasn't a cleverer
regex; it was checking every --- occurrence across the real corpus and confirming
that all of them were table rows before narrowing the rule.
After extraction, the system compares the top-level heading numbers it can see in the source against the sections it actually produced. The obvious design is a coverage ratio with a threshold. I rejected it.
There is no such thing as an optional Article. A set of Articles of Incorporation that dropped one heading of seven scores 85.7% — comfortably above any sensible ratio cutoff, and missing a provision that could decide a dispute. So the gate is absolute: any missing top-level number fails, triggering a bounded re-extraction and, past that, a parse-status flag a human has to clear.
A second pass catches the subtler case — a section that exists for a heading number but whose body is just an echo of its own heading. That check needed its own fix after the naive version false-flagged six legitimate sections whose bodies genuinely were short. It now combines an absolute character floor with a ratio test, and the code documents the residual false-positive class it knowingly accepts.
Search here runs partly on embeddings — each section converted into a list of numbers that places it in space, so passages about similar things land near each other. Governing documents fight that, because they are saturated with the association's own name and its instrument vocabulary. Every section says "Maplewood Commons" and "Declaration" and "Covenants." That repetition dominates the dense vector and compresses everything into a narrow similarity band — measured at 0.348 to 0.571 across an entire corpus, which is not enough spread for semantic ranking to mean anything.
So the index-side de-noiser strips self-reference boilerplate, document-type title phrases,
and OCR debris like recording stamps and print-portal footers — from the embedding
input only. Stored body_text is never touched, because that is the
verbatim string a citation quotes. It also separates single-word boilerplate from multi-word
title phrases, so a content-capable word like "covenants" is stripped only when it forms part
of the document's own name rather than when it means something.
The pre-flight classifier runs before extraction, so the authority hierarchy is seeded at the moment of upload rather than retrofitted afterward.
Not all governing documents carry equal legal weight. A state statute outranks a declaration. A declaration outranks bylaws. Bylaws outrank rules and board resolutions. That ordering is the domain's central fact, and encoding it is the difference between a document search tool and a governance system.
| Rank | Tier | Retrieval bonus |
|---|---|---|
| 1 | State statute | +0.060 |
| 10 | Declaration / CC&Rs | +0.060 |
| 20 | Bylaws · Articles of Incorporation | +0.040 |
| 30 | Rules · policies · board motions | +0.020 |
| 99 | Other | 0 |
That last line is the whole design. The naive version of this feature ranks by authority and produces a system that confidently returns the declaration for a question the declaration never addresses. The coded rule is narrower and correct: the highest-authority document that actually covers the topic wins. Authority breaks ties among relevant candidates. It never promotes an irrelevant section over a relevant one.
This is the piece no general-purpose engineer would anticipate, and it took me a real governance background to see. An amendment to a declaration carries the declaration's authority — but it is stored as its own document, and the naive schema ranks it as a lesser instrument than the thing it modifies. That produces a system that prefers superseded language over current law.
An amendment is stored at rank 30 like any other secondary instrument. At retrieval time,
the system resolves its amends_document_id, fetches the parent's rank, and
scores the amendment at the parent's authority plus a +0.020
bonus — so amended text edges out the original it replaced. A missing parent logs a warning
and falls back to the amendment's own rank rather than failing the query.
Inheritance is a retrieval-time computation rather than a stored value on purpose: the lineage can be corrected, re-linked, or re-parented without a migration, and the stored record stays a faithful description of what the document is rather than what it currently outranks.
Because the authority bonus is applied by threshold rather than by exact match, an inherited rank lands in the parent's bonus band automatically. Orphaned amendments — those referencing a parent that isn't in the corpus — are surfaced to the board rather than silently mis-ranked, and a separate detector reads the corpus for references to amendments that were never uploaded at all.
A board asks whether they can put up a fence. Somewhere in eight hundred pages, one paragraph decides it. The job is finding that paragraph — and the trap is that the passage which sounds most like the question is frequently not the passage that governs it.
Most search systems rank by resemblance and stop. In a governance corpus that is the wrong answer, because the passage most similar to the question is frequently not the passage that legally controls it. A rule adopted in 2015 may describe parking in plainer, more answer-shaped language than the declaration — and lose to it in any court in the country. Similarity and authority are different axes, and retrieval has to carry both.
Every candidate scores as
semantic × 0.65 + keyword × 0.35 + authority_bonus + amendment_bonus,
capped at 1.0. Semantic similarity is how close two passages sit in that
number-space, measured by pgvector inside Postgres; keyword coverage is a straightforward
word-match pass over the same text.
The two bonus terms are where the domain lives, and both are small enough to lose to a real
relevance gap.
Keyword coverage carries a floor of 0.60 — but only for deciding whether a section is eligible to be considered at all. Ranking uses the raw coverage value. That distinction sounds pedantic and was responsible for a real precision failure: letting the floored value into the ranking math injects a flat constant into every candidate's score, compressing the field into a band roughly 0.10 wide that the actual semantic spread cannot separate. Definitions sections — which mention every term in the document — rode that band straight to rank #1. The floor gates inclusion. Raw coverage orders the results. Two jobs, two values.
A funnel narrows the candidate pool to six sections before the model is called. Left alone, hybrid ranking will happily fill all six slots from a single document — which makes cross-document conflict structurally invisible, because the model never sees the two sides. The funnel groups candidates by authority rank, identifies which tiers are genuinely relevant, reserves one slot per relevant tier, then fills what's left in hybrid order. It never pads to six with noise.
Architectural-control provisions are catch-alls: no exterior alteration or other structure or improvement without prior written approval. That language governs an EV charger, a satellite dish, or a pickleball court without naming any of them, which means it matches neither lexically nor semantically when someone asks about an EV charger. The question returns a false silence against a corpus that actually answers it.
The fix expands the query — both the embedded text and the keyword terms — toward architectural-control vocabulary, and expands nothing else. The ranking math is untouched. Lowering the relevance threshold was tried first and disproved: it added hallucinations and cost accuracy.
Signage, flag content, and political speech deliberately never trigger expansion,
so questions in that territory still return honest silence rather than being pulled toward a
catch-all that would look like an answer. The trigger regexes are word-bounded with the care
that implies — \bpool\b never matches "carpool," "court" fires only with a sport
qualifier so it never catches "courtesy," and flagpole is included as a structure
while a bare flag is excluded as speech.
And because expansion only ever changes the query, it can only surface a catch-all that already exists. It cannot manufacture one. That property is why the feature was safe to ship at all.
The most important thing this system can say is "your documents do not address this." Getting that right took three layers and one embarrassing bug.
A pre-model gate evaluates whether anything retrieved actually clears a relevance bar, and distinguishes off-topic questions from governance-adjacent ones the corpus simply doesn't cover — different situations that deserve different answers. That gate used to read the floored keyword score, which meant its "is there any keyword signal" test was always true, and silence could never fire whenever the keyword search returned any row at all. It now judges raw coverage against an explicit threshold.
Above it sits an insufficient-evidence rule in the prompt with unusually careful boundaries: an adjacent section is not coverage — an officer's duty to keep records is not an owner's right to inspect them — but a governing definition is, and so is an allocation or a stated mechanism. And above that, a precedence rule: a statute or preemption caveat may qualify a rule that exists, but it may never manufacture one that doesn't.
raw_keyword_coverage.Two documents say different things about the same subject. The obvious move is to decide which one outranks the other and declare the other void. That move, applied confidently, can tell a board its own voting method is invalid — and I shipped it before I understood why.
When two documents address the same subject differently, the obvious behavior is to rank them by authority and declare the loser overridden. I built that. It was wrong, and the way it was wrong is the most instructive thing in this project.
Consider a declaration stating that each unit shall have a single vote and bylaws stating that voting is on a percentage-interest basis. Ranked by authority, the declaration wins and the bylaws are "overridden and of no legal effect." That output is confident, well-cited, and would invalidate the association's actual voting method.
The two provisions do not conflict. One describes indivisibility — a unit gets one vote, not three. The other describes weight — how much that vote counts. They are answering different questions and they operate together. A system that resolves this by precedence produces a board that stops counting votes correctly.
The conflict rule now tests reconcilability first and only ranks by authority once genuine incompatibility is established. Two provisions that can both be true are both operative, and the answer explains how they fit together instead of declaring a winner. Only a real contradiction — where following one means violating the other — triggers precedence.
This shipped as a ratified architectural decision after an A/B — running both versions against the same questions and comparing the results — and the honest scope of that result is stated further down this page: the specific defect it fixed is confirmed stable across three runs with no safety regression, and the aggregate pass-rate movement around it is inside the noise band and is not claimed.
Getting hierarchy right is not one mechanism. It is a stack, and the language model is deliberately not the load-bearing part of it.
| Layer | Mechanism | Model? |
|---|---|---|
| Conflict precheck | detects multi-document coverage before generation | none |
| Conflict-tier guarantee | pulls one section per tier past the send threshold so both sides are visible | none |
| Resolution injection | controlling document computed in code, injected as established fact | none |
| Reconcile-before-override | the prompt rule, with the voting example as its worked case | prompt |
| Inversion detector | scans output for the controlling doc named within ±80 chars of "overridden" / "no legal effect"; one corrective retry, then human review | none |
| Post-generation net | sets the conflict flag programmatically via heading-keyword overlap when the model missed it | none |
The computation that matters — which document controls — happens in code before the model is
called, and enters the prompt as a fact rather than an instruction to be interpreted. Two
rounds of escalating prompt language taught me why: the more explicit the template
instruction, the more confidently the model misapplied it. A prompt is a request. A
MIN(rank) over an integer column is a guarantee.
Architecture Note 14 — prompts are the last resort, not the architecture →
The hardest design problem here was never extraction or ranking. It was trust. Specifically: how do you get a board member who has been skeptical of technology their entire adult life to act on an AI-generated answer?
The answer I arrived at was counterintuitive. Don't hide the uncertainty. Publish it. Show how the answer was built, how confident the system is in each element, and why. A layperson shouldn't need to understand retrieval to evaluate the output — they should be able to read a plain-language scorecard and make their own call.
Every answer carries two independent grades. Answer Confidence grades the answer itself. Corpus Readiness grades the document set it was drawn from. They are orthogonal on purpose: an answer can be strong against a corpus that is thin — meaning the system answered well from what it has, and what it has is incomplete. A single blended number smears those two facts into one misleading figure.
Splitting them lets a board read the real situation: this answer is solid, but you are missing three documents that could change it.
The scoring engine is not in this codebase. It is transparent-confidence, an open-source npm package I extracted and published under Apache-2.0 with zero runtime dependencies, consumed here at arm's length through a thin adapter. It scores eight dimensions — three always on, five opt-in — and renormalizes weights so the 0–100 score stays comparable regardless of which subset is active.
BoardPath scores four of them from real signal and explicitly declares the rest not-assessed. Not defaulted, not quietly averaged in at a neutral value — reported as unmeasured. A confidence system that invents a number for a dimension it cannot observe is doing the exact thing the whole feature exists to prevent, and the adapter would rather show a gap than fill it.
Extracting the scorer into a package was also the honest test of whether it was really domain-agnostic. It was; the governance-specific parts stayed behind, and what came out is general enough that the boundary is enforced by publication rather than by intention.
A theme runs through every part of this build, and it isn't specific to AI: when a rule matters, encode it somewhere it cannot be forgotten. Not a comment. Not a convention. Not an instruction in a prompt. A constraint that fails loudly when violated.
Enforcement is the part of community association management most likely to end up in front of a judge, and the question is always the same: can the association prove what it did and when. So the violations module has no language model in its spine at all.
Every event appends to a hash chain: each entry carries a fingerprint computed from the entry
before it, so altering any past record breaks every fingerprint after it. Tampering becomes
arithmetic rather than a matter of trust —
sha256(prev_hash ‖ 0x1F ‖ canonical(event) ‖ 0x1F ‖ timestamp) — computed
twice, in TypeScript and in SQL, and cross-checked byte-for-byte on insert.
Enforcement runs at three levels: UPDATE and DELETE are revoked from
every application role including the service key; a trigger raises unconditionally so even the
table owner cannot silently mutate a row; and a single append function holds a per-violation
advisory lock, reads the chain tip, and computes the next link. Two unique constraints make a
concurrent append lose rather than fork the chain.
The attorney fact-packet assembled from that chain is built with no model deciding what goes in it. Selective-enforcement exposure — the doctrine that sinks associations in court — is detected by comparing this violation's escalation timeline against every comparable one, which is a query, not a judgment. The value here is that the record is provably complete, and a probabilistic system cannot make that promise.
There are two answering engines in this product and they must never touch. The document engine answers from a specific association's corpus with authority ranking, hierarchy resolution, and citations. The advisor engine answers general community-management questions from a curated knowledge base with no association data in scope at all.
The separation isn't a flag. The advisor's retrieval module talks to exactly one search surface — a global knowledge-base table with no association column — and never calls the document search, never reads the documents tables, and never applies authority rank, hierarchy ranking, or the amendment bonus. The only thing it shares with the document engine is commodity plumbing: the same embedding model and the same database client. The code notes, correctly, that an embedding model is not an authority signal.
Every tenant table carries row-level security — a rule enforced by the database itself, so a query for another association's records comes back empty regardless of what the application asked for. It is keyed on a membership lookup rather than directly on a user id, through a security-definer function that exists specifically to prevent policy recursion on the membership table itself.
But the honest framing is the one the migration itself uses: RLS here is defense-in-depth, not the primary gate. The service key bypasses row-level security by design, so every API route is separately gated by an explicit membership check. RLS exists because the anonymous key ships in the browser and a policy is the last thing standing if application-layer auth is ever wrong.
Both of these were self-found, before any external user was exposed to them. I'm including them because a case study that only describes the parts that went well is a sales page, and because how a solo builder handles finding their own defect is more diagnostic than the defect.
A self-directed security pass turned up an API surface reachable without authentication, and — worse — a hardcoded service-role JWT sitting in code I had written. A service-role key bypasses row-level security entirely. That is the credential you least want in a repository.
The fix was not a patch on the exposed route. It was the tenancy model described above: row level security enabled across every tenant table with membership-scoped policies, the key rotated and moved, and — because RLS is bypassed by the service key by design — an explicit membership check added as the primary gate on every API route. The migration that shipped it documents its own residual watch-items rather than declaring the problem closed: one table with a deliberately public read policy that is enumerable, and two with RLS enabled but no policy, which denies everything.
Pre-launch, no customer data existed, and no user was ever exposed. I could have quietly fixed it. But the useful part isn't that a mistake happened — it's that the response was structural rather than local. The exposed route was a symptom of not having a real tenancy boundary. Patching the route would have left the actual defect in place and made me feel like I'd solved something.
The advisor engine answers from a curated knowledge base of community-association guidance. Much of that knowledge base was drafted with AI assistance, and before any of it could be used publicly I ran a full statutory-integrity sweep against it.
The sweep found roughly 129 incorrect legal claims across about 99 articles — fabricated statute citations, provisions stated backwards from what the law actually says, and references to rules that had been repealed or withdrawn. All four correction passes were written and merged.
That content read well. It was fluent, plausible, well-organized, and wrong often enough to be dangerous in a domain where being wrong about a statute is the whole risk. Nothing about the writing signaled which claims were invented. That is precisely the failure mode this entire product is built to prevent, and I found it in my own knowledge base.
The standing rule that came out of it: AI-authored legal content requires statute-level verification before any public or binding use. The corrected corpus went to a professional handoff of 41 legal-substance items for CMCA and attorney sign-off, and that review still gates the advisor engine's public availability. It has not been waived because the sweep went well.
Testing this is harder than testing ordinary software, because the failure mode does not look like a failure. There is no crash and no error message — there is a fluent, well-cited, confident paragraph that happens to be wrong. Someone has to be able to tell the difference, repeatedly, without reading all eight hundred pages themselves.
A governance answer that is wrong with confidence is worse than no product. So the testing apparatus is not a suite bolted on near launch — it is its own build, roughly the size of a small product, and it is the part of this project I would defend hardest in a technical interview.
Its guiding constraint: the language model is consulted last, and only for what genuinely requires judgment. Most verdicts in this harness are computed, not graded.
Every question carries an oracle — a first-class record of what a correct response must and must not do. Not a reference answer to fuzzy-match against, but a set of assertions: which of five behaviors is expected (answer, refuse, clarify, state-silent, flag-conflict), which citations must appear, which claims are forbidden, which caveats are required, and what the hierarchy resolution must be — controlling document, overridden documents, and the reason. Oracles carry authorship and human verification provenance as columns, because who checked this and when is part of the evidence.
Behavior and forbidden-claim detection are critical checks that drive a verdict to outright failure. Citation and hierarchy correctness can only downgrade a result to partial. That asymmetry is deliberate: answering a silent topic at all is a different category of wrong than citing the right rule by number instead of by name.
Contract validators apply deterministic hard-defect rules with no model in the loop: high confidence asserted on a silent topic, a hierarchy conflict flagged without a resolution note, counsel review recommended where nothing warrants it, a substantive answer given to an out-of-scope question. Oracle comparison matches each answer against its contract by structured check rather than by asking a model whether it looks good. Retrieval traces record whether the oracle's required section actually surfaced in the top 1, 3, 6, and 10 candidates — exposing ranking failures that answer text alone would hide. Only then does a nine-axis judgment evaluator grade what's left: correctness, grounding, citation precision, authority handling, silence handling, completeness, calibration, boundary behavior, and usefulness — each on its own rubric, never collapsed into a holistic score.
Character-span precision and recall are computed on LegalBench-RAG methodology against committed gold spans, with a self-validating check on the gold itself wired into the fixture suite. There is no MRR and no NDCG in this harness, and I don't claim either.
Questions are generated in three passes — corpus topology, then questions with draft oracles, then adversarial injection and forbidden-claim authoring. They are then handed to a separate auditor running a different model and a deliberately skeptical prompt, so the system that writes a test is never the system that approves it.
Before a question is allowed to assert that the corpus is silent on a topic, a deterministic pre-LLM step extracts the topic keywords and searches the actual document sections. If matching sections exist, the question is hard-rejected regardless of what the LLM auditor concluded.
That check exists because of a specific burn: three questions in a stress set were designed as silent-topic tests, and the corpus turned out to cover all three. Every result they produced was measuring nothing. A silent-topic test whose premise is false doesn't fail loudly — it passes, and quietly certifies the wrong behavior.
Answer keys are cleared through decorrelated dual review: two independent verifiers dispatched in parallel with no coordination, each pulling primary sources itself, each emitting a verdict per claim with its source and rationale. Both must agree and both must be grounded, or the item escalates to me.
The load-bearing clause is that dual review produces a verdict, never the edit. A verifier cannot flip a verification flag, change an oracle value, or touch a live seed. Alongside it sits the standing rule for the whole measuring apparatus: the lane that measures the product never edits the product. A failing eval is a routed finding with a lifecycle — not-yet-routed, routed, landed, verified or regressed — tracked in a cross-lane ledger. It is never a quiet weakening of the assertion that failed.
The measurements this program produces are less interesting than what it does when a measurement looks bad. Every one of these was a case where the obvious read was "the product regressed" and the actual answer was "the ruler is wrong."
| What looked wrong | What was actually wrong |
|---|---|
| Hierarchy handling failing 8 of 8 | A raw document-name substring test that failed any answer citing the controlling provision by section number instead of by name. The strict number was a measurement floor, not a gap. |
| Amendment citations scoring partial | Proven a scorer artifact against live citation data — the model cited the correct paragraph and the ruler under-credited it. Six partials flipped to pass; three were correctly left alone; three negative fixtures added to keep the fix from over-crediting. |
| Advisory accuracy at 46.2% | Oracle section hints that never resolved to real section ids, so correct answers landed as partial. Ground-truthed four "0%" topics, found all four substantively correct and cited, and published the number as a floor rather than a result. |
| A de-noise change regressing 93.3% → 86.7% | Two detector false-positives on byte-identical answer text. Not the flag under test. |
| A locked benchmark reporting 32 of 32 verified | A fixture-to-database parity diff found only 4 verified in the database, and one question's stored expectation stale. Refused to blind-flip the verification flags, because that would have stamped the wrong expected behavior as verified. |
| "Is the judge too harsh?" | Audited and ruled fair. Each persistent failure classified as generation, retrieval, or scorer artifact — and a prior finding of my own was disproved in the process and recorded as disproved. |
| A built, green retrieval improvement | Its A/B failed. Section-reached dropped 9 points. The branch was parked and the decision record marked rejected rather than shipped on intuition. |
| A top-priority hardening fix I had queued | See below. I disproved my own premise and refused to ship it. |
I had queued a hardening task at top priority, on the belief that a scoring component was a third language-model detector carrying a known non-determinism defect. The work was scoped, prioritized, and ready to build.
It was wrong. That component is a pure synchronous function with no model call, verified at the source before anything was merged. Wrapping it in majority voting would have called the same deterministic function three times, returned the same value three times, and cost three times as much to do it.
A green majority-vote wrapper would have installed a decoration of rigor over a component that was never the source of the problem — and hidden the real defect behind it. The red-team question this program asks of every eval is could this pass while the real behavior is broken? That fix would have made the answer yes.
What shipped instead was the opposite: the predicate extracted verbatim so fixtures import the real function rather than a drift-prone copy of it, then pinned by a 28-assertion fixture that re-scores 25 verdicts 25 times and asserts zero drift — plus a deliberate bite check that seeds randomness into the branch to confirm the suite actually fails when the property is violated. A determinism test that cannot fail is not a test.
A cross-encoder is a second, slower model that re-reads the question against each candidate section and re-orders them — a specialist brought in to fix the ordering after the fast search has done its work. This is the clearest thing the evaluation program ever bought me, and it cost a feature I had already shipped and was proud of.
An early retrieval benchmark on a twenty-question gold set said something specific: the right section was almost always in the candidate pool, and almost never first. Recall was fine. Rank was the failure.
Fixing document fragmentation — embedding heading and body together — pushed recall to ceiling and made rank worse, which is the kind of result worth stopping for. The diagnosis: the 0.60 keyword floor was injecting a flat constant into every candidate's score, compressing the field into a band about 0.10 wide that the roughly 0.16 semantic spread could not separate. Definitions sections, which mention every term in the document, rode that band to the top.
| Stage | Recall | In pool | Ranked #1 |
|---|---|---|---|
| Baseline | 75% | 90% | 30% |
| After de-fragmentation | 95% | 100% | 15% |
| After cross-encoder rerank | 100% | 100% | 75% |
Rank #1 went from 3 of 20 to 15 of 20 for about 400ms of added latency. Along the way the funnel had to stop fighting the reranker: it reserved the first slot for the highest-authority tier, which forced a statute definitions section scoring −7.4 ahead of the correct operative provision scoring 15.93. The fix was to let tier diversity govern membership while the reranker governs position. A separate provider A/B was thrown out entirely because a free-tier rate limit returned 429 on 19 of 20 calls and silently fell back — that number was recorded explicitly as the hybrid baseline, not as a result for the provider it appeared to measure.
On a real-OCR corpus, rerank OFF beat rerank ON. Pass rate 71.4% without it versus 57.1% with it. Controlling-section-reached 75.0% without versus 62.5% with.
The reranker had only ever been validated on the clean synthetic corpus. Real governing documents are scanned, OCR'd, and noisy in ways a purpose-built test corpus is not, and the cross-encoder's judgments degraded on exactly the material the product exists to handle.
So it is off by default in production. The module is preserved, fail-open, and dormant behind an opt-in flag for a future A/B — but no request takes that path unless the caller asks for it.
Turning it off was safe because the underlying defect had been fixed at the source in the meantime. The tie-band the reranker was hired to break doesn't exist anymore, because ranking now reads raw keyword coverage instead of the floored value. The reranker was compensating for a scoring bug. Fix the bug and the compensation becomes overhead that makes real-world results worse.
A 5× improvement in rank #1 is a great slide. It was also measured on the wrong corpus, and I would have shipped it permanently on the strength of that number if the evaluation program hadn't been built to re-measure on real data before trusting a win.
The module is still in the repository and still fail-open — no key, timeout, HTTP error, or malformed response returns null and the caller keeps hybrid order — and its own contract line says what it is: a ranking aid, never a gate. A component that can only improve an ordering and never block an answer is one you can safely leave in place after deciding not to use it.
Every product page in this category quotes an accuracy percentage. This one used to quote five. I took them down, because I could not answer the only question that matters about a number like that: if you ran it again tomorrow, would you get the same figure?
An earlier version of this page published five accuracy figures: answer accuracy on two corpora, a controlling-section rate, zero authority-rank errors, zero high-confidence hallucinations, and 100% prompt-injection resistance. They came off. The reasoning is more useful to a technical reader than the numbers were.
Not every measurement in this program has the same epistemic status, and the line between them is the same line that runs through the whole architecture.
| Measurement class | Reproducible? | Why |
|---|---|---|
| Retrieval evals — recall, in-pool, rank #1 | byte-identical | Run with the model out of the loop entirely. A repeat run returns the same bytes. |
| Controlling-section-reached | byte-identical | Set intersection on section ids. Verified identical across two independent A/Bs. |
| Contract validators | deterministic | Pure boolean and substring logic, no model call. |
| Answer accuracy, pass rates | variant | Three of the four inputs to the behavior matcher are model-emitted, so a re-run is a fresh generation, not a re-scoring. |
| Hallucination and injection counts | variant | Detector-based, majority-voted, and fail-open on error paths. |
| Judgment axes | variant | A language model grading language-model output. |
I originally attributed run-to-run instability to judge variance. That was wrong, and the correction is recorded in the decision record rather than quietly edited: the flips are generation variance. Because most of the matcher's inputs regenerate on every run, re-running a benchmark produces new answers to score, not a new score for the same answers. The practical consequence is that mitigation means repeating generations, not repeating scorings — which changes how every future A/B in this project has to be run.
A follow-up measurement refined it: on clear-cut cases the flag held stable 24 out of 24 draws. The wobble concentrates in genuinely borderline questions. That same record states plainly why the opposing arm of the experiment was not run — the measured arm was already at ceiling, so the comparison was arithmetically unavailable.
| Claim | Why it's gone |
|---|---|
| Answer accuracy percentages | Generation-variant, with observed bands wide enough that a single-run delta of one or two questions is inside the noise. My own decision record says not to bank these rates. |
| Zero authority-rank errors | The metric has no writer. The column is declared in the schema and read by the analyzer, and nothing in the codebase ever sets it. Every report printed 0.0% because the value is always null. It could not have been nonzero. |
| Zero high-confidence hallucinations | Both contributing detectors fail open — a timeout resolves to "not detected." A run where the detector was down is indistinguishable from a clean one, and one documented run's zeros were catch-block defaults rather than measurements. The fix is written and held pending review, because it changes what two safety gates persist. |
| 100% prompt-injection resistance | Two defects. The check is a substring tripwire on five literal phrases, so an injection that succeeds without emitting one of them scores as a pass. And the aggregate returns 1.0 when a run contains zero injection questions — against a gate requiring ≥ 1.0, meaning a run with no injection tests certifies perfectly. |
One number survives the standard, and it is the one that answers what's the variance on that with zero: controlling-section-reached — did the section that should decide the question actually reach the model. Retrieval-side, computed by set intersection over section ids, demonstrated byte-identical across two independent A/B runs.
On real-OCR corpora it runs in the neighborhood of 64%, and it belongs here as a ceiling rather than an accomplishment. No amount of generation quality can exceed it: if the controlling section never reaches the model, no answer-side skill recovers the answer. Retrieval is the binding constraint on this system today, and my own eval notes say so in those words. Quoting that number without naming the corpus would be misleading — clean synthetic and real scanned documents are different problems.
A number goes on this page when I can answer what is the variance on that without flinching. Everything else waits — not until it looks better, but until the measurement is reproducible enough to defend. When the current QA arc closes, the answer-side numbers return as a band across repeated generations rather than a point estimate, which is a stronger claim than the one I took down.
There is a version of this section that quotes 89% and moves on. It would be more impressive and less true, on a product whose entire proposition is that it tells you when it doesn't know.
All of the above exists to make one thing possible: a volunteer board answering its own governance questions without a management company. The application is organized into seven areas, and the shape of that navigation is itself a domain argument — it maps to the jobs a board has, not to the data model underneath.
| Surface | What it does | Status |
|---|---|---|
| The Boardroom | The answering workspace — question to cited answer with a confidence scorecard, plus meeting mode, motion recorder, transcript ingest, and inline drafting | shipped |
| Documents | Ingestion, classification, amendment linking, orphan detection, consolidated current-state view | shipped |
| Violations | Zero-LLM enforcement spine, hash-chained audit log, selective-enforcement detection, attorney fact packet | shipped |
| Meetings | Agendas, packets, minutes, and a multi-step annual-meeting wizard covering notice math, quorum, proxies, and balloting | shipped |
| Governance | Six tabs — conflicts, obligation calendar, amendments, topic briefs, corpus health, corporate standing | shipped |
| Closing | Estoppel and resale certificates with a per-state statutory fee-cap engine; board-side readiness and fulfillment | board side |
| History | Chronicle — decision log, point-in-time answers, amendment lineage, orientation reports | built, partly unwired |
Chronicle is the institutional-memory layer — the answer to why do we do it this way? It holds a searchable record of board decisions, full amendment lineage for any provision, point-in-time queries that retrieve from the corpus as it stood on a past date, an orientation report for newly seated members, and a governance health score across five dimensions.
It is built and available to every board, not held back as a separate product. It is also not fully wired: several of those surfaces exist, function, and hit real APIs, but do not yet have a navigation path to reach them. That's a wiring gap on the way to the full demo, and I'd rather describe it accurately than let a feature list imply a click-through that isn't there yet.
The Boardroom answers questions. The deeper problem in self-managed governance is that boards don't know which questions to ask, or when. The election notice deadline passes. The insurance certificate lapses. A 1994 provision quietly stops matching current state law.
Steward is the answer to that: a prioritized action stack assembled from obligations, missing documents, unresolved conflicts, and unbriefed meetings, reordered by the role of whoever is looking — a treasurer sees financial obligations first, a secretary sees meeting items first. When nothing is urgent it proposes the forward-looking work a well-run board does between crises. And it goes to the board rather than waiting: a digest runs on a Monday/Wednesday/Friday schedule and emails the same stack out, so a deadline gets found before it passes instead of after.
Steward maintains, boards approve. It proposes, drafts, and alerts. Every action that changes anything still requires explicit board approval. It is an advisor, never an actor — which is the same boundary the violations module draws, the same one the confidence scorecard draws, and the same one the hierarchy gate draws.
A separate general-advisor engine — the one firewalled from all association data — is built and API-complete but not yet surfaced in the interface. Its public availability is gated behind the professional review of the knowledge base described earlier, and that gate hasn't moved because the sweep went well.
A limited demo is live and public with no signup, running against a purpose-built corpus. The full demo lands early September, and founding boards are being onboarded now ahead of a public launch in the fall. There are no paying customers yet and no revenue claim on this page.
The product handles the governance workflow end to end: ingestion and classification, scored extraction with a completeness gate, authority-weighted hybrid retrieval, amendment lineage with inheritance, citation-grounded generation, deterministic conflict resolution, two-axis confidence, enforcement with a provable audit trail, correspondence, and institutional memory across board terms.
Retrieval is the binding constraint, and I know precisely where. A live probe on a real condo corpus traced a class of comparative governance-structure questions that fail because the keyword preprocessing strips the very nouns that carry the question's meaning — the words board, declarant, and declaration are boilerplate in a pets question and are the entire payload in a question about how a 2009 amendment changed board composition. A rule that is correct in the normal case and wrong on a legitimate edge class. The fix is scoped and measured; it is not merged, so it is not on this page as a win.
The QA arc that produced most of this page's measurement discipline is still running. Two safety-gate changes are written and held for review specifically because they alter what blocking gates persist. When that closes, the answer-side numbers come back as a defensible band.
Every architectural decision on this page bends toward one person: the board member who has been skeptical of technology their entire adult life and has been right to be. Not the one who trusts AI — the one who wants to verify it before acting. The two-axis confidence card, the citation chain, the deterministic hierarchy gate, the append-only audit log, the refusal to publish a number I can't reproduce — those are all the same design, applied at different layers.
Fourteen years of doing this work by hand is what told me which corners could not be cut. The tools to build it properly only arrived recently.
Architecture Note 04 — I spent 14 years solving this problem without the right tools →