Veridi

The problem

Fact-checking at scale breaks in predictable ways. Single-pass LLM verification often produces plausible-sounding but wrong answers. Retrieval-augmented approaches inherit source bias and struggle with adversarial prompts. And most verification pipelines are uncalibrated: there may be a confidence score, but being ungrounded, it doesn’t reliably reflect accuracy.

Veridi is our answer to those three failures. It’s an agentic fact-checking methodology and implementation pipeline, designed so verification is structured, auditable, and calibrated, rather than a single LLM call wearing a lab coat.

The pipeline

Seven agents (A1 through A7) run on FastAPI with the Anthropic SDK and a local SQLite store. Each has a defined role, a handoff contract with the next agent, and a documented failure mode. A two-tier model dispatch uses a cost-efficient workhorse (Sonnet 4.6) for primary processing and invokes a high-capability advisor (Opus 4.6) only on uncertain or high-risk claims. The tier registry is vendor-neutral, so escalation logic references tiers rather than model names; switching providers is a configuration change, not a refactor. Four verification depth tiers let a caller trade latency for rigor. Routing across nine domain specialists keeps each agent’s context focused. Twelve gaming countermeasure vectors block known manipulation patterns. Every output is Brier-scored, so stated confidence is measurable against observed accuracy.

What I’ve built

Architecture, engineering, and methodology, in roughly that order. The technical layer is a seven-agent FastAPI pipeline with explicit scaffold contracts and a cost model that accounts for API, storage, and compute per claim. The methodology layer is a ~21,000-line corpus across 59 documents spanning all three systems (Veridi, Pragma, Praxis): twenty-two domain specialists, a four-tier source hierarchy, thirty-two gaming countermeasure vectors, and the Brier calibration framework. The product specification sits at 1,968 lines in version 1.4, covering failure modes, cost, and validation protocol; iteration continues.

What this demonstrates

The product specification calls out seven applied-AI engineering skills evidenced by the build. Each has a verifiable home in the codebase and documentation:

Specification precision. Every agent has explicit input and output formats, hard constraints, and enumerated edge cases. Confidence calibration distinguishes tier ceiling, evidence-quality assessment, and field coefficient annotation, three concepts that naive specifications conflate.
Evaluation design. Per-agent test cases at three levels (common, edge, adversarial) drawn from the golden and adversarial test suites. Thirteen production monitoring metrics with alert thresholds. Quality-drift detection with a graded response protocol. Brier calibration with a feedback loop into methodology updates.
Decomposition for delegation. Seven agents with explicit handoff points and per-boundary rationale. The fact-check orchestrator (A3) decomposes further into seven tightly coupled reasoning stages within a single LLM call, because cross-stage context is critical. The Brier accumulator (A7) is deterministic code, not an LLM, because its task doesn’t need one.
Trust boundary design. A fourteen-row trust map classifies sub-tasks by cost of error, reversibility, frequency, verification method, and oversight level. Graduated oversight: low-risk claims flow through the workhorse only; medium-risk get advisor review; high-risk get advisor and human review. Auto-escalation from Standard to Full tier when gaming flags fire.
Cost economics. Tier-proportional methodology loading (Quick loads roughly 12K tokens, Forensic roughly 80K). Blended per-claim cost amortizes advisor calls across all claims, not only escalated ones. Break-even analysis versus human analyst rates. Vendor portability cost quantified across migration effort, feature loss, and re-validation.
Failure pattern recognition. Seven named failure modes mapped to specific system locations with detection and mitigation: context degradation, specification drift, sycophantic confirmation, tool selection errors, cascade failure, silent failure, and follow-up context pressure.
Context architecture. Tier-gated methodology loading is a context architecture decision, not just a cost one. The advisor runs on a shorter context than the workhorse (no tool-use history), making it less susceptible to context degradation. The vendor-neutral tier registry applies the same decoupling principle to model choice.

Validation

The original validation report covered 97 claims (96 pass, 1 partial, 0 fail) across eight domains, eleven attack vectors, and four languages (Japanese, Turkish, Chinese, Hindi). A 10-claim Security Suite has since been added, bringing the behavioral regression set to 105. Cross-model evaluation completed. The same protocol runs across prompt and scaffold changes, so silent degradation is detectable before it ships.

Where it stands

The invite-only beta is live at app.veridi.org. The landing at veridi.org is public; the app is gated by invite code, issued case by case. Source is available on request.

What’s next

Opening signup as capacity allows, expanded validation coverage, and entity formation as a Canadian cooperative.