Verita AI: production LLM systems that have to show their work

A vision-LLM audit pipeline

When a remote contractor bills for a day of work, what was that day? The answer sets what people get paid, so "the model thinks it looks fine" does not survive a dispute.

I built the pipeline that reads session frames from screen recordings with a vision model, checks each frame against that project's task policy, and buckets the time: genuine work, AI-assisted work, off-task activity, and frames the model could not read. Guideline violations are flagged from the same read. The output is a sealed ledger, priced and recomputable from the underlying rows, so a disputed figure can be traced instead of argued about.

One session through the pipeline. The task policy is per project, so the same screen can be clean on one and off-task on another.

Choosing the read model was an evaluation problem, not a preference. I benchmarked candidates across providers through OpenRouter on a golden set of human-labelled sessions, comparing precision, recall and cost per session. The pipeline reached 80% precision and recall on that set, and has sealed 264 sessions in production.

An LLM data-annotation platform

Distributed annotators produce training data here, and the client's, reviewer's and operator's views must never leak into each other. I built the core: five-role access control with an explicit capability matrix, a workflow state machine with audited transitions and calibration routing, screening and qualification gates before an annotator can claim work, an append-only audit log, and 10+ annotation widget types on the React side.

Two decisions I would defend anywhere. The permission model is tested by a table-driven matrix over every endpoint and every persona, with a completeness guard that fails when an endpoint is added without an assertion, because permission bugs live in the endpoint nobody remembered to test. And audit coverage is derived from the URL configuration rather than a hand-kept list, so the check proves all 54 endpoints are covered instead of trusting the list. Over this work the test suite grew from 210 to 1,239 tests.

The ingestion path took real time too. Remote images are fetched with private address ranges blocked, so a crafted URL cannot turn the server into a proxy for the internal network, then re-hosted to S3 through presigned uploads. Enqueueing moved off the request cycle into asynchronous SQS batches after 1,000-image imports began timing out. That path delivered 6,756 processed images to a client, with geo-derived metadata backfilled.

A rubric verifier, and a testing standard

For an AI interviewing product, I delivered the second generation of the finance-interview rubric verifier in six days: LLM-judge claim validation against the rubric, shadow-verifier gating so a new version runs on production traffic without changing outcomes, and hard execution time limits so a slow evaluation fails predictably instead of hanging.

The lasting contribution was a standard rather than code. A test only counts once a deliberate defect in the source has shown it can fail, and the defect itself has to be confirmed as landed, because a mutation that silently did not apply produces a test that looks proven and is not. Tests that pass against broken code are worse than no tests. They transfer confidence without earning it.

SOC 2 Type II, during a live audit window

I led readiness while the observation period was already running, which removes the option of fixing things quietly. Control coverage moved from 154 to 165 of 173, which is 95%.

• A pgAudit and storage-encryption migration across 11 production PostgreSQL databases, in Terraform rather than applied by hand.
• GuardDuty, CloudTrail and file-integrity monitoring deployed and wired to real alerting, which is how I found the file-integrity rules had been installed in a form that never fired.
• 40+ evidence artifacts for the auditor, and remediation of authenticated penetration-test findings.
• A backup and access review that surfaced backup jobs silently failing on missing role permissions.

The recurring lesson: most compliance failures are not missing controls. They are controls configured once and never verified as working. The interesting work is proving the alarm fires, not installing the alarm.

The security pipeline

I built a pipeline that runs five scanners (gitleaks, OpenGrep, Bandit, Trivy, CodeQL) and hands the merged output to a reasoning model for triage: deduplicating across engines, dropping false positives, and ranking by exploitability. On the production codebase it reduced 345 raw findings to 33 verified vulnerabilities, nine of them critical: remote code execution, SSRF, IDOR and exposed credentials.

The generalised version is open source, with no findings and no customer code in it. The 345 to 33 result belongs here, on the internal run; the public repo ships the method, not the numbers. Read that case study.

What I found

The most useful things I found were not features. Each was silent, and each was caught by asking a question the system was not asking itself.

Outcome, and what it does not prove

What it does not prove. 80% on a golden set is not 80% in production; the set is finite and production drifts from it. 1,239 tests says nothing about which would fail, which is why the mutation standard exists. 165 of 173 controls is readiness, not the auditor's opinion. And none of it is public; the numbers are my résumé's and this page is my account of them.