How the vvaharness engine is built

The mental model

VVAH turns a source tree into a set of verified, scored, and optionally remediated-and-validated findings. Three ideas shape the whole design:

1. Threat-model before you analyse. Stages 1–3 build a map of the attack surface — call graph, entry points, sinks, STRIDE threats — and use it to decompose the repo into ranked work units. Deep analysis (S4) only ever sees focused chunks, never "here is the whole repo, find bugs."
2. Every stage boundary is a typed, checkpointed artifact. No raw dict crosses a boundary — each step consumes and emits a pydantic model, serialized to a SQLite state store as JSON, never pickle. The goal: "a $50 run that dies at step 3 stays debuggable," and --resume skips everything already done.
3. Triage speed is the bottleneck, not discovery. The pipeline invests heavily in filtering — deterministic prefilter gates, adversarial per-finding verification, multi-axis dedup — so what reaches a human is small and confirmed. The headline metric is Mean Time to Adapt (MTTA): time from AI-discovered exploitability to a validated fix.

The same flow, drawn as the Visa whitepaper presents it — inputs on the left, the per-stage skills the harness applies on the right, machine-readable artifacts on the outputs edge:

Module layout

The installed package is vvaharness/. The driver lives in orchestrator/, the analysis stages in pipeline/stages/, and everything the stages call — transport, config, redaction, contracts — sits alongside.

vvaharness/
  cli.py              console entry point — setup / doctor / estimate / gc / scan / remediate / validate
  orchestrator/       pipeline driver: entry, scan, batch, preflight, checkpoints, store (SQLite), cmdb …
  manifest.py         run-level run_manifest.json — version, roles, config hash, target git SHA, timing
  models.py           pydantic contracts — ContextPackage, Finding, FinalReport, …
  config/             loader (${ENV} expansion, local override, step1 overlays)
    profiles/         default.yaml (all-CLI) · sdk.yaml (voting on) · full.yaml (multi-backend)
  pipeline/stages/    s1_preprocess … s8_chain, s11_validate wrapper
  remediation_agent/  Step 10 — proposes & applies a minimal fix, writes per-finding DTOs
  validation/         Step 11 — Claude Agent SDK adversarial panel over the DTOs
  backends/           transport: llm (dispatch) · sdk · agent_sdk · oai · claude_cli · localtools
  report/            enrich (CVSS env, CMDB, MD→SARIF) · cvss · cwe · redact
  injectors/         cve_feed · design_controls (optional context loaders)
  util/              environment · tokens · metrics · errlog · prompts · json_extract · status
  lang/              language hints — EXT_TO_LANG, LANG_HINTS, SPECIALIST_HINTS

Entry point & orchestration

cli.py is the installed vvaharness command. Before dispatching it checks the interpreter against Requires-Python from package metadata (never a hardcoded floor) and best-effort loads .env. A bare invocation prints help and exits — it deliberately does not default to scan, so a mistyped command never produces a junk manifest.

The single-repo driver

scan_repo() runs the eleven stages. Each follows one shape: load checkpoint → skip if present under --resume → else run under status + token phase → save checkpoint → honor --stop-after. Key mechanics:

• State lives outside the scanned repo. run_id is a SHA-256 of the resolved path; checkpoints go to $VVAHARNESS_STATE_DIR (default ~/.vvaharness/state/). A hostile target can't pre-plant --resume state, and --resume is refused outright if the state dir resolves inside the repo.
• Stages 5+6+7 share one checkpoint — a 5-tuple under the s7 key — because the prefilter gates are near-zero-cost and re-run cheaply against the S4 checkpoint.
• Non-fatal degradation is deliberate. A failed threat model or auto-step1 warns and continues; an S10 preflight failure disables only S10.
• Artifacts land under <repo>/security-scan/ — timestamped report.md, report.sarif, and errors.jsonl. HEAD is pinned into report.git_sha so S10 refuses to patch a moved tree.

Batch mode

Batch is strictly sequential — the only parallelism lives inside the S4/S6 thread pools, never across repos. Cloning is hardened against git-option injection (reject - refs and ./.. dest names, -- separators), scrubs tokens from output, and binds each checkout to its source ref via a marker kept outside the workspace so a stale or pre-seeded directory is never silently reused.

Checkpoints & the run manifest

The store is WAL-mode SQLite with runs and checkpoints tables. The load-bearing invariant: payloads are JSON via pydantic, never pickle — a tampered checkpoint degrades to a step re-run, never CWE-502 RCE on resume. manifest.capture() writes run_manifest.json only when a scan actually ran, recording config_sha256 and config_local_sha256 (the effective config), the per-role model map, and the target git SHA — with argv scrubbed in two passes.

The pipeline, stage by stage

Data flows as in-memory pydantic objects through scan_repo, checkpointed at each boundary. Prompts are assembled from shared fragments in util/prompts.py plus per-stage system strings and, for S4, the language and specialist bodies from lang/hints.py.

Data contracts

models.py defines every stage boundary. The invariant: no raw dict crosses a boundary, and every strict field populated from LLM JSON carries a field_validator(mode="before") that degrades one off-schema value to a safe default instead of killing the run.

Finding.canonical_key(line_bucket) — (file, line_start // bucket, vuln_class) — is the identity used for both S4 voting and S7 dedup; bucketing the line absorbs the small jitter between runs. Every model→prompt renderer neutralizes Markdown injection, since these strings mix operator and model-generated content.

The transport layer

Every model call goes through backends/llm.py, which resolves the per-role {id, via, …} config node and dispatches on via:. Two public calls exist — prompt() (single-shot, no tools) and agentic() (a tool-using session). Roles are swapped in config alone.

Remediation

remediation_agent/ proposes and (in fix mode) applies a minimal patch per verified finding, through the main dispatcher (role models.remediate). Standalone and in-pipeline paths adapt to the same per-finding loop. Flow: locate the newest report → select top-N by CVSS → policy pre-gate → snapshot files → model call (the agent edits the tree directly) → post-gate → write artifacts.

<repo>/security-remediation/<NN_slug>/
  remediate_report.json    the DTO — canonical hand-off contract
  evidence/triage.json     the agent's RemediationVerdict + meta (redacted)
  evidence/summary.md      human summary (redacted)
  evidence/diff.patch      git diff, or a synthesized unified diff for non-git

The DTO carries status, the verbatim finding, the patch, and an empty validation block — remediation lays out the shape; Step 11 fills the scores. Policy gates are opt-in and fail-closed: a policy that fails to parse denies everything. The pre-gate can deny with no model spend; the post-gate inspects what actually changed on disk and reverts any edit to a forbidden path (CWE-22-hardened). Resume authenticates by content hash, closing the "existence-implies-done" gap for reordered state.

Validation

validation/ grades each remediation DTO with an adversarial Claude Agent SDK panel. It ships its own harness abstraction; the only hard cross-links to the rest of the system are the shared checkpoint store and the DTO wire contract.

Discovery is deterministic and spends no model budget: glob the DTOs, skip malformed ones, select the validatable statuses. Per finding, the repo is staged into an ephemeral hardlinked copy (.git excluded) and the panel launches: security-architect and penetration-tester always, plus a conditional cross-repo-analyzer only when a fix spans 2+ repos. Per-CWE bypass hints from validator_hints.yaml are injected into the shared launch prompt.

Host-authoritative scoring

The load-bearing decision: the agents synthesize gate statuses into synthesized_gates.json, but the host recomputes the numeric verdict in-process with the canonical engine, keyed strictly by finding id. The agent's self-reported score is never trusted; a missing gate file fails closed to UNVERIFIABLE.

Multipliers are pass=1.0 / partial=0.5 / fail=0.0 / skip=0.0 (skips drop from both numerator and denominator). Below a 0.50 coverage floor the result is UNVERIFIABLE. Thresholds: ≥ 0.80 Fixed, ≥ 0.50 Partially Fixed, else Not Fixed. The verdict transitions the DTO status — Fixed → validated, Not/Partially → validation_failed (re-validatable), UNVERIFIABLE → needs_review.

Reporting & enrichment

report/enrich.py owns the Markdown↔SARIF boundary and CVSS enrichment. The Markdown report is the interchange format; enrich.py parses it back with a line-oriented state machine, and an idempotency guard ensures re-enrichment never double-appends a score.

• CVSS base follows FIRST.org §7 — report/cvss.py is the strict library; enrich.py carries a second, more tolerant parser (a known duplication worth consolidating).
• VulContextSeverity folds CMDB context into the score: a non-externally-facing app downgrades Modified Attack Vector N → A, and CR/IR/AR come from the PCI/PAN/PII flags, under a fixed SAST temporal preset.
• CMDB join merges an application row with its parent when flags are blank, normalizes app ids, and warns on genuine post-normalization collisions.
• CWE mapping trusts a parsed CWE-NNN token, else falls back deterministically from the vuln class — never an LLM-emitted CWE blindly.

Cross-cutting concerns

Security-relevant invariants

The whole point of a security scanner is that it can be pointed at hostile code. The design makes several promises worth stating in one place:

Non-negotiable architectural security practices

The Visa whitepaper closes with twelve architectural and design practices it treats as non-negotiable for critical infrastructure — drawn from Visa's Mythos deployments and Project Glasswing, on one premise: frontier AI collapses discovery-to-exploit from weeks to minutes, so trust can never be assumed. They are the targets VVAH both enforces in its own internals (a scanner is pointed at hostile code) and hunts for in the code it scans. Each is quoted, then mapped to where it lives in this codebase.

References

• Visa whitepaper — Frontier AI: A New Era of Cyber ResilienceProject Glasswing, Visa Cybersecurity, June 2026. Source of the four-phase / eleven-stage diagram and the twelve non-negotiable practices.corporate.visa.com/…/project-glasswing.pdf
• Visa announcement — Visa Releases Its AI-Powered Cyber Defense System to Open Sourcecorporate.visa.com/…/visa-cybersecurity-mythos-project-glasswing.html
• Reference implementation — visa/visa-vulnerability-agentic-harnessgithub.com/visa/visa-vulnerability-agentic-harness
• Anthropic — Project Glasswinganthropic.com/project/glasswing