Testing and trust

Trust in an evidence tool has to be earned mechanically, not claimed. This page describes how SBOMFlow is tested, what the checks actually verify, how the product behaves when inputs go wrong, and how you can re-run the verification yourself. All figures below were produced by running the listed commands against the current source tree; re-run them to check us.

Current verified figures#

What the test suite actually covers#

• Parser tests — positive fixtures for every supported manifest, lockfile, and embedded build format, plus negative and malformed fixtures for each.
• Malformed-input behaviour — broken JSON/YAML/CSV/archives must produce a cataloged warning with the exact path, never a crash and never silence.
• Resource-bound tests — oversized archive members, deep nesting, and decompression limits are enforced and surfaced as warnings.
• Memory budgets are enforced, not just measured — a per-push CI lane runs a deterministic synthetic scan and fails on a gross regression against a committed budget: peak process memory (ru_maxrss) and a per-phase peak that names which phase grew. The thresholds carry large cross-machine headroom, so the lane alarms on a real regression rather than on runner variance.
• Identity tests — version epochs, vendor suffixes, prereleases, range specifiers, container digests, and same-name/different-origin components must survive unchanged; a structural-garbage guard is tested against both garbage and legitimate unusual values.
• Determinism tests — repeat runs under a fixed --as-of must be byte-identical, including the evidence-bundle ZIP. Determinism is asserted across machines, not just reruns: an environment matrix re-runs a full analysis under four timezone × locale combinations (UTC through Kiritimati and Anchorage; the C locale through a comma-decimal one) and requires every produced artifact — HTML report, PDF summary, issues.csv, and the JSON surfaces — to have identical SHA-256 digests. Source guards keep it that way by construction: library code never touches the locale, the renderers use no locale-aware number formatting or locale-dependent strftime directives, and they never read the wall clock.
• Time-bomb guard — a test fixture pinned to a near-future date passes for months, then the wall clock crosses it and the suite goes red with no code change and no obvious cause. A guard scans every test for future-dated literals and requires each one to be a shared far-future fixture, built relative to the run, or on a reviewed list stating why it cannot expire. The guard is itself time-independent: it compares literals against fixed constants, never against "now", so its own verdict cannot drift either.
• Boundary tests — reviewer status defaults, VEX justification enforcement, self-approval rejection, gate/VEX consistency, and draft watermarks are asserted directly.
• Integrity tests — the audit-log hash chain, artifact hashes, and cross-artifact references are validated, and corrupted or tampered outputs must fail validation.
• Crash-consistency tests — every consequential human-decision write is crash-tested by exhaustively enumerating every interruption point in its transaction protocol (and interruption pairs for concurrent operations), asserting after each schedule that state is old-complete or new-complete, the audit log carries at most one event per operation, and recovery either converges or refuses visibly.
• Stateful property testing (optional dev lane) — beyond the enumerated schedules, a separate make hypothesis-stateful lane drives two model-checked state machines over randomized operation sequences: decision writes, idempotent retries, crashes and recoveries against a real store, and the full waiver lifecycle (create, renew, revoke, expire under a pinned clock, gate evaluation). The lane runs a fixed, derandomized profile so results are reproducible; it needs one optional dev-only package and is never part of the zero-dependency make test baseline — a baseline test enforces that separation.
• Error-contract tests — every CLI error code renders with a fix and a docs link, and every code has a matching section in the error reference.
• Docs and site checks — every relative documentation link must resolve, and the public build is checked against a fail-closed allowlist so internal material cannot be published.

How CVE and advisory matching is tested#

Matching a component to a CVE is where a scanner earns or loses your trust: a false positive wastes a reviewer's afternoon, and a false negative is the one that matters. Both directions are tested, and both failure modes are handled conservatively rather than guessed.

• Against false positives — a withdrawn advisory (OSV withdrawn) or a rejected one (NVD Rejected) is excluded from active findings and recorded in the scan's provenance so the exclusion stays auditable. A disputed advisory is not dropped: it stays active and is flagged for a human. If an advisory's withdrawn value is itself malformed, the finding stays active and warns (advisory_withdrawn_malformed) — bad data never silently removes a finding.
• Against false negatives — pinned known-affected oracles run the full local matching path (package URL → version → advisory affected-range) against a pinned advisory snapshot, and assert that each oracle's affected version matches while its paired fixed version does not.
• Against quiet guessing — when two versions cannot be ordered under a component's ecosystem scheme, or an advisory range cannot be resolved locally, the finding is surfaced for human verification with lowered match confidence. An unresolvable advisory range also carries the advisory_range_unresolved warning. It is never silently asserted as affected, and never silently dropped.
• Component identity — a package-URL conformance corpus pins how versions, epochs, vendor suffixes, and qualifiers are parsed, so the coordinate a CVE is matched against does not drift between releases.

A CVE match is an observation, never a decision. Suppressing a finding — not_affected, fixed, or a waiver — comes only from human review, never from the matcher.

Synthetic stress testing (TestBuz)#

Beyond unit tests, SBOMFlow is exercised against TestBuz: a fully synthetic, deterministic corpus of ten distinct connected-device manufacturer archetypes — from a 19-person sensor company to multinational network, maritime, metering, EV-charging, medical, robotics, and industrial estates — with realistic build systems, supplier evidence, release rooms, deliberately malformed files, and planted operational inconsistencies.

Runs against this corpus are checked by an executable acceptance contract (complete workflow coverage across all ten businesses) and a per-fact semantic evaluation that fails on any silent omission, false merge, false positive, missing provenance, or overclaim.

What the acceptance contract does not check. It runs on an interpreter with no optional packages installed at all, so it proves the standard-library-only baseline on every machine — and the surfaces that need an optional package are not exercised there. In particular, JSON-Schema validation of emitted SBOMs against the vendored official CycloneDX and SPDX schemas runs in its own lane (make conformance), which installs the schema validator into a throwaway environment and validates against the pinned official schemas. The acceptance run states this in its own output rather than leaving a green result to be read as wider than it is. The reason is cost, and it is measured: schema-validating the largest corpus outputs takes minutes per output, which does not belong in a per-change gate. That cost is tracked as a defect in its own right, not treated as settled.

AI surface hardening and its measured budgets#

The AI scan runs on the default analyze/audit path, so every hostile shape below reaches it on an ordinary run with no flag. Three adversarial corpus cases cover them, each with the control that keeps a refusal-only suite honest:

Resource limits are measured, not assumed — the numbers below come from running the shipped code against a generated tree, not from reading the source:

Memory is measured with tracemalloc, never the process-wide RSS high-water mark, which is contaminated by whatever ran before it. No budget here is a wall-clock assertion: several agents work this repository concurrently, so the performance budget is expressed in work — bytes per read, entries walked — which is a property of the algorithm rather than of the machine's load. Elapsed times are recorded for a human and asserted on by nothing.

A traversal budget that fires is never silent: the walk stops or prunes, names the budget in bounds_hit, and the inventory is marked PARTIAL. A depth bomb prunes its own branch without erasing its siblings, so planting one deep chain cannot turn into an inventory-erasure primitive.

Latest full validation (July 2026)#

The most recent end-to-end validation pass ran every tier of this laboratory and then went further than "the commands exited zero":

• Business-scale advisory controls. One estate's full scan (about 4,900 components) is fed a deterministic offline advisory snapshot using the official ecosystem naming conventions, with positive and negative controls: three advisories that must match through the version-range engine, and one fixed-before-shipped advisory that must not match — so "producing more findings" can never fake a better score. A known-exploited entry must surface first in the issue queue with the exploited flag and the matching triage uplift, while unreviewed findings stay under_investigation (the reviewer boundary is asserted, not assumed).
• Every artifact re-checked from outside. All generated outputs across all ten estates were re-parsed and reconciled independently: product identity agreement, count reconciliation across SBOM/VEX/issue surfaces, every bundle and sharing-pack hash recomputed, no forbidden conformity language, and no leaked secrets or machine-specific paths on shareable surfaces.
• Hostile and offline pressure. A network tripwire fails any accidental connection attempt; mutation self-checks deliberately corrupt outputs and require the oracles to catch each corruption, proving the assertions are not vacuous.

The acceptance contract currently passes all 20 cases and all 22 feature groups. The full offline suite total is stated once, at the top of this page, and guarded there — repeating it here is how it went stale before.

This process found real defects — which is its job. Two matter to users and are fixed with regression coverage in the current release:

• Maven advisory coordinates from real-world advisory data (group:artifact) could fail to match the equivalent component identity recorded from build files (group/artifact), silently missing advisories. Matching is now separator-consistent, with no-overmatch guards.
• Very large recognized lockfiles (over the general text-scan limit) previously produced no components and no explanation. Recognized structured inputs now parse up to a much higher dedicated cap, and anything beyond it is hashed and surfaced with a documented resource warning and a concrete remedy — never silently dropped. See known limitations for the current caps and a measured large-input example.

Be clear about what this is and is not:

• It is synthetic. Every company, product, and file is fictional; sample advisories are labelled non-real.
• It is not real-customer validation, and it is not proof of conformity or universal format compatibility.
• Its purpose is reproducible engineering stress testing — large realistic inputs, hostile file shapes, and honest scoring of what was detected, warned, or left to human review.

How SBOMFlow behaves when things go wrong#

Telling whether an audit was changed#

SBOMFlow is built to answer "was this audit changed after the fact?" honestly. Every audit is tamper-evident: the audit log is an append-only hash chain and each artifact carries a SHA-256, so if the recorded evidence was changed, sbomflow validate fails (exit 4) and new reviewer decisions are refused. To see what changed between two audits of successive releases, compare their release records — drift lists the components, findings, and gaps that changed.

Verify it yourself#

All of these run offline from a source checkout:

make test                 # full offline suite
make demo                 # end-to-end offline evidence pack
make wave4-check          # importer/integration gauntlet
make validate-artifacts   # structural validation of generated outputs
make check-docs-links     # every relative docs link resolves
make check-docs-publication  # fail-closed public-content leak check
make install-check        # clean-venv wheel install + installed CLI run

The TestBuz acceptance contract and per-fact semantic evaluation run offline from a source checkout as part of the same trust gauntlet, writing their results to a fresh output directory you choose.

Optional developer lanes sit beside the baseline without gating it: make hypothesis-stateful runs the stateful property-testing machines described above (it installs one dev-only package into an isolated virtual environment on first use; every later run is offline).

What we do not claim#

No test suite proves the absence of defects, and a synthetic corpus cannot guarantee every future customer input. SBOMFlow does not claim perfect accuracy, universal format support, or legal conformity — and any claim on this site that cannot be traced to running code and tests is a bug. Real, sanitised customer artifacts remain the standard we validate against next.

Next: capabilities · security & privacy · what's new