Testing and trust

Trust in an evidence tool has to be earned mechanically, not claimed. This page describes how SBOMFlow is tested, what the checks actually verify, how the product behaves when inputs go wrong, and how you can re-run the verification yourself. All figures below were produced by running the listed commands against the current source tree; re-run them to check us.

Current verified figures#

What the test suite actually covers#

• Parser tests — positive fixtures for every supported manifest, lockfile, and embedded build format, plus negative and malformed fixtures for each.
• Malformed-input behaviour — broken JSON/YAML/CSV/archives must produce a cataloged warning with the exact path, never a crash and never silence.
• Resource-bound tests — oversized archive members, deep nesting, and decompression limits are enforced and surfaced as warnings.
• Memory budgets are enforced, not just measured — a dedicated lane runs a deterministic synthetic scan and fails on a gross regression against a committed budget for peak process memory and for a per-phase peak that names which phase grew. The thresholds carry large cross-machine headroom, so the lane alarms on a real regression rather than on runner variance.
• Identity tests — version epochs, vendor suffixes, prereleases, range specifiers, container digests, and same-name/different-origin components must survive unchanged; a structural-garbage guard is tested against both garbage and legitimate unusual values.
• Determinism tests — repeat runs under a fixed --as-of must be byte-identical, including the evidence-bundle ZIP. Determinism is asserted across machines, not just reruns: an environment matrix re-runs a full analysis under four timezone × locale combinations (UTC through Kiritimati and Anchorage; the C locale through a comma-decimal one) and requires every produced artifact — HTML report, PDF summary, issues.csv, and the JSON surfaces — to have identical SHA-256 digests. Source guards keep it that way by construction: library code never touches the locale, the renderers use no locale-aware number formatting or locale-dependent strftime directives, and they never read the wall clock.
• Time-independence guard — determinism is asserted across time as well as locale: no test can expire with the wall clock, and the guard that enforces it is itself time-independent, so its own verdict cannot drift either.
• Boundary tests — reviewer status defaults, VEX justification enforcement, self-approval rejection, gate/VEX consistency, and draft watermarks are asserted directly.
• Integrity tests — the audit-log hash chain, artifact hashes, and cross-artifact references are validated, and corrupted or tampered outputs must fail validation.
• Crash-consistency tests — every consequential human-decision write is crash-tested at every interruption point in its transaction protocol (and at interruption pairs for concurrent operations), asserting after each that state is old-complete or new-complete, the audit log carries at most one event per operation, and recovery either converges or refuses visibly.
• Stateful property testing (optional dev lane) — beyond the enumerated schedules, a separate lane drives randomized operation sequences — decision writes, idempotent retries, crashes and recoveries against a real store, and the full waiver lifecycle — under a fixed, reproducible profile. It needs one optional dev-only package and is never part of the zero-dependency baseline.
• Error-contract tests — every CLI error code renders with a fix and a docs link, and every code has a matching section in the error reference.
• Every example printed by the CLI is executed, including from an installed wheel — every sbomflow … line an operator can copy off a --help page, a sbomflow help page, the sbomflow examples output or the generated CLI reference pages is one example in the command catalog, and the suite runs every one of them in a fresh directory with the network poisoned. Each must return the exit it declares and be seen to do what it says — write the files it names (a JSON one must parse) or print the text it names — so an exit code alone never counts as proof. Every example for a command that serves any requirement regime runs a second time against a product configured with requirement_regime: none and must keep its exit. make cli-examples-check then builds the wheel, installs it into a clean offline environment and runs the same examples through the installed sbomflow, reporting N of N with N counted from the catalog. This is a statement about SBOMFlow's own tests: it says the documented commands work as printed, and nothing about any product they are run on.
• Docs and site checks — every relative documentation link must resolve, and the public build is checked against a fail-closed allowlist so internal material cannot be published.

How CVE and advisory matching is tested#

Matching a component to a CVE is where a scanner earns or loses your trust: a false positive wastes a reviewer's afternoon, and a false negative is the one that matters. Both directions are tested, and both failure modes are handled conservatively rather than guessed.

• Against false positives — a withdrawn advisory (OSV withdrawn) or a rejected one (NVD Rejected) is excluded from active findings and recorded in the scan's provenance so the exclusion stays auditable. A disputed advisory is not dropped: it stays active and is flagged for a human. If an advisory's withdrawn value is itself malformed, the finding stays active and warns (advisory_withdrawn_malformed) — bad data never silently removes a finding.
• Against false negatives — pinned known-affected oracles run the full local matching path (package URL → version → advisory affected-range) against a pinned advisory snapshot, and assert that each oracle's affected version matches while its paired fixed version does not.
• Against quiet guessing — when two versions cannot be ordered under a component's ecosystem scheme, or an advisory range cannot be resolved locally, the finding is surfaced for human verification with lowered match confidence. An unresolvable advisory range also carries the advisory_range_unresolved warning. It is never silently asserted as affected, and never silently dropped.
• Component identity — a package-URL conformance corpus pins how versions, epochs, vendor suffixes, and qualifiers are parsed, so the coordinate a CVE is matched against does not drift between releases.

A CVE match is an observation, never a decision. Suppressing a finding — not_affected, fixed, or a waiver — comes only from human review, never from the matcher.

Synthetic stress testing#

Beyond unit tests, SBOMFlow is exercised against an in-house laboratory of ten synthetic connected-device manufacturer estates — from a small sensor company to multinational industrial estates — with realistic build systems, supplier evidence, release rooms, deliberately malformed files, planted operational inconsistencies, held-out answer keys, and positive and negative controls.

Runs against this corpus are checked by an executable acceptance contract (complete workflow coverage across all ten businesses) and a per-fact semantic evaluation that fails on any silent omission, false merge, false positive, missing provenance, or overclaim.

What the acceptance contract does not check. It runs on an interpreter with no optional packages installed at all, so it proves the standard-library-only baseline on every machine — and the surfaces that need an optional package are not exercised there. In particular, JSON-Schema validation of emitted SBOMs against the vendored official CycloneDX and SPDX schemas runs in its own conformance lane, which installs the schema validator into a throwaway environment and validates against the pinned official schemas. The acceptance run states this in its own output rather than leaving a green result to be read as wider than it is. The reason is cost, and it is measured: schema-validating the largest corpus outputs takes minutes per output, which does not belong in a per-change gate. That cost is tracked as a defect in its own right, not treated as settled.

AI surface hardening and its measured budgets#

The AI scan runs on the default analyze/audit path, so every hostile shape below reaches it on an ordinary run with no flag. Three adversarial corpus cases cover them, each with the control that keeps a refusal-only suite honest:

Resource limits are measured, not assumed — the numbers below come from running the shipped code against a generated tree, not from reading the source:

Memory is measured per allocation, never as the process-wide high-water mark, which is contaminated by whatever ran before it. No budget here is a wall-clock assertion: performance budgets are expressed in work — bytes per read, entries walked — which is a property of the algorithm rather than of the machine's load, so they hold on any machine. Elapsed times are recorded for a human and asserted on by nothing.

A traversal budget that fires is never silent: the walk stops or prunes, names the budget in bounds_hit, and the inventory is marked PARTIAL. A depth bomb prunes its own branch without erasing its siblings, so planting one deep chain cannot turn into an inventory-erasure primitive.

Latest full validation (July 2026)#

The most recent end-to-end validation pass ran every tier of this laboratory and then went further than "the commands exited zero":

• Business-scale advisory controls. One estate's full scan (about 4,900 components) is fed a deterministic offline advisory snapshot using the official ecosystem naming conventions, with positive and negative controls — advisories that must match and one that must not — so "producing more findings" can never fake a better score. Unreviewed findings stay under_investigation (the reviewer boundary is asserted, not assumed).
• Every artifact re-checked from outside. All generated outputs across all ten estates were re-parsed and reconciled independently: product identity agreement, count reconciliation across SBOM/VEX/issue surfaces, every bundle and sharing-pack hash recomputed, no forbidden conformity language, and no leaked secrets or machine-specific paths on shareable surfaces.
• Hostile and offline pressure. A network tripwire fails any accidental connection attempt; mutation self-checks deliberately corrupt outputs and require the oracles to catch each corruption, proving the assertions are not vacuous.

The acceptance contract passes in full on the current release. The full offline suite total is stated once, at the top of this page, and guarded there — repeating it here is how it went stale before.

This process found real defects — which is its job. Those that matter to users are fixed with regression coverage in the current release and recorded in what's new; one example:

• Very large recognized lockfiles (over the general text-scan limit) previously produced no components and no explanation. Recognized structured inputs now parse up to a much higher dedicated cap, and anything beyond it is hashed and surfaced with a documented resource warning and a concrete remedy — never silently dropped. See known limitations for the current caps and a measured large-input example.

Be clear about what this is and is not:

• It is synthetic. Every company, product, and file is fictional; sample advisories are labelled non-real.
• It is not real-customer validation, and it is not proof of conformity or universal format compatibility.
• Its purpose is reproducible engineering stress testing — large realistic inputs, hostile file shapes, and honest scoring of what was detected, warned, or left to human review.

How SBOMFlow behaves when things go wrong#

Telling whether an audit was changed#

SBOMFlow is built to answer "was this audit changed after the fact?" honestly. Every audit is tamper-evident: the audit log is an append-only hash chain and each artifact carries a SHA-256, so if the recorded evidence was changed, sbomflow validate fails (exit 4) and new reviewer decisions are refused. To see what changed between two audits of successive releases, compare their release records — drift lists the components, findings, and gaps that changed.

Verify it yourself#

Everything below runs offline against an installed build:

• Run sbomflow analyze twice over the same tree with the same --as-of; the artifacts are byte-identical.
• Run sbomflow validate on an output directory, then edit one artifact and run it again: exit 4, naming the mismatch.
• Run sbomflow verify-bundle --layered on a bundle, then on a copy with one member altered: the content layer fails and the bundle is NOT TRUSTED.
• Run sbomflow doctor to see the local environment facts a run relies on.

The synthetic laboratory's acceptance contract and per-fact semantic evaluation run offline as part of the same trust gauntlet; they are available under the private-review process.

How the v0.5.5 build was verified#

v0.5.5 is available to approved private testers. The build handed to them is a certified private v0.5.5 tester build with no known release-blocking defects, subject to the documented restrictions. This section states what that sentence rests on, and — equally important — what it does not.

Release identity#

These values come from one generated source, so no two published pages can disagree about them.

What was established#

• Every mandatory release check ran and passed. None was skipped, and a skipped check would never have counted as a pass.
• Three independent builds of the same tree produced a byte-identical wheel, compared by SHA-256 — among them builds from two separate exports of the tree that had no part in producing the release artifact.
• A clean-machine install ran with the index and dependency resolution both switched off. That is why "zero required runtime dependencies" is a behaviour here rather than an assertion about a metadata field.
• Zero network markers in the install transcript. The sweep looked for five markers, and each of the five was first calibrated against a transcript of an install that did use the network, so the zero is a measurement rather than a pattern that never fires.
• 501 of 501 wheel entries were inspected. The examined count and the total are the same number, which is the only reason a finding count of zero carries information.

What was not established#

• The build is not signed. There is no signature on the wheel.
• There is no continuous-integration attestation. No hosted build service produced one for this release.
• A checksum is not a signature. It proves the bytes you received match the bytes the build produced; it does not establish who built them.
• Reproducibility here is one machine, one interpreter, one operating system. All three builds ran in the same environment. This is same-environment reproducibility and is not a cross-platform or toolchain-independent claim.
• An optional trust layer whose extra was absent is not claimed to have run. A check that could not run reports that it was skipped; it never reports a pass.
• Native Windows is unproven. The native-Windows check runs and has never passed — see the compatibility matrix. The verified platforms are Linux, macOS on Apple Silicon, and Windows through WSL.
• "No known release-blocking defects" is not "error-free." It is a statement about what was known when the build was certified.
• None of it is a verdict about your product. No conformity, certification, safety or security conclusion is expressed or implied.

Every number above is stated with the count it was measured out of, and every absence was calibrated against a case where the same check fires.

What we do not claim#

No test suite proves the absence of defects, and a synthetic corpus cannot guarantee every future customer input. SBOMFlow does not claim perfect accuracy, universal format support, or legal conformity — and any claim on this site that cannot be traced to running code and tests is a bug. Real, sanitised customer artifacts remain the standard we validate against next.

Next: capabilities · security & privacy · what's new