Recovery playbooks

Things go wrong in the middle of real release work: a laptop dies between a reviewer's decision and its audit record, a disk hiccup truncates a file, an old evidence pack meets a newer SBOMFlow. This page is the operator's map for those moments — one diagnose → recover path per failure, using only commands that already exist.

Every tested playbook below is exercised end-to-end by the offline test suite: the failure state is really created, the documented diagnosis commands are really run, and the documented recovery is really verified. A recipe here cannot rot into something that no longer works.

Two rules hold everywhere on this page:

• The engine never guesses. Recovery completes only steps whose outcome is provable from recorded digests; anything ambiguous is retained and named for a human. Nothing here creates a reviewer decision, accepts evidence, or claims conformity.
• Exit codes are the contract. 0 healthy, 1 an enforced gate said no, 2 usage/input, 4 integrity or structural failure, 5 needs attention. See exit codes and sbomflow help exit <code> offline.

Attach diagnostics to any bug report with the sanitized, offline support bundle (config shape and warning counts, never your source or findings):

sbomflow support-bundle ./evidence --yes

PB-01 — Interrupted decision transaction

Symptom. A decision command (triage, evidence review, waiver, approval, claim) crashed or was killed mid-write. Every consequential decision runs as a crash-recoverable transaction: a write-ahead journal is prepared, the decision file is written, the audit event is appended, and only then is the journal completed and removed. A crash leaves a discoverable journal instead of a silently half-recorded decision.

Diagnose. List the journals, then classify each against the real state of the decision file and the audit log:

sbomflow decisions transactions --store ./evidence
sbomflow decisions verify-transactions --store ./evidence

verify-transactions exits 5 when anything needs a human, 0 when the store is clean. A recoverable row means the decision landed but its audit event did not (or nothing was applied and the attempt can be safely aborted). Narrow large stores with --operation-type, --target, or --status.

Recover. Target-scoped and dry-run first, exactly as printed by the diagnosis:

sbomflow decisions recover --store ./evidence --operation-id <operation-id> --dry-run
sbomflow decisions recover --store ./evidence --operation-id <operation-id>

Recovery completes only the provably missing step: the audit event is reconstructed from the recorded decision and verified against the digest bound before the crash, then appended exactly once (the event carries recovered: true). An attempt that never applied is aborted clean. You do not have to run this eagerly — the next decision write on the same target heals it automatically first.

Never hand-edit or hand-delete a journal that verify-transactions can still classify. It is the only record linking an applied decision to its pending audit event.

PB-02 — Corrupt or truncated decision journal

Symptom. A journal file itself is unreadable — torn by a crash mid-write, or damaged on disk. Because an unreadable journal's target cannot be trusted, all new decision writes on that store are refused (fail closed) until a human resolves it; the refusal message points back here.

Diagnose.

sbomflow decisions verify-transactions --store ./evidence
sbomflow decisions transactions --store ./evidence --status corrupt

The corrupt row names the exact journal file and the parse error. Rows can also be manual_intervention_required, state_mismatch, or audit_mismatch — readable journals whose recorded digests contradict the observed state. Those stay quarantined in place with the mismatch named; the engine will not auto-repair by guessing which side is right.

Recover. This is a manual-intervention path by design:

1. Confirm the parts the engine can verify are intact — the decision files and the hash-chained audit log:

``bash sbomflow validate ./evidence ``

1. Investigate how the journal was damaged (crash timing, disk, permissions), and preserve a copy of the unreadable file with your incident notes.
2. Only after that investigation, move the unreadable journal out of the store's journal directory (.decision-txn/ inside the output directory) and quarantine it with your notes — this is the human act the engine refuses to automate. Re-run verify-transactions to confirm the store accepts decisions again.
3. Re-issue the affected human decision so the store carries a complete, audited record of it:

``bash sbomflow review ./evidence --triage <finding-id> --status under_investigation --reviewer you@example.com ``

Never let the engine (or a script) guess: an unreadable journal is evidence about an interrupted human action, not noise to be cleaned on sight.

PB-03 — Audit-chain mismatch

Symptom. sbomflow validate (or any decision command, which refuses to run over a broken chain) reports that the audit log's hash chain does not verify. Exit code 4, error E043.

Diagnose.

sbomflow validate ./evidence

Validation localizes the break instead of just declaring it: a defect confined to malformed JSON on the final line with every prior entry chained intact is reported as consistent with a truncated/incomplete tail (a crash mid-write), NOT mid-chain tampering — while altered event content, a broken link, or a stripped chain is reported at its exact entry position as tampering. The distinction is the whole diagnosis: a torn tail is an interrupted write; a mid-chain break means a recorded entry changed after the fact.

Recover. Treat a mid-chain break as tampering until explained:

1. Do not record anything new into the directory — SBOMFlow already refuses.
2. Restore audit-log.jsonl from your last known-good copy (the evidence store, or an archived reviewer bundle: the bundle records the chain head and entry count as an external anchor, and sbomflow store verify re-checks stored releases), then re-run sbomflow validate to confirm.
3. Document the incident: when the break was found, what the localization said, which copy you restored from. The chain proves that something changed, not who changed it — see the honest limits in security & privacy.

Never silently re-chain or hand-repair hashes. A rewritten-but-consistent chain is exactly what the anchor comparison exists to catch.

PB-04 — Stale decision or expired waiver

Symptom. A reviewer decision was recorded against an earlier state of the product (the component's version has changed since), or a time-boxed gate waiver has lapsed. Neither is auto-invalidated or auto-renewed: the record stands as history and the surfaces below prompt a human to look again.

Diagnose. The review queue marks prior decisions that need re-confirming (prior decision … component version changed since; re-confirm), the gate output lists waivers by status, and the freshness horizon answers "what lapses next?" in one dated report:

sbomflow review ./evidence
sbomflow report freshness ./evidence

An expired waiver appears in release-gate.json under waiver_status.expired (with negative days_until_expiry) and on the horizon as kind: waiver, status: expired — and the item it covered blocks the gate again, which is the point of an expiry.

Recover. Both paths are recorded human actions with history preserved:

• Stale decision: re-review it. A new decision on the same finding is a first-class supersession — a distinct audited operation that records the exact predecessor it replaces; the earlier decision remains in the file and the audit log.
• Expired waiver: renew or let it lapse. A renewal needs a strictly later expiry and a fresh reason (silently re-waiving with the stale reason is refused), and the previous term is kept in the waiver's renewal history:

sbomflow review ./evidence --renew-waiver-finding <waived-finding-id> --reason "risk re-assessed; new expiry approved by security review" --expires-at 2027-06-30T00:00:00Z --reviewer you@example.com

Then re-run the gate so the release reflects the renewal:

sbomflow analyze . --output ./evidence --waivers ./evidence/waivers.json

Never auto-renew, and never edit a recorded decision in place — the lapse-then-human-renewal loop is what makes a waiver an exception rather than a permanent hole.

PB-05 — Stale vulnerability snapshot

Symptom. Findings look clean but the advisory data they were matched against is old. SBOMFlow makes this visible instead of letting run time masquerade as freshness: every run's vuln-source-health.json records, per source, whether it was usable (state) and how old its data is (freshness), against a documented per-source staleness threshold.

Diagnose. Read vuln-source-health.json in the output directory: a snapshot past its threshold reports state: stale / freshness: stale with its age in days, and the run raises the cataloged stale_vulnerability_data warning naming the source (see the warning catalog). A source with no readable date reports freshness: unknown — never fresh. To make CI fail on it, promote the warning: --strict=stale_vulnerability_data exits 5 after all artifacts are written.

Recover. Supply newer data — staleness is never "fixed" by re-running:

• Offline (preferred): point the run at a newer pinned snapshot:

``bash sbomflow analyze . --output ./evidence --kev-file kev.json ``

(the same applies to --nvd-file, --epss-file, and --osv-snapshot).

Never treat the run timestamp as data freshness, and never block on staleness by default — a stale source was genuinely consulted; its age is a reported fact for you to act on.

PB-06 — Unsupported schema version

Symptom. An output directory produced by an older SBOMFlow carries artifact schema versions the current tooling has since evolved past.

Diagnose. Name the family and version of every artifact, then dry-run the migration:

sbomflow upgrade inspect ./evidence
sbomflow upgrade plan ./evidence

inspect prints family=… version=… per artifact; plan reports exactly which files would change, from which version to which, flagging any lossy step. Neither writes anything.

Recover. Migrate to a copy — never in place — and validate it:

sbomflow upgrade migrate-copy ./evidence --dest ./evidence-migrated
sbomflow upgrade validate ./evidence

(point upgrade validate at the migrated directory). The source directory is untouched, and the migrated copy carries upgrade-manifest.json — the receipt recording every applied step, the original content hashes, and any lossy-step warnings. Migrations transform representation only: they never invent evidence, reviewer decisions, or authority.

Never hand-edit artifacts to a newer shape — an edit invalidates recorded digests and turns a schema question into an integrity question (PB-03).

PB-07 — Evidence gap for a requirement

Symptom. A CRA requirement in scope for your product class has no observed evidence behind it. Gaps are first-class records, not absences: each carries a stable gap_id, the requirement, the signal that was not observed, and a concrete recommendation.

Diagnose. The gap register lives in evidence-pack.json under evidence_gaps, cross-referenced per requirement in cra-coverage.json (gap_ids). Enforce it and read the reasons:

sbomflow analyze . --output ./evidence --fail-on-gaps
sbomflow explain ./evidence --gate
sbomflow explain ./evidence --gap <gap-id>

A blocked run exits 1 with reason code GATE_EVIDENCE_GAPS listing the gap ids; explain --gap prints the requirement, the official reference, and what would satisfy the signal.

Recover. Close the gap with real inputs, then let a human accept:

1. Supply the evidence the recommendation names (for example, a monitored SECURITY.md disclosure policy in the product tree), or record manufacturer-supplied Annex I input files, and re-run the analysis — the gap disappears from the register only because the signal is now observed.
2. A human reviews the new evidence item; observation is never acceptance:

``bash sbomflow review ./evidence --accept vulnerability_disclosure_policy --reviewer you@example.com ``

1. If the evidence genuinely cannot land before the release, a reviewer may record a time-boxed waiver with a reason and expiry (see PB-04) — visible in the gate output, never a resolution of the gap.

Never infer acceptance. See observed vs reviewed — the engine observes; humans decide.

PB-08 — Failed approval quorum

Symptom. The release gate blocks because required sign-off roles are not each covered by an active, attributed approval (revoked and expired approvals do not count; separation of duties refuses self-approval of one's own release).

Diagnose. The gate names the policy and every unmet role, machine-readably (reason_code: GATE_MISSING_APPROVALS in release-gate.json, the unmet roles in its ids) and human-readably:

sbomflow analyze . --output ./evidence --require-role product-security --fail-on-missing-approvals
sbomflow explain ./evidence --gate

Recover. Obtain the attributed approval from the named role — there is no actorless way to satisfy a quorum:

sbomflow approve ./evidence --role product-security --reviewer approver@example.com
sbomflow analyze . --output ./evidence --require-role product-security --fail-on-missing-approvals --approvals ./evidence/approvals.json

The re-run exits 0 once every required role carries an active approval. Each grant, revocation, and override is a crash-consistent, audited operation (PB-01 covers an interrupted one).

Never fake quorum by editing the ledger — approvals are attributed human records, and an emergency --override is itself a recorded, attributed, time-boxable exception, not a bypass.

PB-09 — Unverified signature or unauthorized signer (deferred)

Status: deferred — the signature-verification surface is being reworked; this playbook lands, tested, with that work.

Until then: treat any bundle whose verification fails as untrusted, keep it for investigation, and confirm you hold the right public key before retrying. sbomflow verify-bundle exits 3 on a failed signature and never "partially trusts" a bundle.

PB-10 — Invalid bundle content (deferred)

Status: deferred — same rework as PB-09; the content-dimension playbook lands, tested, with it.

Until then: a failed content check means the bundle no longer matches its manifest. Re-export from the canonical output directory and investigate the mismatch; never pass a bundle whose extra or altered files are hidden.

PB-11 — Broken post-market projection (deferred)

Status: deferred — the post-market case workflow is experimental; its recovery playbook ships when that surface stabilizes.

Until then: case projections are rebuilt from the canonical hash-chained case journals, never edited in place.

See also

• Troubleshooting — symptom-first index for everything that is not a recovery scenario.
• Reviewer runbook — the normal-path decision workflow these playbooks return you to.
• Exit codes and the error reference — the stable contract every command above follows.
• CI runbook — branching on reason codes in pipelines.