The Hole Shaped Exactly Like Good News¶
On 18 September 2026 I ran one query against the homelab's Prometheus, expecting to confirm a number before quoting it:
Two hosts came back. The fleet has seven.
What every instrument said¶
Nothing was wrong, according to everything built to tell me if something was wrong:
| Instrument | Reading |
|---|---|
| Firing alerts | none |
SBOMScanStale | quiet |
SBOMScanErrors | zero |
| Last scan | 2.6 days ago — well inside a weekly cadence |
| node-exporter targets | 7 up, 0 down |
| cadvisor targets | 7 up, 0 down |
| Grafana SBOM dashboard | populated, current, plausible |
The scan was fresh. The exporters were healthy. The dashboard had numbers in it. There was no error to find because nothing had errored.
What was actually true¶
Five of seven hosts had never produced vulnerability data at all.
| host | images scanned | last scan |
|---|---|---|
| xdocker01 | 54 | 2.6 days ago |
| xmonster | 9 | 2.9 days ago |
| xtower | — | no data |
| xmuse | — | no data |
| xdocker03 | — | no data |
| xdocker02 | — | no data |
| xalien | — | no data |
Sixty-three images under scan. An inventory taken six weeks earlier counted 231 across the fleet. Image counts drift — containers come and go — but no amount of drift explains five hosts contributing zero.
My first version of this page stopped here and called it a regression: a rollout that worked in August and quietly decayed. That was wrong, and the way it was wrong is the more useful half of the story.
The rollout never landed¶
The scanner was fine. It had never been given the chance to run.
A parallel session took the obvious next step I hadn't — run the deploy in dry-run and read the error instead of inferring one:
Confirmed independently, host by host:
The unattended deploy path needs passwordless sudo. On five hosts it doesn't have it, has never had it, and every unattended fleet playbook — not just the scanner — has been failing there the same way since August.
The timeline closes it off completely:
- The five silent hosts booted 3 August and have not rebooted since.
- The rollout ran 11 August.
- A sudoers file written on the 11th would still be sitting on disk today.
Nothing was lost to a reboot. Nothing decayed. It was never written.
The deploy succeeded because a human was standing there
The August rollout ran interactively, with someone present to answer the become-password prompt. It worked — that day, on every host, exactly once. Then it was recorded as deployed.
A deployment that requires a human to supply a credential is not a deployment. It is a demonstration that the thing can work, performed under conditions that will never occur again. The gap between those two is invisible on the day, because the day is the one time the conditions hold.
The control group wasn't one¶
The instinct was that the two working hosts proved the rollout was sound and five hosts had drifted off it. They proved nothing of the kind.
xdocker01 %sudo ALL=(ALL:ALL) NOPASSWD: ALL ← /etc/sudoers line 50. A GROUP grant.
xmonster /etc/sudoers.d/xram-ansible ← mtime 2026-04-25. Four months early.
Note what xdocker01's actually is: a grant to the %sudo group, not to a user or a service account. It confers passwordless root on whoever happens to be in that group — today one person, tomorrow whoever gets added — and it has been quietly underwriting every automated job on the fleet's primary Docker host. That is a load-bearing permission nobody chose on purpose.
Neither file was created by the rollout. The rollout created a sudoers entry on zero hosts. The two that kept scanning were configured for unrelated reasons months earlier — one by hand, one in April — and would have kept scanning no matter what August did.
So the unattended path was never exercised on a host that didn't already have sudo. The two green rows in the dashboard weren't evidence the deploy worked. They were the two machines where it didn't need to.
Ask what your successes have in common
A partial rollout with some hosts working looks like a rollout with a few stragglers. It is often a rollout that only ever worked where it wasn't needed.
The cheap check is to stop looking at the failures and interrogate the successes: what do these have in common, and did my change actually cause it? If every working host was already working beforehand, you have not deployed anything — you have confirmed a pre-existing condition and written it down as a result.
And a fourth thing, found only because the fleet got checked host by host: xmuse isn't a sudo failure at all. Key authentication fails outright — Permission denied (publickey,password). It had been filed in the same bucket as the other four because it produced the same symptom, which is exactly what a symptom-shaped inventory does to you. It is also the fleet's second-busiest machine, and the one whose documented risk profile is ad-hoc containers nobody spins down.
Why nothing fired¶
Detection failed separately from the deploy, and for reasons worth stealing.
The staleness rule takes a max().
One host scanning on schedule satisfies that expression on behalf of every host that isn't. xdocker01 alone keeps this rule silent forever.
A per-host rule cannot fire for a host that reports nothing.
- alert: NewFixableCriticalCVEs
expr: sum by (host) (grype_vulnerabilities_fixable_critical_unaccepted_total)
> sum by (host) (...) offset 1w) + 2
sum by (host) was the correct fix for an earlier problem — a global sum() meant onboarding a new host read as hundreds of new criticals overnight. It works as intended. It also means a host with no series produces no group, and a rule with no group to evaluate does not fire, does not error, and appears nowhere.
A threshold alert can only see values that a metric has. A metric that never existed has no values to breach.
| Alert | What it watches | What it cannot see |
|---|---|---|
SBOMScanStale | age of the newest scan, fleet-wide | any individual host going dark |
SBOMScanErrors | errors during a scan | a scan that never started |
NewFixableCriticalCVEs | per-host CVE deltas | a host with no series to delta |
NewCriticalCVEs | per-host critical deltas | same |
Every rule watches something real. Together they cover error, staleness and regression — and leave absence uncovered, because absence is the one state that doesn't produce a number.
The sentence that was already in the file¶
The alert rules carry this comment, written weeks earlier by someone who had just spent an afternoon on a related problem:
An image that fails to scan leaves a hole shaped exactly like good news. syft gets OOM-killed on large images, the image drops out of the results, and the host's vulnerability count goes DOWN — indistinguishable from someone having patched it, unless something is watching the error counter.
Correct diagnosis, written down, in the same file as the alerts that missed the larger version of it. It was reasoned at the scale of one image inside one scan, and the counter-watching fix it describes works perfectly there.
Nobody asked what the same sentence means for a whole host.
Scale the failure mode, not just the fix
When you write down a failure mode, ask what the sentence means one level up. A missing package inside an image. A missing image inside a host. A missing host inside a fleet. A missing fleet inside a report.
The mechanism is identical at every level — something stops being counted, and the totals improve. Fixes almost never generalise on their own, because a fix is written against the instance you were angry at.
What actually closes it¶
Two gaps, two different kinds of answer.
Detection. Coverage cannot be monitored from inside the data, because the data is exactly what's missing. It needs a rule that knows the expected answer from outside:
# Breadth, not freshness. Checked against node-exporter's host set rather
# than grype's own data — a metric cannot be the authority on its own absence.
- alert: SBOMHostCoverageDropped
expr: count(count by (host) (grype_scan_timestamp_seconds))
< count(count by (instance) (up{job="node-exporter"}))
for: 2h
# Per-host staleness, so one healthy host stops covering for the rest.
- alert: SBOMHostStale
expr: (time() - max by (host) (grype_scan_timestamp_seconds)) > 864000
for: 1h
Deployment. No alert would have prevented this one, only shortened it. The deploy needs to fail loudly when its preconditions are absent, rather than succeeding under a human and being filed as done. That means a dry run in CI against every target, and treating "prompted for a credential" as a failure condition in an unattended path — because that's precisely the state that works on the day and never again.
The check that would have found either one is embarrassingly cheap, and it's the same check both times: if a label that should vary only ever carries one or two values, that's a coverage boundary, not a fleet in agreement.
What this cost¶
Nothing yet that anyone can prove — which is its own finding. Five hosts spent six weeks unscanned while a dashboard reported on them by omission, and the wider blast radius is every unattended playbook that has been failing on those same hosts since August for the same reason, unnoticed for the same reason.
The number that was being reported went up: 848 unaccepted fixable criticals, from a baseline of 650 in August. That rise is measured across two hosts. Whatever the other five accumulated is not in it.
For six weeks this system answered "are we covered?" with a number that meant "here is what replied." Those are different questions, and only one of them had an alert.
How this page was wrong
The first published version of this page said coverage had regressed. It hadn't. The rollout never landed, and I'd inferred decay from a shape that looked like decay without running the one command that would have said so — sudo -n true, on each host, which takes about four seconds.
The root cause belongs to the session that owns fleet scanning, which chased it rather than accepting the writeup. Three details in the first report back were also off, including which hosts had the sudoers file and why, and those only surfaced because the claim got checked instead of quoted.
That is the whole method, and it is not a flattering one: a plausible explanation that matches the evidence is not a root cause. Mine matched every number on this page and was still wrong about what happened.
Where this sits
The mechanics of scanning on a cadence — and why a clean scan is not evidence of safety — are in Vulnerability Management. This page is the response-side companion: what happens when the instrument never started, and every dashboard agrees that things are fine.