Vulnerability Management

The shape of this work is a funnel with a cadence, not a checklist with an end.

Why the same artifact needs scanning forever¶

You built the image in March. It's pinned to a digest, it's byte-identical today, and nobody has touched it.

Whether it's safe to run is a completely different question in August than it was in March, and the reason has nothing to do with your code. Someone published research. A CVE got assigned. A vendor confirmed. A matcher added a rule.

The artifact is frozen. The knowledge isn't. So the scan isn't a thing you did — it's a thing you do, on a schedule, forever, to artifacts you already shipped.

Cadence¶

Most teams have this exactly backwards, because CI is where scanning was easiest to add.

The first row is the one that's usually missing. A scan wired into CI runs when you change something, which means an artifact you haven't touched in five months has been evaluated exactly once, against a vulnerability database that no longer exists.

Your scanner's silence is not evidence¶

There's a pipeline between "a vulnerability exists" and "your tool tells you," and every stage adds delay:

researcher finds it
  └─ embargo / coordinated disclosure        days to months
      └─ CVE ID assigned                     hours to weeks
          └─ published to NVD                often same day
              └─ NVD enrichment (CPE, CVSS)  days to weeks  ← the long pole
                  └─ matcher/DB adds a rule  hours to days
                      └─ your next scan      up to your cadence

A clean scan means nothing was known and matched at the time it ran. It does not mean you're not vulnerable. During the first days of a major disclosure, the gap between "everyone is talking about it" and "scanners reliably detect it" is real and it is where the damage happens.

This is why on-demand scanning matters and why you should be able to answer "do we run X?" without a scanner at all — from an SBOM inventory, by grep, in minutes. The scanner is the routine instrument. Inventory is what you use when the routine instrument is behind.

Triage at scale¶

The workflow below assumes one vulnerability. Your scanner will hand you twelve hundred.

That's not a tooling failure — it's the correct output of pointing a matcher at a real estate. The discipline is the funnel that turns it into a list a person can act on this week:

1,202  raw findings across all scanned artifacts
  ├─ −412  not actually present (backport false positives, see below)
  ├─ −301  not reachable (dev dependency, unused code path, disabled feature)
  ├─ −286  no fix available yet → accepted-risk register with a review date
  └─   203 real, fixable
        ├─ −168  swept by routine base-image rebuild + dependency bump
        └─    35 need a human decision
              └─ 6  this week

Every arrow is a decision, and each one needs to be recorded, not just made. An undocumented filter is indistinguishable from negligence six months later, and it's how the same 412 findings get re-triaged every quarter by someone who doesn't know they were already dismissed.

The two filters doing the most work are reachability and the backport problem. Both are below.

The funnel tells you what survives, not what to do¶

Read it again and notice what it doesn't produce. It ends at six things this week — a count. Nobody can act on a count. And the row doing the most work, −168 swept by routine base-image rebuild, is written as something that happens to you, passively, as though those findings evaporate on a schedule. They don't. Somebody rebuilds something.

The missing step is a transpose. The funnel is organised by finding; work is organised by action. Turning one into the other is the difference between a report and a plan, and it's mostly a GROUP BY:

The unit of remediation is not the vulnerability. It's the command that clears it.

Twelve hundred findings is not twelve hundred decisions. It is usually a dozen, and most of the mass sits in two or three.

What that looks like on a real fleet¶

From this homelab, unedited:

848 unaccepted fixable criticals across 52 images
  top  5 images = 33%
  top 10 images = 48%
  top 20 images = 72%

A long tail, which is what everyone expects, and not the interesting part. This is the interesting part — eight images sitting at exactly the same count:

open-source-licensing-guide     24
building-software-guide         24
keep-your-teeth                 24
honeybadger/web                 24
come-sit-with-me                24
brams-journey                   24
openresearchdataplanner         24

Eight identical numbers are never eight problems. All of them are FROM caddy:2-alpine, and the tag had moved without anything re-pulling it. 192 findings — 23% of everything unaccepted on the fleet — behind one docker build --pull.

Sorted by finding, that's 192 rows across eight projects, and it reads like eight weeks of work. Sorted by action, it's one line, and it needs no review because the tag doesn't change:

That is a page somebody reads in two minutes and forwards without rewriting. The list of 848 is not, and no amount of sorting by CVSS makes it one.

The third row was already done when it was written¶

That strikethrough is not an edit for honesty's sake. It is the failure mode this whole section has to survive, and it happened on first contact.

uptime-kuma was bumped 2.5.0 → 2.5.4 on 16 September, three days before that table was published. The container on the host is running 2.5.4. Git declares 2.5.4. Prometheus reports 2.5.0 with 47 findings, because the weekly scan ran against the image that was there before the redeploy — and it will keep reporting it until the next scan.

Nothing malfunctioned. Every component is behaving exactly as designed, and the plan was still wrong, because:

A remediation plan inherits the staleness of the scan it was built from. On a weekly cadence, any row can be up to seven days out of date — which means a plan generated from scan data alone will routinely ask people to do work that is already finished.

This is not a cosmetic problem. It is the specific way these documents die. A reader who checks one row, finds it already done, and concludes the page is stale will not check the other three — and the 192-finding row, which is real and costs nothing, dies alongside the mistake. One false row discredits the document, not just itself.

Declared, running, scanned¶

The fix is cheap and it isn't more scanning. Three sources already know three different things, and a plan should be built from all three rather than the slowest one:

Validate every row against the first two before emitting it. If the running image no longer matches the image the finding was filed against, the row is already resolved — drop it, or better, keep it marked done so the reader can see the plan is tracking reality rather than lagging it.

And the disagreements between those three are themselves findings. An image running something git doesn't declare is drift. An image git declares that nothing is running is a stale pin. Both are invisible to a scanner, which only ever sees the third column.

Two shapes that look identical and aren't¶

One more thing the first draft of that table got wrong: it listed a moving tag and a pinned version as though they were the same kind of work.

Both surface in a scanner as N findings on an image. They are not the same row, they don't go to the same person, and grouping them together is how the cheap one gets stuck behind the expensive one's review cycle.

Three things separate a plan from a report¶

1. Grouped by remediation, sorted by how much each action clears.
2. Each group carries its actual command, so handing it to whoever deploys is forwarding, not translating.
3. Decisions write back. A no becomes a dated entry with an expiry, so it drops out of next month's plan instead of being re-triaged by someone who doesn't know it was already dismissed.

The third one is the funnel's own rule — every arrow needs to be recorded, not just made — applied to the arrows nobody writes down, because "we'll get it in the next rebuild" feels like a decision that doesn't need recording right up until the rebuild doesn't happen.

Severity is not priority¶

CVSS is the number everyone sorts by and it's the wrong one to sort by alone. It describes the vulnerability's properties in the abstract — not whether anyone is exploiting it, and not whether it matters where you run it.

A CVSS 9.8 in a code path you don't call, on a host with no ingress, is not an emergency. A CVSS 6.5 on the CISA KEV list, reachable, on your edge, is tonight.

Practical sort order: KEV first, then EPSS above ~10%, then CVSS — filtered throughout by reachability and exposure. That single reordering usually cuts the "urgent" pile by an order of magnitude and makes it more accurate, not less.

The backport problem¶

This is the single largest source of false positives in enterprise Linux environments, and it will eat your credibility with whoever you're asking to do the patching.

Red Hat, Debian, SUSE, and Ubuntu backport security fixes without changing the upstream version number. openssl-1.1.1k-12.el8 contains fixes that upstream shipped in 1.1.1t. A scanner matching on version strings sees 1.1.1k, checks its database, and reports every CVE fixed between k and t.

All of them are wrong.

# Ask the vendor, not the version string
rpm -q --changelog openssl | grep -i CVE-2023-0286
# Debian/Ubuntu
zcat /usr/share/doc/openssl/changelog.Debian.gz | grep -i CVE-2023-0286

The fix is to make your scanner vendor-aware — Trivy and Grype both consume distro security trackers (RHSA, DSA, USN) and will use vendor fix status when configured to, rather than naive version comparison. If you're seeing hundreds of findings against a patched RHEL host, this is almost always why.

Getting this wrong doesn't just create noise. It teaches the people receiving your reports that your reports are wrong, which is much harder to undo than the misconfiguration.

The response workflow¶

Once something survives the funnel:

1. Triage — are we actually affected? Do we use the component, the affected version, and the affected functionality? A flaw in a logging library's UDP handler doesn't touch you if you only write files.

2. Assess — what's the blast radius? Which systems, what data, what's reachable from there. This is where your inventory does the work; if you're discovering topology now, you're already behind.

3. Decide.

4. Remediate. Prefer the routine mechanism — a base image rebuild or dependency bump that sweeps dozens at once — over a targeted patch that fixes exactly one. Emergency changes should be rare enough to stay special.

5. Verify what's deployed, not what's merged. This is where the loop actually closes, and where it most often doesn't. A merged PR is not a deployed fix. Re-scan the running artifact and confirm the finding is gone.

6. Document. What you decided, why, and when it gets revisited.

When you can't fix it¶

Sometimes there's no patch, or the upgrade breaks something you can't break right now. Accepting that risk is a legitimate engineering decision.

Accepting it permanently by neglect is not, and that's the default outcome unless you build against it.

Tools¶

# The pairing that matters: inventory once, match repeatedly
syft packages myimage:sha256-abc... -o cyclonedx-json > sbom.json
grype sbom:./sbom.json          # re-run this daily; the SBOM never changes

That second command is the whole chapter in two lines. The SBOM is generated once and never changes. The answer changes every day.

Quick Reference¶

Triage order¶

1. KEV — actively exploited, confirmed
2. EPSS > 10% — likely to be exploited soon
3. CVSS — severity, as a tiebreak only
4. Filter all of the above by reachability and exposure

Cadence checklist¶

• Deployed artifacts scanned daily, not just on commit
• Scanner is vendor-aware for distro backports
• Can answer "do we run X?" from inventory in minutes, without a scanner
• Filter decisions recorded, not just made
• Every accepted risk has an owner and a review date
• Verification re-scans the running artifact