Skip to content

The same brass engine twice: on the left sealed under glass and still, on the right running hot and venting steam

Build Time vs Run Time

The idea: A build-time gate answers a question about the past. A runtime monitor answers a question about the present. They can be pointed at the identical artifact and disagree, and both be correct.


The same bytes, two answers

Scan a container image in March. Clean — no critical findings, pipeline green, ship it.

Scan the same image in December. Same registry, same tag, same digest, not one byte different. Seventeen criticals.

Nothing happened to the image. Nobody rebuilt it, force-pushed it, or replaced it. Both scans ran correctly and both reported the truth. What changed was everything around the artifact: a researcher published, a CVE got assigned, a vendor confirmed, an exploit landed in a Metasploit module, and a matcher somewhere added a rule.

This is the whole chapter. The artifact is immutable. Its safety is not.

Every practice in this book falls out of taking that seriously.

The canonical case

In 2019 you added log4j-core-2.11.1.jar to a project. It was a good choice — the standard logging library for the JVM, maintained, ubiquitous, boring in the way you want infrastructure to be boring. Any review would have approved it. Any scanner would have passed it.

On December 9, 2021, the same file — same SHA, never modified, never rebuilt — became the most consequential vulnerability of the decade.

Nobody made a mistake in 2019. The decision was correct when it was made and it stayed correct until it didn't, and the moment it stopped being correct had nothing to do with anyone on your team.

If your security model is "we evaluated it carefully before we shipped it," log4shell is the counterexample that ends the argument.

Why gates feel like security

A gate is enormously satisfying to build.

It has a finish line. It produces a boolean. It goes green. You can put it in CI, watch it block a bad PR once, and feel the system working. It's the kind of work that ends.

A monitor never ends. It produces a stream instead of a verdict, its output grows whether or not anything is wrong, and it demands attention on a schedule that doesn't respect your calendar. Nobody has ever felt the satisfaction of finishing a monitor.

So teams build the gates, watch them go green, and reasonably conclude they've done security. This isn't stupidity — it's an incentive gradient. Gates are cheap, bounded, and legible to management. Monitors are expensive, unbounded, and their success looks like nothing happening.

The gate is the easier half and the half with a clear ending, which is exactly why it gets built first and mistaken for the whole job.

The test

Here's how to sort any control you own:

The one question

If nobody ever rebuilds or redeploys, will this control ever produce a different answer than it does today?

  • No → it's a gate. It evaluated a moment. It is finished.
  • Yes → it's a monitor. It's evaluating the present, forever.

Run it against your actual pipeline and the results are usually uncomfortable.

Control Gate or monitor Why
License compatibility check Gate The license text doesn't change. Answer stable for years
Signature / provenance verification Gate Either the artifact came from that pipeline or it didn't. Settled at build
npm ci against a committed lock file Gate Excellent gate. Resolves exactly what you tested. Says nothing about next Tuesday
Pinning a base image to a digest Gate Guarantees you get the same bytes. Makes no claim those bytes are safe
Secret scanning on a diff Gate This commit did or didn't contain a key
SBOM generation Gate A description of what you built, at the moment you built it
SBOM matching against a vuln database Monitor Same SBOM, new answers, forever
Vulnerability scanning in CI Gate, disguised as a monitor It scanned against that day's database and never ran again
Scheduled re-scan of deployed artifacts Monitor The point is that it re-runs
Certificate expiry checking Monitor The cert doesn't change; the date does
Egress monitoring on a running host Monitor Behavior is a present-tense property
Config drift detection Monitor The baseline is a gate; detecting departure from it is not

The row worth staring at is vulnerability scanning in CI. Nearly every team has it. Nearly every team believes it covers them. It is a gate that produces monitor-shaped output, which is the most misleading combination available — it emits CVE findings, which feel like ongoing surveillance, while actually being a photograph taken on the day of the build.

What changes while the artifact doesn't

Three categories, and they fail differently:

What's known. A CVE gets published. NVD enrichment lands weeks later. A proof of concept appears. CISA adds it to the Known Exploited Vulnerabilities catalog and it stops being theoretical. Your artifact was always vulnerable; the world just found out. This is the log4shell shape.

What's reachable. You didn't change the service, you changed its neighborhood. A firewall rule opened. A new integration handed it a credential it didn't have before. It moved from a lab subnet to one that routes. The code is identical and the blast radius tripled.

What's true elsewhere. A vendor in your chain got breached and your credentials were sitting in their logs. A maintainer's account was taken over. A certificate expired. This is the class where nothing in your estate changed at all, and you are nonetheless compromised — see the LiteLLM compromise, where organizations that had done nothing wrong that week found out in July about credentials taken in March.

Gates cover none of these. They can't — they ran before any of it happened.

The inverse mistake

Having established that gates are over-trusted, the opposite error deserves equal billing, because it's the one that kills real monitoring programs.

Treating a monitor's output like a gate's verdict.

A gate says yes or no and you act on it. A monitor produces a stream, and a stream needs entirely different handling: thresholds, deduplication, suppression that expires, and an explicit decision about what you will not act on.

Point a scanner at a fleet, treat all 1,202 findings as blocking, and within two weeks nobody reads the output. The monitor still runs. It still works. It is now decorative, and you have a green checkmark next to a thing you aren't doing — which is worse than no monitor at all, because it occupies the slot where a real one would go.

A monitor without a triage discipline isn't security, it's a subscription to bad news. That discipline is Operations, and its runbook form is CVE triage.

Half-life

The useful refinement: gates don't stay true for the same length of time.

Gate Half-life What it means practically
License check Years Re-check on dependency change, not on a timer
Provenance / signature Stable But re-verify what's deployed, not just what you built
Lock file resolution Weeks to months Until a dependency you use gets an advisory
Vulnerability scan Days Assume it's stale almost immediately
Reachability assessment Until the next network change Which nobody tells you about

So for every gate you own, two questions:

  1. What's its half-life?
  2. What monitor covers it after that?

If the answer to the second one is nothing, you don't have a gap in your monitoring — you have a control that quietly expired and is still displaying its last result.

The line to remember

A scan that passed is not evidence. It's a timestamp.

It tells you that at one moment, against the data available then, nothing was found. That statement gets less true every single day, and nothing will notify you when it stops being true at all.

Which is the entire reason this book exists separately from the supply chain guide. That book is about making the gate as good as it can be — and it should be good, because a bad gate makes everything downstream harder. This one is about the fact that a perfect gate still only ever answers a question about the past.


Nothing changed, and that was the problem

The version of this I keep living through isn't dramatic. Nobody gets breached. Somebody just asks a question I can't answer.

"Is that host still on the vulnerable version?" And I know exactly what we deployed, because it's pinned and in git and the pipeline was green. What I don't know is what that pin means now. The pipeline that told me it was fine ran in March and has not had an opinion since.

The habit that fixed it wasn't a tool. It was re-reading my own green checkmarks and asking what date they were actually from. Most of them turned out to be historical documents I'd been treating as live telemetry.

A pipeline this guide's own author broke reported green for three months while the thing it was supposed to be doing was completely broken. Same lesson from the other direction: green means nothing failed in the way you were watching for. It has never meant more than that.