Secrets Management

The premise of this chapter: you will not prevent every leak, so design for the one you don't catch.

Assume the leak¶

The gate-side advice — don't hardcode, don't commit, don't bake into layers — is all correct and all insufficient, because it only covers the leaks you cause.

Consider how credentials actually escaped in the incidents of the last two years:

• A compromised GitHub Action read secrets out of the runner's memory and printed them to public workflow logs
• A poisoned vulnerability scanner ran inside CI — as it was designed to — and harvested every credential in reach
• A malicious PyPI package executed at interpreter startup and swept ~/.aws/credentials, ~/.kube/config, and the environment
• A vendor with a copy of your API key got breached, and you found out from a news article

In none of those did anyone commit a secret to git. Every one of those organizations could have had flawless authoring hygiene and identical outcomes.

So the operational question isn't how do I stop secrets escaping. It's:

When a credential escapes without my knowledge, how much damage can it do, for how long, and how will I find out?

Three properties answer that: scope, lifetime, and observability. They're the whole chapter.

The inventory you probably don't have¶

Start here, because everything downstream depends on it and almost nobody can do it cold.

Can you list every credential in your environment, what it grants, and what would break if you revoked it right now?

Most people can't. They can list the ones they think about — the cloud keys, the database password — and then discover during an incident that there's a service account from 2023 in a cron job, a deploy key on a repository nobody maintains, and a personal access token belonging to someone who left.

The artifact you want is boring:

The columns that matter are What it reaches and Lifetime, and the value of the table is entirely in how uncomfortable it is to fill in honestly. Every row you can't complete is a row you'd be guessing about at 2 a.m.

Lifetime is the control that matters most¶

If you fix one thing, fix this.

A long-lived credential converts any single leak, ever, into indefinite access. A credential that expires in fifteen minutes converts the same leak into a fifteen-minute problem — and, crucially, does so without anyone noticing, deciding, or responding.

That's the key property: short lifetimes work even when your detection fails completely. Every other control in this chapter assumes someone eventually notices. Expiry doesn't.

Prefer identity over secrets. OIDC federation between CI and cloud providers, Kubernetes workload identity, and cloud instance roles all replace "a string that grants access forever" with "prove who you are, get something that expires." When it's available, take it — it removes the thing an attacker steals rather than protecting it better.

The remaining long-lived credentials are then a short, deliberate list rather than an accident, and that list is what your rotation schedule is for.

Scope: know the blast radius before it matters¶

Secrecy is a control that fails silently. Scope is a control that holds after secrecy fails.

A leaked credential scoped to write one object storage bucket is an incident. The same leak on a credential with account-wide admin is a very different week. The difference was decided months earlier by whoever provisioned it, usually by taking the default.

Practical rules:

• One credential per consumer. Shared credentials make revocation a negotiation — you can't kill it without knowing everything that breaks.
• Scope down to the operation, not the service. "Write to this bucket" beats "S3 access." "Push to this one package" beats "publish."
• Separate read from write, and both from admin. Most workloads need one.
• No standing admin. If a human needs it, make it something they request and it expires.
• Kubernetes service accounts default to more than you think. A token that can create pods in a namespace is a foothold, not a read credential — which is exactly what the LiteLLM payload used it for.

Rotation as a standing practice¶

Rotation has a bad reputation because most teams only ever do it under duress, discover it's harder than expected, and conclude it's not worth it.

It's worth it, and the reason is counterintuitive: the point of scheduled rotation isn't hygiene, it's rehearsal. A rotation you've done four times calmly is a rotation you can do at 3 a.m. with an active incident. One you've never done is a research project you'll be starting at the worst possible moment.

Order matters, and getting it wrong locks you out. Before rotating anything, know:

1. What issues this credential? If you rotate the issuer's credential first, you may not be able to issue the replacement.
2. What consumes it? Every consumer needs the new value before the old one dies.
3. Is there an overlap window? Two valid credentials at once turns a hard cutover into a rolling one. If the provider supports it, always use it.
4. What's the rollback? Know how to put the old one back before you kill it.

The general safe shape:

1. Issue new credential alongside the old      (both valid)
2. Update consumers, one at a time             (verify each)
3. Confirm zero use of the old credential      (last-used telemetry)
4. Revoke the old credential
5. Verify nothing broke, with the old one gone

Step 3 is the one people skip, and it's the one that prevents the outage. If you can't observe whether the old credential is still being used, you're guessing about step 4 — which is the next section.

Observability: noticing use¶

This is the half the gate-side chapter has no concept of, and the half that turns secrets management from a filing exercise into an actual monitor.

What you want to be able to answer:

That last one is underrated. Failed authentication attempts against a credential you already killed are among the highest-signal, lowest-noise alerts available — legitimate consumers get updated, so continued attempts mean somebody has a copy from before.

Most providers expose last-used timestamps (AWS IAM credential reports, GitHub token last-used, registry audit logs). Very few teams look at them on a schedule. Reading that report quarterly and revoking everything untouched for 90 days is maybe an hour of work, and it shrinks your inventory faster than any policy.

When one leaks¶

You will usually learn about this from someone else, and later than you'd like — a vendor advisory, a news article, a note from a colleague, or an FBI FLASH advisory in July about credentials taken in March.

The sequence:

1. Revoke first, investigate second. Understanding the full attack is not a prerequisite for killing the token. Do not let analysis delay revocation.
2. Assume it was used. Not "might have been." If it was readable, treat it as read. Scope your response to what the credential could do, not to what you can prove it did.
3. Scope the blast radius. What did this credential reach? What credentials were reachable from what it reached? Leaked credentials chain — a CI token that reaches a registry that holds an image that carries another credential is three problems.
4. Rotate outward from the leak. The compromised credential, then anything it could have read, then anything reachable from there. This is where your inventory earns its existence.
5. Audit the exposure window. Provider logs from the earliest possible compromise, not from when you found out. Those dates are usually months apart.
6. Check for persistence. A credential is often used to establish access that outlives it — a new user, an added SSH key, a webhook, an OAuth grant, a scheduled job. Revoking the original credential does nothing about any of those.

The public-repo case remains as bad as it ever was. If a secret reached a public repository even briefly, assume harvest. Bots scan continuously; the window between push and harvest is measured in seconds. Removing it from history does not remove it from clones, forks, or caches. Revoke, don't tidy.

Storage, briefly¶

The gate-side chapter covers this properly; here's the runtime-relevant summary.

The thing to notice: every row except the last one is a place where a credential sits, waiting to be read by whatever else is running there. That's the argument for the last one.

Quick Reference¶

The three properties¶

Incident sequence¶

• Revoke before investigating
• Assume used, not "may have been used"
• Scope blast radius, including credentials reachable from it
• Rotate outward from the leak
• Audit logs from earliest possible compromise, not from discovery
• Hunt persistence — users, keys, grants, jobs, webhooks
• Verify what was published, if it could publish

Detection tools¶