Skip to content

Brass corridor lined with pipes and lamps, a large alarm bell above a door standing open at the far end, red light spilling across the wet floor

Incident Response

What you do when the answer is no.

Incident response is mostly a scoping problem wearing a technical costume. The hard part is almost never "how did they get in" — that usually falls out of a vendor writeup within a day. The hard part is what did this reach, and that question is answered by inventory work you either did months ago or are now doing at 2 a.m. with a flashlight.


The order that matters

Most small teams get this wrong in the same sequence, so it's worth stating plainly:

  1. Contain, then investigate. Understanding the malware completely is not a prerequisite for revoking the token it stole.
  2. Preserve before you clean. The instinct to delete the artifact destroys the evidence that tells you what it did. Copy first.
  3. Scope before you rotate. Rotating in the wrong order can lock you out of the systems you need to rotate next.
  4. Assume the credential is gone. Not "assume it might be." If it was readable, treat it as read.
  5. Write it down while it's happening. Your memory of hour two will be wrong by hour six, and the postmortem is where the actual value is.

Planned chapters

Chapter What it covers
The First Hour A concrete sequence for the moment you learn something is wrong, written for someone who is alone and has other responsibilities
Scoping the Blast Radius Turning "a CI runner was compromised" into a specific list of credentials, hosts, and downstream artifacts. The inventory questions you should be able to answer in minutes
Rotation Under Fire Dependency order, the bootstrap problem, and how to rotate without locking yourself out. Includes the credentials people consistently forget
Evidence and Forensics What to preserve, how, and for how long — proportionate to a homelab or small team, not a courtroom
Communications and Disclosure Who to tell, when, and how much. Includes the institutional version, where the answer involves people whose job title contains the word "counsel"
Writing the Postmortem Blameless in practice rather than as a slogan, and how to write one that changes something

You will be alone and it will be inconvenient

Institutional incident response assumes a team, a rota, and someone whose entire job is this. Most people reading this have none of those. You'll find out on a Saturday, from a Mastodon post, while you're doing something else.

Everything here is written for that case. Where a practice only makes sense with a staffed SOC, it says so, and offers the version that fits one tired person with a laptop.

Related from the supply chain side: The Script That Ate Itself is a postmortem about being confidently wrong, which is the most common failure mode in an investigation and the one worth reading before you need it.