Writing a One-Page Incident Runbook for a Small Team

Copy this into a shared document, a repo file or an on-call tool. Fill every bracket before the document counts as done.

INCIDENT RUNBOOK: first 30 minutes
Owner: [name] | Last tested: [date] | Review due: [date]

1. DECLARE
   Incident commander: [name] [phone] [backup: name + phone]
   Declare in: [channel or bridge link]
   Severity: SEV1 = users cannot complete a core action; SEV2 = degraded,
   workaround exists; SEV3 = internal only.

2. CALL TREE (call in this order, log the time of each call)
   1. [Name, role, phone]      reason: [owns the system]
   2. [Name, role, phone]      reason: [owns the data layer]
   3. [Name, role, phone]      reason: [customer-facing contact]
   4. [Vendor or provider support line, number, account ID]

3. FIRST THREE CHECKS (stop when one gives you an answer)
   Check A: [exact URL, pre-filtered for the last 60 minutes]
   Check B: [exact command, copy-paste ready]
   Check C: [exact query or status page URL]

4. CLOCK (write actual times next to each)
   00:05 acknowledged in the channel by a named person
   00:15 decision logged: escalate or hold
   00:30 wider group notified (template message below)

5. ESCALATE WHEN
   [condition 1] [condition 2] [condition 3]

6. ROLLBACK / MITIGATE
   [exact command or console path] Expected result: [what you should see]

7. STATUS MESSAGE TEMPLATE
   "[Service] is [down/degraded]. Customers [impact]. Started [time].
   We are [action]. Next update by [time]."

Two blanks deserve extra care. The first is severity, because a team that argues about severity at minute four has lost four minutes. Define severity by observable customer impact, not by internal anxiety: whether users can complete a core action. The second is the status message template, because writing prose under stress is slow and tends to produce messages with no time commitment.

The "stop when one gives you an answer" instruction in section 3 is deliberate. Three checks are not a checklist to complete; they are a short decision tree. If check A shows the database connection pool is exhausted, you skip B and C and go to mitigation. Teams that treat first checks as a sequence to finish often spend the whole 30 minutes confirming a cause they already found.

A caller tree is only useful if it has phone numbers, not handles. Chat handles are fine for the first attempt, but the runbook exists for the case where the usual channel is exactly what is broken, or where the person is away from the desk.

Build it with three rules.

First, order by system ownership, not seniority. The first call should be the person who can restart or roll back the thing that is failing. An engineering manager who has to ask someone else is a second call.

Second, include a reason next to each name. "Owns the payments database" tells a tired responder whether the call is even relevant. A list of five names with no reasons gets used as a notification list, which produces a five-person conference call and no action.

Third, include the vendor path with the account ID written down. Provider support lines ask for the account identifier, and hunting for it mid-incident is a classic time sink. Write the number, the account ID, and the support tier next to each other.

The caller tree also has a backup line for every primary. A single unbacked page is a single point of failure; escalation policy guidance calls out eliminating single points of failure and assigning ownership as core design goals in this escalation policy guide.

Small teams usually find that three to five numbers is enough. If you are writing more than five, the list has become a directory. Keep the directory elsewhere and keep the tree to the people who can change the state of the failing system.

The test for a first check is simple. Could a tired person who has never seen this alert run it in under two minutes without opening another document and without writing a query from scratch?

That rules out "look at the logs" and "check the dashboard." It allows a pre-filtered URL, a copy-paste command, and a status page link. On-call templates that skip this detail are the ones that fail in practice, because the dashboard link was changed and nobody updated the runbook as described in this on-call runbook write-up.

A workable set of three checks for a small service is: one customer-facing signal, one infrastructure signal, one dependency signal.

# Customer-facing signal: recent 5xx share, last 60 minutes
# (example using a log search API; use your own provider's tool)
curl -s "$LOG_API/query" \
  -H "Authorization: Bearer $READONLY_TOKEN" \
  -d '{"service":"payments","window":"60m","filter":"status>=500"}' \
  | jq '.error_rate, .count'

# Infrastructure signal: is the database accepting connections
psql "$REPLICA_DSN" -c "select count(*) from pg_stat_activity;"

# Dependency signal: what the provider says right now
curl -s "$PROVIDER_STATUS_URL" | jq '.status.indicator'

Two practical notes. Use a read-only token so the commands in the runbook cannot make things worse, and store that token in the same place the on-call engineer already gets credentials, not in the document. And write the expected output next to each command: "error_rate above 0.05 means customers are affected" is the sentence that turns a number into a decision.

Link to dashboards rather than embedding screenshots. A screenshot goes stale silently; a link either works or fails loudly, which is the better failure.

Escalation is where small teams drift most. The instinct is to keep trying, because escalating feels like admitting defeat, and because the person who owns the system would rather solve it alone than wake a colleague. The clock removes that judgment call.

The three milestones that work for a small team:

The 5-minute acknowledge exists to kill the silent gap where an alert fires and everyone assumes someone else has it. The 15-minute decision exists to stop heroic solo debugging: if the first checks did not produce a fix path in 15 minutes, the odds that the next 15 will be better are low. The 30-minute notification exists because the cost of telling stakeholders early is small, and the cost of telling them after they have discovered the issue themselves is reputation.

Notifying early is also consistent with the guidance that an incident response plan should be invoked even when an incident is suspected to be a false alarm, on the grounds that near misses drive improvement in CISA's small business cyber guidance. The same logic applies to the clock: acknowledge fast, decide fast, notify fast, and accept that some of those acknowledgements will be for noise.

Set escalation triggers as observable conditions, not feelings. "We have made no change that reduced the error rate in 15 minutes" is a trigger. "It feels bad" is not. Two or three triggers are enough:

• Customer-facing error rate above an agreed threshold for 10 minutes.
• A mitigation attempted and failed.
• A second service or a second customer segment affected.

Write them as conditions with numbers, and define who decides when a condition is met: the incident commander, alone. Consensus at minute 16 is not a plan.

Here is the same template filled in for a small team running a payments API. Names and numbers are placeholders.

INCIDENT RUNBOOK: payments API, first 30 minutes
Owner: Dana R. | Last tested: [drill date] | Review due: [date]

1. DECLARE
   IC: Dana R., 555-0142 (backup: Priya S., 555-0187)
   Channel: #inc-payments (bridge link pinned)
   SEV1: checkout returns 5xx for more than 5% of requests for 10 minutes
   SEV2: card authorizations degraded, retry path works

2. CALL TREE
   1. Dana R. 555-0142   owns the payments API service
   2. Priya S. 555-0187  owns the database and connection pool
   3. Marco L. 555-0199  owns customer support, sends status messages
   4. Payments provider support, 555-0100, account ID [ID]

3. FIRST THREE CHECKS
   A. Error-rate dashboard, pre-filtered to 60 minutes: [URL]
   B. psql "$REPLICA_DSN" -c "select count(*) from pg_stat_activity;"
      Expected: under 80. Above 150 means pool exhaustion.
   C. Provider status page: [URL]

4. CLOCK
   00:05 Dana acknowledged in #inc-payments
   00:15 decision logged: error rate flat, no mitigation yet, hold
   00:30 Marco posts status to customers and support

5. ESCALATE WHEN
   error rate over 5% for 10 minutes after one mitigation attempt
   connection count over 150 after a restart
   a second region shows the same errors

6. MITIGATE
   Restart the API worker group: [console path]
   Expected: error rate returns under 1% within 4 minutes

7. STATUS TEMPLATE
   "Card payments are failing for some customers since [time]. Retries may
   succeed. We are restarting part of the payment service. Next update by
   [time]."

Walking the clock with this fill:

At minute zero the alert fires on the error-rate condition. At minute two Dana acknowledges and starts check A: the dashboard shows 5xx at 9% for the last 12 minutes, concentrated on card authorization. That is SEV1 by definition, so the decision at minute 15 is effectively made already, but the clock keeps it honest: Dana writes "escalating, SEV1, provider check pending" in the channel.

Check B returns a connection count of 173, well above the 80 baseline and above the 150 trigger. That single number answers the question, so checks beyond A and B are skipped. Dana calls Priya at minute six, because the trigger condition (connection count over 150) is met regardless of the clock.

Priya identifies a stuck worker group holding connections. The mitigation is a restart of that group, run from the console path in section 6. Expected result is written down, so at minute nine Dana can compare: error rate down to 3% and falling. At minute 12 it is under 1%.

At minute 15 Dana logs the hold decision: "mitigated, monitoring." At minute 30 Marco posts the status message to customers and support, using the template with a next-update time.

Total time to mitigation: under ten minutes, because three numbers and two phone calls replaced an hour of discussion. The runbook did not make the team smarter. It removed the discovery work from the worst possible moment.

Note what the worked example leaves out: no root cause analysis, no postmortem, no customer credit decisions. Those belong later and in a different document. The one page covers only the first 30 minutes.

An untested runbook is a hypothesis. The cheapest test is a 30-minute tabletop drill, and it doubles as training for whoever is on call next.

Run it like this:

1. Pick one scenario from your incident history and one from a scenario you fear, such as a provider outage. Write a one-paragraph alert as the starting point.
2. Put the incident commander role on someone who is not the usual owner. The usual owner knows too much to expose gaps.
3. Start a timer. Read each section aloud. Every time someone says "where is that?" or "who is that?", mark the line. That is a defect.
4. Forbid improvisation. If the runbook says to call a number, call it. Numbers that ring out or route to a dead extension are findings worth having in a drill, not at 3 AM.
5. Stop at 30 minutes regardless of whether the scenario "resolved." Record how many clock milestones were hit on time and how many lines needed a fix.

Facilitators should be explicit that tabletop exercises exist to build reflexes for the real event, which is why leadership attendance is part of the guidance in CISA's small business guidance. Small teams get the same benefit with far less ceremony: one scenario, one timer, one page of notes.

Keep the runbook current with three triggers for review:

That third row is the one teams forget. A migration, a dashboard rebuild, a support tier change or a staff departure all invalidate a line. Attaching the runbook check to the change ticket, rather than to a calendar, is the difference between a document that is current and one that is merely reviewed.

If your team is small enough that the runbook is one person's memory with extra steps, the fix is the drill, not more prose. Two drills a year with three to five people is enough to keep a one-page document honest, and it costs less than one hour of engineering time per drill.