SRE Runbooks

30 runbooks covering the most common alert categories an SRE encounters. Each follows the same structure: severity → what this means → diagnostic steps → common causes & fixes → escalation → post-incident.

When an alert fires in Mattermost, its post includes a runbook_url annotation pointing at the matching page here. Click the link in chat, you land on the runbook, you work the alert.

Categories

Compute & containers — High CPU Usage, High Memory Usage, CPU Throttling High, Pod CrashLoopBackOff, Pod Not Ready, Image Pull BackOff, Pods Unschedulable, Deployment Replicas Unavailable, Node Not Ready.

Application — High HTTP 5xx Error Rate, High API Latency, Service Endpoint Down, Request Rate Anomaly.

Database — Database Connectivity Loss, Database Replication Lag, Database High Latency, Postgres Connections Near Max.

Storage — Persistent Volume Full, Disk Fill Rate High.

Networking — Ingress High 5xx, Certificate Expiring Soon, DNS Resolution Failure.

Observability — Prometheus Scrape Target Down, Alertmanager Notification Failure.

Security — Unexpected Container Image, API Server Auth Failure Spike, Privileged Container Started, Interactive Shell in Container, RBAC Privilege Escalation, Security Tooling Down.

Use the sidebar on the left to navigate. Use the search box up top to jump straight to a runbook by name or by error string.

Authoring conventions

If you're writing a new runbook, copy runbooks/TEMPLATE.md as the starting point. Fill in every section — don't leave placeholder text that might get read mid-incident. Use real kubectl commands with real namespaces, not <NAMESPACE> placeholders. The on-call wants to copy-paste.