High CPU Usage

!!! danger "Severity: Warning → Critical at sustained levels" Target response: 15 minutes for warning, 5 minutes for critical. Sustained CPU pressure causes user-visible latency, request queuing, and eventual request timeouts.

What this alert means

A container, pod, or node has been running above 80% CPU utilization for 10+ minutes. The alert fires when:

sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total[5m])
) > 0.8

This is per-container CPU as a fraction of allocated limit, not a fraction of node capacity. A pod with a 2-core limit using 1.6 cores sustained hits this threshold, even if the node has 30 idle cores.

Sustained CPU above the limit causes:

• CFS throttling — Linux's CPU scheduler caps the container at its limit, pausing threads until the next 100ms accounting window. Visible in container_cpu_cfs_throttled_seconds_total.
• Request latency — threads waiting on the scheduler can't serve requests, so p95/p99 latency spikes.
• Memory pressure — when CPU-bound code runs longer, it holds allocations longer, pushing the heap up.

The alert is warning-severity by default; if a container is over 95% for 20+ minutes, a sibling rule (ContainerCPUCritical) fires at critical-severity.

Quick diagnostics

Three commands to run before reading further:

# WHERE: shell with kubectl context set to the affected cluster.
# WHAT: top 20 pods cluster-wide by CPU, sorted descending.
# READ: if the alert's <pod> is at the top with millicores in the
#   thousands, that confirms the source. If the alert's pod is mid-
#   list, look at neighbors that may be CPU-starving it (noisy
#   neighbor on the same node). The CPU column is current usage,
#   not limit — compare against the pod's CPU limit (from
#   `kubectl describe pod`) to know if you're at the cap.
kubectl top pods -A --sort-by=cpu | head -20

# WHERE: Grafana → Explore (Prometheus data source) or Prometheus /graph.
# WHAT: CFS throttle time per container per 5 minutes, as a rate.
#   CFS throttling = Linux scheduler capping the container at its
#   CPU limit. The container's threads literally stop running until
#   the next 100ms accounting window. That IS the user-visible
#   latency cause for CPU alerts.
# READ: 0 = no throttling, healthy. >0.1 (rate of 100ms+ throttled
#   per second) means the container is throttled 10%+ of the time —
#   raise the CPU limit or shrink the workload. Filter to the
#   specific failing container by appending a label selector:
#     rate(container_cpu_cfs_throttled_seconds_total{namespace="<namespace>",pod="<pod>"}[5m])
rate(container_cpu_cfs_throttled_seconds_total[5m])

# WHERE: shell with kubectl context set.
# WHAT: last 5 rollouts of the deployment in <namespace>. <namespace>
#   is filled in by AM at alert time; replace `<deployment-name>`
#   with the actual deployment from the alert's pod (the pod name
#   typically has the deployment name as a prefix).
# READ: if REVISION N was created in the last ~30 minutes and the
#   CPU alert started after, you've found the cause. Roll back with
#     kubectl rollout undo deployment/<deployment-name> -n <namespace>
#   If no recent deploys, the cause is upstream (traffic spike,
#   downstream slowness, or a config change applied via a different
#   path like a ConfigMap reload).
kubectl rollout history deployment -n <namespace> --limit 5

Severity & urgency

Diagnostic steps

Run these in order. Stop as soon as the cause is obvious.

1. Confirm the alert is current

The Alertmanager UI shows actively firing alerts. The Mattermost post that linked you here might be stale (group_wait + repeat_interval delays).

# From any host with kubectl access — list the firing pods
kubectl top pods -n <namespace> --sort-by=cpu | head -10

If the alert says <pod-name> but kubectl top shows it under-budget now, the spike has passed. Move to step 5 (find the cause of the historical spike).

2. Check container limits vs actual usage

kubectl describe pod -n <namespace> <pod-name> | grep -A2 "Limits:"

Expected output shape:

Limits:
  cpu:     2
  memory:  4Gi
Requests:
  cpu:     500m
  memory:  1Gi

If Limits.cpu is unset, the container can saturate the node — the CPU alert is a symptom of missing limits, not a sized-too-small container.

3. Look for CFS throttling

# Throttled time as a fraction of total runtime
kubectl exec -n <namespace> <pod-name> -- \
  sh -c 'cat /sys/fs/cgroup/cpu.stat'

You're looking at throttled_time vs usage_usec. If throttled_time / usage_usec > 0.1, the container is being throttled heavily — Linux is pausing it to enforce the CPU limit.

4. Inspect the workload itself

# Top processes inside the container
kubectl exec -n <namespace> <pod-name> -- top -bn1 | head -20

If one PID dominates CPU, you've found the offender. If it's the main process broadly, you have a workload-shape problem (see causes A and B below).

5. Check recent deployment and traffic patterns

# Was there a recent deploy?
kubectl rollout history deployment -n <namespace> <deployment-name>

# Traffic against the affected service over the last hour:
# Open Prometheus and query:
#   sum by (pod) (rate(http_requests_total{namespace="<namespace>"}[5m]))

A traffic spike at the same time as the CPU spike suggests load. A CPU spike without traffic change suggests a code regression.

Common causes & fixes

A. Recent deployment with a regression

Common patterns: an N+1 query introduced in a new release, a regex catastrophic backtrack, a tight loop without a sleep, accidentally disabled caching.

B. Real traffic increase

If you autoscale via HPA, check that the HPA's metric is actually firing:

kubectl get hpa -n <namespace>
kubectl describe hpa -n <namespace> <hpa-name>

A common bug: HPA points at the wrong metric (e.g., cpu but limits aren't set, so CPU% calculation breaks).

C. Missing CPU limits

Don't set limits much higher than requests — the gap is where CFS throttling lives. Aim for limits ≤ 2× requests as a starting heuristic.

D. Plugin / dependency hot loop (Mattermost-specific)

E. Garbage collection pressure (Go/Java workloads)

For Go workloads, GODEBUG=gctrace=1 reveals GC frequency. For JVM, check GC logs. If GC dominates, more memory is usually the answer — the CPU spike is downstream of a memory pressure problem.

Escalation

If unresolved within the target response time:

1. Platform on-call — @platform-oncall in #mm-incidents. PagerDuty service: mattermost-platform.
2. Cloud team — if multiple unrelated services are affected at the same time, suspect node-level pressure. PagerDuty service: cloud-platform.
3. Mattermost vendor support — if the cause is suspected in the Mattermost binary itself (unusual). Open a P1 ticket at support.mattermost.com.

Severity ladder:

Post-incident

After the immediate fix lands:

1. File a follow-up issue with root cause and what was changed. Use the team's incident template.
2. Update this runbook if the cause wasn't already covered — the most underused improvement loop in SRE is "I just fixed this; let me add it to the playbook for the next person." Open a merge request against this repo.
3. If a rollback was needed, file a regression bug against the service that shipped the offending change. Don't let a regression sit unfixed just because the rollback worked.
4. Consider whether the alert thresholds are right. A real incident that fires below your threshold (you found out from a user, not from PagerDuty) is a tuning signal.

Required Prometheus labels

The Quick diagnostics commands above use <label> placeholders that Alertmanager fills in from each alert's labels at delivery time. For this runbook to render copy-paste-runnable commands, your Prometheus rule must emit:

• namespace — the Kubernetes namespace of the failing deployment
• pod — referenced in the PromQL filter example for narrowing throttle-rate analysis; useful but not required for the basic command to run

When a label is missing, the rendered command shows <no value> in that slot — still readable, just not auto-runnable. Add the label to your rule and reload Prometheus.

Related runbooks

• Pod Not Ready — when high CPU causes the readiness probe to fail
• Container OOM-killed (TODO)
• HPA not scaling (TODO)

Appendix: useful PromQL queries

Find the top 10 CPU-consuming containers in the last 5 minutes:

topk(10,
  sum by (namespace, pod, container) (
    rate(container_cpu_usage_seconds_total[5m])
  )
)

CPU usage as a fraction of limit (the alert's actual expression):

sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total[5m])
)
/
sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="cpu"}
)

CFS throttling rate — fraction of time the container is being throttled:

rate(container_cpu_cfs_throttled_periods_total[5m])
/
rate(container_cpu_cfs_periods_total[5m])