Sample Prometheus alerting rules shipped with the plugin — one working rule for every runbook (31 rules across 30 runbooks). Copy them into your Prometheus, tune the thresholds, and alerts route straight to the matching runbook in chat.
⬇ Download prometheus-rules.yaml — add it to your Prometheus rule_files: glob, or wrap the groups: block in a PrometheusRule CRD if you run the Prometheus Operator.
Every threshold andfor:duration below is a demo-cluster placeholder — tune to your traffic and SLOs before trusting them in production. Therunbook:label on each rule is the join key that routes it to a runbook.
# Sample Prometheus alert rules covering all 30 canonical runbooks
# (including the security category) shipped by the mattermost-plugin-alertmanager plugin.
#
# WHERE THIS FILE LIVES
# ---------------------
# In the plugin source: samples/prometheus-rules.yaml
# In the plugin tarball: bundled under the same path.
# To install: copy or symlink into a directory that matches the
# `rule_files:` glob in your prometheus.yml, then reload Prometheus:
# curl -X POST http://prometheus:9090/-/reload
#
# HOW TO VALIDATE THESE RULES BEFORE TRUSTING THEM
# -------------------------------------------------
# 1. Syntax: `promtool check rules prometheus-rules.yaml`
# 2. Routing: in any Mattermost channel with plugin receivers, run
# `/alertmanager validate --simulate runbook=high-cpu-usage` (or
# any other slug). The plugin walks Alertmanager's loaded route
# tree against those labels and reports which receiver an alert
# with that label set would actually dispatch to. Useful for
# catching dead routes BEFORE the rule fires for real.
# 3. End-to-end: pick one rule, lower its threshold so it fires
# immediately, watch the bound channel for delivery, then revert.
#
# HOW THIS FILE MAPS TO THE PLUGIN
# --------------------------------
# Each rule below emits a `runbook: <slug>` label. The plugin's
# /alertmanager add command creates Alertmanager receivers named
# `<slug>--<channel>` and a matching routes block that matches on
# the `runbook` label. An alert with `runbook: high-cpu-usage`
# fires into receivers like `high-cpu-usage--alerts` and lands in
# the bound Mattermost channel.
#
# LABEL CONTRACT WITH THE RUNBOOKS
# --------------------------------
# Each runbook's `## Required Prometheus labels` section names the
# labels its Quick diagnostics commands need to render with
# pre-filled values (vs `<no value>` placeholders). The expressions
# below are written to emit those labels — verified runbook-by-runbook
# against each footer. Where the underlying metric doesn't carry a
# required label natively, an inline comment names the kube-state-
# metrics join or relabel needed to produce it.
#
# Stack assumed below:
# - kube-state-metrics (for pod / deployment / node phase metrics)
# - cadvisor (for container_* CPU and memory metrics)
# - node_exporter (for node_filesystem_* / node_memory_*)
# - blackbox_exporter (for probe_*)
# - kubelet (for kubelet_volume_stats_*)
# - postgres_exporter / mysql_exporter (for pg_* / mysql_* metrics)
# - nginx ingress controller (for nginx_ingress_*)
#
# WHAT YOU NEED TO ADJUST BEFORE THIS IS PRODUCTION-READY
# -------------------------------------------------------
# 1. Thresholds and `for:` durations — every value below is a
# placeholder calibrated for a small demo cluster. Real
# environments need thresholds tuned to traffic shape and SLOs.
# 2. PromQL series names — assume kube-state-metrics, cadvisor, and
# node_exporter use upstream-default names. Adjust if your
# scrape config renames them via metric_relabel_configs.
# 3. Severity labels — the plugin doesn't enforce severity routing
# out of the box. Add severity-based routes to alertmanager.yml
# if you want sev1 to page and sev2 to chat-only.
# 4. The `runbook_url` annotation is supplied automatically by the
# plugin via the runbook-host helper. These sample rules only
# declare the slug — the plugin renders the full URL based on
# your Mattermost SiteURL.
#
# Format is the standard Prometheus rule_files: format. See
# https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/
# for the schema. Drop this file into a `rule_files` glob in your
# prometheus.yml.
groups:
# =========================================================
# COMPUTE — the workhorse runbooks. CPU, memory, pod, node.
# =========================================================
- name: compute
interval: 30s
rules:
# HighCPUUsage — labels required: namespace, pod.
# Container-level CPU as a fraction of the container's quota.
# `container=""` filters out the pod-level aggregate emitted
# by cadvisor (which would double-count). The 0.85 threshold
# is "container using > 85% of its CPU limit for 10m".
- alert: HighCPUUsage
expr: |
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m])
)
/
sum by (namespace, pod) (
kube_pod_container_resource_limits{resource="cpu"}
) > 0.85
for: 10m
labels:
severity: warning
category: compute
runbook: high-cpu-usage
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} CPU > 85% of limit for 10m"
description: "Sustained high CPU may indicate runaway processes, undersized limits, or a noisy-neighbor pod."
# HighMemoryUsage — labels required: namespace, pod.
# Working-set memory as a fraction of the container's limit.
# Working-set is the closest proxy to "what counts against OOM
# killer" — RSS plus active page cache, minus inactive cache.
- alert: HighMemoryUsage
expr: |
sum by (namespace, pod) (
container_memory_working_set_bytes{container!="",container!="POD"}
)
/
sum by (namespace, pod) (
kube_pod_container_resource_limits{resource="memory"}
) > 0.90
for: 10m
labels:
severity: warning
category: compute
runbook: high-memory-usage
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} memory > 90% of limit for 10m"
description: "High memory pressure can trigger OOM kills and degrade scheduler behavior."
# PodCrashLoopBackOff — labels required: namespace, pod.
# kube_pod_container_status_restarts_total has `container` too;
# sum it away so the alert fires per-pod not per-container.
- alert: PodCrashLoopBackOff
expr: |
sum by (namespace, pod) (
rate(kube_pod_container_status_restarts_total[5m])
) * 60 * 5 > 3
for: 5m
labels:
severity: critical
category: compute
runbook: pod-crashloopbackoff
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} restarting > 3x in 5m"
description: "Crashlooping pods indicate a bad image, missing config, or failing liveness probe."
# PodNotReady — labels required: namespace, pod.
# `condition="false"` series is 1 when the pod is NotReady.
- alert: PodNotReady
expr: |
sum by (namespace, pod) (kube_pod_status_ready{condition="false"}) > 0
for: 10m
labels:
severity: warning
category: compute
runbook: pod-not-ready
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} not Ready for 10m"
description: "Pod has been in NotReady state long enough to suggest a real problem, not a transient blip."
# DeploymentReplicasUnavailable — labels required: namespace, app.
# The base metric carries `namespace` and `deployment`. The
# runbook needs `app` (e.g., the value of app.kubernetes.io/name)
# so the diagnostic commands can query by app label. Join with
# kube_deployment_labels to pull `label_app` and relabel it
# via `label_replace` into a plain `app` label.
- alert: DeploymentReplicasUnavailable
expr: |
label_replace(
(
kube_deployment_status_replicas_unavailable > 0
)
* on (namespace, deployment) group_left(label_app_kubernetes_io_name)
kube_deployment_labels,
"app", "$1", "label_app_kubernetes_io_name", "(.+)"
)
for: 10m
labels:
severity: warning
category: compute
runbook: deployment-replicas-unavailable
annotations:
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} (app={{ $labels.app }}) has unavailable replicas"
description: "Desired replica count not met. Check pod events and scheduler / resource constraints."
# NodeNotReady — labels required: node.
- alert: NodeNotReady
expr: |
kube_node_status_condition{condition="Ready",status="true"} == 0
for: 5m
labels:
severity: critical
category: compute
runbook: node-not-ready
annotations:
summary: "Node {{ $labels.node }} not Ready for 5m"
description: "A NotReady node is a candidate for cordon + drain. Investigate kubelet, networking, and disk pressure."
# CPUThrottlingHigh — labels required: namespace, pod, container.
# CFS throttling as a fraction of scheduler periods. Catches
# latency from hitting the CPU *limit* even when average usage
# looks modest — invisible to HighCPUUsage.
- alert: CPUThrottlingHigh
expr: |
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/
sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m])) > 0.25
for: 10m
labels:
severity: warning
category: compute
runbook: cpu-throttling-high
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) CPU-throttled > 25% for 10m"
description: "Throttled by the CFS scheduler — raise the CPU limit or the request/limit ratio."
# ContainerOOMKilled — labels required: namespace, pod, container.
# Distinct from a crashloop: the kernel OOM-killer reaped the
# container for exceeding its memory limit. Reuses the
# high-memory-usage runbook (same remediation).
- alert: ContainerOOMKilled
expr: |
increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) > 0
for: 0m
labels:
severity: critical
category: compute
runbook: high-memory-usage
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) was OOMKilled"
description: "Container hit its memory limit and was killed. Raise the limit or fix the leak."
# PodImagePullBackOff — labels required: namespace, pod.
# Container can't pull its image (bad tag, missing/expired pull
# secret, registry issue). The pod never starts — vs a crashloop
# where it starts then dies.
- alert: PodImagePullBackOff
expr: |
max by (namespace, pod) (
kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}
) > 0
for: 5m
labels:
severity: critical
category: compute
runbook: image-pull-backoff
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
description: "Bad tag, missing/expired imagePullSecret, or registry outage — the rollout is blocked."
# PodsUnschedulable — labels required: namespace, pod.
# Scheduler can't place the pod: capacity, taints, affinity, or an
# unbound PVC. Requested capacity that never arrives.
- alert: PodsUnschedulable
expr: |
max by (namespace, pod) (kube_pod_status_unschedulable) > 0
for: 10m
labels:
severity: warning
category: compute
runbook: pods-unschedulable
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is unschedulable for 10m"
description: "No node fits — check requests vs capacity, node selectors/affinity, taints, and PVC binding."
# =========================================================
# APPLICATION — request-path SLO indicators.
# =========================================================
- name: application
interval: 30s
rules:
# HighHTTPErrorRate — labels required: namespace, service.
# http_requests_total is whatever your application emits;
# assume it carries `service` and `namespace` already (via
# Prometheus relabel from kubernetes_sd or app instrumentation).
# If your app only emits `service`, add `namespace` via
# metric_relabel_configs at scrape time.
- alert: HighHTTPErrorRate
expr: |
sum by (namespace, service) (rate(http_requests_total{code=~"5.."}[5m]))
/ sum by (namespace, service) (rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
category: application
runbook: high-http-error-rate
annotations:
summary: "Service {{ $labels.namespace }}/{{ $labels.service }} 5xx rate > 5% for 5m"
description: "Sustained 5xx errors indicate a real availability incident, not a single bad request."
# HighAPILatency — labels required: namespace, app.
# Grouping by `app` matches what the runbook's kubectl
# commands query (`-l app=<app>`). Assumes the
# http_request_duration_seconds metric carries an `app`
# label — most instrumentation libraries (prometheus-client,
# otel) let you set this via constant labels.
- alert: HighAPILatency
expr: |
histogram_quantile(0.95,
sum by (namespace, app, le) (rate(http_request_duration_seconds_bucket[5m]))
) > 1.0
for: 10m
labels:
severity: warning
category: application
runbook: high-api-latency
annotations:
summary: "Service {{ $labels.namespace }}/{{ $labels.app }} p95 latency > 1s for 10m"
description: "Tail latency degradation. Check downstream dependencies and DB query times."
# ServiceEndpointDown — labels required: namespace, service.
# blackbox_exporter's probe_success is keyed by `instance`.
# To produce namespace/service, configure the blackbox scrape
# to relabel from kubernetes_sd_configs:
# metric_relabel_configs:
# - source_labels: [__meta_kubernetes_service_namespace]
# target_label: namespace
# - source_labels: [__meta_kubernetes_service_name]
# target_label: service
# Once those labels exist on probe_success, this rule fires
# per-service rather than per-probe-target URL.
- alert: ServiceEndpointDown
expr: |
max by (namespace, service) (probe_success == 0)
for: 5m
labels:
severity: critical
category: application
runbook: service-endpoint-down
annotations:
summary: "Service {{ $labels.namespace }}/{{ $labels.service }} probe failing for 5m"
description: "Synthetic probe failing — either the service is dead or the network path is broken."
# RequestRateAnomaly — labels required: namespace, service.
# Compares current 5m rate against same-time-yesterday's 5m rate.
# `offset 1h` is short for demo purposes; switch to `offset 1d`
# to compare against weekly seasonality in production.
- alert: RequestRateAnomaly
expr: |
abs(
sum by (namespace, service) (rate(http_requests_total[5m]))
- sum by (namespace, service) (rate(http_requests_total[5m] offset 1h))
)
/ sum by (namespace, service) (rate(http_requests_total[5m] offset 1h)) > 0.5
for: 15m
labels:
severity: warning
category: application
runbook: request-rate-anomaly
annotations:
summary: "Service {{ $labels.namespace }}/{{ $labels.service }} request rate diverges >50% from one hour ago"
description: "Could be traffic loss (upstream broken) or a sudden surge (DDoS, viral content, retry storm)."
# =========================================================
# DATABASE — replication, latency, reachability.
# =========================================================
- name: database
interval: 30s
rules:
# DatabaseConnectivityLoss — labels required: instance, namespace, app.
# The base `up` series is per-(instance, job). `namespace` and
# `app` come from the application side — the metric used here
# (pg_up via postgres_exporter) only carries `instance`. The
# runbook's `app` and `namespace` placeholders refer to the
# CONNECTING application, not the DB itself. Set these via
# static rule labels if you scope this rule per-team, or via
# metric_relabel_configs that pull from your service catalog.
- alert: DatabaseConnectivityLoss
expr: |
up{job=~".*postgres.*|.*mysql.*"} == 0
for: 2m
labels:
severity: critical
category: database
runbook: database-connectivity-loss
annotations:
summary: "Database {{ $labels.instance }} unreachable for 2m"
description: "DB exporter cannot scrape the DB. Check the DB process, networking, and credentials."
# DatabaseReplicationLag — labels required: instance.
# pg_replication_lag_seconds from postgres_exporter naturally
# carries `instance` (the replica being measured).
- alert: DatabaseReplicationLag
expr: |
pg_replication_lag_seconds > 30
for: 5m
labels:
severity: warning
category: database
runbook: database-replication-lag
annotations:
summary: "Postgres replica {{ $labels.instance }} lag > 30s for 5m"
description: "Replica is behind primary. Risk of stale reads and increased failover RPO."
# DatabaseHighLatency — labels required: instance.
# Aggregate by instance so each DB host fires independently —
# a single laggy replica shouldn't be hidden by a healthy
# primary's faster median.
- alert: DatabaseHighLatency
expr: |
histogram_quantile(0.95,
sum by (instance, le) (rate(pg_stat_statements_mean_time_seconds_bucket[5m]))
) > 0.5
for: 10m
labels:
severity: warning
category: database
runbook: database-high-latency
annotations:
summary: "Postgres {{ $labels.instance }} p95 query time > 500ms for 10m"
description: "Slow queries usually mean missing indexes, table bloat, or a noisy-neighbor workload."
# PostgresConnectionsNearMax — cluster-level (aggregate, no labels).
# In-use connections as a fraction of max_connections. New clients
# get refused at 100% — Mattermost writes fail. Critical at 95%.
- alert: PostgresConnectionsNearMax
expr: |
sum(pg_stat_activity_count) / on() pg_settings_max_connections > 0.8
for: 5m
labels:
severity: warning
category: database
runbook: postgres-connections-near-max
annotations:
summary: "Postgres connections > 80% of max_connections"
description: "Approaching the connection ceiling. Add a pooler (PgBouncer) or raise max_connections with RAM to back it."
# =========================================================
# STORAGE — volume fill, disk fill rate.
# =========================================================
- name: storage
interval: 30s
rules:
# PersistentVolumeFull — labels required: namespace, pod.
# kubelet_volume_stats_* carries `namespace` and
# `persistentvolumeclaim`. To resolve PVC → pod, join with
# kube_pod_spec_volumes_persistentvolumeclaims_info which
# carries `namespace`, `persistentvolumeclaim`, and `pod`.
- alert: PersistentVolumeFull
expr: |
(
kubelet_volume_stats_used_bytes
/ kubelet_volume_stats_capacity_bytes * 100
) * on (namespace, persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info > 90
for: 5m
labels:
severity: critical
category: storage
runbook: persistent-volume-full
annotations:
summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} (pod {{ $labels.pod }}) > 90% full"
description: "Volume close to capacity. Expand the PVC or clean up data before writes start failing."
# DiskFillRateHigh — labels required: instance.
# node_filesystem_avail_bytes naturally carries `instance`,
# `mountpoint`, `device`, `fstype`. The mountpoint filter
# focuses on root — drop it to catch fills on all volumes.
- alert: DiskFillRateHigh
expr: |
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
for: 30m
labels:
severity: warning
category: storage
runbook: disk-fill-rate-high
annotations:
summary: "Node {{ $labels.instance }} root disk projected to fill in < 4h"
description: "Linear extrapolation says the disk will fill within four hours. Act before it hits 100%."
# =========================================================
# NETWORKING — ingress, certs, DNS.
# =========================================================
- name: networking
interval: 30s
rules:
# IngressHigh5xx — emits `ingress` (no runbook footer constraint).
- alert: IngressHigh5xx
expr: |
sum by (ingress) (rate(nginx_ingress_controller_requests{status=~"5.."}[5m]))
/ sum by (ingress) (rate(nginx_ingress_controller_requests[5m])) > 0.05
for: 5m
labels:
severity: critical
category: networking
runbook: ingress-high-5xx
annotations:
summary: "Ingress {{ $labels.ingress }} 5xx rate > 5% for 5m"
description: "Ingress-level 5xx indicates upstream service failure or backend pool exhaustion."
# CertificateExpiringSoon — labels required: instance.
# probe_ssl_earliest_cert_expiry naturally carries `instance`
# (the probe target hostname:port).
- alert: CertificateExpiringSoon
expr: |
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
for: 1h
labels:
severity: warning
category: networking
runbook: certificate-expiring-soon
annotations:
summary: "TLS cert on {{ $labels.instance }} expires in < 14 days"
description: "Lead time for renewal. Don't wait until the day-of."
# DnsResolutionFailure — emits `instance` (no runbook footer
# constraint). Uses blackbox probe DNS module + the generic
# probe_success series. Either lookup-time-zero (the probe
# didn't try) or probe_success=0 (it tried and failed) counts.
- alert: DnsResolutionFailure
expr: |
probe_dns_lookup_time_seconds == 0 or probe_success{job="dns"} == 0
for: 5m
labels:
severity: critical
category: networking
runbook: dns-resolution-failure
annotations:
summary: "DNS lookups failing for {{ $labels.instance }} for 5m"
description: "DNS failure cascades into every dependent service. Check CoreDNS / upstream resolvers."
# =========================================================
# OBSERVABILITY — meta-alerts on the monitoring stack itself.
# =========================================================
- name: observability
interval: 30s
rules:
# PrometheusScrapeTargetDown — labels required: namespace, service.
# `up == 0` per-target. Under kubernetes_sd_configs the scrape
# config will (with default kubernetes-sd relabel rules) attach
# __meta_kubernetes_service_namespace and
# __meta_kubernetes_service_name — surface those as `namespace`
# and `service` via relabel_configs at scrape time so this rule
# produces the labels the runbook expects.
- alert: PrometheusScrapeTargetDown
expr: |
up == 0
for: 5m
labels:
severity: warning
category: observability
runbook: prometheus-scrape-target-down
annotations:
summary: "Prometheus cannot scrape {{ $labels.namespace }}/{{ $labels.service }} ({{ $labels.job }}/{{ $labels.instance }}) for 5m"
description: "Missing scrape target = blind to that subsystem. Either the target is dead or the scrape config is wrong."
# AlertmanagerNotificationFailure — emits `receiver` natively.
- alert: AlertmanagerNotificationFailure
expr: |
rate(alertmanager_notifications_failed_total[5m]) > 0
for: 5m
labels:
severity: critical
category: observability
runbook: alertmanager-notification-failure
annotations:
summary: "Alertmanager notification failures for receiver {{ $labels.receiver }}"
description: "Alertmanager cannot deliver. If THIS fires, the whole alerting pipeline is degraded — investigate the receiver target (Mattermost webhook, PagerDuty, etc.)."
# =========================================================
# SECURITY — supply-chain, auth, runtime, and policy signals.
# Several of these depend on tooling BEYOND the stack above and
# are valid-but-silent until it's present:
# - PrivilegedContainerStarted -> Kyverno / Gatekeeper policy metrics
# - InteractiveShellInContainer -> Falco (falco_events)
# - RBACPrivilegeEscalation -> Falco k8s-audit plugin
# - SecurityToolingDown -> those sensors' own /metrics targets
# Adjust the allowlist regex and job names to YOUR environment.
# =========================================================
- name: security
interval: 30s
rules:
# UnexpectedContainerImage — labels required: namespace, pod.
# Any running container whose image is OUTSIDE the approved
# registry allowlist. The regex below is an example — replace it
# with your own registries before trusting this rule.
- alert: UnexpectedContainerImage
expr: |
count by (namespace, pod, container, image) (
kube_pod_container_info{image!~"^(mattermost/|registry\\.mattermost\\.com/|ghcr\\.io/mattermost/).*"}
) > 0
for: 5m
labels:
severity: warning
category: security
runbook: unexpected-container-image
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} runs unapproved image {{ $labels.image }}"
description: "Image from outside the approved registry allowlist — possible supply-chain issue or rogue workload."
# APIServerAuthFailureSpike — cluster-level (no per-pod labels).
# A surge of 401/403 at the API server: credential abuse, a
# stolen/expired token, or a broken RBAC change hammering the API.
- alert: APIServerAuthFailureSpike
expr: |
sum(rate(apiserver_request_total{code=~"401|403"}[5m])) > 1
for: 10m
labels:
severity: warning
category: security
runbook: apiserver-auth-failure-spike
annotations:
summary: "kube-apiserver auth failures (401/403) elevated for 10m"
description: "Sustained authn/authz denials — investigate the source identity and any recent RBAC changes."
# PrivilegedContainerStarted — requires Kyverno or Gatekeeper.
# Stock kube-state-metrics can't see securityContext.privileged;
# this keys off a policy engine's violation counter instead.
- alert: PrivilegedContainerStarted
expr: |
sum by (namespace, policy, rule) (
increase(kyverno_policy_results_total{rule=~".*privileged.*|.*run-as-non-root.*", policy_result="fail"}[10m])
) > 0
for: 0m
labels:
severity: critical
category: security
runbook: privileged-container-started
annotations:
summary: "Privileged/root container policy violation in {{ $labels.namespace }}"
description: "A container started privileged / root / hostPath — container-to-node escape risk."
# InteractiveShellInContainer — requires Falco (falco_events).
# Falco's built-in "Terminal shell in container" rule. Label names
# (k8s_ns_name / k8s_pod_name) follow Falco's metric conventions.
- alert: InteractiveShellInContainer
expr: |
sum by (k8s_ns_name, k8s_pod_name) (
rate(falco_events{rule="Terminal shell in container"}[5m])
) > 0
for: 0m
labels:
severity: warning
category: security
runbook: interactive-shell-in-container
annotations:
summary: "Shell spawned in {{ $labels.k8s_ns_name }}/{{ $labels.k8s_pod_name }}"
description: "Interactive shell inside a running container — hands-on-keyboard activity worth confirming."
# RBACPrivilegeEscalation — requires Kubernetes audit logs via Falco.
# High-privilege RBAC changes: cluster-admin bindings, wildcard
# roles. A classic post-compromise persistence step.
- alert: RBACPrivilegeEscalation
expr: |
sum by (k8s_ns_name) (
rate(falco_events{rule=~"Create ClusterRoleBinding.*|Attach to cluster-admin.*|K8s Role.*wildcard.*"}[5m])
) > 0
for: 0m
labels:
severity: critical
category: security
runbook: rbac-privilege-escalation
annotations:
summary: "High-privilege RBAC change detected"
description: "cluster-admin binding or wildcard role created/modified — treat as a compromise indicator until cleared."
# SecurityToolingDown — the meta-alert. If Falco / Kyverno / Gatekeeper
# stop reporting, every tooling-dependent rule above goes silently
# blind. This is the rule that tells you the others can't fire.
- alert: SecurityToolingDown
expr: |
up{job=~"falco|kyverno.*|gatekeeper.*"} == 0
for: 5m
labels:
severity: critical
category: security
runbook: security-tooling-down
annotations:
summary: "Security sensor {{ $labels.job }}/{{ $labels.instance }} is down for 5m"
description: "A runtime/policy/audit sensor stopped reporting — tooling-dependent security alerts are now blind."