Database Connectivity Loss

!!! danger "Severity: Critical" Target response: 5 min. The application can't reach its database. Writes (post, login, file upload) fail server-wide. Reads may or may not work depending on replica config.

What this alert means

The application reports zero active database connections while its pods are still healthy enough to be scraped by Prometheus. The alert fires when:

mattermost_db_master_connections_total{namespace="mattermost"} == 0
  and on (namespace, pod)
up{namespace="mattermost", job=~"mattermost.*"} == 1

The and on (namespace, pod) up == 1 clause means: only fire when the app pod itself is healthy. If the whole pod is down, a different alert (PodNotReady / scrape target down) is the right one.

Connection loss has several causes — DB outage, credential rotation, network policy change, connection pool exhaustion. Diagnosing narrows from "is the DB up?" to "is the network path clean?" to "are the credentials still valid?"

Quick diagnostics

Three commands to run before reading further. These work whether your Postgres is RDS, Azure Database for PostgreSQL, Cloud SQL, a self-managed VM, or an in-cluster StatefulSet — they connect via $DATABASE_URL rather than assuming a specific topology.

# WHERE: shell with psql installed. <instance> is filled in by AM
#   at alert time with the failing DB host. Auth comes from
#   ~/.pgpass or the PGUSER/PGPASSWORD env vars. sslmode=require
#   is correct for managed DBs (RDS, Azure DB, Cloud SQL) which
#   require TLS; works against in-cluster too.
# WHAT: minimal connectivity probe — does the DB accept connections
#   and run a trivial query?
# READ:
#   returns "1" → healthy.
#   connection refused → DB process down OR network blocked.
#   timeout → DB reachable but unresponsive (locked, max_connections
#     hit, internal crash).
#   FATAL: password authentication failed → auth issue, check
#     credentials or rotated password.
#   FATAL: no pg_hba.conf entry for host → your IP isn't allowed
#     (security group / pg_hba.conf rule needed).
psql "host=<instance> sslmode=require" -c "SELECT 1;"

# WHERE: shell with psql. Same connection params as above.
# WHAT: active connection count vs the DB's max_connections setting.
# READ: active near max → you're connection-saturated; new
#   connections fail with "FATAL: too many connections." Usual
#   causes: app-side connection leak (pool not returning connections),
#   missed pgbouncer config, or undersized max_connections for
#   the workload. Raise max_connections OR fix the leaker; both
#   require a different runbook follow-up.
psql "host=<instance> sslmode=require" -c "SELECT count(*) AS active, current_setting('max_connections') AS max FROM pg_stat_activity;"

# WHERE: shell with kubectl context set (works when the APP runs
#   in k8s, even if the DB doesn't). <namespace> is filled in by
#   AM at alert time; <app> needs to match the calling app's label.
# WHAT: last 200 lines from the application pods, filtered to
#   connection-failure patterns.
# READ: common patterns the grep catches:
#   "could not connect" → DB unreachable from app's network
#   "connection refused" → port/firewall issue
#   "FATAL" → DB-side rejection (usually auth)
#   "password authentication failed" → credentials mismatch
#   Spot-check timestamps: if errors started within the alert's
#   window, the alert is current. If they trail off, the issue
#   already resolved and the alert is stale.
kubectl logs -n <namespace> -l app=<app> --tail=200 | grep -iE "could not connect|connection refused|FATAL|password authentication failed"

Severity & urgency

Diagnostic steps

1. Confirm the DB itself is up

If it's a managed DB (RDS, Cloud SQL, Azure DB):

# Open the provider console — look for the instance status
# Look for: maintenance windows, automated failovers, alarms on the DB itself

If it's a self-hosted DB on the same cluster:

kubectl get pods -n <db-namespace>
kubectl logs -n <db-namespace> <db-pod> --tail=200

A DB that's down is its own alert and its own runbook — escalate to the DB team if so.

2. Test connectivity from the app pod

kubectl exec -n mattermost deploy/<mattermost-deployment> -- \
  sh -c 'nc -zv $MM_SQLSETTINGS_DATASOURCE_HOST 5432'

Three possible outcomes:

• succeeded — L4 reachable. Issue is application-layer (credentials, pool exhaustion).
• refused — DB host is up but rejecting connections. Likely listener config or firewall.
• timeout — Network path is broken. NetworkPolicy or routing issue.

3. Check for recent credential rotation

# Recent change to the DB credential secret?
kubectl get secret -n mattermost <db-secret-name> -o jsonpath='{.metadata.resourceVersion}'

# When did the deployment last restart?
kubectl get pods -n mattermost -l app=mattermost \
  -o custom-columns='POD:.metadata.name,STARTED:.status.startTime'

If the secret's resource version is more recent than the pod's start time, credentials were rotated but the app didn't restart to pick them up. The app is still trying the old credentials.

4. Check NetworkPolicy changes

# Any recent NetworkPolicy changes in the mattermost or db namespace?
kubectl get networkpolicy -n mattermost
kubectl get networkpolicy -n <db-namespace>

# Recent NetworkPolicy events:
kubectl get events -n mattermost --field-selector reason=NetworkPolicyUpdated \
  --sort-by='.lastTimestamp' | tail -10

If a NetworkPolicy was applied recently, it might be blocking egress to the DB.

5. Inspect connection pool state

# Mattermost-specific metric: master pool state
curl -sS http://<mattermost-pod>:8067/metrics | grep mattermost_db_master

Look for mattermost_db_master_connections_total and mattermost_db_master_connection_attempts_total over time. If attempts are climbing but connections_total stays at zero, the app is trying and failing.

Common causes & fixes

A. DB failover or maintenance window

B. Credentials rotated without app restart

C. NetworkPolicy / firewall change blocking egress

D. Connection pool exhaustion

E. DB at max_connections globally

F. DNS resolution failure for DB hostname

G. TLS handshake failure (rare)

Escalation

If unresolved within 5 minutes:

1. Database team — @dba-oncall in #mm-incidents. PagerDuty service: data-platform. The DB team owns DB-side issues (capacity, failover, slow queries).
2. Platform on-call — @platform-oncall. PagerDuty service: mattermost-platform. Owns the app-side state (credentials, pool config, restart authority).
3. Network team — if NetworkPolicy or routing issue. PagerDuty service: network-ops.

Severity ladder:

Post-incident

1. Postmortem — a database outage is a Tier 1 incident. Full timeline, impact assessment, contributing factors, action items.
2. Update this runbook if the cause category was novel.
3. Process changes — did the DB team notify the app team about the upcoming maintenance window? Was the credential rotation coordinated? Was the NetworkPolicy change reviewed?
4. Resilience changes — should the app retry harder on connection loss? Should there be a circuit-breaker before user-visible errors? Should reads route to a replica during master outages?

Required Prometheus labels

The Quick diagnostics commands above use <label> placeholders that Alertmanager fills in from each alert's labels at delivery time. For this runbook to render copy-paste-runnable commands, your Prometheus rule must emit:

• instance — the DB hostname (e.g., postgres-prod.internal.example.com, db.rds.amazonaws.com)
• namespace — the Kubernetes namespace of the connecting application
• app — the application label of the connecting service (typically the value of app.kubernetes.io/name or your team's app label convention)

When a label is missing, the rendered command shows <no value> in that slot — still readable, just not auto-runnable. Add the label to your rule and reload Prometheus.

Related runbooks

• Database High Latency — when DB is reachable but slow
• Database Replication Lag — when replica reads diverge
• DNS Resolution Failure — when cluster DNS breaks
• High HTTP 5xx Error Rate — the downstream effect of DB loss

Appendix: useful PromQL queries

Connection count over time per pod:

mattermost_db_master_connections_total{namespace="mattermost"}

Failed connection attempts (Postgres-side):

rate(pg_stat_database_conflicts_total[5m])

Time since last successful connection (per pod):

time() - mattermost_db_master_last_successful_connection_timestamp