Configuration

End-to-end setup: plugin settings, what the plugin generates, and what your alertmanager.yml needs around it.

Plugin settings

System Console → Plugins → Alertmanager exposes 5 settings. All are optional — the plugin runs with defaults on a fresh install.

WebhookHost

The host+port the plugin writes into rendered api_url: values. Defaults to ServiceSettings.SiteURL when empty.

Set this when Alertmanager reaches Mattermost over a different network path than end users:

Format: <scheme>://<host>:<port> — no trailing slash, no path. The runbook URLs embedded in alert post templates continue using SiteURL since those are clicked by users in browsers, not consumed by Alertmanager.

WebhookRotationDays

When non-zero, the background reconciler DMs sysadmins about receivers whose webhook hasn't been rotated within this many days. Default 0 = feature disabled.

Reminder-only — the plugin never auto-rotates. Per-receiver opt-in mandatory via on flag on /alertmanager add. See ROTATION.md for the full playbook.

Recommended values:

Values below 30 produce noise unless your secret-rotation policy actually rotates that frequently.

AlertManagerCABundle

PEM-encoded CA certificates used when the plugin queries Alertmanager's REST API (/alertmanager status, alerts, silences, expire_silence, route simulation). Concatenate multiple certs in one block. Leave empty to trust only system CAs.

This setting only affects the plugin's outbound calls TO Alertmanager. The webhook URLs Alertmanager POSTs to are Mattermost-side and use whatever TLS Mattermost is configured with.

-----BEGIN CERTIFICATE-----
<your CA>
-----END CERTIFICATE-----

AlertManagerAllowedCIDRs

SSRF destination allowlist for the plugin's outbound Alertmanager calls. A team admin supplies the Alertmanager URL and the Mattermost server then connects to it (inventory probes, validate, alerts/silences/status), so an unrestricted URL is an SSRF primitive. The guard runs at connect time against the resolved IP, so a hostname that resolves to a blocked address (incl. DNS rebinding) is caught, not just literal IPs.

Blocked by default (no allowlist): loopback (127.0.0.0/8, ::1), link-local incl. cloud metadata 169.254.169.254, multicast, unspecified (0.0.0.0), and all private/internal ranges — RFC1918 (10/8, 172.16/12, 192.168/16), CGNAT (100.64/10), and IPv6 ULA (fc00::/7). With an empty setting, only public destinations are reachable.

Requirement — Alertmanager must be directly reachable from the Mattermost server. The plugin connects to Alertmanager directly and deliberately ignores HTTP_PROXY / HTTPS_PROXY on the Mattermost process: through a proxy the SSRF guard would only see the proxy's address while the proxy fetched the real (possibly blocked) destination. Routing Alertmanager API calls through a corporate/egress proxy is therefore not supported — if the Mattermost server has no direct network path to Alertmanager, alerts/status/validate and the inventory probes will fail. (In-cluster and same-network deployments are direct, so this only affects egress-restricted, proxy-only topologies.)

Because private is blocked by default, an in-cluster Alertmanager on a private IP or *.svc.cluster.local will NOT work until you list its CIDR here. This is the deliberate "no internal SSRF by default" posture — set the allowlist to the network your Alertmanager runs in:

10.0.0.0/8

An explicit entry re-enables an otherwise-blocked address (e.g. 127.0.0.1/32 for a same-host Alertmanager), except link-local / cloud-metadata / multicast / unspecified addresses, which are never reachable even if an entry would match them. A catch-all 0.0.0.0/0 or ::/0 is rejected outright — it would defeat the guard, so name your actual networks instead. A malformed setting where no entry is usable (all invalid, or only /0) fails closed: if a previous good allowlist is live it is kept (a typo can't silently widen egress); if none is (e.g. first load / server start), all Alertmanager egress is denied until the setting is fixed — it never falls back to allowing public destinations.

MetricsToken

When set, exposes Prometheus-format metrics at /plugins/com.mattermost.alertmanager/metrics. Prometheus scrapes the endpoint using this token in the Authorization: Bearer <token> header. Leave empty to disable the endpoint entirely (returns 404 when unset).

This field is secret: true, so once saved it shows blank in System Console and can't be read back. If you lose the value (or need to rotate it), don't guess — run the slash command below, which mints a new token, stores it, and reveals it once in an ephemeral reply along with a ready-to-paste scrape_config:

/alertmanager metrics-token generate   # sysadmin-only; rotates the token
/alertmanager metrics-token            # status only — does not rotate or reveal

Or generate one yourself and paste it into the setting:

openssl rand -hex 32

Configure Prometheus to scrape with the same token:

scrape_configs:
  - job_name: mattermost-alertmanager-plugin
    metrics_path: /plugins/com.mattermost.alertmanager/metrics
    authorization:
      type: Bearer
      credentials: <your token>
    static_configs:
      - targets: ['<mattermost-host>:443']

AssembledYAMLTTLHours

The bot DMs sysadmins the assembled alertmanager-receivers.yml + alertmanager-routes.yml files after every /alertmanager add and /alertmanager rotate all --overdue. Those files contain channel-bound webhook URLs (bearer tokens by URL).

This setting controls how many hours the DM'd files persist before the auto-delete janitor removes them. Default 72. Set to 0 to disable auto-delete (files persist forever — not recommended for production).

Internal: receiver list storage

The receiver list is stored in the plugin KV store (key alertconfigs:v1), not in the plugin config. It holds webhook IDs and any Alertmanager basic-auth passwords, and the KV store — unlike plugin config — is not returned by GET /api/v4/config, so those secrets aren't exposed to delegated System Console roles (sysconsole_read_plugins). It is managed entirely by slash commands; there is no System Console field to hand-edit it.

Breaking change / upgrade note. Earlier builds stored this list in the plugin config map (alertconfigsjson). The move to the KV store is a clean break with no automatic migration — after upgrading, /alertmanager list starts empty and you re-create receivers with /alertmanager add. The old config value is ignored (and the underlying Mattermost webhooks it referenced are orphaned — delete them under System Console → Integrations → Incoming Webhooks if you're not re-adding). This is intentional; the plugin is not deployed in long-lived installs that would need a data migration.

The stored JSON shape is:

[
  {
    "name": "high-cpu-usage--alert-slo-channel",
    "team": "testing",
    "channel": "alert-slo-channel",
    "alertManagerURL": "http://alertmanager:9093",
    "webhookID": "<MM webhook ID>",
    "webhookHostOverride": "",
    "lastRotatedAt": "2026-06-11T12:34:56Z",
    "lastReminderAt": "0001-01-01T00:00:00Z",
    "rotationRemindersEnabled": false
  }
]

Alertmanager URL format

alertManagerURL is the Alertmanager base URL — the plugin appends /api/v2/... itself when it queries the backend. Consequences:

• A path prefix works. Alertmanager behind a reverse proxy with --web.external-url=https://mon.example.com/alertmanager is fine; the appended path lands under the prefix.
• A query string or fragment is rejected — including a bare trailing ? or #. Either terminates the base URL early and silently truncates the appended API path onto a different endpoint, so /alertmanager add refuses them.
• Scheme must be http:// or https://, a host is required, and the host and any path are held to a strict character grammar (blocks shell/CSV injection, since the URL renders into a copy-paste curl and an exported CSV).
• No embedded credentials — set the basic-auth user/password fields on the receiver entry instead.

A value that bypasses these (a direct KV write, say) is neutered on load, not rejected: a hard failure would stop the whole plugin loading. The bad URL is blanked, disabling only that receiver's alerts/status/silences queries — alert delivery is unaffected (it runs through the Mattermost webhook, not this URL).

What the plugin generates

When you run /alertmanager add <team> <channel> <am-url> [set] [on]:

1. Resolves the destination channel (auto-creates if missing).
2. Mints an ephemeral PAT for the calling sysadmin, uses it to create one Mattermost incoming webhook per receiver in the chosen set, revokes the PAT.
3. Stores each receiver in plugin config, named <runbook-slug>--<team-slug>-<channel-slug>. Stamps LastRotatedAt = now. If on was passed, sets RotationRemindersEnabled = true.
4. Renders two YAML fragments and DMs them to the calling user:
   • alertmanager-receivers.yml — paste under receivers:
   • alertmanager-routes.yml — paste under route.routes:

The plugin handles slack_configs blocks and the matching routes: entries. Hook IDs are baked into the rendered api_url values; the plugin remembers them so future /alertmanager rotate or /alertmanager remove can find the right webhook to act on.

What your alertmanager.yml needs around the plugin's output

The plugin generates the receivers: and the route.routes: block. You provide the surrounding structure:

# REQUIRED — global defaults
global:
  resolve_timeout: 5m         # how long after last notification before "resolved"

# REQUIRED — top-level routing
route:
  receiver: default-fallback   # catch-all for unrouted alerts (must exist below)
  group_by: ['alertname', 'cluster']
  group_wait: 30s              # delay before first notification
  group_interval: 5m           # delay between notifications for same group
  repeat_interval: 4h          # repeat if still firing (4h+ for prod)

  routes:
    # <-- PASTE FROM alertmanager-routes.yml HERE -->

# REQUIRED — receivers section
receivers:
  # REQUIRED — catch-all default referenced by `route.receiver` above.
  - name: default-fallback

  # <-- PASTE FROM alertmanager-receivers.yml HERE -->

# OPTIONAL — inhibit rules (suppress lower-severity when higher is firing)
inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: ['alertname', 'cluster']

Required fields

Recommended for production

Optional

Custom (non-runbook) receivers

The /alertmanager add flow binds receivers to shipped runbooks: the plugin emits a runbook=<slug> route and your Prometheus rule sets that runbook label so the alert lands in the right channel with the runbook's diagnostics inline.

Sometimes you have an alert with no runbook — a bespoke rule you wrote that doesn't map to the shipped catalog. For that, use add-custom:

/alertmanager add-custom <team> <channel> <am-url> <name>

This creates a single receiver named <name>--<team>-<channel> with its own Mattermost webhook, tracked and rotated like any other receiver. Two things differ from add:

1. No runbook content. The chat post is the raw alert — there's no runbook behind the custom name, so no diagnostics/links are rendered.
2. You wire the route manually. The plugin can't know which labels your custom alert carries, so it does not generate a matcher. It's on you to route alerts to the receiver.

Wiring the route (the one manual step)

Run /alertmanager export. The DM'd config includes the receiver block plus a commented route stub:

# Custom (non-runbook) receivers — created via /alertmanager add-custom.
# The plugin cannot know which labels your custom alerts carry, so no
# matcher is generated. Uncomment each block and fill in the matcher(s)
# your Prometheus rules actually emit, then paste under `route.routes:`.
#  - matchers: [ <your labels, e.g. alertname="MyCustomAlert", severity="critical"> ]
#    receiver: my-name--my-team-my-channel
#    continue: true

Uncomment it, set the matcher(s) to labels your rule emits (e.g. alertname, severity, team), and paste under route.routes: in your alertmanager.yml alongside the plugin-generated runbook routes. Keep continue: true so the route composes cleanly with the others.

Silent-failure caveat: because the route is manual, the plugin can confirm the receiver is loaded in Alertmanager (the inventory "OK" badge) but it cannot confirm any route actually targets it. If you never add the matcher, the receiver exists and looks healthy yet receives zero alerts. After wiring, use /alertmanager validate --simulate <labels> to confirm an alert with your labels dispatches to the custom receiver.

Prometheus Operator (--format=crd)

/alertmanager export --format=crd emits the custom receiver into the AlertmanagerConfig (so its Secret + receiver exist), but — like the file format — generates no route for it. The plugin's generated CR gates entry on the runbook label a custom alert doesn't carry, so a custom receiver can't be reached through this CR without edits: add your own sub-route under spec.route.routes and widen the parent route's matchers to admit your alert's labels (the export leaves a commented stub showing where). For anything beyond a quick tweak, route the alert to the receiver from your own top-level Alertmanager config instead.

Naming rules

The custom <name> must be lowercase [a-z0-9_-], contain no -- (the reserved <slug>--<team>-<channel> separator), and must not collide with a runbook slug or a category set name (all, compute, application, …). The assembled <name>--<team>-<channel> is capped at 190 characters.

What your Prometheus rules need

For the plugin-generated routes to dispatch alerts to the right receiver:

- alert: HighCPUUsage
  expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace, pod) > 0.8
  for: 10m
  labels:
    severity: critical
    runbook: high-cpu-usage      # ← matches a plugin receiver's base slug
  annotations:
    summary: "Pod CPU above 80% for 10 minutes"
    description: "..."

The runbook label is what the plugin-generated routes match on. The value must equal the receiver's base slug (e.g., high-cpu-usage, not high-cpu-usage--alert-slo-channel). Routes auto-translate the base slug → suffixed receiver name.

Alerts without a runbook label fall through to route.receiver (the default fallback).

Required Prometheus labels per runbook

Each shipped runbook documents the labels its Quick diagnostics section expects in a "Required Prometheus labels" footer. Most expect at least namespace and pod (for compute, application, storage runbooks) plus the runbook-specific label like instance / job / service. Security runbooks differ — several use apiserver / Falco labels (e.g. k8s_ns_name, job) instead. See the corresponding runbooks/*.md file, or run /alertmanager docs requirements for the full per-alert metric / label / tooling matrix.

If a label is missing on an incoming alert, the template falls back to leaving the placeholder text in place — operators see <namespace> rather than a substituted value. Validate with:

/alertmanager validate --simulate runbook=<slug> namespace=<ns> pod=<pod>

That walks AM's route tree against your label set without firing a real alert.

samples/prometheus-rules.yaml ships a complete rule set covering all 30 runbooks with the correct label patterns. Use it as a starting point.

Receiver naming convention

The plugin names every receiver <runbook-slug>--<team-slug>-<channel-slug>:

high-cpu-usage--sre-alert-slo-channel
high-cpu-usage--sre-oncall-critical
database-connectivity-loss--platform-dba-team-channel

The team slug is part of the name because channel names are unique only per team — town-square exists in every team. Without it, the same runbook delivered to same-named channels in different teams would collide (Alertmanager requires globally-unique receiver names).

The -- after the runbook slug is the parse boundary — runbook filenames never contain doubles, so the runbook slug is always recoverable. The <team>-<channel> tail uses a single hyphen and is identity/display only (team and channel are also stored as separate fields); global uniqueness is enforced by the plugin rejecting duplicate names on save, not by the separator.

Fan-out: one alert, multiple channels

To deliver the same runbook to multiple channels (e.g., CPU alerts to both team channel AND oncall pager), run /alertmanager add once per destination:

/alertmanager add testing alert-slo-channel http://alertmanager:9093 compute
/alertmanager add testing oncall-critical http://alertmanager:9093 compute

The plugin creates 6 receivers in each channel, all channel-suffixed:

• high-cpu-usage--alert-slo-channel
• high-cpu-usage--oncall-critical
• (and the rest of the compute set, in both)

/alertmanager export (or the next /alertmanager add DM) produces a routes block like:

routes:
  - matchers: [runbook="high-cpu-usage"]
    receiver: high-cpu-usage--alert-slo-channel
    continue: true                               # keep evaluating after match
  - matchers: [runbook="high-cpu-usage"]
    receiver: high-cpu-usage--oncall-critical
    continue: true

continue: true on every plugin-generated route makes AM keep evaluating routes after a match, so the alert hits both receivers and both channels get the post. The plugin sets this unconditionally — each runbook's matcher is unique, so continue only changes behavior in the fan-out case, where it fixes the otherwise-dead second route.

Channel auto-creation

The destination channel doesn't have to exist before you run /alertmanager add. If missing, the plugin creates it as an open channel under the named team. The bot user (@alertmanagerbot) is the channel creator. The plugin doesn't add anyone else to the channel — that's a Mattermost admin concern after creation.

Multiple Alertmanagers

The plugin supports multiple AM backends as independent registrations:

/alertmanager add testing alerts-east http://alertmanager.east.example.com:9093 compute
/alertmanager add testing alerts-west http://alertmanager.west.example.com:9093 compute

Each gets its own channel binding. /alertmanager status (when invoked in either channel) queries the AM URL bound to that channel's receivers. Independent error handling — failure on one doesn't affect the other.

For one Mattermost serving multiple Alertmanagers reachable via different network paths (e.g., one MM serving K8s clusters in different VPCs), use the per-receiver host override:

/alertmanager add testing alerts-east http://am.east:9093 compute --webhook-host=http://mattermost.east.svc:8065
/alertmanager add testing alerts-west http://am.west:9093 compute --webhook-host=http://mattermost.west.svc:8065

The override takes precedence over the global WebhookHost setting at YAML render time for those specific receivers.

HA Alertmanager (peers of the same logical instance)

Every peer fires its configured webhooks independently for the same alert group. The Mattermost incoming webhook receiver doesn't deduplicate by default, so identical posts may appear if multiple peers send simultaneously.

Mitigations from least to most invasive:

• Set external_labels identical across peers so they group the same way at the AM side and one peer becomes the primary notifier
• Front Mattermost with a deduplication proxy (out of scope for this plugin)
• Configure only one peer to send webhooks (loses redundancy)

Validating your configuration

Three commands cover the common verification paths:

# Cheap, read-only — checks AM reachable + receivers loaded in AM
/alertmanager validate

# Route simulation — walks AM's loaded route tree against a label set,
# reports which receivers would catch the alert. No alert fired.
/alertmanager validate --simulate runbook=high-cpu-usage severity=critical

# Diff your channel's assembled YAML against what AM has loaded,
# with schema validation via prometheus/alertmanager/config.Load
/alertmanager export --diff-against-loaded

The --diff-against-loaded mode catches undefined-receiver references, malformed matchers, and route tree errors before the operator pastes new YAML and reloads AM.

The admin inventory page at /plugins/com.mattermost.alertmanager/admin/inventory provides the same simulation as a GUI form (Mode / Type / Target / Channel / Severity dropdowns) plus the org-wide inverse-drift view (receivers in AM that the plugin doesn't track).