Metrics & monitoring

Send Prometheus Alertmanager alerts to on-call with CallHeim.

Alertmanager webhook receiver (firing/resolved groups). CallHeim maps the payload, attaches repeats to the incident that is already open, and pages whoever is on call for the service it belongs to.

prometheussource names also accepted: prom

Alerts fired by Prometheus's Alertmanager become CallHeim alerts: repeats of the same alert collapse onto one incident, and the service the integration is bound to pages whoever is on call for it. CallHeim includes a payload mapping for Alertmanager’s webhook format. Alertmanager can also be pointed straight at this same URL with source=alertmanager if you run it without a separate Prometheus server in front.

How the alert reaches on-call

  1. Step 1Prometheus evaluates an alerting rule and hands it to Alertmanager, which groups it with other alerts sharing the same labels.
  2. Step 2Alertmanager POSTs the group as one JSON body to your CallHeim ingest URL, on the interval its route config sets.
  3. Step 3CallHeim splits the group into one alert per member and maps each alert's severity label.
  4. Step 4A repeat of an alert (by its own Alertmanager fingerprint) attaches to the incident that is already open for it, and alerts from one group that fire together can share an incident.
  5. Step 5The integration's bound service selects an escalation policy, which pages the on-call tier.

Setting it up

  1. 01

    Add a webhook_configs receiver to Alertmanager's configuration.

    Create the Prometheus Alertmanager integration in CallHeim first to get the exact ingest URL.

  2. 02

    Set send_resolved: true on the receiver.

    This makes Alertmanager POST again when alerts resolve, which is what resolves the incident — see Recovery below.

  3. 03

    Route alerts to the receiver.

    Add a route (or route tree) that matches the alerts you want paged to the receiver name you configured.

  4. 04

    Reload or restart Alertmanager to apply the config.

Example payloadJSON
{
  "version": "4",
  "groupKey": "{}:{alertname=\"HighRequestLatency\"}",
  "status": "firing",
  "receiver": "callheim",
  "commonLabels": { "alertname": "HighRequestLatency", "severity": "critical" },
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "HighRequestLatency", "severity": "critical", "instance": "api-1:9090" },
      "annotations": { "summary": "High request latency on api-1", "description": "p99 latency is 850ms (> 500ms) for 5m" },
      "startsAt": "2026-06-27T10:00:00Z",
      "fingerprint": "a1b2c3d4e5f6"
    }
  ]
}
Example payload in Alertmanager's webhook_config format; check the Alertmanager documentation for the current schema.

What CallHeim reads from the payload

Payload fields and what each one maps to on the incident
Payload fieldMaps toNote
annotations.summary / labels.alertname / annotations.descriptionincident title
labels.severityseveritycritical→P1, error or high→P2, warning→P3, info→P4; a severity label set to "page" also maps to P1, and anything else maps to P3.
annotations.description / summary / messageincident body
alerts[].fingerprintidentity signalAlertmanager's own per-alert fingerprint is stable across a firing→resolved pair, which is what lets a later resolved notification find the same incident.

How CallHeim processes it

Routing

Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged.

Deduplication

A repeat of the same alert attaches to the incident that is still open for it in the same service, triggered or acknowledged, with no new incident and no new page. When the sender supplies its own dedup key (a PagerDuty dedup_key, an Opsgenie alias, a Datadog alert id, an Alertmanager fingerprint, or a dedupKey that is more than the alert title), this holds however late the repeat arrives. When the identity comes from the e-mail subject or the alert title, the repeat attaches only if the incident was triggered in the last 24 hours; after that it opens a new incident and pages. A repeat at P1 or P2 is not folded into an open P3, P4 or P5 incident: it opens a new incident and pages. Once an incident is resolved or closed, the next repeat opens a new incident. The fingerprint is Alertmanager's own per-alert fingerprint when present, else the group key, else the alert name and instance — so the same alert re-firing collapses even across a resolve/fire pair. When the fingerprint is present it is the sender's own key, so a repeat attaches at any age.

Recovery

When Alertmanager sends a resolved notification, CallHeim resolves the incident. If several alerts from one group are on the same incident, it stays open until every one of them has recovered. Checked with a real Alertmanager (v0.28.1) on 26 September 2026, including groups where one alert recovers while another is still firing. An incident that also holds an alert which never sends a recovery stays open until a person resolves it.

Troubleshooting

Common problems and how to fix them
ProblemFix
An incident stays open after an alert resolved.Expected while another alert on the same incident is still firing — it resolves when the last one recovers (see Recovery above). If every alert has recovered, check that send_resolved: true is set on the receiver, or resolve it in CallHeim.
HTTP 400 on send.Check the Alertmanager version — the webhook payload schema (fields like values) changed between major versions.
Alerts page with no summary text.The alerting rule has no summary or description annotation, so the title falls back to the bare alertname.
No alerts ever arrive.Confirm the route actually matches to this receiver — Alertmanager's default route can silently absorb alerts that do not match a more specific route first.
HTTP 401 once you enabled signing.A static header from Alertmanager's http_config can't satisfy a per-request HMAC signature — leave CallHeim's signing off for this integration unless you add a proxy in front that can compute and add the header per request.

Security

Alertmanager's webhook receiver (http_config, http_headers) can send only a static header value, never a valid per-request HMAC-SHA256 signature of a changing body. The integration URL is the credential: keep it secret, and if it is exposed, disable the integration and create a new one. A signing proxy you run in front of the sender can add an X-CallHeim-Signature header; without one, do not set a signing secret, because every unsigned delivery would be rejected.

Vendor documentation checked

The thresholds it passes through

Repeat, sender-supplied dedup key
attaches, any age
Repeat, title or subject only
attaches, incident under 24 h old
Flap threshold
4 transitions / 600s
Title correlation
similarity ≥ 0.6, same source and service
Group window default
600s

All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup rule and the flap settings are fixed. A P1 or P2 repeat on an open P3 to P5 incident opens a new incident and pages. How alerts are processed →

CallHeim

Point Prometheus Alertmanager at CallHeim and see what it does with your alerts.

Start on the free trial, connect Prometheus Alertmanager and send a test alert to see the incident it opens.