Metrics & monitoring
Send Prometheus Alertmanager (direct) alerts to on-call with CallHeim.
Alertmanager pointed straight at CallHeim (no Prometheus relay). CallHeim maps the payload, attaches repeats to the incident that is already open, and pages whoever is on call for the service it belongs to.
Alerts pushed straight into Alertmanager — from Cortex, Thanos Ruler, Mimir's ruler, VictoriaMetrics' vmalert, or a script posting to Alertmanager's own HTTP API — become CallHeim alerts: repeats of the same alert collapse onto one incident, and the service the integration is bound to pages whoever is on call for it. CallHeim includes a payload mapping for Alertmanager’s webhook format. This is the direct-receiver catalog entry: if you run a Prometheus server evaluating rules in front of Alertmanager, the Prometheus Alertmanager guide is the more specific one to read — the wire format and this integration's setup are otherwise identical.
How the alert reaches on-call
- Step 1A rule engine other than Prometheus itself — Cortex, Thanos Ruler, Mimir's ruler, VictoriaMetrics' vmalert — or a script calling Alertmanager's own API evaluates a condition and posts the alert straight to Alertmanager, or someone posts one with amtool.
- Step 2Alertmanager groups it with other alerts sharing the same labels, exactly as it would a Prometheus-originated alert.
- Step 3Alertmanager POSTs the group as one JSON body to your CallHeim ingest URL, on the interval its route config sets.
- Step 4CallHeim splits the group into one alert per member, maps each alert's severity label, and stores the identity namespaced under am: so it cannot collide with a Prometheus-routed alert.
- Step 5The integration's bound service selects an escalation policy, which pages the on-call tier.
Setting it up
- 01
Create the Alertmanager (direct) integration in CallHeim.
This catalog entry has its own ingest URL, already tied to the alertmanager parser, so no query parameter is needed.
- 02
Add a webhook_configs receiver to Alertmanager's configuration, pointed at that URL.
The same alertmanager.yml receivers block as any other webhook_config; nothing about the alert's origin changes the syntax.
- 03
Set send_resolved: true on the receiver.
Makes Alertmanager POST again when alerts resolve, which is what resolves the CallHeim incident — see Recovery below.
- 04
Route the alerts you push directly into Alertmanager to this receiver.
A route (or route tree) matching the labels your ruler or script sets — the same routing mechanism Alertmanager always uses, regardless of who created the alert.
- 05
Reload or restart Alertmanager to apply the config.
{
"version": "4",
"groupKey": "{}:{alertname=\"ReplicationLagHigh\"}",
"status": "firing",
"receiver": "callheim-direct",
"commonLabels": { "alertname": "ReplicationLagHigh", "severity": "critical" },
"alerts": [
{
"status": "firing",
"labels": { "alertname": "ReplicationLagHigh", "severity": "critical", "instance": "pg-replica-2" },
"annotations": { "summary": "Replica lag on pg-replica-2 exceeds 60s", "description": "Pushed by a custom checker script, not a Prometheus rule" },
"startsAt": "2026-06-27T10:00:00Z",
"fingerprint": "b7c8d9e0f1a2"
}
]
}What CallHeim reads from the payload
| Payload field | Maps to | Note |
|---|---|---|
| annotations.summary / labels.alertname / annotations.description | incident title | |
| labels.severity | severity | critical→P1, error or high→P2, warning→P3, info→P4; a severity label set to "page" also maps to P1, and anything else maps to P3. The mapping is identical to the Prometheus-routed integration. |
| annotations.description / summary / message | incident body | |
| alerts[].fingerprint | identity signal | Namespaced under am:, so a direct Alertmanager alert and a Prometheus-routed one sharing the same underlying fingerprint never collapse onto each other if a tenant runs both integrations. |
How CallHeim processes it
Routing
Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged.
Deduplication
A repeat of the same alert attaches to the incident that is still open for it in the same service, triggered or acknowledged, with no new incident and no new page. When the sender supplies its own dedup key (a PagerDuty dedup_key, an Opsgenie alias, a Datadog alert id, an Alertmanager fingerprint, or a dedupKey that is more than the alert title), this holds however late the repeat arrives. When the identity comes from the e-mail subject or the alert title, the repeat attaches only if the incident was triggered in the last 24 hours; after that it opens a new incident and pages. A repeat at P1 or P2 is not folded into an open P3, P4 or P5 incident: it opens a new incident and pages. Once an incident is resolved or closed, the next repeat opens a new incident. The fingerprint is Alertmanager's own per-alert fingerprint when present, else the group key, else the alert name and instance — namespaced under am:, distinct from the Prometheus integration's own prom: namespace even when both would otherwise see the identical alert. When the fingerprint is present it is the sender's own key, so a repeat attaches at any age.
Recovery
When Alertmanager sends a resolved notification, CallHeim resolves the incident, exactly as for the Prometheus-routed integration. If several alerts from one group are on the same incident, it stays open until every one of them has recovered. Checked with a real Alertmanager (v0.28.1) on 26 September 2026, including groups where one alert recovers while another is still firing. An incident that also holds an alert which never sends a recovery stays open until a person resolves it.
Troubleshooting
| Problem | Fix |
|---|---|
| An incident stays open after an alert resolved. | Expected while another alert on the same incident is still firing — it resolves when the last one recovers (see Recovery above). If every alert has recovered, check that send_resolved: true is set on the receiver, or resolve it in CallHeim. |
| HTTP 400 on send. | Check the Alertmanager version — the webhook payload schema (for example the values field) changed between major versions, independent of who is pushing alerts into it. |
| Alerts from a script or ruler never reach CallHeim. | Confirm the pushed alert actually matches a route pointing at this receiver — an alert with no matching route falls through to Alertmanager's default route, same as a Prometheus-originated one would. |
| A directly-pushed alert has no summary text. | Whatever created the alert (script, ruler) did not set a summary or description annotation on it — CallHeim falls back to the bare alertname. |
| HTTP 401 once you enabled signing. | Alertmanager's http_headers sends a fixed value, not a per-request signature — it can't satisfy CallHeim's HMAC check. Leave signing off for this integration unless you add a proxy in front that can compute and add the header per request. |
Security
A current Alertmanager accepts a custom header through the webhook receiver's http_config (http_headers), but that value is static, and a static header can never be a valid per-request HMAC-SHA256 signature of a changing body. The integration URL is the credential: keep it secret, and if it is exposed, disable the integration and create a new one. A signing proxy you run in front of the sender can add an X-CallHeim-Signature header; without one, do not set a signing secret, because every unsigned delivery would be rejected.
Vendor documentation checked
- Prometheus: Alertmanager webhook_config — checked 2026-09-24
The thresholds it passes through
- Repeat, sender-supplied dedup key
- attaches, any age
- Repeat, title or subject only
- attaches, incident under 24 h old
- Flap threshold
- 4 transitions / 600s
- Title correlation
- similarity ≥ 0.6, same source and service
- Group window default
- 600s
All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup rule and the flap settings are fixed. A P1 or P2 repeat on an open P3 to P5 incident opens a new incident and pages. How alerts are processed →
Point Prometheus Alertmanager (direct) at CallHeim and see what it does with your alerts.
Start on the free trial, connect Prometheus Alertmanager (direct) and send a test alert to see the incident it opens.