Metrics & monitoring

Send Grafana alerts to on-call with CallHeim.

Grafana unified alerting contact point (8+) and legacy. CallHeim maps the payload, attaches repeats to the incident that is already open, and pages whoever is on call for the service it belongs to.

grafanasource names also accepted: grafana-alerting

Grafana alert rules become CallHeim alerts: repeats of the same alert collapse onto one incident, and the service the integration is bound to pages whoever is on call for it. CallHeim includes a payload mapping for Grafana’s webhook format.

How the alert reaches on-call

  1. Step 1A Grafana alert rule evaluates and its state changes; unified alerting groups it with other firing alerts sharing the same labels, the way Alertmanager does.
  2. Step 2Grafana POSTs the group as one JSON body to your CallHeim ingest URL from its webhook contact point.
  3. Step 3CallHeim splits the group into one alert per member and maps each member's severity label (or, on a legacy pre-8 payload, its state).
  4. Step 4A repeat of that alert (by its own Grafana fingerprint) attaches to the incident that is already open for it.
  5. Step 5The integration's bound service selects an escalation policy, which pages the on-call tier.

Setting it up

  1. 01

    Add a Webhook contact point in Alerting > Contact points.

    Create the Grafana integration in CallHeim first to get the exact ingest URL.

  2. 02

    Paste the ingest URL into the URL field.

    Grafana stores and displays this URL in plain text when the contact point is read back — treat it the same way you would any other credential.

  3. 03

    Leave the HTTP method on POST and the payload on the default JSON body, unless you need custom fields.

  4. 04

    Attach a notification policy to the contact point.

    Match it to the alert rules you want paged.

Example payloadJSON
{
  "status": "firing",
  "title": "[FIRING:1] DiskSpaceLow",
  "message": "Disk usage is 92%",
  "commonLabels": { "alertname": "DiskSpaceLow", "severity": "warning" },
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "DiskSpaceLow", "severity": "warning", "instance": "db-1" },
      "annotations": { "summary": "Disk space low on db-1", "description": "Only 8% free" },
      "fingerprint": "ggg111"
    }
  ]
}
Example payload in Grafana's unified-alerting webhook format; check Grafana's own documentation for the current schema.

What CallHeim reads from the payload

Payload fields and what each one maps to on the incident
Payload fieldMaps toNote
alerts[].annotations.summary / top-level title / labels.alertnameincident titleUnified alerting (Grafana 8+).
labels.severityseverityRead as a general severity word, not a fixed enum — an unrecognized value defaults to P3.
annotations.description / top-level message / valueStringincident body
alerts[].fingerprintidentity signalFalls back to the alert name with no fingerprint.
ruleName / state (legacy, pre-8)title / severityalerting→P2, no_data→P3, ok or paused→P4.

How CallHeim processes it

Routing

Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged.

Deduplication

A repeat of the same alert attaches to the incident that is still open for it in the same service, triggered or acknowledged, with no new incident and no new page. When the sender supplies its own dedup key (a PagerDuty dedup_key, an Opsgenie alias, a Datadog alert id, an Alertmanager fingerprint, or a dedupKey that is more than the alert title), this holds however late the repeat arrives. When the identity comes from the e-mail subject or the alert title, the repeat attaches only if the incident was triggered in the last 24 hours; after that it opens a new incident and pages. A repeat at P1 or P2 is not folded into an open P3, P4 or P5 incident: it opens a new incident and pages. Once an incident is resolved or closed, the next repeat opens a new incident. The fingerprint is Grafana's own per-alert fingerprint when present, else the alert name (or, on the legacy payload, the rule id). When the fingerprint is present it is the sender's own key, so a repeat attaches at any age.

Recovery

Grafana's unified alerting sends the same Alertmanager-shaped group envelope, and CallHeim reads each alert's own status in it. An incident that holds several alerts stays open while any of them is still firing.

Troubleshooting

Common problems and how to fix them
ProblemFix
A resolved alert stays open.Check whether it shares a notification policy grouping with an alert that is still firing — see Recovery above.
One incident holds several alerts from a group.Expected: each alert in the group is processed on its own, and alerts with similar titles from the same source can be grouped onto one open incident. Use a narrower notification-policy grouping to keep them apart.
HTTP 400 on send.A custom payload template on the contact point produced invalid JSON — validate it in Grafana's template preview before saving.
No alerts arrive.Confirm the notification policy actually routes to this contact point; an unmatched alert falls through to the default policy.
HTTP 401 once you enabled signing.Grafana's custom header on a webhook contact point is a fixed value, not a per-request signature — it can't satisfy CallHeim's HMAC check. Leave signing off for this source unless you add a proxy in front that can compute and add the header per request.

Security

Grafana's webhook contact point can send a custom header, but only a static value, never a valid per-request HMAC-SHA256 signature of a changing body. The integration URL is the credential: keep it secret, and if it is exposed, disable the integration and create a new one. A signing proxy you run in front of the sender can add an X-CallHeim-Signature header; without one, do not set a signing secret, because every unsigned delivery would be rejected.

Vendor documentation checked

The thresholds it passes through

Repeat, sender-supplied dedup key
attaches, any age
Repeat, title or subject only
attaches, incident under 24 h old
Flap threshold
4 transitions / 600s
Title correlation
similarity ≥ 0.6, same source and service
Group window default
600s

All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup rule and the flap settings are fixed. A P1 or P2 repeat on an open P3 to P5 incident opens a new incident and pages. How alerts are processed →

CallHeim

Point Grafana at CallHeim and see what it does with your alerts.

Start on the free trial, connect Grafana and send a test alert to see the incident it opens.