Logging

Send Grafana Loki alerts to on-call with CallHeim.

Loki ruler alerts via Alertmanager-shaped webhook. CallHeim maps the payload, attaches repeats to the incident that is already open, and pages whoever is on call for the service it belongs to.

lokisource names also accepted: grafana-loki

Grouped alerts

A group is split into one alert per member. An incident that holds several alerts stays open while any of them is still firing.

Setting it up

Create an Integration in CallHeim and choose Grafana Loki from the Catalog to get its ingest URL. Add that URL as a webhook receiver in the Alertmanager that Grafana Loki sends its alerts to. The snippet beside this is the receiver configuration, with {{WEBHOOK_URL}} replaced by your real URL.

Grafana Loki cannot sign its requests in a format CallHeim verifies, so the integration URL is the credential. Keep it secret, and if it is exposed, disable the integration and create a new one. Do not set a signing secret unless a signing proxy you run signs each request. With a secret set, every unsigned delivery is rejected.

CallHeim includes a payload mapping for Grafana Loki’s webhook format.

Grafana Loki documentation →

Setup snippetloki
# alertmanager config used by the Loki ruler:
receivers:
  - name: callheim
    webhook_configs:
      - url: "{{WEBHOOK_URL}}"
        send_resolved: true
Check Grafana Loki’s current documentation for the exact fields.

After it arrives

What CallHeim does with Grafana Loki alerts.

Log-derived alerts carry long, near-identical titles. A repeat of the same alert attaches to its open incident (within 24 hours of the incident opening unless the sender supplies its own dedup key), and alerts with closely similar titles from the same source can be grouped onto one open, unacknowledged incident within ten minutes, at a title similarity of at least 0.6.

CallHeim maps Grafana Loki’s severity or priority field to a P1 to P5 level where the source sends one, and shows a separate, explainable severity suggestion that a person can apply. How severity is suggested →

Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged. Alerting and escalation →

The thresholds it passes through

Repeat, sender-supplied dedup key
attaches, any age
Repeat, title or subject only
attaches, incident under 24 h old
Flap threshold
4 transitions / 600s
Title correlation
similarity ≥ 0.6, same source and service
Group window default
600s

All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup rule and the flap settings are fixed. A P1 or P2 repeat on an open P3 to P5 incident opens a new incident and pages. How alerts are processed →

CallHeim

Point Grafana Loki at CallHeim and see what it does with your alerts.

Start on the free trial, connect Grafana Loki and send a test alert to see the incident it opens.