Metrics & monitoring
Send Kubernetes Events alerts to on-call with CallHeim.
Cluster events via kubernetes-event-exporter webhook (Warning→P2). CallHeim maps the payload, attaches repeats to the incident that is already open, and pages whoever is on call for the service it belongs to.
Needs a forwarder
Kubernetes Events does not post to CallHeim on its own for this. You set up a forwarder, rule or script that sends the event to your CallHeim URL; the setup steps show what to send.
Setting it up
Configure kubernetes-event-exporter with a webhook receiver pointed at the URL, and a route that sends events to that receiver.
Create an Integration in CallHeim and choose Kubernetes Events from the Catalog to get its ingest URL, and use that URL as the target of your forwarder. CallHeim’s ingest URL takes a JSON request body. The snippet beside this shows the request it expects, with {{WEBHOOK_URL}} replaced by your real URL.
Kubernetes Events cannot sign its requests in a format CallHeim verifies, so the integration URL is the credential. Keep it secret, and if it is exposed, disable the integration and create a new one. Do not set a signing secret unless a signing proxy you run signs each request. With a secret set, every unsigned delivery is rejected.
CallHeim includes a payload mapping for Kubernetes Events’ webhook format.
# kubernetes-event-exporter config:
route:
routes:
- match:
- receiver: callheim
receivers:
- name: callheim
webhook:
endpoint: "{{WEBHOOK_URL}}"
# Without a route the exporter sends nothing to the receiver.
# To send Warning events only, add this beside match:
# drop:
# - type: "Normal"After it arrives
What CallHeim does with Kubernetes Events alerts.
Threshold alerts flap at the boundary. Where a source sends a recovery event CallHeim recognizes, a series that changes state four times inside ten minutes is grouped instead of re-paging, and the underlying incident stays open and visible.
CallHeim maps Kubernetes Events’ severity or priority field to a P1 to P5 level where the source sends one, and shows a separate, explainable severity suggestion that a person can apply. How severity is suggested →
Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged. Alerting and escalation →
The thresholds it passes through
- Repeat, sender-supplied dedup key
- attaches, any age
- Repeat, title or subject only
- attaches, incident under 24 h old
- Flap threshold
- 4 transitions / 600s
- Title correlation
- similarity ≥ 0.6, same source and service
- Group window default
- 600s
All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup rule and the flap settings are fixed. A P1 or P2 repeat on an open P3 to P5 incident opens a new incident and pages. How alerts are processed →
Point Kubernetes Events at CallHeim and see what it does with your alerts.
Start on the free trial, connect Kubernetes Events and send a test alert to see the incident it opens.