Logging

Send Splunk alerts to on-call with CallHeim.

Saved-search webhook alert action. CallHeim maps the payload, attaches repeats to the incident that is already open, and pages whoever is on call for the service it belongs to.

splunk

Splunk saved-search alerts become CallHeim alerts: repeats of the same saved search collapse onto one incident, and the service the integration is bound to pages whoever is on call for it. CallHeim includes a payload mapping for Splunk’s webhook format.

How the alert reaches on-call

  1. Step 1A Splunk saved search runs on its schedule and its trigger condition matches.
  2. Step 2Splunk's webhook alert action POSTs a JSON body (the search name and the first result row) to your CallHeim ingest URL.
  3. Step 3CallHeim reads the search name as the identity signal; severity comes only from a field your search populates, defaulting to P3.
  4. Step 4A repeat of the same saved search attaches to the incident that is already open for it, if that incident was triggered in the last 24 hours.
  5. Step 5The integration's bound service selects an escalation policy, which pages the on-call tier.

Setting it up

  1. 01

    Add the CallHeim ingest URL to the webhook allow list.

    Splunk Cloud Platform 8.2203+ and Splunk Enterprise 9.0+ refuse to POST to a URL that is not allow-listed first.

  2. 02

    Add a Webhook action to the alert.

    From the saved search's alert actions (Save As Alert, or Edit Actions on an existing one), add Webhook and enter your CallHeim ingest URL.

  3. 03

    Populate a severity field in the search itself.

    Splunk has no built-in severity concept for a search result — add an eval such as severity="critical" (or urgency, or priority) to the search if you want CallHeim to read anything other than the P3 default.

Example payloadJSON
{
  "sid": "scheduler__admin__search__RMD5abc_at_1751020800_42",
  "search_name": "Failed logins > 100 in 5m",
  "result": {
    "count": "137",
    "host": "auth-prod-1",
    "severity": "high",
    "message": "137 failed logins from 10.0.0.5"
  },
  "results_link": "https://splunk.example/app/search/..."
}
Example payload from Splunk's webhook alert action; check Splunk's own documentation for the current fields.

What CallHeim reads from the payload

Payload fields and what each one maps to on the incident
Payload fieldMaps toNote
search_name / search / result.search_nameincident title
result.severity / result.urgency / result.priority / top-level severityseverityRead as a general severity word (for example "critical" or "high"); defaults to P3 when none of these is populated by the search.
result.message / result._raw / results_linkincident body
search_name, scoped by result.host or result.sourceidentity signalsid is the per-run scheduler id and is deliberately not read, so repeated firings of the same saved search collapse instead of each run opening a new incident.

How CallHeim processes it

Routing

Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged.

Deduplication

A repeat of the same alert attaches to the incident that is still open for it in the same service, triggered or acknowledged, with no new incident and no new page. When the sender supplies its own dedup key (a PagerDuty dedup_key, an Opsgenie alias, a Datadog alert id, an Alertmanager fingerprint, or a dedupKey that is more than the alert title), this holds however late the repeat arrives. When the identity comes from the e-mail subject or the alert title, the repeat attaches only if the incident was triggered in the last 24 hours; after that it opens a new incident and pages. A repeat at P1 or P2 is not folded into an open P3, P4 or P5 incident: it opens a new incident and pages. Once an incident is resolved or closed, the next repeat opens a new incident. Identity is the saved search's name (scoped by a host or source field in the result row if present), so repeated firings of the same search collapse even though each run has a different sid. Because the identity is the search name, the 24-hour limit applies: a repeat joins the open incident only if it was triggered in the last 24 hours. While the incident is open, a repeat that attaches to it does not page again.

Recovery

Splunk has no recovery event at all for this integration: a saved search fires each time its condition matches and simply stops firing when it no longer does — there is no distinct "cleared" notification the way an uptime check or a metrics threshold sends one. A person resolves the incident once the underlying condition is known to be gone.

Troubleshooting

Common problems and how to fix them
ProblemFix
The webhook never fires.The URL is not on the webhook allow list yet — add it in Splunk's alert action settings before the action can send anything.
Every alert lands at P3.The search has no severity, urgency or priority field — add one with an eval in the search itself.
The title is the raw search text, not the search name.search_name was empty in the delivered body; check the saved search itself has a name set.
Repeats from the same search open separate incidents.The earlier incident had already been resolved or closed, it was triggered more than 24 hours ago (the identity is the search name, so the 24-hour limit applies), or the result row's host/source value changed between runs, which changes the identity scope.
HTTP 401 once you enabled signing.Splunk's built-in webhook alert action has no field for a custom header, so it cannot send X-CallHeim-Signature. Leave signing off unless you route the webhook through a script that can add it.

Security

Splunk's built-in webhook alert action does not expose a custom-header field. The integration URL is the credential: keep it secret, and if it is exposed, disable the integration and create a new one. A signing proxy you run in front of the sender can add an X-CallHeim-Signature header; without one, do not set a signing secret, because every unsigned delivery would be rejected.

Vendor documentation checked

The thresholds it passes through

Repeat, sender-supplied dedup key
attaches, any age
Repeat, title or subject only
attaches, incident under 24 h old
Flap threshold
4 transitions / 600s
Title correlation
similarity ≥ 0.6, same source and service
Group window default
600s

All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup rule and the flap settings are fixed. A P1 or P2 repeat on an open P3 to P5 incident opens a new incident and pages. How alerts are processed →

CallHeim

Point Splunk at CallHeim and see what it does with your alerts.

Start on the free trial, connect Splunk and send a test alert to see the incident it opens.