Splunk Alerts to On-Call

Splunk saved-search alerts become CallHeim alerts: repeats of the same saved search collapse onto one incident, and the service the integration is bound to pages whoever is on call for it. CallHeim includes a payload mapping for Splunk’s webhook format, built and tested against sample payloads.

How the alert reaches on-call

Step 1A Splunk saved search runs on its schedule and its trigger condition matches.
Step 2Splunk's webhook alert action POSTs a JSON body (the search name and the first result row) to your CallHeim ingest URL.
Step 3CallHeim reads the search name as the identity signal; severity comes only from a field your search populates, defaulting to P3.
Step 4Repeats of the same saved search within five minutes collapse onto one incident.
Step 5The integration's bound service selects an escalation policy, which pages the on-call tier.

Setting it up

01
Add the CallHeim ingest URL to the webhook allow list.
Splunk Cloud Platform 8.2203+ and Splunk Enterprise 9.0+ refuse to POST to a URL that is not allow-listed first.
02
Add a Webhook action to the alert.
From the saved search's alert actions (Save As Alert, or Edit Actions on an existing one), add Webhook and enter your CallHeim ingest URL.
03
Populate a severity field in the search itself.
Splunk has no built-in severity concept for a search result — add an eval such as severity="critical" (or urgency, or priority) to the search if you want CallHeim to read anything other than the P3 default.

Example payloadJSON

{
  "sid": "scheduler__admin__search__RMD5abc_at_1751020800_42",
  "search_name": "Failed logins > 100 in 5m",
  "result": {
    "count": "137",
    "host": "auth-prod-1",
    "severity": "high",
    "message": "137 failed logins from 10.0.0.5"
  },
  "results_link": "https://splunk.example/app/search/..."
}

Example shape written by us, matching Splunk's documented webhook alert action payload; check Splunk's own documentation for the current fields.

What CallHeim reads from the payload

Payload fields and what each one maps to on the incident
Payload field	Maps to	Note
search_name / search / result.search_name	incident title
result.severity / result.urgency / result.priority / top-level severity	severity	Read as a general severity word (for example "critical" or "high"); defaults to P3 when none of these is populated by the search.
result.message / result._raw / results_link	incident body
search_name, scoped by result.host or result.source	identity signal	sid is the per-run scheduler id and is deliberately not read, so repeated firings of the same saved search collapse instead of each run opening a new incident.

How CallHeim processes it

Routing

Each alert source is bound to a service, and the service’s escalation policy (or its team’s) sets who is paged.

Deduplication

Repeats of the same alert collapse onto one incident while they keep arriving within five minutes of each other. A repeat that arrives after a longer quiet gap opens a new incident. Identity is the saved search's name (scoped by a host or source field in the result row if present), so repeated firings of the same search collapse even though each run has a different sid.

Recovery

Splunk has no recovery event at all for this integration: a saved search fires each time its condition matches and simply stops firing when it no longer does — there is no distinct "cleared" notification the way an uptime check or a metrics threshold sends one. A person resolves the incident once the underlying condition is known to be gone.

Troubleshooting

Common problems and how to fix them
Problem	Fix
The webhook never fires.	The URL is not on the webhook allow list yet — add it in Splunk's alert action settings before the action can send anything.
Every alert lands at P3.	The search has no severity, urgency or priority field — add one with an eval in the search itself.
The title is the raw search text, not the search name.	search_name was empty in the delivered body; check the saved search itself has a name set.
Repeats from the same search open separate incidents.	They arrived more than five minutes apart, or the result row's host/source value changed between runs, which changes the identity scope.
HTTP 401 once you enabled signing.	Splunk's built-in webhook alert action has no field for a custom header, so it cannot send X-ItOnCall-Signature — leave signing off unless you route the webhook through a script that can add it.

Security

CallHeim verifies an HMAC-SHA256 signature on each source’s requests once you enable signing on that integration (the sending tool must be able to sign). Splunk's built-in webhook alert action does not expose a custom-header field, so it typically cannot send X-ItOnCall-Signature — leave signing off and rely on the ingest URL as the credential.

Vendor documentation checked

Splunk: Use a webhook alert action — checked 2026-09-24

The thresholds it passes through

Dedup window: 300s
Flap threshold: 4 transitions / 600s
Title correlation: similarity ≥ 0.6, same source and service
Group window default: 600s

All defaults are published. You can turn title correlation off or change its threshold, and set the window on your own noise rules; the dedup window and the flap settings are fixed. How alerts are processed →

Send Splunk alerts to on-call with CallHeim.