Skip to main content
Version: 1.4

Configuring Notification Channels

Notifications carry a raised alarm the last mile, to a person. When a detection rule's actions raise an alarm, a per-tenant policy routes it by severity to the channels you configure: email over SMTP, a webhook, or a native Slack, Microsoft Teams or PagerDuty channel. Policies add throttling and escalation of unacknowledged alarms.

This machine-to-human path is deliberately separate from the machine-to-machine outbound connectors. Connectors carry payloads to systems; notifications carry alerts to people, with recipients and per-severity routing. For how alarms are raised in the first place, see Event Processing & Alarms.

To manage policies and quiet hours from the web console instead, see Notification policies in the user guide.

Status

Available. You manage channels and policies over the notification-management GraphQL API. Reading requires the notification:read authority; creating or changing anything requires notification:write.

Channels​

A channel is a delivery endpoint your tenant configures: an instance of a channel type, plus its connection config. Query notificationChannelTypes for the types the platform defines. Today smtp, webhook, slack, teams and pagerduty ship with working adapters; each type's available flag says whether its adapter has landed.

A channel splits its settings in two:

  • config: the non-secret connection settings, as a JSON document (SMTP host/port/from; webhook URL/method/headers and how it authenticates).
  • secret: the credential, such as the SMTP password or a webhook auth token. It is stored in the platform's envelope-encrypted secret store and is write-only. You submit it on create, and it is never returned on read; the channel exposes only a hasSecret boolean.

On an update, the secret field behaves as follows:

  • Omit it to leave the existing secret unchanged. You never need to re-send it.
  • Send a non-null value to replace it.
  • Send null or an empty string to clear it.
  • On a webhook channel whose config declares bearer or header auth, clearing the secret is refused. Switch auth to none in the same request if you mean the endpoint to be anonymous.

If your client binds one variable per field, an unsupplied variable arrives as an explicit null and clears the secret. On a webhook channel that declares bearer or header auth, that same null makes the whole update fail instead, even one that only meant to rename the channel. Send the whole request as a single variable and leave the secret key out.

Create an SMTP channel​

mutation {
createNotificationChannel(request: {
token: "ops-email",
name: "Operations email",
channelType: "smtp",
config: "{\"host\":\"smtp.example.com\",\"port\":587,\"from\":\"alerts@example.com\",\"username\":\"alerts\",\"security\":\"starttls\"}",
secret: "<smtp password>",
enabled: true
}) { token channelType hasSecret enabled }
}

An SMTP channel that sets username must have a secret. A delivery to one that has none is refused on its first attempt, before the platform connects to the mail server, and is not retried; it is counted with reason="credential" (see below). This is checked at delivery, not when you save the channel.

Create a webhook channel​

A webhook channel POSTs the rendered notification to a URL. Create it the same way, with channelType: "webhook" and a config carrying the url, an auth mode, and optionally method and extra headers. The only accepted method is POST, which is also the default; any other method is refused when you save the channel.

auth is required and says how the channel authenticates:

authWhat is sentsecret
noneNo credential header. Use this when the URL itself carries the credential, as a Slack incoming webhook's does.Must not be set
bearerAuthorization: Bearer <secret>Required
headerThe secret in the header named by authHeader, prefixed by authScheme and a space if you set one. For example, "authHeader":"X-API-Key" sends the raw token, and "authHeader":"Authorization","authScheme":"Token" sends Authorization: Token <secret>.Required

authHeader and authScheme are read only with header. With none or bearer, leave them out: a channel that sets them is refused rather than having them silently ignored.

A channel whose auth and secret disagree is refused when you save it, not when an alarm fires. That covers a missing auth, bearer or header with no secret, and none with a secret. To make a bearer channel anonymous, send auth none and secret: null in the same update. An update that only renames, describes or disables a channel is not checked, so you can always switch a misconfigured channel off; enabling one is checked.

A channel that reaches delivery in that state anyway, for example one saved before auth existed, is not sent. The delivery is refused on its first attempt and not retried, and the notification service logs the tenant, the channel's token and the reason. The refusal is counted on devicechain_notificationmanagement_deliveries_refused_total{reason="credential"}, which you can alert on. The same counter carries reason="egress" for a webhook or SMTP channel whose destination is on an address outbound traffic may not reach, such as a private, loopback or cloud-metadata address. It also carries reason="no_adapter", counted when a channel of a type this build has no delivery adapter for is skipped, so nobody is notified through it. The credential and egress refusals are terminal: the delivery is not retried.

mutation {
createNotificationChannel(request: {
token: "oncall-hook",
name: "On-call webhook",
channelType: "webhook",
config: "{\"url\":\"https://hooks.example.com/alarms\",\"auth\":\"bearer\"}",
secret: "<token>",
enabled: true
}) { token channelType hasSecret enabled }
}

For a Slack incoming webhook, use "auth":"none" and leave secret out. The native Slack channel below formats the message for you, and is the better choice.

The generic webhook text is a plain summary that uses the device token or originator instead of the device name; Slack users should use the native Slack channel.

A webhook channel's JSON body carries the alarm's eventType (RAISED, ESCALATED, DEESCALATED, CLEARED, ACKNOWLEDGED, or RENOTIFIED for a timed re-notification), alarmToken, alarmKey, severity, state, the originator's originatorType and originatorId when known, and the times. When the alarm's device is known it also carries deviceToken, deviceName and deviceTypeToken; a key is left out when its value is unknown (for example a device with no name), never sent empty. Timed re-notifications (RENOTIFIED) and late CLEARED notifications picked up by the scheduler also include these device and originator fields when known. Alarms that were notified before upgrading omit these fields from those later notifications.

Optionally set linkBaseUrl in the webhook channel's config to your console's http or https base URL (https recommended), for example "linkBaseUrl":"https://console.example.com". The payload then includes alarmUrl and, when the alarm names its device, deviceUrl. The base URL must not carry a query string or fragment, because links are built by appending a path. Both keys are absent when linkBaseUrl is unset. If a stored webhook has an invalid linkBaseUrl, delivery skips it, omits both links and logs a warning.

Native Slack, Teams and PagerDuty channels​

The three native types speak each service's own format instead of a generic JSON body. They share three rules:

  • The credential is the secret. For Slack and Teams it is the webhook URL (which is itself a credential, so it is stored encrypted and never returned, not kept in config); for PagerDuty it is the Events API v2 routing key. There is no auth setting. A native channel with no secret is refused when you save it, and a delivery to one that has none is refused on its first attempt and not retried, counted with reason="credential".
  • config is optional and strict. An unknown key is refused when you save the channel, so a misspelled setting cannot be silently ignored. linkBaseUrl is the http or https address of your console (https recommended), with no query string or fragment; when set, the message carries an Open alarms button or link to <linkBaseUrl>/alarms.
  • Delivery goes through the same outbound guard as a webhook. A destination on a private, loopback or cloud-metadata address is refused and not retried (reason="egress"), and the instance's egress allowances apply exactly as for a webhook. A rule's recipients are ignored: the webhook or the routing key decides where the message goes.

Each message names the alarm, its severity, its state, the device it came from, the metric and value that raised it, and when it was raised. While an alarm is open and unacknowledged, a hint says to acknowledge it in the console to stop further notifications; a cleared or acknowledged alarm omits the hint.

Create a Slack channel​

A Slack channel posts a Block Kit message to a Slack incoming-webhook URL: a header, a grid of the alarm's fields, the acknowledge hint and an optional Open alarms button. The message's colour bar follows the severity (red for CRITICAL, orange for MAJOR, amber for MINOR and WARNING, grey for INDETERMINATE), turning green when the alarm clears and blue when it is acknowledged. Text from the alarm is escaped so it cannot inject a Slack mention or link.

mutation {
createNotificationChannel(request: {
token: "ops-slack",
name: "Operations Slack",
channelType: "slack",
config: "{\"linkBaseUrl\":\"https://console.example.com\"}",
secret: "https://hooks.slack.com/services/T000/B000/XXXXXXXX",
enabled: true
}) { token channelType hasSecret enabled }
}

The secret must be an https URL.

Create a Microsoft Teams channel​

A Teams channel posts an Adaptive Card to the webhook URL of a Teams Workflows (Power Automate) flow that uses the When a Teams webhook request is received trigger and posts the card to a chat or channel. The retired Office 365 connector webhooks are not supported. The card carries the same facts as the Slack message, with the title coloured by severity, and an Open alarms action when linkBaseUrl is set.

mutation {
createNotificationChannel(request: {
token: "ops-teams",
channelType: "teams",
secret: "https://prod-00.westus.logic.azure.com:443/workflows/...",
enabled: true
}) { token channelType hasSecret enabled }
}

Create a PagerDuty channel​

A PagerDuty channel sends Events API v2 events, and maps the alarm's lifecycle onto the incident's. Every event carries dedup_key set to the alarm's token, so the whole life of one alarm is one incident:

Alarm eventPagerDuty action
raised, escalated, de-escalated, or re-notified by a policy's escalationtrigger (a repeat on an open incident updates it instead of opening another)
acknowledgedacknowledge
clearedresolve

The alarm's severity maps to PagerDuty's: CRITICAL to critical, MAJOR to error, MINOR and WARNING to warning, anything else to info. For the policy to resolve incidents, route the cleared and acknowledged transitions through the same channel as the raise.

config accepts region (us, the default, or eu, which selects events.eu.pagerduty.com), source (the event's source; devicechain by default) and linkBaseUrl.

mutation {
createNotificationChannel(request: {
token: "oncall-pagerduty",
channelType: "pagerduty",
config: "{\"region\":\"eu\",\"source\":\"plant-3\"}",
secret: "<routing key>",
enabled: true
}) { token channelType hasSecret enabled }
}

In the console​

Notification Channels, next to Notification Policies in the sidebar, lists, creates, edits and deletes channels of every type with a form for each. The credential box is write-only: on an existing channel it shows that a credential is stored, and leaving it empty keeps it.

A webhook channel can also sign its deliveries so the receiver can verify them. It is off unless you ask for it, by adding "signing":true to the channel's config once you have generated a signing secret for it. A channel set to sign that holds no secret is refused, not sent unsigned, and counted with reason="signing". See Signing Outbound Webhooks.

Chat messages and the webhook summary name the device by its name (or its token when it has no name), and list its device type. When the channel has a linkBaseUrl and the alarm names its device, the message also carries an Open device button or link to <linkBaseUrl>/devices/<device token>; with no linkBaseUrl, or for an alarm that does not name its device, there is no device link.

Policies​

A policy decides which raised alarms are delivered, to whom, and through which channels. It carries a set of rules. Each rule maps:

  • a severity: CRITICAL, MAJOR, MINOR, WARNING, INDETERMINATE, or "*" for any;
  • to a channel, named by token;
  • with a JSON array of recipients that the adapter interprets: email addresses for SMTP; it may be empty for a webhook.

The severity here is uppercase because it is the alarm's severity. A detection rule's own authoring severity is lowercase (major) and is uppercased when the alarm is raised, so a notification rule always matches the uppercase form. A rule whose severity is not one of those values (a lowercase major, say) is refused at write, rather than stored as a rule that could never match.

Policies are tenant-wide

deviceTypeToken is not honoured yet, and a policy that sets it is refused at write. Leave deviceTypeToken unset. See Device-type scoping for why.

Notify when an alarm clears​

By default a policy tells its channels only when an alarm is raised or escalates. Set notifyOnClear: true on a policy to also tell them when an alarm that the policy paged clears. The clear goes to the channels and recipients that actually delivered a page for that alarm, among those the policy routes, whatever severity the alarm has when it clears (an alarm raised CRITICAL, lowered to MAJOR and then cleared resolves on the channel that paged it). Its eventType is CLEARED, which PagerDuty maps to a resolve. It is off by default, so a policy that existed before the option behaves as it did.

mutation {
updateNotificationPolicy(token: "ops", request: { notifyOnClear: true }) { token notifyOnClear }
}

A clear is informational, so it is handled differently from a page:

  • It goes only to destinations that delivered a page in that cycle. A policy that never paged the alarm (its notification was throttled, held for quiet hours, or the policy was created after the raise) sends no clear.
  • It is sent once per cycle, after the clear has been recorded, so no "still unresolved" escalation follows it. If no channel takes it, it is retried; once any channel has taken it, it is not repeated. A clear that is processed before its alarm's page has been recorded is sent right after that page, and one that could not be sent then is picked up by the notification scheduler's next passes for up to 30 minutes.
  • It is not throttled, does not count as a notification, and does not start escalation.
  • A clear that happens during a policy's quiet hours is dropped, not held: a "cleared" released hours later would report old news.
  • Enabling the option does not announce historical clears: the clear must be after the option was enabled. A crash after claiming a clear but before sending can lose that announcement; it is not an exactly-once delivery guarantee.
  • notifyOnClear cannot be cleared with null; send true or false.

Device-type scoping​

Scoping a policy to a device type needs a cross-service lookup from the alarm's originator to its device type, which has not landed. Until it does, the dispatcher skips a scoped policy rather than applying it tenant-wide and over-notifying. Refusing the write is deliberate: a policy that accepted the field would return success and then deliver nothing at all.

Throttling and escalation​

Two more settings shape delivery:

  • throttleSeconds is the minimum gap between notifications for the same alarm, so a flapping condition does not flood a channel. A change that escalates the alarm's severity is not throttled. null means no throttle.
  • escalateAfterSeconds and maxEscalations: when set (> 0), an alarm that stays unacknowledged and uncleared for that long after its last notification is re-notified, up to the cap. If maxEscalations is unset or 0, a service-wide default cap of 5 applies. A null/0 escalateAfterSeconds disables escalation for the policy.

Escalation is safe to run on several replicas: each escalation tier is claimed before it is sent, so exactly one replica delivers it. Each alarm has one shared escalation clock and tier. If several escalating policies match, the shortest window sets the cadence, and every cap counts against the shared tier.

mutation {
createNotificationPolicy(request: {
token: "default-routing",
name: "Default alarm routing",
throttleSeconds: 300,
escalateAfterSeconds: 900,
maxEscalations: 3,
enabled: true,
rules: [
{ severity: "CRITICAL", channelToken: "oncall-hook", recipients: "[]" },
{ severity: "*", channelToken: "ops-email", recipients: "[\"ops@example.com\"]" }
]
}) { token enabled rules { severity channel { token } } }
}

On update, the request's rules replaces the policy's existing rule set. Omit rules to leave the stored rules untouched; sending null or [] leaves the policy with no rules. As with a channel's secret, a client that binds one variable per field sends an unsupplied rules as null and empties the rule set, so send the whole request as a single variable. Naming an unknown channel token fails the whole write.

Two people editing the same policy can overwrite each other: a field both of them change keeps only the last save, and because a rules list replaces the whole set, the second save drops every rule the first one wrote. To prevent that, pass expectedUpdatedAt with the policy's updatedAt as you read it (or as your previous update returned it). If the policy has changed since, the update is refused with "notification policy was modified by another writer; reload and try again" and nothing is written. Reload, reapply your change, and send it again.

Quiet hours​

Quiet hours keep a policy from paging people at the wrong time, such as overnight. While a policy is inside its quiet hours it holds its notifications instead of sending them, and it sends them when the quiet stretch ends. Each policy has its own quiet hours, so a night-shift policy and a day-team policy can follow different schedules.

Quiet hours are not maintenance windows. A maintenance window belongs to a device type and says when disruptive commands may start. Quiet hours belong to a notification policy and say when that policy holds its pages. They use the same slot format and the same time-zone and daylight-saving rules, but they are otherwise unrelated, and neither affects the other.

A policy gets quiet hours from three fields. They are null on a policy without quiet hours.

FieldMeaning
quietHoursTimezoneThe IANA time zone the slots are read in, such as Europe/Berlin. Local and unknown names are refused. The policy's own zone is always used, never the server's.
quietHoursSlotsA JSON string holding 1 to 56 weekly slots, in the same document as a maintenance window. Each slot has a day (MONDAY to SUNDAY), a start (24-hour HH:MM, wall-clock time in the zone) and a durationMinutes (1 to 10080, elapsed time).
quietHoursBypassSeverityCRITICAL, MAJOR, MINOR, WARNING or INDETERMINATE, or null. Alarms at or above this severity are sent even during quiet hours. null holds every severity. It is allowed, and has no effect, on a policy with no quiet hours.

Set quietHoursTimezone and quietHoursSlots together, or leave both null.

mutation {
createNotificationPolicy(request: {
token: "night-shift",
name: "Night shift pages",
enabled: true,
quietHoursTimezone: "Europe/Berlin",
quietHoursSlots: "[{\"day\":\"MONDAY\",\"start\":\"22:00\",\"durationMinutes\":480},{\"day\":\"TUESDAY\",\"start\":\"22:00\",\"durationMinutes\":480}]",
quietHoursBypassSeverity: "CRITICAL",
rules: [
{ severity: "*", channelToken: "ops-email", recipients: "[\"ops@example.com\"]" }
]
}) { token quietHoursTimezone quietHoursSlots quietHoursBypassSeverity }
}

This policy is quiet from 22:00 to 06:00 on Monday and Tuesday nights, in Berlin time. A CRITICAL alarm in that stretch is still sent at once. Any other alarm waits until 06:00.

updateNotificationPolicy is a partial update, and the three quiet-hours fields carry three states: omit one to keep it, send null to clear it, send a value to set it. The server checks the fields together after applying the update, so to remove quiet hours, clear quietHoursTimezone and quietHoursSlots in the same request (and quietHoursBypassSeverity too, if you want it gone).

mutation {
updateNotificationPolicy(
token: "night-shift",
request: { quietHoursTimezone: null, quietHoursSlots: null }
) { token quietHoursTimezone quietHoursSlots }
}

What happens during quiet hours​

  • Held, then sent. When a raised or escalated alarm matches a policy that is inside its quiet hours, and the alarm's severity is below the bypass severity (or no bypass is set), the notification is held, not dropped. The policy sends it through its rules when the quiet stretch ends. The escalation scheduler releases held notifications on its pass, which runs once a minute by default, so a release can be up to a pass late. A release interrupted by a restart of the notification service is not lost: it is sent a few minutes later. In rare cases that means the same page is sent twice.
  • Acknowledged or cleared drops it. A held notification is dropped instead of sent if the alarm was acknowledged or cleared after the event that was held.
  • One held notification per alarm per policy. If the same policy holds a later raised or escalated event for the same alarm, the later one replaces the earlier. The page in the morning describes the alarm as it last was, and an alarm that flaps overnight costs one page, not one per flap.
  • The bypass severity sends at once. An alarm at or above the bypass severity is sent immediately, even inside quiet hours. An alarm whose severity is not one of the known tiers is always sent.
  • Throttling comes first. A notification the policy's throttle suppresses is suppressed, not held.
  • Escalation waits. While a policy is in quiet hours, its escalation re-notifications wait for alarms below the bypass severity. They resume after the stretch ends: an alarm still unacknowledged and uncleared is re-notified on the first pass after it. Quiet hours do not use up escalation tiers.
  • Each policy is independent. A policy without quiet hours, or one that is not quiet right now, still pages immediately for the same alarm. A held notification is still sent at the end of the stretch even if another policy has already paged someone, because the recipients can differ.
  • A fresh page answers the held one. When the policy itself pages for the alarm, because its quiet hours have ended or the alarm reached the bypass severity, what it was holding for that alarm is dropped rather than sent afterwards.
  • Edits re-evaluate what is held. Changing a policy's quiet hours or bypass severity, or enabling or disabling it, is applied on the next pass. A held notification is sent if the policy no longer holds it, or held until the new end if it still does. A disabled or deleted policy drops what it was holding.
  • A failed release is retried. A release that reaches no channel, because every delivery failed, is retried after 1, 2, 4 and 8 minutes, 5 attempts in all. It is then dropped, counted and recorded as a dead letter, the same way as an alarm that reached nobody when it was raised.

Time zones and daylight saving​

Slots are wall-clock times in the policy's quietHoursTimezone, so quiet hours follow daylight saving time with the zone. A slot whose start falls in a spring-forward gap, a time that does not exist, opens at the end of the gap. A slot whose start falls in a fall-back overlap, a time that happens twice, opens at the first occurrence. A slot's duration is elapsed time, so a long slot across a clock change ends an hour earlier or later on the wall clock.

A slot may run past midnight. A slot that runs past the end of Sunday wraps into Monday.

What the server refuses​

Every violation is answered with extensions.code INVALID_VALUE, and nothing is written.

  • quietHoursTimezone and quietHoursSlots are not set together.
  • The zone is Local or an unknown name.
  • A slot has an unknown day, a start that is not 24-hour HH:MM, or a durationMinutes outside 1 to 10080, or the list does not hold 1 to 56 slots.
  • Two slots overlap in the week as written. Slots that only touch are fine, and a Sunday slot that runs past midnight is checked against Monday.
  • The slots together cover the whole week (10080 minutes). To silence a policy entirely, disable it instead.

Watching it​

The counter devicechain_notificationmanagement_quiet_hours_total counts what quiet hours did, with the label outcome:

outcomeCounted when
heldA notification was held rather than sent.
releasedA held notification was sent when its stretch ended.
dropped_resolvedA held notification was dropped because the alarm was acknowledged or cleared.
dropped_policyA held notification was dropped because its policy was disabled or deleted.
dropped_exhaustedA held notification was dropped after 5 releases reached no channel. It is also recorded as a dead letter.
supersededA held notification was dropped because the policy paged a newer event for the same alarm.

In the console, quiet hours are edited on the Monitor → Notification Policies page; see Notification policies.

Delivery ledger​

Every attempt to send an alarm's notification through a channel is recorded in the tenant's delivery ledger, so "was this alarm actually paged, to whom, and why not?" has an answer. Query it with notificationAttempts (requires notification:read):

query {
notificationAttempts(criteria: { pageNumber: 1, pageSize: 20, alarmToken: "<alarm token>" }) {
results {
attemptedAt trigger policyToken channelToken recipients
status attempt maxAttempts latencyMillis errorCode errorMessage
}
}
}

Each record is one try, newest first. A channel allowed three tries that failed twice and then delivered leaves three records: FAILED (attempt 1), FAILED (attempt 2) and SENT (attempt 3). Filter by alarmToken, channelToken or status, and by time with since and until (RFC 3339).

statusMeaning
SENTThe channel accepted the notification.
FAILEDA failure a retry might fix: a timeout, an error status, a refused connection.
REFUSEDRefused for good, so no retry followed: the destination is one outbound traffic may not reach (EGRESS_BLOCKED), or the channel's credential configuration is refused (CREDENTIAL_REFUSED).
NOT_ATTEMPTEDPlanned but never tried: the channel's delivery secret could not be read (SECRET_UNAVAILABLE), or the time allowed for the alarm's deliveries ran out before this channel (BUDGET_SPENT).

trigger says what caused the delivery: ALARM (an alarm state change), ESCALATION (an unacknowledged alarm was re-notified) or QUIET_HOURS_RELEASE (a notification held by quiet hours was sent when the quiet period ended). New status, trigger and errorCode values may be added in later releases, so treat one you do not recognise as informational.

errorMessage is the provider's error text with every URL replaced by <url>, because a Slack or Teams webhook URL is itself a credential, and the ledger is readable by anyone with notification:read. The message itself is not stored: subjectHash (a SHA-256 of the subject line) shows whether two attempts carried the same one.

Each channel also reports deliveryHealth { lastSuccessAt lastFailureAt consecutiveFailures }, read from the ledger. consecutiveFailures counts the failed and refused attempts since the channel last delivered. NOT_ATTEMPTED records do not count against a channel, since they describe something the platform did not do, and neither does a FAILED record with errorCode: BUDGET_SPENT: the time allowed for all of the alarm's deliveries ran out while that attempt was in flight, so the platform cut it short. deliveryHealth is null for a caller without notification:read, such as a channel returned by a mutation to a caller who may write but not read.

Two properties to keep in mind:

  • The ledger never delays a notification. Records are written after the attempt and off the delivery path. If the database cannot take a record, the record is dropped and counted on devicechain_notificationmanagement_ledger_writes_total{outcome="failed"} (or outcome="dropped" when too many writes are already waiting). A record proves an attempt was made; a missing record does not prove one was not.
  • Recipients are personal data. Records are deleted after 7 days (the same window as resolved notification state), or after AttemptRetentionSeconds in the notification-management configuration. They are also erased with the tenant when it is deleted.
  • Size it before raising the retention. The ledger holds one record per attempt: roughly the alarm transitions notified, times the channels each one reaches, times the tries per channel, plus escalations. A tenant paging 10,000 times a day through 3 channels keeps about 210,000 records at the 7-day default, and every record carries its recipient list.

Verify the path end to end​

  1. Create a channel (as above). Confirm enabled: true on the result, and that hasSecret is true for an SMTP channel with a username or a webhook declaring bearer or header, and false for a webhook declaring none.
  2. Create a policy whose rules map the severities you care about to that channel.
  3. Raise a real alarm. Trip a detection rule on a test device (see Event Processing & Alarms) and confirm the email or webhook call arrives.
  4. Inspect delivery state. The service keeps a read-only per-alarm record of what it has done. Query notificationStatesByAlarmToken(alarmTokens: [...]), or search with notificationStates. Check firstNotifiedAt and notifyCount, and, once the alarm has sat unacknowledged past the escalation window, escalationLevel.
  5. Read the delivery ledger. notificationAttempts(criteria: {pageNumber: 1, pageSize: 20, alarmToken: "..."}) lists every attempt for the alarm with its outcome. A FAILED or REFUSED record carries errorCode and the provider's errorMessage; see Delivery ledger.

Acknowledging or clearing the alarm stops further escalation. The state row records acknowledgedAt/clearedAt alongside the notification history. If the alarm clears and its condition returns, the platform re-opens the same alarm, and its state row starts over for the new cycle: escalation runs again until that cycle is acknowledged or cleared.