跳到主要内容
版本:开发版

告警手册

Chart 附带的每一条告警都有一个 runbook_url,指向本页中对应的条目。每个条目说明告警的严重程度、告警在告诉你什么,以及该怎么处理。正文就是告警自身的描述,因此通知与本页不会互相矛盾。

只有当 Alertmanager 配置了接收器,告警才会送达真人。请参阅将告警路由给真人,用 dcctl install --alert-webhook、--alert-email 或 --alert-config 进行设置。

devicechain.area-availability​

InstanceAreaUnavailable​

严重程度: critical · 规则组: devicechain.area-availability

an area of this instance has no pod available

含义与处理方式. Deployment ‹deployment› wants ‹value› pod(s) and has had none available for ten minutes, so that area is not serving. Because its pods are gone, the area also stops reporting its own readiness and metrics, and the instance overview's readiness and scrape panels will not show it: this alert and the panel for areas with no pod available are what do. Describe the deployment and its pods: kubectl describe deployment and kubectl get pods in the instance's namespace show a failing image pull, a pod that cannot be scheduled for want of CPU or memory, a failed readiness probe, or a crash before ready (kubectl logs --previous). If the area was scaled to zero by mistake, scale it back; an area you turned off on purpose has zero wanted pods and does not raise this.

devicechain.broker​

BrokerCertificateExpiringSoon​

严重程度: warning · 规则组: devicechain.broker

the broker's TLS certificate expires within 30 days

含义与处理方式. The broker server named by the pod label presents a TLS certificate that expires in less than 30 days. When it expires the broker refuses TLS connections from devices and from this instance's own services. The certificate is renewed by dcctl upgrade, which re-issues it inside its last 30 days and swaps it in without restarting the broker when only the certificate changed, so running dcctl install and then dcctl upgrade for this instance now, both with the same dcctl release, clears this alert. If this keeps firing after an upgrade, the broker has not picked the new certificate up; check the server's log, and restart the server pod if it did not switch.

BrokerCertificateExpiryCritical​

严重程度: critical · 规则组: devicechain.broker

the broker's TLS certificate expires within 7 days, or already has

含义与处理方式. The broker server named by the pod label presents a TLS certificate that expires in less than 7 days, or that has already expired. Once it has expired the broker's TLS connections fail for devices and for this instance's own services. Run dcctl install and then dcctl upgrade for this instance now, both with the same dcctl release: it re-issues the certificate under the authority the instance keeps. An instance that does not keep its authority (one bootstrapped by an early release) cannot re-issue it in place and has to be rebuilt; dcctl upgrade says so when that is the case. If an upgrade has run and this still fires, the broker has not picked the new certificate up; restart the server pod.

BrokerLeadershipImbalanced​

严重程度: warning · 规则组: devicechain.broker

one broker server leads far fewer streams than the others

含义与处理方式. At least one broker server leads fewer than half the streams an even share would give it, so the others carry the replication and acknowledgement work for it. Only the streams that dcctl is allowed to move are counted: this instance's replicated streams other than the device-ingest ones (the broker's MQTT streams, the device capture stream and every stream an ingest area writes), which are never rebalanced because a leader change on them can lose messages already acknowledged to a device. After a rolling restart of the broker the server restarted last leads nothing, and at the load this was measured under that doubled publish acknowledgement latency and cost a few percent of HTTP ingest. dcctl upgrade evens leadership out after it rolls the broker, so this fires after a restart done any other way (a node drain, a pod loss, a manual rollout), or when that step was skipped or warned. Run dcctl broker rebalance <instance> to even it out; each move is a brief election for one stream. If it keeps returning, check the broker pods for repeated restarts.

BrokerMqttClientAuthLost​

严重程度: warning · 规则组: devicechain.broker

the broker is not requesting device client certificates

含义与处理方式. The broker server named by the pod label is not asking MQTT devices for a client certificate, so devices that authenticate by certificate cannot connect. Devices that use a token or a password are not affected. Either the server is running but its MQTT listener stopped requesting certificates (a configuration reload that did not keep the setting, or the broker's own check of its listener keeps failing; its log says which), or the server is not running the DeviceChain broker image, for example after a rollback to the upstream nats image. Check the server's image with kubectl get pod -o jsonpath='{.spec.containers[0].image}'. On the DeviceChain image, restart the server pod, and report the log lines around the last configuration reload. On the upstream image, certificate devices stay refused until the DeviceChain image is restored.

devicechain.command-delivery​

CommandBatchReserveRefusals​

严重程度: warning · 规则组: devicechain.command-delivery

The delivery-machinery reserve is refusing fleet writes the tenant's ceiling would have admitted

含义与处理方式. Fleet writes are being refused that WOULD have fitted against the tenant's own undelivered-command ceiling, and were refused only by the headroom the delivery machinery reserves for itself. This is the reserve doing its job — nothing is broken and every caller was told — but a rate sustained for this long means the ceiling and the reserve disagree about what a normal fleet write is. Raise the offending tenant's ceiling, or lower the reserve, deliberately. These counters carry NO tenant label — by design, since a tenant is unbounded cardinality — so the Command Delivery dashboard can tell you which codes and which bound, never which tenant. To find the tenant, read the recorded command batches: one that fanned out partially records a per-device HELD_CEILING_EXCEEDED refusal, and a batch refused whole records nothing at all, so a tenant seen here may have left no batch behind.

CommandClaimsStranded​

严重程度: critical · 规则组: devicechain.command-delivery

Commands are stranded in SENT and may time out against innocent devices

含义与处理方式. At least one command was left reading SENT after both its publish and its release failed. It was never delivered. What happens next depends on the device's transport, and only one of the two answers is safe. On LwM2M the stranded reconciler re-arms it within about ten minutes, so it is delivered when the device next wakes — check devicechain_commanddelivery_command_delivery_stranded_recovered_total{disposition="PARKED"} to confirm that happened. A disposition of CANCELLED there means the opposite: the command's batch had been called off, so the row was cleaned up rather than delivered. On MQTT nothing retries it and it expires as TIMEOUT, which records the failure against the DEVICE. So treat every MQTT TIMEOUT reported in this window as unattributed until this is resolved, and do not send engineers to inspect the devices named. Look for a broker outage overlapping a database outage: it takes BOTH to strand a claim, which is why this counter is normally flat at zero.

CommandPresenceReadErrors​

严重程度: warning · 规则组: devicechain.command-delivery

The command presence gate is failing open

含义与处理方式. The dispatch sweep cannot read the presence projection, and the gate fails OPEN by design — so commands are being dispatched WITHOUT the absence check, to devices nobody confirmed were there. Nothing else reports this: dispatch looks healthy and the holds-placed counter goes to zero, which reads as "no devices are absent" rather than "nothing is being checked". Check the device-state area and the presence projection's own health.

LpwanAckReportsAnomalous​

严重程度: warning · 规则组: devicechain.command-delivery

Confirmed LoRaWAN downlink reports do not match pending commands

含义与处理方式. For 10 minutes, network-server acknowledgements have named no pending downlink or disagreed with its correlation fields. Commands may reach their deadline despite a device acknowledgement. Check the Command Delivery dashboard, integration device mapping and network-server queue item identifiers; inspect outbound-connectors logs before resending a command.

LpwanAckSettlementFailing​

严重程度: warning · 规则组: devicechain.command-delivery

Confirmed LoRaWAN downlink reports cannot settle commands

含义与处理方式. For 10 minutes, confirmed-downlink reports have failed in the store or response path, or could not be read. A device may have acknowledged while its command remains pending. Check outbound-connectors logs, database writes and the broker response stream; inspect pending commands before retrying them.

LpwanDownlinkTrackingFailing​

严重程度: warning · 规则组: devicechain.command-delivery

LoRaWAN commands cannot be tracked or answered reliably

含义与处理方式. For 10 minutes, the LoRaWAN adapter has failed to claim, look up, track or answer commands. pending_failed means the network server accepted a confirmed downlink but its pending acknowledgement could not be stored. The device may still receive it. Check outbound-connectors database and broker health and logs; reconcile the network-server queue before resending.

devicechain.container-restarts​

InstanceContainerRestarted​

严重程度: warning · 规则组: devicechain.container-restarts

a container in this instance was restarted

含义与处理方式. Kubernetes restarted the container named by the pod and container labels in the last 15 minutes. A DeviceChain service exits on purpose when it cannot do its job, and the reason is in the log of the container that exited: run kubectl logs --previous on the pod and read the last error line, usually after 'A component has declared this process unfit to continue'. A read loop that failed for two minutes with no answer from the message broker names its stream and its last error there. Several services restarting together point at the broker: check that its servers are running and run dcctl ha verify. If the restarted container is a broker server (dc-nats-*), read its previous log too; its streams take a few minutes to be current again. A container whose last state is OOMKilled needs a higher memory limit. Restarts in the first minutes of an install or upgrade are expected. Messages the container had read from the broker but not acknowledged are delivered again, but requests a device or client had in flight to it fail and must be retried, and every restart lengthens the outage it happened in.

devicechain.database-backup​

BackupDestinationAlmostFull​

严重程度: critical · 规则组: devicechain.database-backup

The backup destination is almost full

含义与处理方式. ‹persistentvolumeclaim› is over 85% full. When it fills, WAL archiving starts failing — and failed archiving does not stall commits, it accumulates WAL on the DATABASES' volumes until those fill and PostgreSQL stops. So this ends as a database outage whose cause points nowhere near a database. Grow the volume first. Otherwise remove archives left behind by instances that no longer exist, move to an external destination, or shorten a store's recovery window in the OpenTofu configuration (backup_retention_tsdb for an instance's event store, backup_retention_rdb for the relational store).

BackupDestinationFillingFast​

严重程度: critical · 规则组: devicechain.database-backup

The backup destination will be full within the hour at its current rate

含义与处理方式. ‹persistentvolumeclaim› is more than 65% full and filling fast enough to run out of space within the hour. When it is full, WAL archiving fails for EVERY database archiving into it, and each primary keeps its unshipped WAL on its own volume until that fills and PostgreSQL stops. Likely causes, in order: an instance ingesting faster than the store was sized for; archives left in the store by instances that no longer exist (list the bucket and compare with the instances that are running); a scheduled base backup that stopped, so nothing is being pruned. Grow the volume, remove what no running instance owns, or move backups to an object store you run yourself.

PostgresHasNeverArchivedWAL​

严重程度: critical · 规则组: devicechain.database-backup

PostgreSQL has never successfully archived a WAL segment

含义与处理方式. ‹pod› has archived no WAL segment, ever. The first base backup forces a segment switch, so a working cluster leaves this state within minutes of creation — an hour in it means archiving has never worked here, not that the database is quiet. Note this records no FAILURE either, so the failing-archival alert cannot see it: an archiver that hangs rather than errors looks exactly like this.

PostgresNoRecentBaseBackup​

严重程度: warning · 规则组: devicechain.database-backup

N

含义与处理方式. ‹pod› has taken no successful base backup N. WAL archiving alone restores NOTHING — a recovery replays WAL forward from a base backup, so an ever-growing archive with no recent base is a longer and longer replay onto older and older state. Retention pruning has stopped too: obsolete backups are deleted only after a new backup completes.

PostgresWALArchiveBacklog​

严重程度: critical · 规则组: devicechain.database-backup

PostgreSQL is holding WAL it has not shipped to the backup destination

含义与处理方式. ‹pod› has ‹value› WAL segments (16 MiB each) waiting to be archived. They stay on this database's own volume until they ship — the replicas do not help, because a standby does not keep the primary's unshipped WAL — and when the volume fills, PostgreSQL stops and does not fail over. If the WAL archiving failing alert is also firing, the destination is refusing writes: check that the object store is reachable and has space. If it is not, the archiver is slow or hung, or ingest is producing WAL faster than it ships: check the backup sidecar's logs and the object store's free space, and grow the database volume if the backlog keeps rising.

PostgresWALArchivingFailing​

严重程度: critical · 规则组: devicechain.database-backup

PostgreSQL WAL archiving is failing

含义与处理方式. ‹pod› last FAILED to archive a WAL segment more recently than it last succeeded, so this database is no longer being backed up. Nothing will stall and nothing will crash: writes continue and every health check stays green. Two things are accumulating — the recovery gap, and unshipped WAL on the database's own volume, which will eventually fill it and stop PostgreSQL. Check the object store is reachable, has space, and still accepts these credentials.

PostgresWALArchivingNotConfigured​

严重程度: critical · 规则组: devicechain.database-backup

A PostgreSQL cluster has no WAL archiver attached and is NOT being backed up

含义与处理方式. ‹pod› is running with archiving switched on and NO backup plugin behind it, so every WAL segment is being "archived" by a command that discards it and reports success. Nothing else will tell you: archived_count rises, no failure is ever recorded, disk does not grow, and every other backup alert stays green. This database cannot be restored to any point in time. Check that the Cluster still carries its barman-cloud entry under spec.plugins and that its ObjectStore exists.

devicechain.database-control-plane​

CNPGClusterNotReady​

严重程度: critical · 规则组: devicechain.database-control-plane

A PostgreSQL Cluster has not been Ready for 15 minutes

含义与处理方式. ‹cluster› in namespace ‹namespace› reports Ready=False (‹reason›). THIS BLOCKS FAILOVER: while a Cluster is not reconciling it is never promoted, so losing the primary leaves healthy, fully caught-up replicas that nothing will elect — the databases look fine right up until the moment they need to move. A phase of "Cluster cannot proceed to reconciliation" points at the CloudNativePG operator or the Barman Cloud plugin rather than at Postgres, and the instance pods will look perfectly healthy throughout.

CNPGClusterStateUnobserved​

严重程度: warning · 规则组: devicechain.database-control-plane

No CloudNativePG Cluster conditions are being collected

含义与处理方式. Nothing is reporting PostgreSQL Cluster conditions in namespace ‹namespace›, so the health of database failover is UNKNOWN rather than good — and while this fires, CNPGClusterNotReady CANNOT fire, because it reads the series this rule says are missing. Check, in order: kube-state-metrics is running; it was started with --custom-resource-state-config-file (OpenTofu's monitoring module sets this from cnpg_cluster_metrics, which the root derives from enable_cnpg); and its ClusterRole can list postgresql.cnpg.io/clusters. A missing RBAC rule shows up only as a forbidden line in the kube-state-metrics log and an endpoint that exports nothing.

CNPGClusterUnusable​

严重程度: critical · 规则组: devicechain.database-control-plane

A PostgreSQL instance is not answering its own metrics collector

含义与处理方式. At least one instance of ‹cluster› in namespace ‹namespace› cannot be queried by its CloudNativePG metrics collector, while the pods and the Cluster's Ready condition may look perfectly healthy — the operator holds reserved superuser connections, so it keeps reconciling either way. This alert cannot say WHICH instance (it aggregates them to keep pod identity out of the alert), so check both possibilities. If it is the PRIMARY, clients are affected and the likeliest cause is CONNECTION EXHAUSTION: every rdb-backed service opens a pool of up to 20 against the primary, and a rolling upgrade briefly doubles the pod count, so demand can exceed max_connections and application services then fail with "remaining connection slots are reserved for roles with the SUPERUSER attribute". If it is a STANDBY, clients are unaffected and this is a failing replica rather than an outage. Run SELECT count(*) FROM pg_stat_activity against SHOW max_connections as a superuser, which still works while this fires, and compare per instance.

CNPGCollectorAbsent​

严重程度: warning · 规则组: devicechain.database-control-plane

No CloudNativePG metrics collector is being scraped

含义与处理方式. Nothing is reporting cnpg_collector_up in namespace ‹namespace›, so whether the PostgreSQL instances there can answer their own metrics collector is UNKNOWN rather than good — and while this fires, CNPGClusterUnusable CANNOT fire, because it reads the series this rule says are missing. Check, in order: the CloudNativePG PodMonitor for the Cluster exists and selects its pods; Prometheus can read the namespace; and the instance pods are running and serving :9187.

CNPGControlPlaneUnavailable​

严重程度: critical · 规则组: devicechain.database-control-plane

A CloudNativePG control-plane deployment has no available replicas

含义与处理方式. ‹deployment› in namespace ‹namespace› has zero available replicas. With the operator down, no PostgreSQL Cluster in this cluster can fail over, scale, or be reconfigured — the running databases keep serving and stop being managed. With the Barman Cloud plugin down, the operator additionally cannot reconcile any Cluster that references it, so the same failover block applies. Expect CNPGClusterNotReady to follow within the quarter hour if this is not resolved.

devicechain.database-snapshot-backup​

DatabaseSnapshotBackupsUnobserved​

严重程度: warning · 规则组: devicechain.database-snapshot-backup

Database snapshot backups are on, but nothing is reporting on them

含义与处理方式. This instance's databases take their daily base backups as volume snapshots, but no snapshot ScheduledBackup is visible to monitoring, so the alerts for a snapshot that failed or for pruning that stopped cannot fire. Check that the snapshot ScheduledBackups exist in the database namespaces, and that kube-state-metrics in the monitoring stack is configured to read CloudNativePG Backups and ScheduledBackups.

DatabaseSnapshotPruningStalled​

严重程度: warning · 规则组: devicechain.database-snapshot-backup

Database snapshot backups are not being pruned

含义与处理方式. The DeviceChain operator has not finished pruning the volume snapshots of ScheduledBackup ‹scheduledbackup› in ‹namespace› for over an hour. CloudNativePG does not delete old snapshots, so while this lasts every daily snapshot is kept: each one a full copy of the database, including data a tenant deletion has since removed, and each costing storage at the cloud provider. Check that the operator is running and is the release the cluster was installed with, and read the ScheduledBackup's events: a retention window it cannot read is refused there by name.

PostgresNoRecentSnapshotBackup​

严重程度: warning · 规则组: devicechain.database-snapshot-backup

No database snapshot backup has completed in 36 hours

含义与处理方式. The ‹cluster› database in ‹namespace› has completed no volume-snapshot base backup in 36 hours against a daily schedule, or has never completed one. Point-in-time recovery still works from the weekly base backup in the backup store and the log since, so nothing is lost yet. Check the events on the ScheduledBackup ‹cluster›-snapshot and on its latest Backups: a VolumeSnapshotClass that was deleted, or that belongs to a different storage driver from the database volumes, fails every snapshot.

devicechain.database-storage​

DatabaseVolumeAlmostFull​

严重程度: critical · 规则组: devicechain.database-storage

A database volume is almost full

含义与处理方式. ‹persistentvolumeclaim› is over 85% full. A PostgreSQL volume that fills does not slow down, it stops the database, and failing over does not rescue it: every replica holds the same data, and when the cause is unshipped WAL it is the primary that fills first. Grow it now: CloudNativePG expands a Cluster's volumes in place where the storage class supports expansion, by raising spec.storage.size (or spec.walStorage.size for a -wal claim). Reclaiming space is mostly not an alternative, because the tables that grow are append-only and stored events point at their rows. To see which tenant is behind it, on the relational primary (dc-rdb): SELECT tenant_id, count(*) FROM "device-management".geo_fence_set_versions GROUP BY tenant_id ORDER BY 2 DESC LIMIT 10; If this is a -wal claim, treat it as a stalled archiver first — unshipped WAL is retained until it ships, and the WAL archiving alerts say why it is not shipping.

DatabaseVolumeCritical​

严重程度: critical · 规则组: devicechain.database-storage

A database volume is over 95% full

含义与处理方式. ‹persistentvolumeclaim› is over 95% full. PostgreSQL stops writing when it fills, and CloudNativePG does not fail over a full disk. Grow the volume now (spec.storage.size, or spec.walStorage.size for a -wal claim; the expansion is online where the storage class supports it), or free space by lowering the event data retention. On a dc-tsdb claim, events the store cannot take are held in the message broker (EventStoreRefusingWrites) and ingest backs off, but that buffer is finite.

DatabaseVolumeFillingFast​

严重程度: critical · 规则组: devicechain.database-storage

An event-store volume will be full within the hour at its current rate

含义与处理方式. ‹persistentvolumeclaim› is more than half full and filling fast enough to run out of space within the hour. When it fills, PostgreSQL stops, and CloudNativePG does not fail over a full disk. If only the primary's volume is climbing while its replicas' are not, the cause is WAL the archiver has not shipped — see the WAL archiving and backup destination alerts, and fix the destination first. If every member is climbing together, it is data: ingest is outgrowing the volume, so grow it (spec.storage.size on the Cluster, where the storage class supports expansion) or shorten data retention.

DatabaseVolumeFillingUp​

严重程度: warning · 规则组: devicechain.database-storage

A database volume is over 75% full

含义与处理方式. ‹persistentvolumeclaim› is over 75% full. Database storage on this platform does not come back on its own: several tables are append-only by design and nothing prunes them — the audit journal, device profile versions, entity group versions, dashboard versions, connector versions, geofence set versions and the geofence geometry archive. Each is kept for a reason: a stored event names the profile or fence-set version it was resolved against, and a dashboard or connector version is what a rollback returns to. Retention settings do not apply to any of them — those prune the time-series store, which is the dc-tsdb claims. Act while there is still room to work in: grow the volume, and find out what is producing versions. On the relational primary (dc-rdb), by tenant: SELECT tenant_id, count(*) FROM "device-management".geo_fence_set_versions GROUP BY tenant_id ORDER BY 2 DESC LIMIT 10; Substitute any other history table for the one named — audit_events in any schema, device_profile_versions or entity_group_versions in that one, dashboard_versions or connector_versions in theirs. It counts rows and names no other column, so it works on all of them; they do NOT share a payload column, and a query naming one would fail on most. Size per table is on the append_only_table_bytes series, which every service exports for its own tables. One tenant re-authoring in a loop is the usual answer. On a dc-tsdb claim, check the data-lifecycle retention settings instead; on a -wal claim, the WAL archiving alerts are where the cause is.

DatabaseVolumeFillsWithinDay​

严重程度: warning · 规则组: devicechain.database-storage

An event-store volume will be full within a day at its current rate

含义与处理方式. ‹persistentvolumeclaim› is on course to run out of space within 24 hours at the rate it has filled over the last two. When it fills, PostgreSQL stops and the platform can no longer store events. Events are held in the message broker meanwhile and new ingest is refused once the broker's backlog limit is reached, but a store that stays stopped for longer than the broker can hold loses data (see EventStoreRefusingWrites). Act now, with a day in hand: grow the volume (spec.storage.size on the Cluster; the volume expands online where the storage class allows it), shorten the data retention, or check that compression is acting (an uncompressed row is about 600 bytes). Size it against stored rows, which calculated metrics multiply by (1 + the formulas that apply to the profile).

devicechain.dead-letter​

DeadLetterStoreLosing​

严重程度: critical · 规则组: devicechain.dead-letter

the dead-letter store is losing records

含义与处理方式. Dead letters reached their final delivery attempt without being written to the store, so failures the platform recorded once are now recorded nowhere. The store's database is the thing to look at — the consumer rides out a blip in process and spends every delivery attempt before giving up, so this means an outage of minutes, not seconds. Look also at the consumer's backlog: what is lost here is a subset of what is late.

DeadLetterWriteLost​

严重程度: critical · 规则组: devicechain.dead-letter

work is being lost with no record of it

含义与处理方式. A service gave up on a message AND could not record it as a dead letter, on a delivery that will not repeat — so the work is gone and nothing anywhere records what it was. This is the one failure on these paths with no evidence left behind, which is why it is separate from the giving-up alerts and fires faster than they do. There are four causes, and the pod's LOST error log line tells them apart. The write failed — the broker is unreachable, refusing, or out of storage, so look at JetStream. Or the service refused its own letter as malformed, which is a defect in that service. Or device-management could not encode the record of an inbound event that failed, so it acknowledged the event and nothing published says it failed, which is also a defect in that service. Or a dead-letter READER — the dead-letter store in user-management, or the command writeback in command-delivery — ran out of delivery attempts on a letter, which may now age out of the stream without having been stored or settled (its last attempt may have committed and lost only its acknowledgement, so this cause can overstate the loss); look at that service's database, where a stored letter settles the question, or its own logs. Note that a FULL dead-letter stream does not cause this — every stream discards oldest, so a full one silently evicts the earliest letters instead, which this alert cannot see.

ReactPoisonDropping​

严重程度: critical · 规则组: devicechain.dead-letter

REACT is giving up on detection actions that keep failing

含义与处理方式. Derived events are hitting the redelivery cap with actions still failing — typically command-delivery or the message bus has been unavailable for several minutes. Every action of each detection was attempted on every delivery; the dead letter names each one that failed on the final attempt, by kind and idempotency key. Nothing replays them, so treat each named alarm, command or connector call as not carried out.

devicechain.device-management​

AgentExecutionsFailing​

严重程度: warning · 规则组: devicechain.device-management

Agent action executions are failing steadily

含义与处理方式. For 10m, agent action executions have failed because the execution transaction could not complete (database or statement timeout, policy service or kill feed unavailable). A lock conflict on the proposal or its alarms is refused as a concurrent approval and counted as refused. Check device-management logs and the Agent actions (preview) row on the Platform board. Dependency changes, policy refusals, a closed kill fence and approver refusals are separate results and do not raise this alert.

DeviceCalloutLatencyHigh​

严重程度: warning · 规则组: devicechain.device-management

Device connects are being decided slowly by the auth callout

含义与处理方式. For 10m the 99th percentile time to decide a ‹path› device connect has been above 250ms (‹value› now). The time runs from a request being picked up to the answer being published. The broker gives up on an answer after 5 seconds and the device is refused, so a callout this slow is on its way to refusing devices. It is nearly always the database behind the credential read (token and password) or the trust-anchor and chain work (x509, x509-jit), or a reconnect storm queueing behind the callout's in-flight limit. Compare callout_in_flight with its limit, callout_refused_busy_total, the device-management database latency, and the log for credential store failures.

DeviceCredentialCacheTooSmall​

严重程度: warning · 规则组: devicechain.device-management

The device credential cache is too small for the fleet

含义与处理方式. On ‹pod›, more than 5% of device credential lookups have pushed another credential out of a full cache for 15m (now ‹value›). The cache is smaller than the set of devices reporting through it, so it is cycling and most checks go to the database. Raise inMemoryCache.credentialCacheMiB for device-management, and raise its memory limit with it.

DeviceLookupCacheTooSmall​

严重程度: warning · 规则组: devicechain.device-management

A device-management per-device cache is too small for the fleet

含义与处理方式. On ‹pod›, more than 5% of lookups in the ‹cache› cache have pushed another entry out of a full cache for 15m (now ‹value›). The cache is smaller than the set of devices reporting through it, so it is cycling and most lookups go to the key-value bucket. Raise inMemoryCache.perDeviceCacheMiB for device-management, and raise its memory limit with it.

DeviceTrustAnchorsExpiringSoon​

严重程度: warning · 规则组: devicechain.device-management

A trust anchor expires within 30 days

含义与处理方式. For 1h, ‹value› enabled trust anchor(s) across the instance's tenants expire within 30 days or already have. When a trust anchor expires, every device whose certificate chains to it is refused. List them with the deviceTrustAnchors query (the notAfter field), and add the replacement CA as a new anchor and pin each device's certificate to it before the date.

DeviceX509ChainFailuresSustained​

严重程度: warning · 规则组: devicechain.device-management

Devices with registered certificates are being refused because their certificate chain does not verify, steadily

含义与处理方式. For 15m device-management has refused more than one certificate connect a second (‹value›/s now) because a registered certificate's chain does not build to its trust anchor, or has expired. Only certificates that are pinned to a credential count: certificates nothing pins and malformed presentations are separate results, so a client outside the tenant cannot raise this alert. Either the devices stopped sending the intermediate certificates that link them to the anchor, the chain or the anchor has expired, or a CA was replaced without the devices being re-registered. The answer on the wire is the same for every cause: check what the device presents with openssl s_client against the broker, and the expiry of the certificate, its intermediates and the trust anchor.

DeviceX509IssuerPinUnavailable​

严重程度: critical · 规则组: devicechain.device-management

Certificate devices are being refused because the auth callout cannot vouch for the broker's servers

含义与处理方式. For 10m device-management has refused certificate connects without a break of two minutes because its issuer pin has no usable list of the broker's live servers: its system-account connection is down, or no refresh has succeeded for three refresh periods. Every certificate device is refused until it recovers; token and password devices are not affected. Check callout_issuer_pin_connected and callout_issuer_pin_refresh_total{result="error"}, the device-management log for the system-account connection, and that infrastructure.nats.auth.sysUser and sysPassword are valid.

DeviceX509JitMisconfigured​

严重程度: warning · 规则组: devicechain.device-management

A certificate provisioning profile cannot register devices because it is misconfigured

含义与处理方式. In the last 10m device-management refused ‹value› certificate registration(s) as ‹result›. jit_provisioning_misconfigured and jit_credential_type_unsupported mean the profile's device type was deleted or it mints a credential type this build does not support; jit_ambiguous means a device's certificate chains to the trust anchor of more than one enabled profile at the same depth and cannot say which was meant. Every device concerned is refused until the profile is fixed or each profile has a trust anchor of its own. The service logs the cause.

DeviceX509JitRefusedSustained​

严重程度: warning · 规则组: devicechain.device-management

Devices are being refused registration on first connect because registration is over its limit

含义与处理方式. For 15m device-management has refused certificate registrations at ‹value›/s, from the ‹result› limit. jit_rate_limited means the tenant's registrations per second are at its governance ceiling; jit_busy means the registration slots of every replica are taken, usually by a factory run on one provisioning profile, whose registrations take turns. Devices back off and retry on their own; raise the tenant's provisioning ceiling or spread the run out. A device that keeps retrying with a rotated certificate, before certificate renewal is available, spends the same allowance: every attempt that passes chain verification is charged to the tenant, including one refused afterwards.

MqttStoreBackoffEngaged​

严重程度: warning · 规则组: devicechain.device-management

New MQTT device connections are being refused: the broker's MQTT message store is nearly full

含义与处理方式. For 2m device-management has refused every new MQTT device connection, because the broker's MQTT message store is above 80% of its bound. Devices already connected keep publishing, and connections are accepted again once the store is below 60%. The platform is reading device publishes more slowly than devices send them: look at event-sources and at what reads inbound-events. Refused devices are told their connection is not authorized (MQTT CONNACK code 5) and must keep retrying.

PayloadDecoderRecompileSustained​

严重程度: warning · 规则组: devicechain.device-management

A device-management replica is compiling payload decoders over and over

含义与处理方式. On ‹pod›, payload decoders have been compiled more than once a second for 15m (now ‹value›/s). A compile happens when a device type's resolution is read and its decoded copy is not held in memory, and a large decoder takes tens of milliseconds, so this is paid on the events themselves. The usual cause is decoders let go for room: the decoders one tenant keeps in use do not fit together in its share of the resolution cache, or the cache is too small. Raise inMemoryCache.profileResolutionCacheMiB, or profileResolutionTenantSharePercent for the tenant with the most decoders, and raise the device-management memory limit with it. Compare kv_cache_local_evictions_total{cache="profile-resolution-by-type"} by reason: tenant_share points at the share, capacity at the cache.

ProvisioningAuthFailuresSustained​

严重程度: warning · 规则组: devicechain.device-management

Device self-registration is being refused for a wrong provision key or secret, steadily

含义与处理方式. For 15m device-management has refused more than one self-registration request a second (‹value›/s now) because the provision key or secret was not valid. Either a fleet is registering with a secret that is wrong, was rotated or belongs to a disabled or expired profile, or someone is probing the endpoint. The answer is the same for every cause, so check the provisioning profile the fleet should be using (enabled, not expired, secret current) and the sources in the ingress log.

ProvisioningMisconfigured​

严重程度: warning · 规则组: devicechain.device-management

A provisioning profile cannot register devices because it is misconfigured

含义与处理方式. In the last 10m device-management refused ‹value› self-registration request(s) against a provisioning profile that cannot work: its device type was deleted, its strategy is not one the platform knows, or it mints a credential type this build does not support. Every device using that profile is refused until the profile is fixed. The service logs the profile's token at ERROR.

ProvisioningRateLimited​

严重程度: warning · 规则组: devicechain.device-management

Device self-registration is refusing requests with 429 because a rate limit is exhausted

含义与处理方式. For 15m device-management has refused self-registration requests with 429 at ‹value›/s, from the ‹result› limit. rate_limited_source means one client address used up its allowance: if it is the ingress controller's address, every device is sharing one allowance because infrastructure.http.trustedProxies does not name the controller. rate_limited_tenant means the tenant's registrations per second are at its governance ceiling. Devices retry on their own after the Retry-After they are told.

ProvisioningSessionKickUnconfirmed​

严重程度: warning · 规则组: devicechain.device-management

A revoked or reset device may still be connected: its broker session was never confirmed ended

含义与处理方式. For at least 5m, the oldest session kick that a provisioning revoke or reset still owes has waited more than 10 minutes (‹value› now). The credentials were withdrawn, so the device cannot reconnect, but an MQTT session it already had open can keep publishing until its token expires (12 hours by default). device-management retries every 30 seconds, so the responder that ends sessions (event-sources) is down or unreachable: check that event-sources is running and its MQTT auth callout is healthy.

X509OcspUnavailableAllowed​

严重程度: warning · 规则组: devicechain.device-management

Certificate connections are being allowed without current OCSP status

含义与处理方式. For 10 minutes, ENFORCE_KNOWN anchors have admitted certificate connections whose OCSP status could not be learned. Revoked certificates may be admitted while the responder is unreachable. Inspect the Ingest dashboard allowance reasons, responder fetch failures and breakers. Restore responder reachability and allowed egress, check node clocks and capacity, and verify deviceManagement.callout.ocsp.enabled and the anchor policy. REQUIRE_CURRENT refuses connections when current status is unavailable; change policy only with the impact on device availability understood.

devicechain.device-state​

DeviceStateDroppingEvents​

严重程度: warning · 规则组: devicechain.device-state

device-state is discarding events it could not apply to live device state

含义与处理方式. In the last 15 minutes, device-state gave up on events after their last delivery attempt and discarded them. Each one was not applied to its device's live state, so that device's connectivity, latest readings or last position can be out of date until a newer event for it arrives. The event history is unaffected. The usual cause is device-state's database refusing or failing its writes for longer than the redelivery window: check its database, and the device-state log for 'Giving up on', which names the write and the device.

devicechain.edge​

Lwm2mIngestDroppingNotifications​

严重程度: warning · 规则组: devicechain.edge

LwM2M ingest is dropping Notify samples

含义与处理方式. For at least 10 minutes, lwm2m-ingest has been dropping the samples of Notify messages because ingest failed or refused them, backpressure included. There is no retry in the Notify path (the next Notify supersedes), so a device that reports rarely loses its reading. Check the platform's message broker and the inbound-events stream's backlog, and the lwm2m-ingest log.

Lwm2mIngestLeaderCountNotOne​

严重程度: critical · 规则组: devicechain.edge

LwM2M ingest is not served by exactly one replica

含义与处理方式. For at least 2 minutes, ‹value› lwm2m-ingest replicas have reported holding the leadership lease; it should be exactly 1. At 0, nobody is reading the CoAP/DTLS transport and every LwM2M device is silently unreachable, although every pod looks healthy. Above 1, two replicas are serving at once. Check the lwm2m-ingest logs for why no replica acquired the lease (typically the platform's message broker is unreachable).

Lwm2mIngestLeaderMetricAbsent​

严重程度: critical · 规则组: devicechain.edge

LwM2M ingest has pods available but exports no leader series

含义与处理方式. For at least 5 minutes, the lwm2m-ingest deployment has had pod(s) available while no replica exports devicechain_lwm2mingest_is_leader, so whether the transport is being read cannot be known: Lwm2mIngestLeaderCountNotOne reads an empty sum as healthy and cannot fire. The usual causes are a broken scrape (a ServiceMonitor or PodMonitor change, a network policy, a metrics port or TLS change) or a build without the metric. Check the scrape target for lwm2m-ingest in Prometheus, then the pods' /metrics, before reading the logs. This does not fire for an area that is not deployed or has no pod available (InstanceAreaUnavailable covers that).

Lwm2mIngestLeaderNotServing​

严重程度: critical · 规则组: devicechain.edge

The LwM2M ingest leader holds the lease but is not serving

含义与处理方式. For at least 5 minutes, ‹pod› has held the LwM2M leadership lease while its CoAP/DTLS read loop is not running. It is stuck building its term (typically reading the asserted-active devices from device-state), and it keeps the lease, so no standby can take over. No LwM2M device is being served. Check this pod's log around 'Acquired LwM2M leadership' and restart it so a standby can acquire.

SparkplugCommandBreakerOpen​

严重程度: warning · 规则组: devicechain.edge

Sparkplug commands to a source were paused

含义与处理方式. In the last 20 minutes the command circuit breaker for source ‹source› (tenant ‹tenant›) opened. It opens when the customer broker drops sparkplug-ingest's session right after a command is published, which almost always means the source's credential may not publish on spBv1.0/<group>/NCMD/+ and spBv1.0/<group>/DCMD/+/+. While it is open, commands to that source fail at once as not delivered; telemetry from the source is not affected. Grant the publish permission on the broker. This alert resolves 20 minutes after the last time the breaker opened, whether or not it has closed: after its pause it is half-open and closes only when the next command is published without a disconnect.

SparkplugCommandResponsesFailing​

严重程度: warning · 规则组: devicechain.edge

Sparkplug commands are failing inside the platform

含义与处理方式. For at least 5 minutes, sparkplug-ingest has been counting commands it could not carry through, with result ‹result›: claim_failed, park_failed and response_failed mean command-delivery could not be reached or refused the call (the command stays SENT or is never answered until its lifetime ends); kv_failed means the store of commands awaiting confirmation could not be written, so the command was not published; publish_failed means a publish may or may not have reached the node. Check command-delivery and the message broker, and the sparkplug-ingest log.

SparkplugCommandsUnconfirmed​

严重程度: warning · 规则组: devicechain.edge

Sparkplug commands are being published but not confirmed

含义与处理方式. For at least 15 minutes, more than 90% of the Sparkplug commands sparkplug-ingest published in the last 30 minutes (at least 20 of them) ended without the node reporting the requested value. Sparkplug has no command acknowledgement, so this is how a command path that does not work looks: the customer broker may be dropping the publishes (check the source credential may publish on the NCMD and DCMD topics), the nodes may be ignoring a host that is not their primary (commands.hostRole), or the confirm window may be shorter than the nodes' reporting delay (commands.confirmWindowSeconds). Commands are recorded as TIMEOUT, not as failed.

SparkplugIngestBrokerUnreachable​

严重程度: warning · 规则组: devicechain.edge

Sparkplug ingest cannot connect to a configured broker

含义与处理方式. For at least 10 minutes, sparkplug-ingest has been failing to connect to a configured Sparkplug broker. Readiness is deliberately not gated on broker reachability, so the pod looks healthy while that source ingests nothing. A bad host, a rejected credential or a broker outage all raise this; the sparkplug-ingest log names the source and the reason.

SparkplugIngestDroppingData​

严重程度: warning · 规则组: devicechain.edge

Sparkplug ingest is abandoning accepted messages

含义与处理方式. For at least 10 minutes, sparkplug-ingest has been abandoning accepted messages: their samples or presence transitions were dropped after the retry budget was spent (device-management or the message broker unreachable), at once under backpressure, or on shutdown. A Sparkplug Host gets no broker redelivery, so each one is lost. Check device-management and the platform's message broker, and the sparkplug-ingest log for 'Dropping Sparkplug samples'.

SparkplugIngestLeaderCountNotOne​

严重程度: critical · 规则组: devicechain.edge

Sparkplug ingest is not served by exactly one replica

含义与处理方式. For at least 2 minutes, ‹value› sparkplug-ingest replicas have reported holding the leadership lease; it should be exactly 1. At 0, no Sparkplug source is being read and every device behind it is silently unreachable, although every pod looks healthy. Above 1, two replicas are connecting the same sources. A pod deployed with no sources configured never takes the lease and reads 0 permanently: configure its sources or leave the service out of the deployment. Otherwise check the sparkplug-ingest logs for why no replica acquired the lease.

SparkplugIngestLeaderMetricAbsent​

严重程度: critical · 规则组: devicechain.edge

Sparkplug ingest has pods available but exports no leader series

含义与处理方式. For at least 5 minutes, the sparkplug-ingest deployment has had pod(s) available while no replica exports devicechain_sparkplugingest_is_leader, so whether the transport is being read cannot be known: SparkplugIngestLeaderCountNotOne reads an empty sum as healthy and cannot fire. The usual causes are a broken scrape (a ServiceMonitor or PodMonitor change, a network policy, a metrics port or TLS change) or a build without the metric. Check the scrape target for sparkplug-ingest in Prometheus, then the pods' /metrics, before reading the logs. This does not fire for an area that is not deployed or has no pod available (InstanceAreaUnavailable covers that).

SparkplugIngestSubscriptionRefused​

严重程度: warning · 规则组: devicechain.edge

A Sparkplug source's broker refused a group subscription

含义与处理方式. For at least 5 minutes, a Sparkplug source has been abandoned and retried because its broker accepted the connection but refused (or never acknowledged) a group subscription, most likely because the source's credential may not read that group. While any group is refused the source ingests none of its groups and never announces ONLINE. Check the broker's access rules for the source's credential against the groups configured for it.

devicechain.event-management​

FailedEventsSustained​

严重程度: warning · 规则组: devicechain.event-management

the platform has been refusing more than one event a second for the same reason for 15 minutes

含义与处理方式. Events are being refused or could not be stored at more than one a second for the reason named by the reason label, and have been for 15 minutes. That can be a single device sending that fast (2 a second from one device is enough to fire it) or many devices failing together: a credential or certificate that is wrong across many devices (unauthenticated), devices that were never registered (device_not_found), a gateway publishing for devices it does not hold (not_gateway_child), or a failing dependency (api_call_failed). The tenants' failedEvents query and the Ingest failures view in the console show which devices and what the platform said, at most ten rows per device an hour with the rest counted. Nothing is re-sent: the events were not stored, and fixing the cause makes the devices' next events succeed.

devicechain.event-processing​

DetectCheckpointsStalledWithBacklog​

严重程度: critical · 规则组: devicechain.event-processing

DETECT is not checkpointing while a backlog exists

含义与处理方式. No committed checkpoint in 15m while undelivered work waits — the single-writer loop is likely stuck (a broker or store outage). A replica that lost a split-brain race does not cause this: it exits to be replaced, and DetectHasNoLeader covers the gap.

DetectConsumerBacklogHigh​

严重程度: warning · 规则组: devicechain.event-processing

DETECT consumer is falling behind

含义与处理方式. The resolved-events durable consumer has >50k undelivered messages for 10m; the DETECT engine is not keeping up with ingest.

DetectFactReconcileFailing​

严重程度: warning · 规则组: devicechain.event-processing

The detection engine cannot compare its rules and devices with device-management

含义与处理方式. event-processing's comparison with device-management (or the tenant list from user-management) is failing; the projection label says which part. Nothing is removed while it fails; what is lost is the repair — a missed rule, device or threshold change stays missed until the comparison succeeds. The tenant and cause are in event-processing's error log. Check device-management and user-management health and the service secret; during a rolling upgrade a message that device-management is running an older build is expected briefly.

DetectFactsRepaired​

严重程度: warning · 规则组: devicechain.event-processing

The detection engine repaired changes it had never been told about

含义与处理方式. event-processing found published rules, active profile versions, device memberships or threshold attributes (the projection label says which) that differed from device-management, and corrected them. Detection is right again; until the repair it was running on the old state. A change still in flight is not counted — only one whose notification never arrived. Recurring means notifications are being lost: see DeviceFactPublishFailing and the broker.

DetectFanoutEvalErrors​

严重程度: warning · 规则组: devicechain.event-processing

DETECT leaf-predicate evaluation errors

含义与处理方式. Rule leaf predicates are erroring during fan-out. The sample is SKIPPED, not treated as a non-match, so firings may be silently missed. This counter is unlabelled and nothing logs the rule id — use the canvas preview on each suspect rule to attribute the error.

DetectFenceArchiveSkew​

严重程度: warning · 规则组: devicechain.event-processing

device-management does not serve the geofence manifest doors

含义与处理方式. event-processing asked device-management for a fence-set manifest and was told the query does not exist, which means the two are running builds from either side of the manifest-delivery change. Expected briefly while an upgrade rolls, since the two services are independent Deployments and neither waits for the other. Sustained means device-management is behind — most likely rolled back on its own — and fence edits are not reaching the detection engine until it rolls forward.

DetectFenceGeometryHashMismatch​

严重程度: critical · 规则组: devicechain.event-processing

Geofence geometry did not match the content address it was served under

含义与处理方式. A geometry document did not hash to the address it was requested under. This is never transient and never benign: an address IS the SHA-256 of the document stored under it, so either device-management served the wrong row, something re-encoded the document in transit, or the archive is corrupt. The affected fences are refused rather than compiled, which is why containment degrades instead of answering wrongly. Investigate before restarting anything — a restart hides the symptom and changes nothing.

DetectFenceGeometryUnresolved​

严重程度: warning · 规则组: devicechain.event-processing

Geofence geometry cannot be read back from device-management

含义与处理方式. A geofence set names fences whose geometry this service could not fetch from device-management's archive. Those fences are installed carrying their error, so containment reports unresolvable for them rather than answering — which also shows up as DetectFanoutEvalErrors. It repairs on the next reconcile sweep if the cause was transient; sustained means the archive read is failing. Check device-management's health and the service-token seam. The tenant and fence are named in event-processing's error log, never in this counter.

DetectHasNoLeader​

严重程度: critical · 规则组: devicechain.event-processing

No replica holds the DETECT partition

含义与处理方式. Nothing has owned the DETECT partition for 2m, so no detection rule is being evaluated for any tenant. The absent() half is not optional: a pod in CrashLoopBackOff publishes no series at all, so a rule written only against the value would go quiet at exactly the moment it should fire.

DetectLeaderIsNotConsuming​

严重程度: critical · 规则组: devicechain.event-processing

A DETECT leader is holding the partition without consuming

含义与处理方式. A replica owns the partition but is not inside a live term and has committed nothing for 10m — it is stuck building a term, which holds the partition against every standby while detecting nothing. The checkpoint term is what keeps this quiet during a long replay, which checkpoints as it goes.

DetectLiveDeliveriesRecovered​

严重程度: warning · 规则组: devicechain.event-processing

Events the broker delivered never reached the detection engine, and were recovered

含义与处理方式. The detection engine found events the broker had handed out that it never received (a connection dropped mid-delivery, or a second engine took part of the stream), read them from the stream and applied them in order. Detection is correct; without this recovery those events would have been skipped. Recurring means connections between event-processing and the broker are dropping under load, or two engines are consuming at once: check the broker, the network between them, and DetectHasNoLeader / replica counts.

DetectLiveGapFillFailing​

严重程度: critical · 规则组: devicechain.event-processing

Live detection has stopped because part of the event stream cannot be read

含义与处理方式. The detection engine received an event after a stretch of the stream it had not seen, and cannot read that stretch from the broker. It has stopped applying live events rather than skip past it: no new detections are produced until the read succeeds, and the backlog grows (see DetectConsumerBacklogHigh). Rule, device and tenant changes are still being applied. Nothing is lost; the stretch is read as soon as the broker answers. Check the NATS broker health and its connection from event-processing; the cause is in event-processing error log.

DetectLoopStalled​

严重程度: critical · 规则组: devicechain.event-processing

The DETECT loop on a live leader has stopped making progress

含义与处理方式. A replica that holds the partition and reports itself live has not completed a pass of its detection loop in over 2m (a healthy loop passes at least once a second). The loop is hung inside a call, most likely against a black-holed database or broker connection, and every other DETECT gauge is sampled on that same loop, so they are frozen at their last values and look healthy. No rule is being evaluated for any tenant. Restart the pod named on the alert. A standby never fires this: it is not leading.

DetectRuleStatWriteFailing​

严重程度: warning · 规则组: devicechain.event-processing

Rule-health counters cannot be written

含义与处理方式. event-processing could not write a batch of rule-health fire counters (one tenant's transaction). Detection is unaffected: the counters are written in the background and a failed batch is dropped, not retried, so alarms still raise and clear. What suffers is the console's per-rule fire count and last-fired time, which fall short and go stale; the fires lost are counted in devicechain_eventprocessing_detect_rule_stat_updates_discarded_total (reason write_failed). event-processing logs one warning per flush that lost batches, with the first cause; check its database connection and that the rule_stats table accepts writes.

DetectShardInvariantViolated​

严重程度: critical · 规则组: devicechain.event-processing

DETECT shards disagreed and a checkpoint was refused

含义与处理方式. The detection shards were not at the same watermark and position at a checkpoint, so the snapshot was refused (nothing wrong was saved). This is a defect, not load. Checkpoints keep failing on the pod named on this alert until it is restarted, which rebuilds the shards from the last good snapshot; restart that pod, then report it with its event-processing log from the first refusal.

DetectShardSaturated​

严重程度: warning · 规则组: devicechain.event-processing

A DETECT shard is busy more than 90% of the time

含义与处理方式. One detection shard has spent over 90% of the last ten minutes applying operations, so it is the limit on how fast detection can run. Compare devicechain_eventprocessing_detect_shard_busy_seconds_total across shards: one far above the rest is a hot key (a very busy device, or a correlation anchor with many members), all of them high means detectShards is too low for the load, or the CPU limit is.

DetectTenantOverStateBudget​

严重程度: warning · 规则组: devicechain.event-processing

A tenant is over its DETECT state budget

含义与处理方式. One or more tenants exceed the per-tenant rule-count, live-key or retained-sample budget for 30m. Find the offender in the event-processing warn log. Check the memory gauges before concluding memory is fine — devicechain_eventprocessing_detect_live_keys stays FLAT for the shape that fills the heap fastest: a long-window rule on one busy device moves detect_retained_samples only. devicechain_eventprocessing_detect_pending_timers counts armed absence/session keys (instance-wide, not per-tenant, and has no budget arm here).

DetectWatermarkLagHigh​

严重程度: warning · 规则组: devicechain.event-processing

DETECT watermark is lagging wall-clock

含义与处理方式. The engine watermark is >5m behind wall-clock for 10m; time-based detections (absence/duration) will fire late.

DeviceFactPublishFailing​

严重程度: warning · 规则组: devicechain.event-processing

Rule, device or attribute changes are not being announced to the detection engine

含义与处理方式. device-management could not publish a detection-rule, device-roster, device-attribute or device-ownership notification (the fact label says which) — a marshal error, a broker refusal or a transport fault. The change itself committed; the detection engine picks it up at its next comparison with device-management, within a few minutes, instead of at once (a device-ownership miss is repaired by the device's next event instead). Sustained means every such change is taking minutes to take effect: look at the broker.

GeoFenceSetPublishFailing​

严重程度: warning · 规则组: devicechain.event-processing

Geofence edits are not being announced to the detection engine

含义与处理方式. device-management could not publish a geofence-set manifest — a marshal error, a broker refusal, or a transport fault. The fence edit itself succeeded and is durable; what failed is telling event-processing about it, so containment holds the previous fence set until the five-minute reconcile sweep repairs it. A sustained rate means every fence edit is taking minutes to take effect. If this started after a broker configuration change, check that infrastructure.nats.streamMaxMsgSize is not set below what a manifest needs — the service logs a warning naming both sizes at startup.

ReactConnectorEgressShedding​

严重程度: warning · 规则组: devicechain.event-processing

Connector actions are being shed at the source for over-quota egress

含义与处理方式. Sustained shedding means a tenant was over its outbound ceiling on the timeline its telemetry reached the platform. A catch-up after a restart no longer counts. Each shed action is dead-lettered with reason shed (kind detection-action) up to a per-tenant budget. Anything over the budget is counted on react_connector_shed_unlettered_total and summarised in one letter per tenant per minute. Inspect them with dcctl dead-letters. Detections re-published when the engine restarts are collapsed by the message bus within 30 minutes. A replay older than that can still shed and letter duplicates of actions already delivered. Raise outboundCallsPerSecond / outboundBurst if the traffic is legitimate.

devicechain.event-sources​

BrokerPresenceLosingTransitions​

严重程度: warning · 规则组: devicechain.event-sources

Broker presence transitions are not reaching the event stream

含义与处理方式. For at least 10 minutes, event-sources has been failing to write device connects and disconnects to the platform's event stream (‹value› in the last 10 minutes). The broker does not resend connection notices, so each one is lost until the next reconciliation pass (every 5 minutes by default) puts the device right; until then a device can read offline while it is connected, or the other way round, and a reconnecting device's held commands are not released early. The cause is the platform's message broker refusing or not answering publishes. Check it, and the event-sources log for 'Failed to emit a broker presence transition', which names each device.

DeviceUplinkMessagesFailing​

严重程度: warning · 规则组: devicechain.event-sources

Device MQTT payloads cannot reach their decoder pipeline

含义与处理方式. For 10 minutes, raw device MQTT uplinks have failed before reaching the decoder pipeline. Missing decoders, invalid or oversized payloads and unknown devices go to failed-decode; plan resolution and publish failures retry through the broker. Inspect the Ingest dashboard result labels, failed-events and event-sources logs; check the profile decoder and device-management or broker availability. PayloadDecoderFailuresHigh covers failures after handoff to the decoder.

ExternalMqttSourceLosingEvents​

严重程度: warning · 规则组: devicechain.event-sources

An external MQTT source is losing events it has already read

含义与处理方式. For at least 10 minutes, external MQTT source ‹source› has been losing events: they were read from your broker and decoded, but publishing them to the platform's event stream failed. That source does not retry, and your broker was already told they arrived, so each one is gone. The cause is on the platform's side, not your broker's: its message broker is refusing or not answering publishes. Check the platform's message broker, and the event-sources log for 'is lost: this transport does not retry', which names the source and tenant of each.

ExternalMqttSourceNotReadByOnePod​

严重程度: warning · 规则组: devicechain.event-sources

An external MQTT source is not being read by exactly one pod

含义与处理方式. External MQTT source ‹source› has been read by ‹value› event-sources pods for 2m; it should be read by exactly one. At 0, nothing is reading it, and whatever your broker delivers meanwhile is lost, because the session is not persistent. The usual cause is that your broker is unreachable or refusing the connection: the pod that took the source over logs why, releases the source and tries again every 15 seconds. Another cause is that event-sources cannot reach the platform's message broker to agree which pod reads the source; its log says so. Above 1, more than one pod reports reading it. A pod that has lost the source drops what it is still delivered rather than storing it, and stops reporting within a lease renewal, so a value above 1 that lasts points at a pod that has lost contact with the platform's message broker.

IngestAppendSkewLarge​

严重程度: warning · 规则组: devicechain.event-sources

The message broker's clock is more than 2 seconds from event-sources'

含义与处理方式. For at least 10 minutes, the lag-credited ingest gate (eventSources.ingestMetering: lagCredit) has measured the message broker's append stamps on stream ‹stream› as more than 2 seconds away from the clock of an event-sources pod. The gate corrects a message's age for that offset only up to 2 seconds, so beyond it a tenant is credited too little after a stall (the broker's clock ahead) or too much (behind). devicechain_eventsources_append_skew_seconds shows the offset each pod measured, positive when the broker's clock is behind. Check time synchronisation (NTP) on the nodes running the message broker and event-sources; the alert clears once the clocks agree again.

IngestShedBeyondLagCredit​

严重程度: warning · 规则组: devicechain.event-sources

Messages were shed at the ingest ceiling while the platform's readers were behind

含义与处理方式. In the last 10 minutes, ‹value› messages read from stream ‹stream› on pod ‹pod› were shed at their tenant's ingest ceiling by the lag-credited gate (eventSources.ingestMetering: lagCredit), and in the same 10 minutes that pod measured its readers of the stream as more than a second behind. The likely cause is a tenant sending above its ceiling while the readers were behind: compare devicechain_eventsources_total_msg_rate_limited, raise event-sources to debug logging to name the tenant, and check its ceiling (ingestReadingsPerSecond, ingestBurst) against what its devices send. The other cause is a platform stall longer than the credit covers (the time the tenant's burst covers, never more than 5 seconds): check the platform's message broker and event-sources for a stall at the same time (devicechain_eventsources_reader_lag_seconds shows how far behind each stream was). If the tenant was within its ceiling, report it: it is the case this metering mode exists to prevent. MQTT messages shed this way were already acknowledged to their devices; MqttIngestShedAfterAcknowledgement fires for them as well.

IntegrationAuthFailuresSustained​

严重程度: warning · 规则组: devicechain.event-sources

Requests to an integration are being refused for a bad token

含义与处理方式. For at least 10 minutes, more than 30 requests in each 10 minutes to an inbound integration have been refused because the bearer token was wrong, the integration does not exist, or the integration's failure budget was spent. If a sender stopped delivering after you rotated an integration's token, update the token in the network server or webhook. If nothing of yours changed, someone who knows the URL is trying tokens: the integration's 'last authentication failure' time in the console shows which one, and the sprayed integration keeps serving senders that are already known.

IntegrationDownlinkReportsDiscarded​

严重程度: warning · 规则组: devicechain.event-sources

Network-server downlink reports are being accepted and discarded

含义与处理方式. For 10 minutes, inbound integrations have discarded downlink reports above their separate report allowance while answering 202. The sender believes they arrived, and commands those reports would settle wait until their deadline. Reduce the report flood and inspect integration traffic and the confirmed-downlink dashboard before resending commands.

IntegrationShedAccepted​

严重程度: warning · 规则组: devicechain.event-sources

Integration uplinks are being accepted and discarded at a tenant's ingest ceiling

含义与处理方式. In the last 10 minutes, uplinks from inbound integrations were discarded because a tenant is over its ingest ceiling. They were answered 202 so the network server would not suspend its webhook, which means the sender believes they arrived and they did not. Raise the tenant's ingest ceiling, or reduce what the integration sends.

IntegrationUndecodedHigh​

严重程度: warning · 规则组: devicechain.event-sources

Most integration uplinks carry nothing to store

含义与处理方式. For at least 30 minutes, more than half of the uplinks arriving over inbound integrations have had no decoded values to store. The network server forwards the raw frame but the device profile there has no payload codec configured (or the codec returns nothing), or a generic sender posts only text. Configure the codec in the network server; the integration's 'last undecoded' sample in the console shows a frame it could not read.

MqttFailClosedDurableMissing​

严重程度: critical · 规则组: devicechain.event-sources

A platform durable on the MQTT message store is missing

含义与处理方式. Durable ‹durable› on the MQTT gateway's message store has been missing within the last 10 minutes. While it is missing, the broker acknowledges device QoS 1 publishes of its kind without keeping them, so each one is lost although the device was told it arrived. event-sources creates it again within 30 seconds, and this alert clears 10 minutes after the last time it was found missing; while it cannot be created, the alert stays. A rolling update does not remove a durable, so it does not raise this alert. Find what deleted it (a command run against the broker's consumers, or a tool that manages them), and check the event-sources log for 'platform durable on the MQTT message store'.

MqttFailClosedStoreNotAsDeclared​

严重程度: critical · 规则组: devicechain.event-sources

The MQTT message store is not as the platform declared it

含义与处理方式. Within the last 10 minutes event-sources found the MQTT gateway's message store changed from its fail-closed configuration, or found another DeviceChain instance on the same broker. A changed setting (the discard policy, the size bound, or the store deleted outright) is put back within 30 seconds, but while it lasted a full store could discard messages it had already acknowledged, or refuse every publish. Another instance on the broker shares the store and its bound, so one instance's backlog refuses the other's devices: give each instance its own broker, or turn infrastructure.nats.mqttFailClosed off. The event-sources log names the setting it put back, or the other instances.

MqttGatewayStreamUnbounded​

严重程度: warning · 规则组: devicechain.event-sources

An MQTT gateway stream has no size limit

含义与处理方式. For at least 10 minutes the broker's MQTT gateway stream ‹stream› has had no size limit and event-sources could not set one. The broker creates these streams without a limit when the first MQTT device connects, and an unbounded one can fill the broker's storage and take the whole instance down. Check that the broker has storage free to reserve the limit and look in the event-sources log for 'Could not bound the MQTT gateway'. It clears on its own once the limit is set.

MqttIngestShedAfterAcknowledgement​

严重程度: warning · 规则组: devicechain.event-sources

MQTT messages were dropped at the ingest ceiling after the device was told they arrived

含义与处理方式. In the last 10 minutes, ‹value› messages from MQTT source ‹source› were dropped at their tenant's ingest ceiling. The broker had already acknowledged each one to its device (PUBACK), so the device will not resend it: these messages are lost, not deferred. Either a tenant is sending above its ceiling, or the platform stalled briefly and the messages queued behind it arrived together; the shed counters carry the shed class but not the tenant, so raise event-sources to debug logging to name the tenant. Compare devicechain_eventsources_total_msg_rate_limited and total_msg_reading_limited, check whether the tenant's ceiling (ingestReadingsPerSecond, ingestBurst) matches what its devices send, and look at the platform's message broker for a stall (a log line saying an internal subscription took too long) at the same time.

devicechain.governance​

ConnectorDispatchRateLimited​

严重程度: warning · 规则组: devicechain.governance

The connectors service is shedding outbound actions the detection engine admitted

含义与处理方式. For at least 15 minutes, ‹job› has shed ‹action› dispatches for being over their tenant's outbound rate. The detection engine meters the same ceiling on the same timeline and sheds over-quota actions before dispatching them, so these were admitted at one end and refused at the other: the two ends disagree about the ceiling. The usual cause is the platform default configured differently for event-processing and outbound-connectors (outboundCallsPerSecond and outboundBurst must match), or one of them metering at the default because it cannot read tenant ceilings (see TenantsMeteredAtPlatformDefault), or sends that are failing and being retried, since every retry is metered again here and only once at the source. Each shed dispatch is dead-lettered with reason rate_limited; inspect them with dcctl dead-letters.

RateLimiterOverflowInUse​

严重程度: warning · 规则组: devicechain.governance

HTTP ingest is receiving traffic for more unconfirmed tenant names than it keeps separate allowances for

含义与处理方式. ‹job› has had more than 1024 unconfirmed tenant names active at once, so every further name shares one allowance. Almost always this means something is posting to many invented tenant paths on the HTTP ingest port. Legitimate tenants that have been confirmed, and every MQTT, NATS and LwM2M tenant, keep their own allowance. Find the source of the requests on the HTTP ingest port.

RateMeteringClockFallback​

严重程度: warning · 规则组: devicechain.governance

Outbound actions are being metered on a fallback clock

含义与处理方式. For at least an hour, ‹job› has metered outbound actions on the ‹source› clock because they carried no trigger time. The detection engine has stopped stamping trigger times, so actions are metered on broker or arrival time and a catch-up after a restart can be shed as a flood again. An upgrade's in-flight backlog clears well inside the hour. Check that event-processing and outbound-connectors run the same release.

ReactShedLettersOverBudget​

严重程度: warning · 规则组: devicechain.governance

Outbound connector actions shed at the source are being summarised rather than recorded one by one

含义与处理方式. For at least 10 minutes, ‹job› has shed ‹action› actions faster than its dead-letter budget records them, so the excess is counted and summarised in one dead letter per tenant per minute instead of recorded individually. A tenant is well over its outbound ceiling. Find it with dcctl dead-letters (reason shed) and check its rules, or raise its outboundCallsPerSecond / outboundBurst if the traffic is legitimate.

TenantsMeteredAtPlatformDefault​

严重程度: warning · 规则组: devicechain.governance

A service cannot read tenant ceilings from user-management and is metering tenants at the platform default

含义与处理方式. For at least 15 minutes, ‹job› has admitted ‹dimension› traffic for tenants whose ceilings it could not read because user-management was unreachable or failing, so it metered them at its platform default. For a tenant whose tier sets a lower ceiling, that admits more than the tier allows. Check user-management's health and this service's logs for 'Failed to refresh per-tenant setting'. Tenants whose ceilings were read before the problem started keep their last-known values. Traffic for tenant names user-management does not know is counted separately (cause=unknown-tenant) and does not fire this alert.

devicechain.jetstream-delivery​

BrokerConnectionDiedSilently​

严重程度: warning · 规则组: devicechain.jetstream-delivery

A service's connection to the message broker died without being closed

含义与处理方式. ‹job› gave up a connection to the message broker ‹value› ago, because the server had stopped answering without closing it, and reconnected. A machine that stops or restarts abruptly, a network partition, or a broker server that stopped reading looks like this. Until it gave the connection up, for up to 40 seconds, and then reconnected, that service could neither publish nor receive: if it is event-sources, HTTP ingest was refused or timed out. nats_connection_dead_total says how often, and whether pings or a stalled write detected it. Check whether a node was lost or restarted: kubectl get nodes, and the cloud provider's VM events, because Kubernetes records nothing for a machine that restarts within its node grace period (roughly 40 to 50 seconds). Run dcctl ha verify. Repeats for the same job with no node event point at the network between that service and the broker.

ConsumerHoldingForDatabase​

严重程度: critical · 规则组: devicechain.jetstream-delivery

A service is holding messages because its database refuses writes

含义与处理方式. ‹job› is holding ‹value› message(s) in its ‹consumer› consumer that its database refused. They are not lost: they stay unacknowledged in the message broker and are processed once the database accepts writes again. That consumer is not making progress meanwhile, so what it does is delayed: device events resolved, alarm edges applied, commands answered, notifications sent or update reports recorded, depending on the consumer. The usual cause is a full volume: check DatabaseVolumeCritical and the service's database claim (kubectl get pvc, and the database pod's log for 'No space left on device'), and expand it (spec.storage.size on the Cluster; online where the storage class allows it). Otherwise look for a read-only primary or an exhausted connection limit. Avoid restarting the service while this fires: a shutdown releases what it is holding and each restart uses one of the five deliveries a message has, so repeated restarts can still exhaust them.

EventPersistBacklogGrowing​

严重程度: warning · 规则组: devicechain.jetstream-delivery

Events are arriving faster than the event store persists them

含义与处理方式. Durable ‹durable› has ‹value› events waiting and the backlog has grown steadily for ten minutes. Either the event store is down or refusing writes (EventStoreRefusingWrites, DatabaseVolumeCritical), or ingest is outrunning persistence: compare the batch-size histogram on the persistence dashboard and the database's CPU and IOPS, and raise persistence.writers or persistence.maxBatch. At 90% of the resolved-events ceiling new events are refused (JetStreamIngestBackpressureEngaged) rather than discarded.

EventStoreRefusingWrites​

严重程度: critical · 规则组: devicechain.jetstream-delivery

The event store is refusing writes; events are being held, not stored

含义与处理方式. event-management is holding ‹value› events its database refused. They are not lost: they stay unacknowledged in the message broker and are stored once the database accepts writes again. Nothing new is being stored meanwhile, and the longer this lasts the larger the backlog behind it; once the backlog fills 90% of the resolved-events stream the platform refuses new events (JetStreamIngestBackpressureEngaged), and MQTT events the broker has already acknowledged wait in the capture stream, which discards its oldest once full. The usual cause is a full volume: check DatabaseVolumeCritical and the event-store claim (kubectl get pvc, and the database pod's log for 'No space left on device'), and expand it (spec.storage.size on the dc-tsdb Cluster; online where the storage class allows it) or lower the event retention. Otherwise look for a read-only primary or an exhausted connection limit. Avoid restarting the service while this fires: a shutdown releases what it is holding and each restart uses one of the five deliveries a message has, so repeated restarts can still exhaust them.

JetStreamCaptureBacklogGrowing​

严重程度: warning · 规则组: devicechain.jetstream-delivery

MQTT events waiting to be forwarded are piling up faster than they leave

含义与处理方式. Durable ‹durable› has not yet forwarded ‹value› of what stream ‹stream› can hold, and at the rate that backlog grew over the last five minutes the stream will be full within twenty. This is the early warning for JetStreamCaptureBacklogWillOverflow, which fires when ten minutes remain. Every event in the backlog was sent by an MQTT device that the broker has already acknowledged, so the platform cannot refuse them: once the stream is full it discards the oldest unforwarded ones and the devices are never told (JetStreamDurableLostUnread). Devices are publishing faster than event-sources forwards. Add event-sources replicas now, each of which brings its own publish window, and look at event-sources CPU and at devicechain_eventsources_jetstream_publish_window_full_seconds_total{writer="capture"}; if JetStreamIngestBackpressureEngaged is firing, the slow consumer is further downstream. To buy time, raise streamMaxBytes together with the JetStream volume.

JetStreamCaptureBacklogWillOverflow​

严重程度: critical · 规则组: devicechain.jetstream-delivery

MQTT events the broker has already acknowledged are about to be lost

含义与处理方式. Durable ‹durable› has not yet forwarded ‹value› of what stream ‹stream› can hold, and at the rate that backlog grew over the last three minutes the stream will be full within ten. Every event in it was sent by an MQTT device that the broker has already acknowledged, so the platform cannot refuse them: once the stream is full it discards the oldest unforwarded ones and the devices are never told (JetStreamDurableLostUnread). Devices are publishing faster than event-sources forwards. If event-sources is not busy on CPU, its publish window is the limit: devicechain_eventsources_jetstream_publish_window_full_seconds_total{writer="capture"} grows by close to a second per second; add event-sources replicas, each of which brings its own window (capture.publishWindow already defaults to its maximum, 1024; raise it only if it was lowered). If inbound-events is refusing new events (JetStreamIngestBackpressureEngaged), the slow consumer is further downstream. To buy time, raise streamMaxBytes together with the JetStream volume, and streamMaxMsgs too if the stream fills by message count first (small payloads); event-sources applies the new ceilings when it restarts.

JetStreamCaptureLeaderChanged​

严重程度: warning · 规则组: devicechain.jetstream-delivery

The MQTT capture stream changed leader while devices were publishing

含义与处理方式. Stream ‹stream› changed leader within the last 15 minutes while events were arriving. While a stream has no leader, which lasts several seconds, the broker's MQTT gateway still acknowledges device QoS 1 publishes into it and stores them nowhere, and nothing counts them: JetStreamDurableLostUnread does not fire for this loss, so MQTT events published during the change may have been acknowledged to their devices and lost. In a 12,000 events per second test one such step-down lost about five seconds of events. Look in the broker logs for 'Leader falling behind, stepping down': that is a leader whose node is overloaded, so check the broker's node CPU and whether the capture leader shares a node with the inbound-events leader. A planned broker restart moves leaders too and may lose events the same way, so schedule one at low traffic. This is a known limitation of MQTT ingest, described with the MQTT transport in the documentation.

JetStreamDurableFallingBehind​

严重程度: warning · 规则组: devicechain.jetstream-delivery

a consumer has stayed more than 10000 messages behind its stream for 15 minutes

含义与处理方式. Durable ‹durable› on stream ‹stream› has had more than 10000 messages waiting for it for the last 15 minutes (‹value› now). Whatever that service produces from the stream is that far behind: for device-state, a device's live state lags its stored events. Compare the consumer's rate with the stream's: if the service is keeping pace but not catching up, give it capacity (for device-state, its projection settings and its database); if it has stopped, JetStreamDurableStalledBehindStream and the service's logs say why. If the stream fills first, messages the consumer has not reached will be lost, except where the stream applies backpressure (inbound-events for device-management, resolved-events for event-management): there new events are refused instead (JetStreamIngestBackpressureEngaged).

JetStreamDurableLostUnread​

严重程度: critical · 规则组: devicechain.jetstream-delivery

a consumer lost messages it never read

含义与处理方式. Durable ‹durable› on stream ‹stream› stepped past messages that were removed before it read them, so they were never processed. If a tenant was being deleted, this is expected: the deletion purged messages the consumer had not reached yet. Otherwise the stream was full and discarding its oldest messages while this consumer was behind: either the stream's ceiling is too small for the traffic, or the consumer is slower than its producer. JetStreamDurableFallingBehind and JetStreamDurableUnreadNearFull, which watch this consumer's backlog, tell the two apart.

JetStreamDurableStalledBehindStream​

严重程度: critical · 规则组: devicechain.jetstream-delivery

a consumer that stopped reading is losing messages it has not read

含义与处理方式. Durable ‹durable› on stream ‹stream› has been handed no messages for at least two minutes, and the stream has already discarded ‹value› messages ahead of it. They will never be delivered. The consuming service is running, since it reports this, but its consumer is not reading: look for message handling stuck on a dependency such as the database, or pods waiting to become ready. If that cannot be fixed quickly, raise the stream's ceiling so it stops discarding.

JetStreamDurableUnreadNearFull​

严重程度: warning · 规则组: devicechain.jetstream-delivery

A consumer is close to losing messages it has not read

含义与处理方式. Durable ‹durable› has not yet read ‹value› of what stream ‹stream› can hold. Once its unread backlog reaches the stream's ceiling, the stream discards its oldest messages to make room, and this consumer will never process the ones it had not reached (JetStreamDurableLostUnread). New messages are still accepted. Find out why the consumer is slow: its service's logs, its database, and JetStreamDurableFallingBehind. If the traffic has outgrown the stream, raise the stream's ceiling (and the JetStream volume with it).

JetStreamIngestBackpressureEngaged​

严重程度: critical · 规则组: devicechain.jetstream-delivery

The platform is refusing new events so as not to lose unread ones

含义与处理方式. Stream ‹stream› is refusing new events, for every tenant: a consumer's unread backlog reached 90% of the stream's ceiling, the events it has already processed that the full stream still holds ahead of its unread ones would be discarded within 30 seconds at the rate the stream is discarding them, or a service writing to it could not measure that backlog for 30 seconds. HTTP devices are answered 503 with a Retry-After and should retry. Events from MQTT devices were already acknowledged by the broker and wait in the capture stream, which discards its oldest once it is full (JetStreamDurableLostUnread). Sparkplug and LwM2M readings, and events from an external MQTT broker, are dropped and counted. JetStreamUnreadBacklogNearFull names the consumer that is behind; when it is quiet, it is the second case, and jetstream_backpressure_history_runway_seconds and the writing service's log name the consumer. If neither names a consumer, the writing service may not be reaching the broker at all: look in its log for "Disconnected from NATS" or a write to the NATS server that "made no progress", and for BrokerConnectionDiedSilently; the refusal then lifts once it has reconnected. The refusal lifts once the backlog is below 80% and, if the history was the cause, once that history would last a minute or the consumer has read everything.

JetStreamOutageRunwayShort​

严重程度: warning · 规则组: devicechain.jetstream-delivery

An event stream's outage runway is running out

含义与处理方式. Durable ‹durable› has not yet processed ‹value› of what stream ‹stream› can hold, and at the rate that backlog grew over the last five minutes the platform will stop accepting new events on the stream, for every tenant, within twenty (it does so at 90%). When infrastructure.nats.streamDesignRate is set, the stream is sized to hold streamRunwayMinutes of events at that many events a second while a consumer or the database is down, and this is that time being used; when it is not set, the stream holds only what streamMaxBytes allows. Find out why the consumer is behind: its service is not running or is crash-looping, its database is unavailable or slow (EventStoreRefusingWrites, ConsumerHoldingForDatabase), or the traffic has outgrown the design rate. If it has, raise streamDesignRate together with the JetStream volume: set instance.config.infrastructure.nats.streamDesignRate in a values file and run dcctl upgrade <provider> <instance> --values <file>.

JetStreamReaderForeignDropped​

严重程度: warning · 规则组: devicechain.jetstream-delivery

The device capture reader dropped messages that no device published

含义与处理方式. Durable ‹durable› on stream ‹stream› was handed messages stored under a subject it does not read, and dropped them without passing them on: ‹value› in the last 15 minutes. Its broker-side consumer reads the whole stream unfiltered, so the reader itself keeps only the subjects devices publish telemetry on. Device publishes never land there under another subject, so a dropped message was put into the stream some other way, most likely by a push consumer on another stream whose delivery subject is a device telemetry subject. Nothing was forwarded and no device event is missing, but whoever can create such a consumer is writing into device intake: look for push consumers with a device telemetry delivery subject (nats consumer ls on each stream) and at which login created them. The reader logs the subject and sequence of the first message it drops.

JetStreamStreamNearFull​

严重程度: info · 规则组: devicechain.jetstream-delivery

A stream of records nothing processes is near its ceiling

含义与处理方式. Stream ‹stream› is over 80% of its ceiling. It holds records for an operator, not messages a service processes, so nothing has read what it holds, and once it is full it discards its oldest records to make room. On failed-decode and connector-dispatch.dead those are the records of messages that failed; on max-deliveries they are notices not yet turned into dead letters (MaxDeliveryRecordsWaiting); on dead-letters, the letters themselves, which fires only while user-management, which stores them, does not report reading them, so check that user-management is running; on failed-events, the records of events the platform refused, which fires only while event-management, which indexes them, does not report reading them, so check that event-management is running. Find what is filling it and fix that: a decoder rejecting a device's payloads, a connector whose destination refuses every send, or (on failed-events) a reader that is down while events keep failing to resolve or store. Raise the stream's ceiling only if the records must be kept for longer.

JetStreamUnreadBacklogNearFull​

严重程度: warning · 规则组: devicechain.jetstream-delivery

A consumer's unread backlog is near its stream's ceiling

含义与处理方式. Durable ‹durable› has not yet processed ‹value› of what stream ‹stream› can hold. At 90% the platform stops accepting new events on that stream, for every tenant, rather than discard events this consumer has not read: HTTP devices are answered 503 with a Retry-After, MQTT devices' events wait in the capture stream, and Sparkplug and LwM2M readings, and events from an external MQTT broker, are dropped and counted. Find out why the consumer is slow: its service's logs, its database, and JetStreamDurableFallingBehind. If the traffic has outgrown the stream, raise the stream's ceiling (and the JetStream volume with it).

MaxDeliveryRecordsWaiting​

严重程度: warning · 规则组: devicechain.jetstream-delivery

Messages that ran out of delivery attempts are waiting to be recorded

含义与处理方式. ‹value› notice(s) of messages whose every delivery attempt ran out have been waiting on ‹stream› for 15 minutes without being turned into dead letters. Each service records its own; one that is down or crash-looping records late, so first check that every service is running and look in its logs for errors from the max-delivery recorder. A notice that stays after every service is healthy names a consumer no running service reads any more (a reader removed by an upgrade); it will not be recorded, and can be deleted from the stream.

ReaderHeldMessagePastAckWait​

严重程度: warning · 规则组: devicechain.jetstream-delivery

A service held a message past its acknowledgement window, so it was redelivered

含义与处理方式. The reader for durable ‹durable› held a message longer than the broker waits for an acknowledgement, so the broker handed it out again. stage=worker: a send was still running when that happened, so the message may have been sent twice -- look for a slow or unresponsive destination behind this service. stage=buffer: the reader dropped a message it had fetched but not yet handed to a worker, and the broker's redelivery is handled instead; nothing was sent twice, but the service is not keeping up.

ReplayCoveredDeliveriesExhausted​

严重程度: warning · 规则组: devicechain.jetstream-delivery

A consumer ran out of delivery attempts because it could not save its checkpoint

含义与处理方式. ‹value› message(s) on ‹stream› ran out of delivery attempts to ‹job› in the last 15 minutes. This consumer re-reads the stream from its own saved checkpoint, so nothing has been lost and no dead letter is written, but its checkpoint has not been saved for longer than the broker keeps redelivering: look for the service's database or broker connection failing. While it is running it keeps what it has read in memory and saves it when the checkpoint succeeds. If it restarts first, it re-reads the stream from the last saved checkpoint, and a message the stream has already discarded cannot be re-read (JetStreamDurableUnreadNearFull warns before that).

devicechain.jetstream-replication​

JetStreamClusterUnused​

严重程度: warning · 规则组: devicechain.jetstream-replication

The NATS broker is clustered but nothing on it is replicated

含义与处理方式. This instance is connected to a CLUSTERED NATS broker while every stream and KV bucket is configured for a single replica, so it is paying for a multi-server cluster and surviving zero node losses. Both halves of the toggle have to move together: the server count is OpenTofu's (var.ha / nats_cluster_replicas) and the replica factor is the chart's (instance.config.infrastructure.nats.streamReplicas). dcctl install --ha sets both from one value. If the single-server topology is what you intended, scale the NATS cluster back down rather than leaving it running unused.

JetStreamLeaseBucketNotReplicated​

严重程度: critical · 规则组: devicechain.jetstream-replication

The partition-lease bucket is not replicated

含义与处理方式. ‹stream› holds the partition-ownership leases that decide which pod may write. On an instance configured for HA it must be replicated: at one replica, losing its node does not just lose the leases, it blocks every standby's Acquire and the lease-fenced failover cannot happen at all. Do not claim lease-fenced HA while this is firing.

JetStreamNotReplicatedAsConfigured​

严重程度: warning · 规则组: devicechain.jetstream-replication

JetStream stream or bucket is not replicated as configured

含义与处理方式. ‹stream› has fewer replicas than this instance is configured for. This instance is NOT highly available for that stream. Both halves of the toggle must agree: instance.config.infrastructure.nats.streamReplicas sets the replica count, and the NATS server cluster (tofu var.ha / nats_cluster_replicas) must be large enough to host it. NOTE this does NOT clear on its own once peers arrive: the replica lift runs only when a service starts and ensures its streams, and a lift that failed mid-scale (JSClusterNoPeersError) is warned and not retried. Once the cluster is whole, restart the affected Deployments to re-run the lift.

JetStreamReplicaPeersDegraded​

严重程度: warning · 规则组: devicechain.jetstream-replication

JetStream replica peers are offline

含义与处理方式. ‹stream› reports more replicas than it has online RAFT peers: a copy is offline or missing. It is labelled replicated but is not: losing the leader may lose data or availability. Check NATS pod health and placement -- three replicas across pods that share one node survive zero node losses.

JetStreamReplicaPeersLagging​

严重程度: warning · 规则组: devicechain.jetstream-replication

A JetStream replica has not caught up

含义与处理方式. ‹stream› has had at least one online replica that was not caught up at any check for the last 30 minutes. Short lag under heavy load is normal and does not raise this. A copy this far behind cannot take over without losing recent messages. Check the NATS pods for CPU or disk pressure; it clears once every copy has been caught up at least once in a 15-minute window.

JetStreamReplicationUnobserved​

严重程度: warning · 规则组: devicechain.jetstream-replication

A service can no longer read a JetStream stream's replication state

含义与处理方式. ‹pod› (‹job›) reported the replication of ‹stream› earlier and has not been able to read it for 15 minutes, although the pod is still running. While this fires, the other replication alerts cannot judge that stream for that pod, so its replication is UNKNOWN rather than good. If it fires for every stream at once, the broker or JetStream itself is unavailable: check the NATS pods and the service's connection logs. If it is one stream, that stream has most likely lost its RAFT leader: check it with nats stream info. It clears when the pod reads the stream again, and also once six hours have passed since its last reading, so a resolved alert is not proof of recovery.

devicechain.notification-management​

NotificationClearPickupFailing​

严重程度: warning · 规则组: devicechain.notification-management

Notification escalation and clear pickup passes are failing

含义与处理方式. For 10 minutes, the notification scheduler has reported hard failures while re-paging alarms or delivering pending clear notifications. Clear notifications may be delayed even though escalation already stopped. Check notification-management logs for announcing pending clears failed, channel reachability and credentials, and database errors. Partial passes that deliver clears and leave a backlog for the next tick do not trigger this alert.

devicechain.payload-decoder​

PayloadDecoderFailuresHigh​

严重程度: warning · 规则组: devicechain.payload-decoder

payload decoders are failing more than 5% of the uplinks they decode

含义与处理方式. Over the last 15 minutes more than one in twenty uplinks run through a payload decoder failed to decode. A decoder that is wrong for a device fails on every uplink from it, so this usually means a decoder was published with a mistake, or a device type started sending frames its decoder does not expect. The failures are failed-events records with the reason DECODE_FAILED, and the profile's decoder health shows the last error and keeps recent failing uplinks to replay in the test panel. The result label of devicechain_devicemanagement_resolve_decode_total says which kind of failure it is.

PayloadDecoderSuppressing​

严重程度: warning · 规则组: devicechain.payload-decoder

payload decode failures are being counted but no longer recorded in failed-events

含义与处理方式. A payload decoder is failing so often that device-management has spent the budget of failed-events records for its device type (a burst of 20, refilled 10 a minute, on each replica): more than 50 failures were counted and not recorded in the last 10 minutes. Nothing is lost silently: the count is in the profile's decoder health as suppressed, and the first failures of the window are recorded with the uplink. Fix or unpublish the decoder; the budget refills by itself once it stops failing.

devicechain.safety-gates​

SafetyGateDisabled​

严重程度: warning · 规则组: devicechain.safety-gates

A safety gate is switched off in a running service

含义与处理方式. For at least 5 minutes, ‹job› has run with its ‹gate› safety gate OFF because the configuration it needs is absent (the service secret, or the endpoint of the service the gate consults). Work the gate exists to refuse is being let through: tenant_lifecycle admits work for deleted tenants, presence lets commands to absent devices be published and dropped, rule_validation lets uncompilable detection rules publish and be skipped at load. The service logged a WARN at startup naming what is missing. Restore the missing configuration and restart the service.

devicechain.sign-in​

CredentialAttemptStoreFull​

严重程度: critical · 规则组: devicechain.sign-in

Password sign-ins are being checked without their per-account backoff

含义与处理方式. The credential-attempt store is full, so user-management is checking password sign-ins WITHOUT slowing down repeated failures on any address that is not already waiting out a delay -- and one that is waiting is protected only until that delay ends, at most 5 minutes. Sign-in itself still works -- refusing it would let whoever filled the store lock every user out -- but guessing is now limited only by the per-request field cap and the cost of each password check. The store fills when sign-ins are sent for a very large number of distinct email addresses inside ten minutes, which almost always means someone is spraying addresses. Find where the sign-in traffic comes from (the ingress logs, and user-management's rate-limited 'credential attempt store is full' warning) and block it upstream. The store drains on its own as entries expire, ten minutes after the flood stops. If the sign-in traffic is legitimate, raise instance.config.infrastructure.nats.kvStateMaxBytes, which sizes every state bucket.

DeviceCredentialAttemptStoreFull​

严重程度: warning · 规则组: devicechain.sign-in

MQTT password connects are being checked without their per-username backoff

含义与处理方式. The device credential-attempt store is full, so device-management is checking MQTT password connects WITHOUT slowing down repeated failures on any username that is not already waiting out a delay -- and one that is waiting is protected only until that delay ends, at most 30 seconds. Connects still work -- refusing them would let whoever filled the store keep every password device off the platform -- but guessing a device's password is now limited only by how fast the broker accepts connects. Every connect, successful or not, holds an entry for ten minutes, so the store fills either when connects are sent for a very large number of distinct usernames (someone spraying usernames: find where the MQTT connects come from and block it upstream) or when a very large fleet reconnects all at once, for example after a broker restart. It drains on its own as entries expire, ten minutes after the wave stops. If the connect volume is legitimate, raise instance.config.infrastructure.nats.kvStateMaxBytes, which sizes every state bucket.

MfaIdentityLocked​

严重程度: critical · 规则组: devicechain.sign-in

An account's password is in use by someone failing its second factor

含义与处理方式. A second-factor lock armed on an account: twenty wrong authenticator codes were evaluated, each after a CORRECT password. Someone has this account's password and is guessing its code. The authenticator stays locked for at most fifteen minutes at a time (the person can still sign in with a recovery code), and a password holder can renew the lock. Find the account in the audit journal (operation mfa_locked, preceded by mfa_failed rows and mfa_challenged rows) and RESET ITS PASSWORD with setPassword: that rotates the account's session epoch, which also voids every outstanding sign-in challenge. Do not reset the second factor first -- while the password is known to someone else that would give them the account (the reset refuses without a new password for this reason). The source addresses are in the ingress logs.

MfaVerificationFailuresSustained​

严重程度: warning · 规则组: devicechain.sign-in

Second-factor codes are being refused at a sustained rate

含义与处理方式. user-management has refused more than one second-factor code every ten seconds, on average, for fifteen minutes (wrong, replayed, or refused by a lock). Each attempt follows a correct password, so this is people typing their codes wrong in large numbers (a clock-skew problem on the nodes, or an authenticator app out of sync) or someone using passwords they obtained. Check that the nodes' clocks are synchronised (authenticator codes are accepted one 30-second step either side), then look at the audit journal for mfa_failed rows grouped by account and at the ingress logs for the source addresses. An account that has locked raises MfaIdentityLocked on its own.

SsoIdpUnreachable​

严重程度: warning · 规则组: devicechain.sign-in

An identity provider cannot be reached for single sign-on

含义与处理方式. user-management has been failing to reach the identity provider of connection ‹connection› (its token endpoint or its signing keys) for ten minutes, so people who sign in through it are being refused. Password sign-in is unaffected. The connection's endpoints are the ones recorded and approved in its discovery snapshot, and every request goes through the egress guard: check that the provider is up, that its address resolves from the cluster, and that the egress allow list (instance.config.infrastructure.egress) permits an on-premises provider's range. If the provider moved an endpoint, run refreshIdpDiscovery and approve the new snapshot.

SsoStateStoreFull​

严重程度: critical · 规则组: devicechain.sign-in

Single sign-on is refusing every sign-in because its state store is full

含义与处理方式. The single sign-on state store is full, so user-management is refusing to begin any sign-in through any identity provider (it fails closed rather than drop the protection that makes a replayed callback useless). Password sign-in still works. The store fills when an enormous number of sign-ins are begun inside ten minutes, which almost always means someone is calling beginSso in a loop: the call is public and is limited per source address and globally, but a distributed sender can still fill it. Find the traffic in the ingress logs and block it upstream; the store drains on its own ten minutes after the flood stops. If the volume is legitimate, raise instance.config.infrastructure.nats.kvStateMaxBytes, which sizes every state bucket.

SsoUnverifiedIdentityRefused​

严重程度: warning · 规则组: devicechain.sign-in

Single sign-on refused a sign-in that an identity provider vouched for

含义与处理方式. Connection ‹connection› had a sign-in refused with outcome ‹outcome›. Nobody was signed in. Each of these outcomes is the platform's own check overriding the provider: refused_unverified_email (the provider did not vouch for the address), refused_unverified_domain (the address is not in a domain asserted for this connection), refused_required_claim (the token lacks a claim the connection requires, for example the Workspace domain or the Microsoft tenant id), or refused_system_role_identity (a tenant's provider asserted an identity that holds a system role). They are usually a misconfigured connection or someone probing one, and occasionally an attempt to sign in as another person. Look at the audit journal for the asserted address, and at the provider's own sign-in log.

devicechain.tenant-purge​

TenantPurgeCoordinatorFailing​

严重程度: warning · 规则组: devicechain.tenant-purge

The tenant-deletion coordinator keeps failing to carry a deletion through a pass

含义与处理方式. The coordinator has failed to take at least one tenant deletion through a pass (re-reading the tenant, writing the ledger, lifting a fence or completing it) and has failed again within every Ns for Ns; a single failure the next pass recovers from does not fire this. This is a fault in the coordinator's own work, not a store holding data: a store's refusal is reported by TenantPurgeStoreFailing and TenantPurgeStoreHolding. User-management's log names the tenant and the step.

TenantPurgeStalled​

严重程度: warning · 规则组: devicechain.tenant-purge

A tenant deletion has been open for more than twice its token hold

含义与处理方式. The oldest open tenant deletion is older than Ns, which is twice the configured token hold (Ns) and therefore well past the point where every mandatory wait has elapsed. The purge is not progressing: some store is not reporting clean, or is reporting clean and then dirty again. The tenant's data has NOT been erased and its token is still reserved. The coordinator's own pass metrics will look healthy throughout, because a tenant it cannot finish is still a tenant it visited without error -- this alert is the only signal that the deletion is stuck. The admin API's tenantDeletions query names the tenant and the store still outstanding.

TenantPurgeStoreFailing​

严重程度: warning · 规则组: devicechain.tenant-purge

A storage system cannot be asked to erase a deleted tenant's data

含义与处理方式. For an hour, the ‹store› store has returned an error to the erase request for at least one open tenant deletion, so the coordinator cannot tell whether that tenant's data there is gone. Stores are rdb (relational data), tsdb (telemetry), broker (streams), kv (key-value buckets), detect (rule state) and blob (object storage); other is a store this alert's vocabulary does not name. The error is in user-management's log ("Tenant purge store did not report clean") and in the admin API's tenantDeletions query. The deletion cannot complete until the store answers.

TenantPurgeStoreHolding​

严重程度: warning · 规则组: devicechain.tenant-purge

A storage system has reported a deleted tenant's data still present for longer than the token hold

含义与处理方式. The ‹store› store has answered the erase request for at least one open tenant deletion with data still present or still arriving, continuously for the length of the token hold (Ns). A store normally goes clean within a few passes; one that does not is either retaining data on purpose (a stream with a retention the purge cannot cut, a bucket that refuses deletes) or being refilled by something still writing for the deleted tenant. The admin API's tenantDeletions query names the tenant and the store's stated reason. TenantPurgeStalled fires later, at twice the hold, without naming the store.

TenantPurgeVisibilityLost​

严重程度: warning · 规则组: devicechain.tenant-purge

Nothing is reporting whether tenant deletions are progressing

含义与处理方式. The tenant-purge gauges have not been scraped for 15m, so TenantPurgeStalled above cannot fire and an open deletion would be invisible rather than green. This is the absent() counterweight: the in-flight gauge is unlabelled and registered at process start, so it reads 0 on a healthy instance rather than going missing -- its absence means user-management is not being scraped at all, not that no deletion is open.

devicechain.token-audience​

OAuthResourceWouldReject​

严重程度: info · 规则组: devicechain.token-audience

A client is asking for tokens bound to a resource this instance does not serve

含义与处理方式. For at least an hour, ‹job› has been receiving token or authorization requests that name a resource this instance does not serve. This compatibility alert can only fire on a v1.2/v1.3 instance running observe; v1.4 refuses that configuration and always enforces token audiences. Those older instances count these requests as would_reject. Enforce mode refuses them (invalid_target, or invalid_grant on a refresh). Find the client from the rate-limited 'OAuth resource' warning in the user-management log, which names its client_id and the grant, and have it leave the resource out or use one this instance serves. See the v1.3 upgrade notes: https://github.com/devicechain-io/devicechain/blob/v1.3.0/docs/docs/deployment/releases-and-upgrades.md#v130-token-audience

TokenAudienceWouldReject​

严重程度: info · 规则组: devicechain.token-audience

A client is sending tokens that enforce mode would refuse

含义与处理方式. For at least an hour, ‹job› has been receiving requests whose access token is bound to a resource other than the one this service serves (and is not a registered MCP resource). This compatibility alert can only fire on a v1.2/v1.3 instance running observe; v1.4 refuses that configuration and always enforces token audiences. Those older instances count these requests as would_reject. Enforce mode refuses them. Find the client from the rate-limited 'token audience' warning in the service log, which names its client_id and the audience class, and have it request a token for the resource it actually calls before upgrading to v1.4. See the v1.3 upgrade notes: https://github.com/devicechain-io/devicechain/blob/v1.3.0/docs/docs/deployment/releases-and-upgrades.md#v130-token-audience

devicechain.update-management​

UpdateAttemptsGoingUnknown​

严重程度: warning · 规则组: devicechain.update-management

A burst of devices went quiet in the middle of an update

含义与处理方式. Five or more update attempts in the last 30 minutes were moved to UNKNOWN because the device stopped reporting during installation or reboot. The update may or may not have been applied. A few are routine; a burst means devices are failing together, often on a faulty image. The threshold of five is an absolute count: edit the rule to suit the size of the fleet. Pause the rollout, check whether the affected devices report again (their inventory resolves the attempt) and, for any that never do, decide with abandonUnknownUpdate.

UpdateDownloadsNotConfigured​

严重程度: warning · 规则组: devicechain.update-management

Devices are asking for update downloads and none can be granted

含义与处理方式. Devices have been asking for a download for at least 15 minutes and update-management refused every request because updates.downloadBaseUrl is not configured, so no assigned update can be fetched. The key is derived from the ingress host when it is left unset; it is empty when it was set to an empty string or the instance has no ingress. Set functionalAreas.update-management.config.updates.downloadBaseUrl to the public URL devices reach the service at, and the pending assignments proceed at the next retry of each device. If downloads are deliberately off, cancel the assignments instead: a device keeps asking for as long as one is current.

UpdateReportsMostlyRefused​

严重程度: warning · 规则组: devicechain.update-management

More than a quarter of device update reports are being refused by the state machine

含义与处理方式. For 15 minutes, over a quarter of the update progress reports devices send have been refused for a reason other than being duplicate, stale or late: a digest that is not the target's, the wrong component, a sequence going backwards, a conflicting or malformed report. That is a device-side bug, usually one firmware build misreporting across a whole fleet. Read devicechain_updatemanagement_update_reports_total by verdict to see which refusal dominates, then the update-management log, which names the device and the attempt. Pause the rollout feeding that fleet while you look.

UpdateRolloutStepsFailing​

严重程度: warning · 规则组: devicechain.update-management

Group rollouts are failing to advance

含义与处理方式. The rollout driver has been failing steps for at least 15 minutes and leaving them for the next pass, so group rollouts are not dispatching to new devices even though they read as running. The causes are a database error during a step, an unreadable compatible-profiles document on the artifact, or device-management being unreachable or not configured for update-management. The stage label of the counter tells a failed step from a failed query for the running rollouts. The update-management log names the failing step.