Skip to main content
Version: Next

Troubleshooting and routine operations

Start with the symptom below. If you already have a firing alert, use the alert runbook for its meaning and response. The Grafana overview and stage boards help locate the first stage where work stops progressing.

Find your symptom​

SymptomFirst checkProcedure
Install or bootstrap stops before services startRun dcctl preflight; read the failed step and Kubernetes events. A claimed cluster may have another operator working on it.Bootstrap, cluster lock, and resuming a failed bootstrap.
Pods are Pending, restarting, or not ReadyDescribe the affected pod for scheduling, image, volume and probe failures. For a restart, inspect the previous container's logs before replacing it.Container restarts and startup refusals in the upgrade notes.
After recovery, user-management refuses startup and other services never become ReadyRead user-management's startup error. A wrong root key cannot open the stored signing key; restarting does not repair it.Root-key recovery and verify escrow.
Sign-in works, but API calls return 401Check the calling client's token audience and the receiving service's refusal logs; a token can be valid yet bound to another resource.Token audience checks.
A user cannot pass the second-factor stepCheck clock synchronization and the account's audit records. If locked after password-authenticated guesses, reset the password before resetting the second factor.Second-factor lockouts.
MQTT times out or fails the TLS handshakeCheck DNS, the host name on the certificate, TCP reachability and the Gateway listener's status.External MQTT troubleshooting.
A device connects but telemetry is missingConfirm the tenant and device, then inspect failed-event reasons and dropped measurements on the device's Connectivity tab.Why isn't my data showing up? and common telemetry causes.
Telemetry is delayed and backlog growsFind the first slow reader on the stage boards; distinguish pending from in-flight messages and check the downstream database and broker.Consumer backlog, ingest backpressure, and tuning.
Commands stay queued or heldInspect the command's state, device presence, maintenance window and command channel before retrying.Command delivery alerts and sending commands.
A rule does not raise an alarmCheck the published profile version, the incoming metric and the detection engine's rule errors and replay state.Detection engine and testing rules.
An alarm exists but no notification arrivesCheck policy matching and channel configuration, then follow the notification stage board and its firing alerts.Notification channels and Grafana boards.
Storage fills, or a volume expansion does not reach the PVCCheck free space and growth in both databases and broker volumes; compare the requested volume size with the PVC.Event-store sizing and expansion and choosing a size.
Backups stop shipping, or recovery is refusedCheck backup alerts, each store's recovery window and root-key escrow; preserve the exact refusal.Backup archiving alerts and disaster recovery.
A deleted tenant is still being erasedIdentify the storage system blocking progress; do not infer completion from its disappearance in the console.Stalled deletion and erasure completion signals.

Collect evidence before changing anything​

Record the time range and time zone, instance, affected tenant, installed version, last successful step and first failing step. In Grafana, select the same time range on each board. Save the alert name and labels, not just a screenshot of its colour.

The following read-only examples use instance acme in namespace dci-acme. Replace the namespace and pod name with yours. Run shell examples in Bash; on Windows, use Git Bash or WSL.

dcctl version
dcctl preflight
kubectl -n dci-acme get pods
kubectl -n dci-acme get events --sort-by=.metadata.creationTimestamp
kubectl -n dci-acme describe pod YOUR_POD
kubectl -n dci-acme logs YOUR_POD --all-containers=true --since=30m

For a container that restarted, also request its previous logs; replace the container name with the one from the pod description. This command reports an error if no previous container exists.

kubectl -n dci-acme logs YOUR_POD -c YOUR_CONTAINER --previous

Logs may already have rotated; missing log output does not prove that no failure occurred. Read structured logs alongside Kubernetes events. Preserve evidence before restarting pods, changing configuration or retrying a write.

Redact before sharing

Remove tokens, passwords, connection strings, root keys, escrow material and private hostnames from logs and command output. Never attach a Kubernetes Secret or a raw configuration dump to a public issue. Use Getting help for reporting channels, including private security reports.

Routine operations​

These are suggested review points, not automated schedules or new platform limits. Choose a cadence that fits your traffic, recovery objectives and maintenance windows.

WhenReview and actionProcedure
Each operating day and after an incidentReview firing alerts, pod restarts, pipeline backlog and free storage. Confirm alerts reach an operator; an empty panel may mean missing metrics.Grafana boards and alert routing.
Each backup cycleVerify that both core and event stores are archiving and that their recovery windows cover your objectives. Check object content and preserved streams separately.Two database backups, backup alerts, and other state.
After bootstrap or key changes, and before a recovery drillVerify the escrow fingerprint; keep the encrypted artifact and its passphrase separately and off the cluster.Verify escrow.
On your recovery-drill scheduleRehearse recovery to a fresh cluster with both stores and the original root key; verify sign-in, secret use and event history. Stream/object backup verification alone is not a restore drill.Recover an instance.
Before an upgrade or configuration changeRead the release's startup refusals and supported upgrade path, review the stored overlay, check backups and use a dry run before applying.Releases and upgrades, versioning policy, and overlay dry run.
After an upgrade or configuration changeCheck rollout readiness, the boards and a representative device's ingest and command outcome; confirm backup shipping continues.Observability and sending commands.
During capacity and certificate reviewsCompare storage growth and retention with available capacity; review expiring trust anchors and plan certificate rotation before expiry.Sizing, tuning, and device certificates.
During access reviews and tenant offboardingReview service accounts and roles; for deletion, verify completion and perform the operator-owned cleanup.Service accounts and tenant deletion.

The stream/object commands are Preview: dcctl backup inspect --verify checks listed bytes, and dcctl restore --plan only plans. There is no restore apply command or coordinated database checkpoint. Follow the freeze and verification requirements in Exporting streams and object content.