Troubleshooting and routine operations
Start with the symptom below. If you already have a firing alert, use the alert runbook for its meaning and response. The Grafana overview and stage boards help locate the first stage where work stops progressing.
Find your symptom
| Symptom | First check | Procedure |
|---|---|---|
| Install or bootstrap stops before services start | Run dcctl preflight; read the failed step and Kubernetes events. A claimed cluster may have another operator working on it. | Bootstrap, cluster lock, and resuming a failed bootstrap. |
| Pods are Pending, restarting, or not Ready | Describe the affected pod for scheduling, image, volume and probe failures. For a restart, inspect the previous container's logs before replacing it. | Container restarts and startup refusals in the upgrade notes. |
| After recovery, user-management refuses startup and other services never become Ready | Read user-management's startup error. A wrong root key cannot open the stored signing key; restarting does not repair it. | Root-key recovery and verify escrow. |
| Sign-in works, but API calls return 401 | Check the calling client's token audience and the receiving service's refusal logs; a token can be valid yet bound to another resource. | Token audience checks. |
| A user cannot pass the second-factor step | Check clock synchronization and the account's audit records. If locked after password-authenticated guesses, reset the password before resetting the second factor. | Second-factor lockouts. |
| MQTT times out or fails the TLS handshake | Check DNS, the host name on the certificate, TCP reachability and the Gateway listener's status. | External MQTT troubleshooting. |
| A device connects but telemetry is missing | Confirm the tenant and device, then inspect failed-event reasons and dropped measurements on the device's Connectivity tab. | Why isn't my data showing up? and common telemetry causes. |
| Telemetry is delayed and backlog grows | Find the first slow reader on the stage boards; distinguish pending from in-flight messages and check the downstream database and broker. | Consumer backlog, ingest backpressure, and tuning. |
| Commands stay queued or held | Inspect the command's state, device presence, maintenance window and command channel before retrying. | Command delivery alerts and sending commands. |
| A rule does not raise an alarm | Check the published profile version, the incoming metric and the detection engine's rule errors and replay state. | Detection engine and testing rules. |
| An alarm exists but no notification arrives | Check policy matching and channel configuration, then follow the notification stage board and its firing alerts. | Notification channels and Grafana boards. |
| Storage fills, or a volume expansion does not reach the PVC | Check free space and growth in both databases and broker volumes; compare the requested volume size with the PVC. | Event-store sizing and expansion and choosing a size. |
| Backups stop shipping, or recovery is refused | Check backup alerts, each store's recovery window and root-key escrow; preserve the exact refusal. | Backup archiving alerts and disaster recovery. |
| A deleted tenant is still being erased | Identify the storage system blocking progress; do not infer completion from its disappearance in the console. | Stalled deletion and erasure completion signals. |
Collect evidence before changing anything
Record the time range and time zone, instance, affected tenant, installed version, last successful step and first failing step. In Grafana, select the same time range on each board. Save the alert name and labels, not just a screenshot of its colour.
The following read-only examples use instance acme in namespace dci-acme.
Replace the namespace and pod name with yours. Run shell examples in Bash; on
Windows, use Git Bash or WSL.
dcctl version
dcctl preflight
kubectl -n dci-acme get pods
kubectl -n dci-acme get events --sort-by=.metadata.creationTimestamp
kubectl -n dci-acme describe pod YOUR_POD
kubectl -n dci-acme logs YOUR_POD --all-containers=true --since=30m
For a container that restarted, also request its previous logs; replace the container name with the one from the pod description. This command reports an error if no previous container exists.
kubectl -n dci-acme logs YOUR_POD -c YOUR_CONTAINER --previous
Logs may already have rotated; missing log output does not prove that no failure occurred. Read structured logs alongside Kubernetes events. Preserve evidence before restarting pods, changing configuration or retrying a write.
Remove tokens, passwords, connection strings, root keys, escrow material and private hostnames from logs and command output. Never attach a Kubernetes Secret or a raw configuration dump to a public issue. Use Getting help for reporting channels, including private security reports.
Routine operations
These are suggested review points, not automated schedules or new platform limits. Choose a cadence that fits your traffic, recovery objectives and maintenance windows.
| When | Review and action | Procedure |
|---|---|---|
| Each operating day and after an incident | Review firing alerts, pod restarts, pipeline backlog and free storage. Confirm alerts reach an operator; an empty panel may mean missing metrics. | Grafana boards and alert routing. |
| Each backup cycle | Verify that both core and event stores are archiving and that their recovery windows cover your objectives. Check object content and preserved streams separately. | Two database backups, backup alerts, and other state. |
| After bootstrap or key changes, and before a recovery drill | Verify the escrow fingerprint; keep the encrypted artifact and its passphrase separately and off the cluster. | Verify escrow. |
| On your recovery-drill schedule | Rehearse recovery to a fresh cluster with both stores and the original root key; verify sign-in, secret use and event history. Stream/object backup verification alone is not a restore drill. | Recover an instance. |
| Before an upgrade or configuration change | Read the release's startup refusals and supported upgrade path, review the stored overlay, check backups and use a dry run before applying. | Releases and upgrades, versioning policy, and overlay dry run. |
| After an upgrade or configuration change | Check rollout readiness, the boards and a representative device's ingest and command outcome; confirm backup shipping continues. | Observability and sending commands. |
| During capacity and certificate reviews | Compare storage growth and retention with available capacity; review expiring trust anchors and plan certificate rotation before expiry. | Sizing, tuning, and device certificates. |
| During access reviews and tenant offboarding | Review service accounts and roles; for deletion, verify completion and perform the operator-owned cleanup. | Service accounts and tenant deletion. |
The stream/object commands are Preview: dcctl backup inspect --verify checks
listed bytes, and dcctl restore --plan only plans. There is no restore apply command
or coordinated database checkpoint. Follow the freeze and verification requirements in
Exporting streams and object content.