Disaster Recovery

Restoring a DeviceChain instance takes two things: a backup of its databases, and the instance's secret-store root key. Almost every backup procedure captures the first and silently omits the second.

This page is about the second one.

Two backups, not one

An instance's data sits on two separate database servers, and they are worth backing up and restoring as two separate operations rather than one:

	Core data (PostgreSQL)	Event data (TimescaleDB)
What	Tenants, identities, devices, profiles, detection rules, dashboards, connectors, last-known state — and every stored secret	Measurements, event history, rollups
Size	Megabytes; it grows with your fleet's configuration	The volume tier; it grows with time
Losing it	The instance cannot be rebuilt	History is gone; the instance still runs
Needs the root key	Yes	No

This is not a policy imposed on one database — it is how the platform already stores things. event-management is the only service that talks to TimescaleDB, and it talks to nothing else; every other service lives entirely on the relational server. There are no cross-writes to keep consistent between them.

Two consequences worth planning around:

• Different schedules. Both stores are archived the same way — a base backup plus a continuous stream of write-ahead log — but they do not want the same base-backup cadence or the same retention. Core data is small and changes when someone changes something. Event data is bulk, append-mostly, and already under a retention policy (data lifecycle) — keeping base backups of chunks the lifecycle reconciler is about to drop is paying twice to store the same rows.
• Different recovery targets. Restoring core data alone gives you a working instance: devices reconnect, detection rules run, commands dispatch, secrets decrypt. Restoring event data backfills history into it. An instance missing its event data is degraded — empty history widgets — not down, so the two halves can carry genuinely different recovery-time targets.

The root key gates the core-data half only. Event data holds no ciphertext, so a TimescaleDB restore needs nothing from this page. Everything below is about core data.

Why the root key needs its own procedure

Every secret DeviceChain stores on your behalf — outbound-connector credentials, SMTP passwords, AI provider keys — is encrypted at rest under a per-secret data key, and each of those data keys is wrapped by one instance-wide root key (the KEK, see Architecture).

That root key lives in the instance's Kubernetes Secret, which means it lives in etcd, and no database backup contains etcd. A PostgreSQL backup archives PostgreSQL; a TimescaleDB backup covers TimescaleDB. Neither contains a single byte of the key.

The consequence is a failure that passes the drill most people actually run:

• Restore the databases in place — into the same cluster — and everything works, because etcd still holds the key. This is the rehearsal that gives false confidence.
• Restore to a fresh cluster — the actual disaster — and the encrypted rows rehydrate perfectly, the restore reports success, and every one of those secrets is permanently unreadable. The new cluster minted a different root key, and the old one is not derivable from anything you still have.

The failure does not surface at restore time. It surfaces later, as an unexplained decryption error, typically long after the backup that could have helped has rotated away.

There is no recovery from a lost root key

The key is 256 bits of randomness and the wrapped data keys are not brute-forceable. If the key is gone, the secrets are gone — a support ticket cannot recover them. This is the one piece of DeviceChain state with no second chance, which is why the escrow below is on by default.

The escrow artifact

dcctl bootstrap writes an encrypted escrow artifact — a small text file containing the root key, sealed under a passphrase you choose:

~/.devicechain/escrow/<instance>-rootkey.escrow

It is a self-describing text file. Opened years later by someone who has never seen one, it explains what it is, what it protects, what happens if it is lost, and the exact command to recover with — without needing this page.

Two properties are worth knowing:

• It is not stored with the instance. It deliberately does not live in ~/.devicechain/<instance>/, because dcctl destroy removes that directory. dcctl refuses an --escrow-file path inside it.
• It carries a key fingerprint in the clear. That is what makes "is this escrow still the right one?" a question you can answer without the passphrase — see verifying.

Choosing a passphrase

Bootstrap takes the passphrase from the first of these it finds:

Source	Use it for
--escrow-passphrase-file <path>	Automation with a secret manager; a trailing newline is stripped.
DCCTL_ESCROW_PASSPHRASE	CI and scripted installs. Set-but-empty is an error, not a fallback.
Interactive prompt	A person at a terminal. Asked twice, to catch a typo now rather than during a recovery.

If none is available and there is no terminal to ask at, bootstrap fails. That is deliberate: the alternative is quietly producing an instance whose secrets die with its cluster.

Store the file and the passphrase separately, and off the cluster

Both in the same place is one compromise away from being no protection at all, and both on the cluster is one disaster away from being no backup at all.

Opting out

For a genuinely disposable instance — a CI run, a demo, a local experiment — pass --no-escrow. --dev implies it.

dcctl bootstrap local scratch --dev            # no escrow, disposable by construction
dcctl bootstrap local scratch --yes --no-escrow

The bootstrap summary then says so in red. Do not use it for anything whose secrets you would miss.

Recovering an instance

Recovery is one command, and it is a command that builds a new instance. The database is recovered from its archive as the cluster is created, before any service connects to it, seeded with the root key from your escrow artifact.

That is why there is no "restore into the running instance" step here. There is no supported way to do that, deliberately: restoring underneath services that have already created their own schemas means dropping tables they hold open and racing their migrations. Recover by rebuilding.

1. Rebuild the instance, recovering the database and the key together.

dcctl bootstrap local my-instance \
  --restore-root-key ~/backups/my-instance-rootkey.escrow \
  --restore-rdb-from dc-rdb

--restore-rdb-from names the archive path inside your backup bucket — the serverName the old instance was writing under, dc-rdb unless you changed it. You will be asked for the artifact's passphrase (or supply it with --escrow-passphrase-file / DCCTL_ESCROW_PASSPHRASE).

Two things this command will not let you do:

• Recover data without the key. --restore-rdb-from on its own is refused. It would rehydrate every row and mint a fresh root key, so the restore would report success and every stored secret would be permanently unreadable — the one failure that is invisible at the moment it happens.
• Recover into a live instance. The flag only takes effect when the database cluster is created. Re-running it against an instance that already exists does nothing at all, rather than half-working.

The recovered instance immediately begins archiving under a new archive path of its own, so it cannot write over the archive it was just born from. The bootstrap summary prints the name it chose.

2. Roll back to a point in time instead, if the disaster was that the data was destroyed correctly — a bad migration, a mistaken bulk delete. Add --restore-rdb-at with an RFC 3339 timestamp strictly before the damage:

dcctl bootstrap local my-instance \
  --restore-root-key ~/backups/my-instance-rootkey.escrow \
  --restore-rdb-from dc-rdb \
  --restore-rdb-at 2026-03-14T09:15:00Z

3. Restore event data separately with --restore-tsdb-from (and optionally --restore-tsdb-at), whenever it suits your recovery-time target. The two stores keep independent timelines on purpose: rewinding telemetry to yesterday does not mean the control plane should be rewound with it. Step 4 does not depend on this.

4. Confirm the stored secrets decrypt — read back a secret-backed object (an outbound connector, a notification channel) through the console or the API. A restore that returns rows is not proof; a value that decrypts is.

5. If you restored event data, check the machinery and not the row count. A recovered event store can hold every row and still have quietly stopped being a time-series database — the tables are there, the queries return, and the thing that is missing is the background work. That store answers queries perfectly for as long as it takes the disk to fill.

Open a shell on the event store's primary — under the operator, psql needs no password there:

kubectl -n dc-system exec -it dc-tsdb-1 -c postgres -- psql -U postgres -d devicechain

Ask it two questions. First, are the event tables still hypertables?

SELECT hypertable_schema, hypertable_name FROM timescaledb_information.hypertables;

Your event tables live in the event-management schema and should all be listed. A table that came back as an ordinary table is the failure this catches, and it is invisible in a schema dump — a plain table and a hypertable look identical there. (Don't look for measurement_rollups. It is a continuous aggregate, and the hypertable backing it is internal, so its absence from this list is normal.)

Second — and this is the one that matters — is the job scheduler actually running on this cluster? Run this, wait a minute or two, and run it again:

SELECT job_id, proc_name, total_runs, last_run_status
  FROM timescaledb_information.job_stats ORDER BY job_id;

total_runs must MOVE. That is the whole check.

Do not judge this by next_start or by a scheduled flag

It is the obvious field to reach for and it cannot tell you anything here. The table behind next_start is an ordinary table, so a physical restore brings it back holding the old cluster's values. A recovered store whose scheduler never started shows every job as scheduled with a perfectly plausible future next_start — and stays that way forever. It looks healthy because the data restored, not because anything will ever run. total_runs is a counter, so watching it advance observes work happening on the cluster in front of you.

The bootstrap finishing is not the database being ready

dcctl reports success once the workloads are up, which can be before the recovering database has finished replaying its archive. If a recovery cannot reach its archive it sits waiting rather than failing, so check the database itself before you believe a restore:

kubectl -n dc-system get clusters.postgresql.cnpg.io

You are looking for Cluster in healthy state. A cluster stuck in Setting up primary has not recovered — most often the archive is unreachable, or --restore-rdb-from names a path that does not exist in the bucket.

Restoring under a different instance name

Perfectly supported — the artifact records the name it was written for and dcctl notes the mismatch rather than refusing. The recorded name is authenticated, so it cannot be edited without invalidating the file.

Verifying your escrow, before you need it

An escrow that no longer matches the running key is indistinguishable from a good one until the day it is used. Check it on an ordinary Tuesday:

dcctl secrets escrow verify ~/backups/my-instance-rootkey.escrow --instance my-instance

This compares the artifact's fingerprint against the key the instance is actually running, needs no passphrase, and exits non-zero on a mismatch — so it composes into a cron job or a CI gate. A mismatch means the instance has no usable escrow, most often because it was re-bootstrapped after the file was written.

To see what an artifact is without opening it:

dcctl secrets escrow show ~/backups/my-instance-rootkey.escrow

What verify does not prove

It proves the artifact names the right key. It does not prove the artifact still opens — that needs the passphrase. Rehearse a real recovery periodically; a fingerprint check is a smoke alarm, not a fire drill.

Re-running bootstrap on a live instance

dcctl bootstrap is idempotent, and re-running it against an existing instance reuses every credential that instance is already running rather than minting new ones. That covers the secret-store root key, the NATS service password, the callout issuer key and the cross-service auth secret. If it cannot determine whether the instance exists, it stops rather than guessing — minting would be the destructive answer.

Why it matters differs by credential. Rewriting the root key makes every stored secret permanently unreadable. Rewriting the broker credentials is recoverable but disruptive: the broker and the services are updated by different mechanisms on different schedules, so fresh credentials open a window in which one side rejects the other, and pods that start inside it fail to reach the broker at all.

One exception

The Grafana OAuth client secret (--grafana-sso) is still re-minted on every run, because its cleartext lives in Grafana's own configuration rather than in the instance config. Both halves are rewritten by the same run, so the effect is a brief window of failing Grafana logins during the rollout.

On a re-run it also reconciles the escrow:

• artifact matches the running key → confirmed, left untouched;
• artifact does not match → the run stops and names it as an orphan;
• no artifact → one is written, so an instance first created with --no-escrow can gain an escrow later.

After dcctl destroy

dcctl destroy removes the cluster and the instance's local state — but not the escrow artifact, which lives outside that directory by design, and which destroy names on its way out.

Keep it for as long as you keep any backup of that instance's databases. It is the only thing that can still read them. Delete it when those backups are gone, and not before.