Releases & Upgrades

DeviceChain ships as a set of prebuilt, versioned container images plus a Helm chart. You do not need to build anything to run it — pull a released version, install the chart, and upgrade in place with zero downtime.

Some versions cannot be upgraded into

Three points in the history require recreating the instance rather than upgrading it:

• v0.9.0 replaced every service's migration chain with a single frozen baseline, so a v0.8.x database meets it and fails on already exists. See The v0.9.0 baseline squash.
• v0.10.0 changed the primary key of the event tables to fix a defect that was silently discarding telemetry. See The v0.10.0 event key change.
• anything built by v0.16.0 or earlier, which recorded no declaration of what the instance is — the record an upgrade now reads to know what to deploy. See Instances built by v0.16.0 and earlier.

If you are on any of them, read the matching section below before you do anything else.

Crossing v0.12.0 needs a few changes first

v0.12.0 upgrades in place, but it changes the topic a device answers a command on, moves one permission, and changes several things whose shape stayed the same. The upgrade will report success either way. Read v0.12.0 — an upgrade that changes contracts before you start.

This applies to any upgrade that crosses v0.12.0, not only one that stops there — going from v0.11.0 straight to a later patch does not skip those changes.

Versioning model

Every release is a single semantic-version git tag (vX.Y.Z). That one version covers everything together — each service image, the operator, the Helm chart, and the dcctl CLI are all published at the same version. There is no per-service version skew to reason about: a deployment is one coherent number.

One command moves all of it together — the operator is not part of the chart, so something outside the chart has to be the thing that moves both. See Zero-downtime upgrades for the procedure.

• Stable releases are vX.Y.Z (e.g. v1.2.0). The :latest tag tracks the most recent stable release.
• Pre-releases are vX.Y.Z-rc.N (e.g. v1.2.0-rc.1). These never move :latest.

Pre-1.0 stability

DeviceChain is pre-1.0

Until v1.0.0, any release — including a patch release — may change APIs, schemas, or behavior without a compatibility shim. This is deliberate: while the data model is still settling, we prefer a clean cutover to carrying a shim we would have to support forever.

Every breaking change is called out at the top of that release's notes. Read them before upgrading. They are the authoritative list; the version number alone does not tell you whether a release is safe for your deployment.

Concretely, before v1.0.0 you should expect that a release may:

• tighten validation, so a request that previously succeeded is now rejected — usually because it was being silently accepted or silently discarded
• change or remove a GraphQL field, rather than deprecating it for a cycle
• alter database schema in ways that a downgrade will not undo
• replace the migration baseline outright, which removes the upgrade path entirely rather than merely making it one-way. When that happens the release notes say so at the top, and the only route forward is to recreate the instance. v0.9.0 and v0.10.0 are such releases
• stop being upgradeable onto from an older instance for a reason that is not the schema at all — the release after v0.16.0 reads a record of what an instance is that earlier releases never wrote, and refuses rather than guessing one

The "upgrade in place with zero downtime" property above describes the mechanics of a rolling upgrade. It is not a promise that your existing API calls keep the same meaning across a pre-1.0 version bump.

Once v1.0.0 ships, this section is replaced by a normal semantic-versioning compatibility promise: breaking changes only in a major version.

Because releases are frequent before GA, the minor version marks a milestone (a significant feature or subsystem landing) and the patch version carries the ongoing cadence of fixes and hardening. A patch release is not automatically a low-risk upgrade during this period — again, the release notes are what tell you.

Images

Images are published to the public GitHub Container Registry under ghcr.io/devicechain-io — for example ghcr.io/devicechain-io/device-management. They are multi-arch (linux/amd64 and linux/arm64) and built on a distroless nonroot base, so they run as an unprivileged user with no shell and a minimal attack surface.

Because the registry is public, no credentials are required to pull released images.

Installing a specific version

Pin the image tag to the release you want:

DC_ROOT_KEY below is the instance's secret-store root key — required by the default profile, generated once with openssl rand -base64 32, and passed unchanged on every install and upgrade. See Deploying with Helm for why.

Substitute a real released tag for <version> — the releases page lists them, and an unreleased value fails at the pull rather than at install time.

helm install dc deploy/helm/devicechain \
  --set instance.id=devicechain \
  --set instance.config.infrastructure.secrets.rootKey="$DC_ROOT_KEY" \
  --set image.tag=<version>

The Helm chart itself is also published as an OCI artifact, so you can install it without a checkout of the repository. The chart is versioned separately from the images and carries no leading v; helm show chart oci://ghcr.io/devicechain-io/charts/devicechain prints the latest, and --version refuses anything that was never published:

helm install dc oci://ghcr.io/devicechain-io/charts/devicechain \
  --version <chart-version> \
  --set instance.id=devicechain \
  --set instance.config.infrastructure.secrets.rootKey="$DC_ROOT_KEY" \
  --set image.tag=<version>

The chart is also listed on Artifact Hub, which shows every published version alongside its default values and rendered templates.

Upgrading a chart-only install

An instance installed with helm install rather than dcctl bootstrap is upgraded with helm upgrade, and it keeps a trap the dcctl path does not have.

dcctl upgrade does not apply to it. That command reads an instance's declaration and its configuration document back out of the cluster, and a chart-only install has neither; it also installs no operator, so there is no second half for anything to move.

The release name below is dc because that is the name the helm install above chose. An instance installed by dcctl bootstrap carries a release named after the instance — devicechain installs as dc-devicechain — so any helm command aimed at one of those needs that name instead.

helm get values dc -n default -o yaml > dc-values.yaml

helm upgrade dc deploy/helm/devicechain \
  -n default \
  -f dc-values.yaml \
  --set image.tag=<new-version>

rm dc-values.yaml   # this file holds your instance's secrets

Carry the values forward — --set image.tag=… on its own will not work

Helm's rule is the trap. An upgrade that passes no values at all reuses the ones already in the release. But the moment you pass any value — including the single --set that changes the version, which is the whole point of an upgrade — Helm starts from the chart's defaults instead, and everything you set at install time is gone. That includes the instance root key, without which the stored secrets of a running instance cannot be read.

Nothing is corrupted when it happens, because the chart refuses to render without the root key:

Error: UPGRADE FAILED: execution error at (devicechain/templates/instance-config.yaml:27:4): instance.config.infrastructure.secrets.rootKey is required: area "notification-management" owns an envelope-encrypted secret store and cannot form its KEK without it, so it would crash-loop. Set it to a base64 256-bit key (openssl rand -base64 32); dcctl bootstrap mints one automatically.

--reuse-values also works, but it silently keeps stale entries when the chart's own defaults move between versions, so prefer writing the values out and passing them with -f, where you can see them.

Zero-downtime upgrades

Upgrading an instance you bootstrapped is one command, and the chart and services are built to roll customers forward without dropping traffic. Four exceptions are documented below: the durable-ingest cutover, which is still an ordinary upgrade but has a visible side effect, and v0.9.0, v0.10.0 and any instance built by v0.16.0 or earlier, which cannot be upgraded into at all. Check the release notes for the version you are moving to before running it:

dcctl upgrade local devicechain --version <new-version>

A release is one version across the service images, the chart, the operator and dcctl, and that command moves all of them together, in the order they have to move in:

1. the operator — its namespace, CRDs, RBAC and controller, applied from manifests embedded in dcctl. It is not part of the Helm chart, so nothing inside the chart can reach it. The whole rendered stream is applied rather than just the controller's image, because the CRDs are in it: a schema left at the version the instance was bootstrapped at silently discards any field a later release added;
2. the configuration document every service reads its credentials and endpoints from, recomposed from this release's chart and written by dcctl, which owns it;
3. the Helm release that runs the services, which rolls them onto the new images and waits for each area to finish.

Run it with --dry-run first if you want to see what it would move. It takes the target cluster from the instance's own record rather than guessing, and says which.

It reads every credential and mints none

dcctl upgrade keeps what the instance is running on: the database owner passwords, the broker's authority and logins, the cross-service secret, the secret-store root key, and the single sign-on client secret. A version change cannot become a credential change.

This is verified rather than asserted. An upgrade of a running instance was checked by comparing a digest of every one of those credentials before and after, and the only thing that had changed was the image tag — on every service, the console and the operator.

An upgrade is not a way to rotate credentials

Because it keeps them by design, it rotates nothing. If you need to change a credential, an upgrade will not do it — and for several of them there is no supported procedure today.

It moves a version, not an instance's shape

The profile, the topology and the enabled functional areas come from the instance's own declaration — what dcctl bootstrap recorded in the cluster — not from flags typed here. Changing what an instance is is a different question with different answers: raising the replica count, for one, does not re-replicate messaging streams that were created at the old one.

Two things are deliberately outside this command as well. It does not run the infrastructure apply, because two of that apply's inputs cannot be recovered from the cluster — the endpoint and bucket names of an off-site backup destination, and the single sign-on client secret's cleartext. And it does not touch the databases beyond letting the services run their own migrations.

What else an upgrade checks

Two things ride along with it, because a version bump is the thing that reliably happens to a live instance and a calendar is not.

The broker's certificate. The messaging broker serves a certificate valid for a year, issued by an authority dcctl mints at bootstrap and keeps in the cluster. An upgrade re-issues that certificate when it is inside its last 30 days, or when it no longer covers every name the brokers dial each other by — which is what scaling an instance up does to a certificate that is otherwise still comfortably in date. The re-issue is under the same authority, so nothing has to re-trust anything, and the broker is restarted so that it actually serves the new certificate rather than holding the old one until something unrelated rolls it. Outside those conditions the check runs and does nothing.

An instance bootstrapped before dcctl kept that authority cannot have its certificate re-issued in place. The upgrade says so and continues rather than failing; recreating the instance is what mints a fresh authority and certificate.

The root-key escrow. Every upgrade checks that the escrow artifact for this instance still protects the key the instance is actually running on. That check needs no passphrase — the artifact records a fingerprint of the key it protects, so matching it against the running one opens nothing.

What it finds	What it does
The artifact protects the running key	Says so, and leaves it alone
The artifact protects a different key	Warns loudly. It most often belongs to an earlier instance of the same name, and restoring from it would recover a cluster that cannot read its own secrets
There is no artifact	Writes one, if you passed --escrow-passphrase-file (or set DCCTL_ESCROW_PASSPHRASE). Otherwise it warns that the only copy of the root key is inside the cluster

This is how an instance first created with --no-escrow gains an escrow later. None of these outcomes fails the upgrade: an escrow problem is about a future disaster and the upgrade in front of it is about the running instance, and an operator who cannot upgrade will work around the check rather than fix it.

This used to be two commands, one of them a helm upgrade

The procedure was: write the current release's values to a file with helm get values, pass them back with -f alongside the new image tag, delete the file because it held your secrets, and then run dcctl upgrade a second time for the operator.

That dance existed only because the Helm release was where the instance's generated credentials lived, and Helm starts from the chart's defaults the moment you pass it any value at all — so an upgrade that did not carry them forward by hand lost them. dcctl owns the configuration document now, the release no longer holds those credentials, and the step that told you to write your secrets to a file simply goes.

It also closes a gap the two-command form had: an upgrade that stopped after the helm half left the new services running against the controller the instance was first bootstrapped with — indefinitely, and with no error to say so.

What makes the rollout safe:

• Surge-before-terminate. Each Deployment uses a RollingUpdate strategy with maxUnavailable: 0 and maxSurge: 1, so a new pod must pass its /readyz readiness probe before an old pod is removed. Capacity never dips during the rollout.
• Graceful shutdown / connection draining. When a pod is asked to terminate it first reports "not ready" (so the Service stops routing new requests to it), waits a short drain window for that change to propagate, and only then finishes in-flight work and shuts down. Configure the window with shutdownDrainSeconds (default 5), kept safely under terminationGracePeriodSeconds (default 30). The two are one budget and the services check it: the drain may take at most half the grace period, because the window only waits — finishing in-flight requests, draining the broker consumers and closing the database pool all happen after it, and the kubelet sends SIGKILL when the grace period expires whether or not that has finished. A larger window is refused when the service starts (and by dcctl bootstrap before it installs anything), not discovered when a pod is already shutting down. Set shutdownDrainSeconds: 0 to skip the drain entirely, which suits a single-instance run with no Service to be pulled out of.
• Coordinated schema migrations. Services run database migrations under a database-level lock, so when several replicas start at once exactly one applies migrations and the rest wait — no races, no duplicate DDL.

Run at least two replicas in production

For true zero-downtime, run replicas: 2 (or more) for each area so the rollout always has a live pod serving traffic. A single replica still has a brief gap while its one pod is replaced. Set it globally with --set replicas=2, or per area under functionalAreas.<area>.replicas. A PodDisruptionBudget is rendered automatically for any area with more than one replica, so node drains can't evict every replica at once.

The v0.9.0 baseline squash

v0.9.0 is the first of the two releases that cannot be reached by upgrading in place (the other is v0.10.0).

Before it, each service's schema was built by a chain of migrations applied in order. v0.9.0 replaces every one of those chains with a single frozen baseline — one migration per service that creates the whole schema as it stands. A database created by v0.8.x has already applied the old chain, so when it meets the baseline it tries to create tables that are already there and fails with already exists. The failure is loud and happens at startup; it does not corrupt anything.

There is no migration path, and before v1.0.0 there will not be one. Carrying a compatibility shim for a schema shape that is still moving is exactly the cost this project has chosen not to take on while every install is still an early one.

To move to v0.9.0, recreate the instance:

# Export anything you need first — this discards the databases.
dcctl destroy local devicechain
dcctl bootstrap local devicechain

Export first — recreation discards your data

The destroy guard protects the databases from an ordinary helm operation, not from a deliberate dcctl destroy. If the instance holds telemetry, device definitions or dashboards you care about, dump them before you start. There is no in-place path that preserves them across this release.

A schema change normally appends a new migration to the baseline, which is an ordinary in-place upgrade. That is the rule, and it holds for almost every release.

This section once promised it would never happen again

It said the squash described "a single release, not a new policy". Then v0.10.0 needed a recreate too, for an unrelated reason. The honest version of the rule is: appending is the norm, and before v1.0.0 a release may still require a recreate when a defect cannot be fixed any other way. Any release that does will say so in its notes and here. Check both before upgrading rather than assuming from the version number.

The v0.10.0 event key change

v0.10.0 is the second release that cannot be reached by upgrading in place, for a different reason from the squash.

An event was identified by the combination of its tenant, device, type and timestamp. That combination is not unique: a device that samples two sensors and publishes each as its own message under one shared timestamp produces two genuinely different events that look identical to the database. The second one was silently discarded — its readings were stored against the first event's record, and once dropped it could never be recognised as a repeat, so every later retry of that message added another copy of its readings.

Any device stamping whole seconds could hit this by emitting twice in one second, and the published .NET SDK stamped whole seconds until this release.

v0.10.0 gives every event, every reading and every relationship record an identity derived from its own content, and makes that identity the key. Storing telemetry correctly means changing the primary key of the largest tables in the system, and those tables are compressed — a database engine will not alter a key on compressed data in place. There is no upgrade path that preserves the existing rows.

To move to v0.10.0, recreate the instance:

# Export anything you need first — this discards the databases.
dcctl destroy local devicechain
dcctl bootstrap local devicechain

The same caution applies as above: recreation discards your telemetry, device definitions and dashboards. Export anything you need before you start.

Two changes to how the API reports time come with it, and neither needs any action:

• Timestamps are now returned at the precision they were recorded — previously they were rounded down to the whole second on the way out, so two readings 200 milliseconds apart came back looking simultaneous. Whole-second timestamps are unchanged on the wire.
• Requests that use a record's updatedAt value to avoid overwriting someone else's edit are now checked at that same precision. Two edits inside one second could previously both pass the check, and the later one silently overwrote a change it had never seen.

v0.11.0 — a normal upgrade again

v0.11.0 is the first release since v0.8.5 that can be reached in place. Its schema change adds three migrations rather than replacing a baseline, so an existing v0.10.0 database is carried forward with its rows intact instead of having to be recreated.

What lands in the database:

• two new tables recording the progress and history of a tenant deletion, and
• two columns on the tenant table tracking its lifecycle state.

Every tenant that already exists is set to the normal, active state as the column is added, so nothing changes for a running instance until you actually delete a tenant.

What was tested, and what was not

Two checks were run before release, and they are worth keeping apart because they measured different things.

The migrations, against a database with data in it. A v0.10.0 schema was built, filled with representative rows, and carried forward. Every one of those rows came through byte-identical, and the resulting schema is identical to a fresh v0.11.0 install — for every functional area, not only the one that changed.

The upgrade itself, on a running instance. A v0.10.0 instance was built from the published v0.10.0 images, given real tenants and identities, then upgraded with the command above. Every service rolled out, row counts across all 67 tables were unchanged apart from the new migration entries and the audit records they wrote, signing in still worked for an account created under v0.10.0, and the new tenant-deletion API answered on the upgraded instance.

Four limits, stated plainly:

• Only the databases were verified. Broker (JetStream) state, object storage and key-value state are not covered by either check.
• PostgreSQL 16 only. Fresh installs are verified on both supported majors; the upgrade path itself was measured on 16.
• The row-by-row comparison came from the first check, not the second. The running instance was verified by row counts, which would not notice a row altered in place rather than removed.
• The web console was left at its v0.10.0 image during the second check, so the v0.11.0 console was not exercised against an upgraded instance.

v0.12.0 — an upgrade that changes contracts

v0.12.0 is reachable in place. Its schema change adds migrations rather than replacing a baseline, so an existing v0.11.0 database is carried forward with its rows intact, and this was measured on a running instance rather than reasoned about.

What it does change is contracts — the MQTT topic a device answers a command on, a handful of GraphQL operations, and the meaning of several things whose shape did not change at all. None of that is visible in an upgrade that reports success, so read this section before you run it.

Do these before you upgrade

1. Update any device that answers commands. The topic a device publishes a command response to is now scoped to that device:

# before
{instanceId}/{tenant}/command-responses
# now
{instanceId}/{tenant}/command-responses/{deviceToken}

The old topic is no longer permitted by the credentials a device is issued, so a device that is not updated will have its responses refused at the broker — it will still receive and act on commands, but the platform will never record that it did, and every one of them will eventually read as timed out.

The reason for the change is that the old topic let any device in a tenant publish a response naming any command, including one issued to a different device. Nothing in the response said who sent it, so nothing could tell. The device token is now part of the topic, which is part of what the broker signs, so a device can only answer for itself.

Upgrade the devices first if you can. Responses sent on the old topic during the changeover are refused, not queued, and a small number of responses already in flight at the moment of the upgrade are dropped rather than delivered.

2. Rename an event source whose id is exactly lwm2m. That value is the one the LwM2M service files its own device presence under, and presence records are matched by exact equality — so your source and that service overwrite each other's rows. event-sources now refuses to start on it, which stops all ingest for the instance.

An id that merely reads as a transport, such as sparkplug:plant-a or lwm2m:site-a, now starts with a warning instead of refusing. Rename those when convenient. In both cases note the trap in renaming: the presence already recorded under the old id is not carried over, and nothing backfills it.

3. Check who reads location history. The queries that return device positions now require the location:read permission rather than event:read. This permission is not in the read-only baseline a viewer receives, so an account that could read position history on v0.11.0 cannot on v0.12.0. Grant it explicitly to the roles that need it.

The same permission now also gates previewing a rule that tests geofence containment. A preview of that kind returns, per device, when it entered and left a region — a read of position however it was asked for — so previewRule requires location:read in addition to the device:read every preview takes. A rule author who could preview every draft on v0.11.0 is refused on containment drafts until that permission is granted. Previews that test no containment are unaffected.

And check who reads it through an AI assistant. The MCP server gained a query_locations tool that returns a device's reported positions, and reaching it takes two grants that are deliberately kept apart. The agent's authorization must include a new location OAuth scope, and the person who authorized it must hold a role granting location:read. Neither alone is enough: the scope is a ceiling on what a token may carry, not a grant of anything.

An agent authorized for read-only alone cannot read position, however much its user holds. That is the point of a separate scope rather than a wider read-only: the consent screen shows a person the raw scope string, so folding position into read-only would have meant an authorization that looks identical before and after while now including where devices — and, often enough, where the people carrying them — have been. Keeping it separate means granting an agent observability is not the same act as granting it location history, and a user can allow one while withholding the other.

An MCP client you have already registered will keep working and will keep being refused position until its authorization request asks for read-only location and the user re-authorizes it. The viewer baseline is unchanged: location:read is still not something a member receives by default. See AI Access (MCP).

4. Check for these GraphQL operations in anything you have written against the API:

Operation	What changed
createCommand	Returns CreateCommandResult! instead of Command!. The command is now under a command field, alongside a rejection field that explains a refusal.
updateDeviceType	Its request argument is now a required DeviceTypeUpdateRequest!, and the semantic changed with it: this is a partial update. An omitted field now KEEPS its stored value instead of erasing it, and an explicit null clears it. So a client that cleared a field by leaving it out must now send null for it — and, in the other direction, renaming a type no longer detaches the profile its devices resolve capabilities through. A client written against the old whole-record behaviour — one that reads the type, then sends every field back — still works and still writes what it sends. token is also gone from the input, so an update can no longer move a type's token. Unrecognised fields in the request are rejected rather than ignored.
assertedActiveDeviceStates	Replaced by assertedDeviceStates, which takes activeOnly and pages through afterId and pageSize.
deviceCredentials, deviceCredentialsById, deviceCredentialsByToken	Now require device:write. For one credential type the readable identifier is the bearer token, so device:read — which every enabled member holds — was enough to open a broker session as any device in the tenant.
locationEvents	Now requires location:read, as above.
geoFenceSetSnapshot, currentGeoFenceSet	Their fences field is now paginated: it takes a required pagination argument and returns results alongside a pagination record, instead of a plain list. Read pages until pageEnd reaches totalRecords. A fence set at the documented limits is larger than a single response can carry, so the list form could not be returned at all for the tenants most likely to ask for it.
Any ...ById(ids: []) query	An empty id list now returns nothing. It used to return the whole table, unpaginated.

5. Stop zero-padding entity ids. An id argument is now parsed as a decimal number and nothing else. It used to be parsed with the base inferred from the literal, so a zero-padded "017" — exactly what a client that formats ids to a fixed width sends — was read as octal and resolved to row 15: the wrong entity, returned successfully with no error to notice. "0x2", "0b101" and "1_0" were accepted the same way. All four forms are now refused outright. Send "17".

6. Expect every service pod to roll, once. The instance configuration document the services are handed now has the coordinate for any functional area this deployment did not enable removed from it. On a deployment without ai-inference — which is every profile except full — that changes the document's bytes and therefore the checksum annotation that rolls pods, so the upgrade restarts every service rather than only the ones whose image moved. It is a normal rolling update and needs nothing from you; it is here so that a full roll does not read as a symptom.

The reason for removing the coordinate is that a hostname for a service nobody deployed was worse than no hostname: the rule authoring surface built its natural-language "Describe" door against it, failed to resolve the name, and reported that the tenant had not consented to external AI routing — blaming a tenant setting for a service the operator never installed. It now says the feature is not enabled on this deployment, which is true.

Changes with no signature change

These are the ones a client cannot detect by looking at the schema.

Updating a device profile clears its location declaration. A profile can now declare that its devices report position, and updateDeviceProfile replaces the whole profile. A client written against v0.11.0 does not send the new field, so updating a profile for any reason — renaming it, editing its description — silently un-declares position for every device on it. The only symptom is that map surfaces go quiet. Send the field, or set the declaration again after any update from an older client.

This no longer applies

updateDeviceProfile has since become a partial update: a request that says nothing about the declaration now leaves it alone, and clearing one takes an explicit null. The advice above is what to do on a v0.12.x or v0.13.x instance; on a current one there is nothing to carry forward. Renaming a profile is a mutation of its own.

Windowed detection rules no longer count buffered readings from outside their window. Repeating, sliding-aggregate and correlation rules used to fold in a reading from any point in the past, which let a rule reading "three readings within ten seconds" fire on readings an hour apart — a store-and-forward device uploading its buffer was the usual trigger. Those rules now discard a reading that arrives after the window it belonged to has passed, as tumbling-window and session rules already did. Expect fewer alarms from those rule kinds on any fleet that uploads in batches, and check detect_late_samples_total to see how much is being discarded. Readings are stored and charted exactly as before; this affects detection only. See running the detection engine.

Geofence geometry is validated more strictly, and stored as written rather than as sent. Three changes, all at the point a fence is created or updated:

• A position must be exactly [longitude, latitude]. A third or later ordinate used to be accepted and ignored.
• The geometry document may carry only the keys the platform reads — kind and geometry at the top level, type and coordinates inside it. Any other key used to be stored and never looked at.
• Coordinates are rewritten into plain decimal notation before being stored. A coordinate sent as 1e-300 comes back as its full decimal expansion. No value is rounded and no fence changes shape, but a document read back is not byte-identical to the one sent.

A fence is also now refused if its stored form exceeds 32 KiB. That is roughly twice the size of a fence using every vertex the platform allows, so ordinary geometry is unaffected; what it refuses is a document whose size comes from notation rather than from shape. The console has always written positions in the accepted form, so fences drawn in the console are unaffected. Existing stored fences are not rewritten and keep working — but one that breaks a rule above will be refused the next time it is saved.

Cancelling a command records CANCELLED. It used to record EXPIRED, which it shared with a command that simply ran out its time. If you branch on EXPIRED to detect your own cancellation, it will no longer be there.

Commands can now sit in HELD or PARKED. A command addressed to a device the platform knows is absent is held rather than published, and one that was dispatched to a device that turned out to be unreachable is parked. Both are waiting, not finished, and both are new — code that treats anything other than QUEUED or SENT as terminal will get this wrong. The full set is now QUEUED, HELD, SENT, PARKED, SUCCESSFUL, FAILED, TIMEOUT, EXPIRED, CANCELLED.

A reading is stored at the instant it was taken. When a message carries many samples, each with its own timestamp — every Sparkplug and LwM2M upload does, and so does any device that buffers while offline — those samples used to be stored at the instant the message arrived. They are now stored at their own. A device uploading an hour of buffered readings writes them across that hour rather than at the moment of upload, so history, charts, retention and detection all see them where they actually belong.

A presence source that stops running now hands its devices back. A device marked ASSERTED used to keep whatever presence it last had, indefinitely: the inactivity sweep skips asserted devices and a data event cannot flip one, so a device that was connected when its source went away read connected forever, and one that was offline had its commands held forever. Broker-asserted MQTT presence now releases the devices it asserted when it is deliberately disabled, or when its NATS system-account credential is missing — returning them to INFERRED without asserting anything about connectivity. On an instance where that applies, expect one state-change event per device, paced, counted under presence_events_total{state="demoted"}, and expect those devices to come back under the ten-minute inactivity sweep. Sparkplug and LwM2M have no automatic release: dcctl presence demote and device-state's new demoteAssertedPresence mutation do it by hand, for any source. The mutation needs a new state:demote permission that no role holds by default. A new gauge, presence_tap_off{reason}, reports whether broker-asserted presence is running at all — something nothing reported before, because a quiet fleet and a tap that never started look identical from outside. See returning a device to inferred presence.

A redelivered reading no longer duplicates its rows. A measurement event's identity is derived from a digest of its own content, and that identity is what makes a redelivery harmless. For a reading carrying more than one metric over a JSON transport, the digest was computed over an order the platform invented rather than one the device sent, so the same reading resolved to a different identity roughly four times in five. When the platform redelivered such a message — which it does routinely, on an unacknowledged publish or a transient write failure — the duplicate was not recognised: the measurement rows were written a second time and the hourly rollups counted them twice. Single-metric readings, and readings arriving over Sparkplug or LwM2M, were never affected. The fix is forward-only: duplicates already written before the upgrade stay where they are, and their rollups stay inflated. If you have charts that looked too high on multi-metric devices, this is why, and they will read correctly from the upgrade onwards.

Every paged list now returns rows in a declared order. Of the platform's 37 list endpoints, 31 named no order at all, which leaves a paged read free to hand the same row out on two pages and never show another one — a real defect that had already been reported twice as a screen reshuffling under an operator. Each list now sorts on a total, unambiguous key. If you have code that depended on the incidental order a particular query happened to return, it will now see a stable one instead, which may not be the same one. One order was chosen deliberately rather than mechanically: device credentials are listed with the most runway left first, because an unbounded read of them feeds credential reuse, and ordering by id would have handed back the credential closest to expiry.

A command answered in plain text now records its answer. A device replying to a command with something that is not JSON — acknowledged, a bare status word — used to fail the write with a database type error and leave the command in SENT, retrying the same doomed write once a minute for the life of the row. The command then timed out against a device that had answered it correctly. Such a response is now stored, losslessly, as a JSON string. Values an API caller supplies are unchanged: those must still be valid JSON, because a caller sending malformed JSON is a caller who should be told so.

Commands to Sparkplug devices now fail immediately instead of being lost. The platform has no command path to a Sparkplug device — those nodes live on your own MQTT infrastructure and nothing bridges the two — and the check that was supposed to refuse such a command was comparing against a value no device ever carries, so it matched nothing and every one of those commands was accepted and then quietly went nowhere. They are now recorded FAILED straight away with that as the reason, and counted under command_delivery_undeliverable_total. Expect commands that used to sit until their TTL and record TIMEOUT to appear as prompt failures instead. See Commands.

A command the platform lost track of is re-armed rather than blamed on the device. A command could reach SENT and then be reached by nothing — the pod that published it dies before recording the outcome — and SENT had no exit except the TTL, which recorded TIMEOUT against a device that was never sent anything. A background pass now finds those and re-arms them to PARKED, so they are delivered on the device's next wake. command_delivery_stranded_recovered_total carries a {disposition} label saying where each one landed. This applies to LwM2M devices only — on plain MQTT a command that appears to have gone nowhere cannot be told apart from one that arrived and whose answer was lost, so the behaviour there is unchanged and command_delivery_stranded_skipped_total{reason="transport"} will show a steady rate that is not a fault. See when the platform loses track of a command.

A rule action the platform cannot ever deliver is dropped instead of retried. When a REACT action is refused for a reason no retry can change — a sendCommand aimed at a device that no longer exists, or at a command outside that device's published vocabulary — it used to be retried to the redelivery limit and then counted as poison, which put an authoring mistake on the same shelf as an infrastructure failure. It is now dropped on the first such refusal and counted under react_actions_permanently_rejected_total, labelled by action type. A standing rate on that counter means a rule is aimed at something its devices cannot accept; the poison counter it used to inflate now means what it says.

A truncated cross-service response is counted. Services read each other's responses up to a fixed 1 MiB cap, and a response over that was silently cut short. It is now counted by devicechain_svcclient_responses_truncated_total, labelled by peer. The reading should be flat at zero; a non-zero one means some service is acting on a partial answer, which is worth knowing about before the symptom reaches a screen.

Input that used to be accepted and now is not

• A notification policy carrying deviceTypeToken. Scoping a policy to a device type is not implemented; the write used to succeed and then deliver nothing at all.
• A notification rule whose severity is not one of the uppercase tiers or *. A lowercase severity used to write, read back unchanged, and never match an alarm.
• An occurredTime of 0001-01-01T00:00:00Z. It is a valid timestamp, and the platform reserves it to mean no time was reported.
• An enqueue that would push a tenant past its held-command ceiling. Commands withheld for an absent device accumulate with no natural brake — a sleeping fleet's backlog can sit for days — and nothing bounded that before. The limit resolves from the tenant's own override, else its tier's, else a platform default of 10,000, and there is no value at any level meaning unlimited. The refusal carries the code HELD_CEILING_EXCEEDED and is the only temporary one the enqueue gate produces: it frees as those devices return. A client that treats every rejection as permanent should special-case it. See how much backlog a tenant may hold.
• An enqueue that would push a tenant past the part of that ceiling reserved for delivery. A share of the limit — 20% by default — is kept for the platform's own command delivery, so a single fleet write cannot consume all of it and leave every automated sendCommand for that tenant refused until the backlog drains. Everything issuing commands on your behalf is bounded by the remainder: the console, the SDKs, dcctl and your own integrations alike. The practical consequence is that a large batch that would have been admitted whole may now be partly refused; where the batch was allowed to fan out partially, its record says which devices did not fit. See part of the ceiling is reserved for delivery.

Bootstrap and the CLI

These reach an instance through dcctl bootstrap and the infrastructure apply rather than through the release, so none of them lands during the upgrade above. They are here because each is a change in what goes wrong.

A broker configuration change now restarts the broker. nats-server cannot hot-reload its authorization-callout block or its JetStream limits, and its refusal is wholesale — it abandons the entire reload, including every unrelated change in the same apply. What that looked like from outside was the worst kind of nothing: the apply reported success, the ConfigMap showed the new values, and the running broker was still on the configuration it booted with, with the only evidence one line inside the broker's own log. Services then failed to authenticate against a ConfigMap that proved their credentials were right. The broker's StatefulSet now carries a hash of its rendered configuration in its pod template, so the server always comes up on the file it was given. The cost is that broker configuration changes now roll those pods, where previously only a chart or image bump did: budget roughly 50–70 seconds per pod, which on a single-server broker is a brief full outage and on three is a rolling restart.

The third-party chart versions are pinned. ingress-nginx and cert-manager were installed at whatever their repository last published, which meant the chart repository was a dependency of planning as well as of applying: when its release-asset host returned 503, the plan failed with an error naming neither the chart nor the network, and it cost two failed bootstraps before the cause was found. They are pinned to 4.15.1 and v1.21.1 respectively — the versions the drilled cluster runs. If you had been relying on picking up a newer one automatically, you now upgrade it deliberately.

A dcctl you built yourself now has a usable default image tag. make -C backend/cli build produced a binary whose default image tag came from the repository's VERSION file — a value no release ever sets and no image was ever pushed under. Every workload landed in ImagePullBackOff, several minutes into a bootstrap that had reported healthy progress the whole way. A locally built dcctl now defaults to dev, which the unpublished-version guard recognises and refuses early with a message you can read, rather than late with one you cannot. A released dcctl was never affected: its tag comes from the release itself.

Configuration

One key moved. maxEventFutureSkewSeconds bounded how far a device-reported timestamp may lead the platform's clock; it was an event-processing setting and is now a device-management one, because the event time is now decided in exactly one place for live detection and replay alike.

A configuration that still sets it under event-processing starts normally and logs a warning naming the new location. The old value is not applied — set it under device-management if you had changed it from the default of 300 seconds.

Nothing was removed from the chart's values, so a v0.11.0 values file applies unchanged.

Drain devices during this one upgrade, or accept a possible detection-engine reset

The bound moved, so for the length of this rollout neither side is holding it. On v0.11.0 only the detection engine bounded a device-reported time; on v0.12.0 only event resolution does. The two services roll as independent deployments, so there is a window where a v0.11.0 event-processing has already been replaced while a v0.11.0 device-management is still publishing — and an event crossing in that window is checked by neither.

What it costs if one arrives with a wildly future timestamp: detection tracks a single time frontier across the whole instance, so that one event advances it and every tenant's pending timers fire at once. Recovering means resetting the engine's snapshot.

This is a one-upgrade boundary, not a standing weakness — once both services are on v0.12.0 they agree permanently, and an instance you destroy and recreate is never exposed. If you are upgrading in place with devices sending, stop device traffic for the rollout, or be prepared to reset the detection snapshot afterwards.

A service that refuses its own configuration now exits non-zero. It used to log "refusing to start" and then terminate with status 0, so the pod reported Completed — exactly what an orderly shutdown reports, and indistinguishable from one at a glance. Those pods will now CrashLoopBackOff instead. Nothing has changed about which configurations are refused; what changed is that the refusal is now visible in kubectl get pods, in a restart count, and to anything that alerts on either. A service that fails to shut down cleanly is reported the same way, for the same reason. If you have an alert that treats a Completed service pod as benign, this is the release where the underlying failure starts reaching you.

v0.12.1 — a patch, nothing to do

v0.12.1 is an ordinary in-place upgrade from v0.12.0. It adds no migration, so the database is untouched, and it changes no API, topic, permission or configuration key — everything the v0.12.0 section above describes is still exactly what you are running.

Two fixes are worth knowing about:

• Status colours in the web console now meet WCAG AA contrast in both light and dark themes. The pending and online badges failed in both themes, and error text failed in dark mode. What the colours mean has not changed — but the filled badges are visibly darker, because that is the only way white lettering on them becomes readable.
• The inactivity monitor no longer reads every device in every tenant into memory on each pass, and no longer issues a database round trip per device it flips; it decides and writes in one statement. Devices go inactive on exactly the same schedule as before — this is a cost change, not a behaviour change — and it matters most on large fleets, and in the moments right after a presence source hands its devices back.

v0.13.0 — geofence limits become part of your plan

v0.13.0 is an ordinary in-place upgrade, and it changes no topic, permission or configuration key.

It does change the database, additively: it creates one table for geofence shapes, adds three nullable columns to the tenant record, and rewrites stored geofence history into the new form once, in place. Nothing is dropped and nothing has to be recreated.

That last step is the one worth knowing about if you already use geofencing. From v0.13.0 a fence's shape is stored once and referred to by its content rather than copied into every version of your fence set, and the upgrade rewrites the fence history you already have so it refers to shapes the same way. Your fences and their history come through unchanged; what changes is how they are stored. The step is safe to re-run and does nothing on an instance that has already had it.

What changes is that the two geofence limits that used to be fixed for everyone — 512 positions in one fence, 100 fences per tenant — are now settings on your plan, joined by a third: a limit on the total positions across your whole fence set. All three keep their previous values by default, so a tenant that has never had them changed is metered exactly where it was and needs to do nothing.

Two things are worth knowing before you upgrade:

• The whole-set limit is new, and its default is what the other two already implied: 51,200 positions, which is 100 fences of 512. So a tenant using geofencing exactly to the documented limits is at the new limit, never over it. The total counts distinct shapes, so two fences drawn identically cost one.
• A change is refused only when it makes a number larger. If an operator later lowers one of your limits below what you already hold, you keep every fence. Editing a fence's name or description, and deleting a fence, always keep working — the check is on growth, not on size. This is what keeps a plan change from stranding fences that were legal when drawn. Making a fence smaller almost always works too; the exception is that the whole-set total counts distinct shapes, so editing one of several identically-drawn fences separates it from the rest and can raise the total even though that fence shrank.

One consequence to plan around: because deleting a fence lowers the stored total, a tenant that is over a limit and deletes a fence cannot recreate it. To move a fence to a different token, create the new one first and delete the old one after — which needs one spare fence slot for the moment both exist.

Operators packaging tiers should know these have real ceilings, because they are not all spent on the tenant alone: the whole-set total is a share of a geometry cache every tenant on the instance draws from, and the fence count bounds an announcement that has to fit one broker message. The refusals name both the number and the setting to raise, and a geofence_cap_refusals_total metric counts them by which limit refused.

v0.14.0 — the packages you build against

v0.14.0 is an ordinary in-place upgrade from v0.13.x. It adds no migration, so the database is untouched, and it changes no API, topic, permission or configuration key. If you only run the platform, there is nothing to do.

What changed is around it — the artifacts you build against, and the CLI you run it with.

The web runtime is published. @devicechain/client, @devicechain/dashboards, @devicechain/widgets and @devicechain/brand are on npm, so embedding a dashboard or a widget in your own application is an install rather than a build against our source tree. The four are released together at one version and pinned to each other. See npm Packages for the install line and the dist-tag policy.

If you were building our widgets from the source tree, one change is yours to make. maplibre-gl is now a peer dependency of @devicechain/widgets: your application supplies the library, its worker URL and its stylesheet, instead of the widget package deciding those for you. That is what makes the package work under a bundler we do not control — but it means a map widget with no host wiring above it now renders an explicit notice rather than a blank canvas, which is the symptom to expect if you upgrade without doing it. The wiring is short and is written out at Rendering a map. Nothing changes on the server.

The .NET and Unity client SDK is published to nuget.org as DeviceChain.Sdk.

dcctl can tell you what it has bootstrapped, and shut all of it down.

# every instance, the cluster it lives in, and whether that cluster is still there
dcctl instances list

# tear down all of them
dcctl destroy --all

🔴 This closes a defect worth acting on, not just knowing about. Until now nothing recorded which cluster an instance had been bootstrapped into — it was derived from the instance name at create time and derived again at destroy time. That derivation is wrong for any instance bootstrapped with --kube-context, and the failure was silent in the worst direction: dcctl destroy asked the provider to delete a cluster that did not exist, which succeeds quietly, removed the local state, and reported the instance destroyed while its actual cluster kept running. If you have ever bootstrapped with --kube-context and destroyed that instance afterwards, its cluster is probably still up. dcctl instances list cannot show you these — the destroy removed the local record, which is precisely the problem — so ask the provider directly (for the local provider, kind get clusters) and delete what you recognise.

From this release the cluster is written down at bootstrap and read back at destroy, and the closing line says which of three things happened: the recorded cluster was deleted, the cluster was already gone and only local state was cleared, or the record could not be trusted and nothing was touched. None of them is the old sentence printed over a cluster still running.

Instances created before this release have no such record and list as no record — destroy will guess the cluster. Destroy still works on them, falling back to the old derivation, so the caveat above continues to apply to them and only to them.

v0.15.0 — updates stop erasing what you did not send

v0.15.0 is an ordinary in-place upgrade from v0.14.x. The new migrations run themselves as the services start, there is nothing to recreate, and no data needs moving by hand.

The breaking changes are in the API and in outbound network access, not in the upgrade itself. If you run the platform and drive it through the console, there is nothing here for you to do. The sections below are for people who call the API directly, who send notifications through something inside their own network, who run the MCP server, or who have customised event-sources configuration.

Update operations no longer replace the whole record

This is the change that affects the most people, and it is the reason this release is marked breaking.

Before, an update replaced the record: any field you left out was erased. Now a field you do not mention is left exactly as it was, and clearing a value takes an explicit null.

The request itself is a new shape that no longer carries the record's own name — updating and renaming are separate operations, and there are now dedicated rename… mutations for the four types that need one. So an application calling the API directly must drop the name from its update requests and regenerate its client code.

A request in the old shape is refused outright with an error naming the field it no longer accepts. It is not half-applied, and it does not fail quietly — which means you find out at the first call rather than from a record that has lost half its contents.

The one case that changes quietly

An application that cleared a value by leaving the field out now keeps the old value instead. Nothing errors; the update simply does less than it used to. If your code relies on omission to clear a field, send an explicit null instead.

Note that not every field accepts null — some are required and refuse it with a named error. Those are fields that could never legitimately be cleared.

One sharp edge worth knowing about

If you build an update request by binding a separate variable per field, a variable you do not supply arrives as an explicit null rather than as an absent field — and explicit null means clear this. On a notification policy's rules that empties the entire rule set and returns success. Bind the whole request object as one variable, or only include the fields you actually intend to change.

The id on a stored event has changed

An event's id is now the event's own identifier, rather than a value assembled from the device token, the event type and the timestamp. Any id you saved from an earlier release will no longer match anything.

The previous form was also not unique: a device reporting two measurements at the same instant produced the same id for both, so any client keeping a normalized cache keyed on it was silently merging those readings into one. If you stored ids, re-read them; if you keyed on them, this is a correctness fix as much as a break.

Outbound connections to private addresses are now refused

Notification webhooks, SMTP relays and connector HTTP calls can no longer reach loopback, private, carrier-grade NAT, link-local or cloud metadata addresses. The check happens at connect time, and a refusal is final — it is not retried.

This is on by default and there is no switch to turn it off.

If your mail relay lives inside the cluster, alarm mail will stop

This is the failure most likely to catch you, because nothing about it looks like a network policy change: notifications simply stop arriving, and the failure is recorded as permanent rather than pending. Allow the specific addresses you use:

instance:
  config:
    infrastructure:
      egress:
        allowedDestinations:
          - 10.96.0.25/32      # the in-cluster SMTP relay

List each destination as its own /32. Destinations on the public internet are unaffected and need no entry.

If you run the MCP server

Two changes need action, and one of them stops the service from starting:

• A resource URL with a trailing slash is now refused at startup. An identifier is compared exactly, so a trailing slash meant tokens were bound to an address that never quite matched. It used to be accepted and then quietly fail to line up; now it fails loudly at boot. Remove the slash.
• The protected-resource metadata has moved to the location the specification defines, with the well-known segment between the host and the path. The chart routes it for you. If you terminate ingress yourself, add a route for the /.well-known/ prefix that does not rewrite the path.

Two configuration keys were removed, and they behave differently

• debug, inside an eventSources entry. Configuration is validated strictly, so leaving this in place stops event-sources from starting, with an error naming the field. Remove it.
• inboundEventBatching and its maxBatchSize / batchTimeoutMs. This one is retired rather than rejected: it is stripped at load with a warning, so the service starts normally. Remove it at your convenience.

The difference is not arbitrary — a key that is retired is one we can still recognise by name, so it can be dropped for you. A key nested inside a list entry cannot be, which is why the first one has to stop the service instead.

Also in this release

Commands are now dispatched the moment they are enqueued rather than waiting for the next sweep, and the sweep interval is configurable if you want to change how often the safety net runs. Dead letters can be read and queried instead of only counted. There is a reporting view you can point a BI tool at. Assets gained parent/child hierarchy and a documented property contract, devices gained a replacement operation, alarms gained bulk acknowledgement, and a tenant can choose the language its console opens in.

The published npm packages and the .NET/Unity SDK carry no source changes in this release. If your own code sends update mutations through them, though, that code is yours to regenerate.

v0.16.0 — devices must name the dispatch they are answering

v0.16.0 is an ordinary in-place upgrade from v0.15.x. One new migration runs itself as command-delivery starts — it adds a column with a default, backfills existing rows in the same statement, and needs nothing from you.

There is one pre-flight worth doing before you upgrade, and one breaking change that affects devices rather than API callers. Beyond those, this release is mostly about services refusing to start on configuration that used to be accepted and quietly ignored — which is safer, and which can stop a pod that has been running happily for months.

Before you upgrade: audit your listener ports for a collision

event-sources runs more than one HTTP listener in a single process — GraphQL on its own port, plus every HTTP event source you have configured. Until now, two of them landing on the same port killed one ingest transport silently: the losing listener died inside a goroutine and was never mentioned again.

Binds are now synchronous and a failure is fatal, so a collision that has been quietly broken for months crash-loops the deployment instead. That is the right behaviour and it is also the one change here most likely to surprise you, because nothing warns you about it today.

Check each source's port against the others and against the GraphQL port, and check that the chart's extraPorts entry agrees with each source's own port. A device-facing port: "0" is also refused now.

Any device that answers a command must echo the dispatch nonce

This is the release's one breaking wire change, and the population it affects is devices built outside this repository — firmware, gateways, anything speaking the command protocol directly.

A delivery envelope carries a dispatchNonce. A device answering that command must now send the same value back in its response envelope. An answer that omits it, or that names a dispatch the command has already moved off, is refused and recorded as a dead letter rather than settling the command.

The reason is a real defect, not tidiness: the same command can legitimately be published more than once — released back to the queue and dispatched again — and without a nonce there is no way to tell which dispatch an answer belongs to. An answer to a superseded dispatch was settling the newer one with the older one's outcome.

How to tell whether this is safe for your fleet

Nothing in the platform can enumerate devices built elsewhere, so the platform counts them for you instead. After upgrading, watch:

• devicechain_commanddelivery_command_delivery_responses_without_nonce_total — answers that named no dispatch at all. Mostly devices that have not been updated, and on a fleet where every device speaks the current contract this should be zero. It counts any answer with no nonce, though, so a duplicate or late answer to an already-settled command lands here too.
• devicechain_commanddelivery_command_delivery_responses_stale_nonce_total — answers naming a dispatch the command has moved off. The usual reading is that commands are being published more than once, which is the defect this change exists to fix; a device replaying an old outbox entry produces it too.

(The doubled command_delivery is not a typo — the series carries the platform namespace and the functional area as prefixes, so the name above is what you paste into a query.)

A refused answer is not discarded. It is written as a dead letter, because the device's report of what it did exists nowhere else — so you can find those answers while you work through the first counter.

If your devices use the .NET/Unity SDK, upgrading the SDK is the whole fix — it carries the nonce for you in both directions. The LwM2M downlink adapter and the device simulator were updated in the same change. The edge agent is unaffected: it sends telemetry and receives no commands.

A related counter arrives alongside them: devicechain_commanddelivery_command_delivery_responses_not_answerable_total counts an answer that named the dispatch its command is on and still could not settle it. No command state in today's vocabulary produces that, so it should read zero; it exists to catch a state being added later without anyone deciding whether an answer may settle it.

Services now refuse to start on things they used to accept

Each of these is a fail-closed correction, and each can stop a pod that previously ran:

What	The condition that now refuses
Secret store	an instance root key that is well-formed but wrong — previously started and failed at the first secret it was asked for
Instance configuration	a misspelled key — previously discarded in silence, with the default applied
Instance configuration	DC_SHUTDOWN_DRAIN_SECONDS still set — the environment variable is gone; the value is infrastructure.shutdown.drainSeconds
Instance configuration	a shutdown drain window longer than half the pod's terminationGracePeriodSeconds
Listeners	two listeners on one port, or a device-facing port: "0"
Any HTTP listener	a port already in use — previously logged from inside a goroutine while the service reported a successful start

The misspelled-key one is worth a moment. A typo in maxSubscriptionMessageBytes measurably halved the effective frame ceiling with nothing logged — the key was dropped and the default applied, which looks exactly like a working configuration. Strict decoding means you find out at startup instead.

Metrics: eleven new series, none renamed or removed

Every series that existed in v0.15.0 keeps its exact name — nothing was renamed and nothing was dropped, including through the change that gave each service its own metrics registry.

What is new: five counters on command-delivery (the two nonce counters above, plus exhausted dispatches, answers a command's state could not accept, and responses lost because a dead letter could not be written), two alarm dead-letter counters on device-management, one early-close counter on event-sources, is_serving on lwm2m-ingest, and the two Sparkplug rebirth counters below.

is_leader changes meaning on lwm2m-ingest, and the docs recommend alerting on it

The gauge is now raised when the replica acquires the lease, rather than after it has finished building its term. A term build takes up to 30 seconds per bound tenant, so a replica that had just won a failover previously reported is_leader=0 for as long as 30 seconds per tenant while actually holding the lease.

If you follow the sum(devicechain_lwm2mingest_is_leader) != 1 alert the deployment guide recommends, that false-leaderless window disappears. The new devicechain_lwm2mingest_is_serving gauge is what now distinguishes "leader, still building its term" from "leader and serving" — the state one gauge could not express. is_leader == 1 with is_serving == 0 sustained is a leader wedged in its build. Note the chart ships no alerting rule for either; this is guidance to author, not a rule you inherit.

sparkplug-ingest gains rebirth_enqueued_total and rebirth_dropped_total. A saturated rebirth queue used to be indistinguishable from an idle one on every series the service exported, because rebirth_requests_total counts successful publishes — so saturation made it stop rising. Read the new pair together: drops climbing while requests hold at a ceiling is fan-out outrunning a healthy publisher; drops climbing while requests stay flat points at the broker connection.

Other behaviour worth knowing about

• A command the platform cannot publish now fails. Previously it cycled between queued and sent on every sweep until its TTL elapsed days later, and then recorded a timeout — which says a device did not answer, when nothing had ever been dispatched. It now stops at a bound (20 attempts by default, about ten minutes at the default 30-second sweep cadence) and records failure naming the platform. The bound is yours to change under functionalAreas.command-delivery.config.maxDispatchFailures.
• A dead letter's reason for a blocked connector destination is now unprocessable rather than exhausted. Update any alert or saved query keyed on the old value; existing records read back unchanged.
• An alarm state change that could not be published is now dead-lettered and counted, and alarm_event_dead_letter_lost_total joins the DeadLetterWriteLost alert.
• An inbound message that cannot be decoded is no longer archived whole. The record points at the original by subject and stream sequence instead.
• GraphQL subscriptions are closed cleanly at shutdown with a 1001 frame, and an inbound frame is now capped — infrastructure.graphql.maxSubscriptionMessageBytes, default 4 MiB. This is the only new chart value in the release, and it has a default.
• Governance refresh is rate-bounded — 50 lookups/sec with a burst of 100, per governed dimension, and a 10-second negative cache after a failed lookup. One resolver keeps roughly 3000 tenants warm without ever reaching the bound. Past it, a tenant that has already been resolved keeps serving its last-known value rather than dropping to the platform default; only a tenant that has never resolved gets the default.
• Separately, a platform-default ingest ceiling of 0 is now floored to 100 messages/sec with a burst of 200, rather than admitting nothing. That is a different axis from the lookup rate above.
• Shutdown is bounded by a budget derived from the grace period, minus the drain window and a two-second margin, and honours cancellation throughout. Consumer read loops back off and then fail the process rather than spinning or retrying forever.
• Two chart details that are easy to trip over. instance.config.infrastructure.metrics.httpPort is retired — a document that still carries it logs a warning naming the key and starts normally, and the chart no longer writes it. And instance.config.infrastructure.shutdown is now written for you by the chart from the top-level shutdownDrainSeconds and terminationGracePeriodSeconds; setting that block by hand makes helm fail the render rather than silently disagreeing with the pod spec. If you supply the instance config through instance.existingSecret, that block is yours to add.
• A pod that ends itself releases its leadership lease on the way out. On lwm2m-ingest both paths that used to exit while still holding it are fixed. The 30-second wait before a replacement can take over is now what follows an abrupt loss — a node failure, a kill -9 — not what follows a pod deciding to stop.

The published packages

The .NET/Unity SDK carries the command-nonce change described above; upgrading it is how a device built on it keeps answering commands.

@devicechain/client, @devicechain/dashboards, @devicechain/widgets and @devicechain/brand have no source changes in this release. One thing to know if you install @devicechain/widgets yourself: its maplibre-gl peer range moves from ^6.6.0 to ^6.7.0. If you pin maplibre-gl at 6.6.x you will see an unmet-peer warning, or an install failure under a package manager that enforces peers strictly. Nothing else about the packages changed.

Instances built by v0.16.0 and earlier

An instance bootstrapped by v0.16.0, or by any release before it, cannot be upgraded onto the release that follows v0.16.0. dcctl upgrade refuses rather than trying.

dcctl bootstrap now records a declaration — a cluster-scoped object saying what the instance is: its profile, its topology, how it is exposed, and which functional areas it runs. dcctl upgrade reads that declaration to know what to deploy, which is what lets one command move a version without being told an instance's shape all over again. Releases up to and including v0.16.0 wrote no such record, so there is nothing for the upgrade to read.

It says so, rather than treating your instance as a name that does not exist:

instance "devicechain" IS in this cluster — named by the DeviceChain Helm releases in this
cluster — and it carries no declaration, so it was built by a release older than the one
that began recording them.

There is no compatibility shim, and before v1.0.0 there will not be one. What the older instance was configured with was never written down in a form this release can read, so a declaration invented after the fact would be a guess applied over a live instance.

To move onto this release, recreate the instance:

# Export anything you need first — this discards the databases.
dcctl destroy local devicechain
dcctl bootstrap local devicechain

Export first — recreation discards your data

The destroy guard protects the databases from an ordinary helm operation, not from a deliberate dcctl destroy. If the instance holds telemetry, device definitions or dashboards you care about, dump them before you start.

The refused upgrade changes nothing

dcctl upgrade reads the instance before it writes anything, so the refusal lands before the first change: the Helm release stays on the revision it was on, the operator keeps running the image it was running, and every row is where it was. Running it to see what it says costs nothing. Both halves of that — the refusal, and the instance being untouched afterwards — are exercised against a real cluster on every release.

Once you are on a release that records a declaration, ordinary in-place upgrades resume. dcctl instances list shows what is declared, and in which cluster.

The one-time durable-ingest cutover

The release that introduces durable MQTT ingest changes how event-sources receives device telemetry: instead of subscribing to the broker as an MQTT client, it consumes a durable capture stream that the broker writes to before it acknowledges the device. This is what stops telemetry being lost when event-sources is down.

Crossing that release once is an ordinary in-place upgrade — but expect a brief window of duplicated telemetry, and plan for it:

• During the rollout the outgoing pod is still ingesting over MQTT while the incoming pod has already begun consuming the capture stream, so messages published in that overlap are ingested by both. The window is bounded by how long the two pods coexist — the incoming pod's startup plus the outgoing pod's drain.
• Events that carry both an altId and a device-supplied occurredTime are unaffected: the write-side dedup key is (tenant, altId, occurredTime), so those duplicates collapse. An event with an altId but no occurredTime does not collapse — the decoder stamps the current time when the device omits one, and the two copies are decoded in different pods at different instants, so they get different timestamps and land as two rows. Telemetry with no altId is not deduplicated at all.
• The overlap is preferred deliberately. The alternative ordering — stopping the old pod before the capture stream exists — loses every message the broker acknowledges in the gap, and that loss is silent: the device is told the message was accepted and it is never stored. A duplicate reading is visible and correctable; a missing one is neither.

Do not set event-sources to Recreate

strategy: Recreate on event-sources produces exactly the lossy ordering above, because it terminates the old pod before the new one creates the capture stream. The chart refuses to render this configuration rather than let it drop telemetry silently. event-sources is not a single-writer service and gains nothing from Recreate — once cut over it can run multiple replicas, which the MQTT-client path it replaces could not.

Data durability

The database tier is intentionally lifecycle-independent from the application. Both databases are provisioned as separate infrastructure with a destroy guard, so upgrading, reinstalling or uninstalling the application never touches them. That is the common case and it is safe.

Removing the database from the infrastructure configuration is a different act

The guard protects each database while it is in the infrastructure configuration. It does not protect one that has been taken out of it: a resource removed from the configuration is no longer covered by rules the configuration declares, and the removal plan will succeed. The database clusters also own their volumes, so removing one takes its data with it rather than leaving an unattached volume behind.

Do not edit the database out of the infrastructure configuration as a way of replacing it.

Upgrading an instance created before the databases moved onto the operator is the one case where this comes up, and it is refused at plan time rather than left to chance. Dump both databases first, then re-run the bootstrap with --allow-legacy-db-removal — which asserts you have handled the data, and verifies nothing. For a local instance, dcctl destroy followed by a fresh bootstrap is simpler and discards the data deliberately.

This is durability of the running volumes — it is not a substitute for scheduled backups and point-in-time recovery, which are provisioned with the production infrastructure. See Deployment & Operator for how the infrastructure and application layers are separated.